Commit graph

63 commits

Author SHA1 Message Date
Eugen Rochko
7a482567f8 Add retention policy for cached content and media (#19232) 2022-09-27 03:08:19 +02:00
Eugen Rochko
7300beadd9 Remove digest e-mails (#17985)
* Remove digest e-mails

* Remove digest-related code
2022-08-25 23:38:22 +02:00
Jeremy Kescher
45fd0464de ip_cleanup_scheduler: Make IP and session retention configurable (#18757) 2022-07-07 03:14:28 +02:00
Eugen Rochko
f4cab3ebad Change search indexing to use batches to minimize resource usage (#18451) 2022-05-18 23:29:14 +02:00
Eugen Rochko
1cd4518c29 Remove IP matching from e-mail domain blocks (#18190)
Clear out e-mail domain blocks created from automatically resolved DNS records
2022-04-29 23:27:03 +02:00
Eugen Rochko
553889bc7c Fix single Redis connection being used across all threads (#18135)
* Fix single Redis connection being used across all Sidekiq threads

* Fix tests
2022-04-28 17:47:34 +02:00
Jeong Arm
851cb96e75 Fix clean_discarded_statuses (#17822) 2022-03-19 09:24:08 +01:00
Eugen Rochko
4d0a2968d9 Change authorized applications page (#17656)
* Change authorized applications page

* Hide revoke button for superapps and suspended accounts

* Clean up db/schema.rb
2022-03-01 16:48:58 +01:00
Claire
d646f23e5d Fix crash in EmailDomainBlockRefreshScheduler (#17649) 2022-02-26 16:28:28 +01:00
Eugen Rochko
e6d2b07ec1 Add trending statuses (#17431)
* Add trending statuses

* Fix dangling items with stale scores in localized sets

* Various fixes and improvements

- Change approve_all/reject_all to approve_accounts/reject_accounts
- Change Trends::Query methods to not mutate the original query
- Change Trends::Query#skip to offset
- Change follow recommendations to be refreshed in a transaction

* Add tests for trending statuses filtering behaviour

* Fix not applying filtering scope in controller
2022-02-25 00:34:14 +01:00
Eugen Rochko
ea5bf21a1d Change e-mail domain blocks to block IPs dynamically (#17635)
* Change e-mail domain blocks to block IPs dynamically

* Update app/workers/scheduler/email_domain_block_refresh_scheduler.rb

Co-authored-by: Yamagishi Kazutoshi <ykzts@desire.sh>

* Update app/workers/scheduler/email_domain_block_refresh_scheduler.rb

Co-authored-by: Yamagishi Kazutoshi <ykzts@desire.sh>

Co-authored-by: Yamagishi Kazutoshi <ykzts@desire.sh>
2022-02-24 17:28:23 +01:00
Eugen Rochko
d979087a98 Fix localization of cold-start follow recommendations (#17479) 2022-02-08 01:53:49 +01:00
Claire
edd7dad3e2 Fix Ruby 2.5 incompatibility (#17465) 2022-02-07 19:57:06 +01:00
Claire
33ea1c9008 Fix Sidekiq warnings about JSON serialization (#17381)
* Fix Sidekiq warnings about JSON serialization

This occurs on every symbol argument we pass, and every symbol key in hashes,
because Sidekiq expects strings instead.

See https://github.com/mperham/sidekiq/pull/5071

We do not need to change how workers parse their arguments because this has
not changed and we were already converting to symbols adequately or using
`with_indifferent_access`.

* Set Sidekiq to raise on unsafe arguments in test mode

In order to more easily catch issues that would produce warnings in production
code.
2022-01-28 00:43:56 +01:00
Eugen Rochko
2fb76550a9 Add notifications for statuses deleted by moderators (#17204) 2022-01-17 09:41:33 +01:00
Eugen Rochko
7b3972c270 Remove IP tracking columns from users table (#16409) 2022-01-16 13:23:50 +01:00
Takeshi Umeda
794d9d267d Fix follow recommendation biased towards older accounts (#17126) 2021-12-13 23:21:14 +01:00
Eugen Rochko
ad73becf3e Add trending links (#16917)
* Add trending links

* Add overriding specific links trendability

* Add link type to preview cards and only trend articles

Change trends review notifications from being sent every 5 minutes to being sent every 2 hours

Change threshold from 5 unique accounts to 15 unique accounts

* Fix tests
2021-11-25 13:07:38 +01:00
Claire
1fcf310c60 Add feature to automatically delete old toots (#16529)
* Add account statuses cleanup policy model

* Record last inspected toot to delete to speed up successive calls to statuses_to_delete

* Add service to cleanup a given account's statuses within a budget

* Add worker to go through account policies and delete old toots

* Fix last inspected status id logic

All existing statuses older or equal to last inspected status id must be
kept by the current policy. This is an invariant that must be kept so that
resuming deletion from the last inspected status remains sound.

* Add tests

* Refactor scheduler and add tests

* Add user interface

* Add support for discriminating based on boosts/favs

* Add UI support for min_reblogs and min_favs, rework UI

* Address first round of review comments

* Replace Snowflake#id_at_start with with_random parameter

* Add tests

* Add tests for StatusesCleanupController

* Rework settings page

* Adjust load-avoiding mechanisms

* Please CodeClimate
2021-08-09 23:11:50 +02:00
Eugen Rochko
2067b0bf34 Add authentication history (#16408) 2021-06-21 17:07:30 +02:00
Claire
78eddfc83c Improve performance of follow recommendation scheduler (#16159)
Express follow_recommendations in terms of account_summaries rather than
accounts, integrate filters that are unconditionally used, and materialize
the resulting view.

This should result in the bulk of the computation being performed only once
instead of **once per recommendation language**.
2021-05-05 22:04:52 +02:00
Eugen Rochko
92b2d926bf Add cold-start follow recommendations (#15945) 2021-04-12 12:37:14 +02:00
Eugen Rochko
9f16bd9c89 Fix configuration for sidekiq-unique-jobs after 7.x upgrade (#15908)
Remove locks from scheduled jobs
2021-03-15 11:17:43 +01:00
ThibG
8d75aa630f Improve account deletion performances further (#15407)
* Delete status records by batches of 50

* Do not precompute values that are only used once

* Do not generate redis events for removal of public toots older than two weeks

* Filter reported toots a priori for polls and status deletion

* Do not process reblogs when cleaning up public timelines

As in Mastodon proper, reblogs don't appear in public TLs

* Clean the deleted account's own feed in one go

* Refactor Account#clean_feed_manager and List#clean_feed_manager

* Delete instead of destroy a few more associations

* Fix preloading

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2020-12-22 23:57:46 +01:00
Eugen Rochko
df8874b24e Fix performance on instances list in admin UI (#15282)
- Reduce duplicate queries
- Remove n+1 queries
- Add accounts count to detailed view
- Add separate action log entry for updating existing domain blocks
2020-12-14 09:06:34 +01:00
Eugen Rochko
da4c730c47 Add IP-based rules (#14963) 2020-10-12 16:33:49 +02:00
Eugen Rochko
e514304a76 Change account suspensions to be reversible by default (#14726) 2020-09-15 14:37:58 +02:00
Eugen Rochko
b864e78db8 Add E2EE API (#13820) 2020-06-02 19:24:53 +02:00
Eugen Rochko
b1c596e8f3 Fix background jobs not using locks like they are supposed to (#13361)
Also:

- Fix locks not being removed when jobs go to the dead job queue
- Add UI for managing locks to the Sidekiq dashboard
- Remove unused Sidekiq workers

Fix #13349
2020-03-31 21:59:03 +02:00
Eugen Rochko
9d3bba9576 Fix expired announcements being re-published (#12964) 2020-01-26 22:43:18 +01:00
Eugen Rochko
f95fd62bac Add announcements (#12662)
* Add announcements

Fix #11006

* Add reactions to announcements

* Add admin UI for announcements

* Add unit tests

* Fix issues

- Add `with_dismissed` param to announcements API
- Fix end date not being formatted when time range is given
- Fix announcement delete causing reactions to send streaming updates
- Fix announcements container growing too wide and mascot too small
- Fix `all_day` being settable when no time range is given
- Change text "Update" to "Announcement"

* Fix scheduler unpublishing announcements before they are due

* Fix filter params not being passed to announcements filter
2020-01-23 22:00:13 +01:00
abcang
f419d968d8 Improve IP cleanup query (#11871) 2019-09-17 08:44:25 +02:00
Eugen Rochko
436266ca27 Change trending hashtags to not disappear instantly after midnight (#11712) 2019-09-02 18:11:13 +02:00
mayaeh
7d13da568e Add tootctl preview_cards remove (#11320)
* Add `tootctl preview_cards remove`

* fix code style

* Remove `Scheduler::PreviewCardsCleanupScheduler` file

* fix code style again
Add exclude case where image_file_name is blank

* Added a function to output confirmation if the specified number of days is less than 2 weeks
2019-07-28 13:48:19 +02:00
Eugen Rochko
f9ed4354b1 Remove WebSub subscriptions (#11303) 2019-07-21 04:08:00 +02:00
Eugen Rochko
ac853dca51 Add periodic removal of older thumbnails for preview cards (#11304) 2019-07-15 07:50:14 +02:00
Eugen Rochko
4931208dd8 Remove Salmon and PubSubHubbub (#11205)
* Remove Salmon and PubSubHubbub endpoints

* Add error when trying to follow OStatus accounts

* Fix new accounts not being created in ResolveAccountService
2019-07-06 23:26:16 +02:00
ysksn
9cad360caa Create Redisable#redis (#9633)
* Create Redisable

* Use #redis instead of Redis.current
2019-02-02 19:11:38 +01:00
Eugen Rochko
c3922e1c3c Fix PublishScheduledStatusWorker being queued with no arguments (#9724) 2019-01-06 09:21:16 +01:00
Eugen Rochko
ae1aaa3b8a Add scheduled statuses (#9706)
Fix #340
2019-01-05 12:43:28 +01:00
Eugen Rochko
6936271536 Improve e-mail digest (#9689)
- Reduce time-to-digest from 20 to 7 days
- Fetch mentions starting from +1 day since last login
- Fix case when last login is more recent than last e-mail
- Do not render all mentions, only 40, but show number in subject
- Do not send digest to moved accounts
- Do send digest to silenced accounts
2019-01-02 10:47:32 +01:00
Eugen Rochko
ad0f37e924 Track historical space stats in PgHero to determine PostgreSQL growth (#8906) 2018-10-09 19:35:14 +02:00
Eugen Rochko
fddae332aa Disable retries on all scheduled tasks (#8413) 2018-08-25 13:28:07 +02:00
Eugen Rochko
fe88a4f42c Get rid of all batch order warnings (#8334) 2018-08-21 12:25:50 +02:00
Eugen Rochko
a03de03ba2 Keep scheduler jobs unique until they're done (#8287) 2018-08-19 15:48:29 +02:00
Eugen Rochko
ca1c696dbd Slightly reduce RAM usage (#7301)
* No need to re-require sidekiq plugins, they are required via Gemfile

* Add derailed_benchmarks tool, no need to require TTY gems in Gemfile

* Replace ruby-oembed with FetchOEmbedService

Reduce startup by 45382 allocated objects

* Remove preloaded JSON-LD in favour of caching HTTP responses

Reduce boot RAM by about 6 MiB

* Fix tests

* Fix test suite by stubbing out JSON-LD contexts
2018-05-02 18:58:48 +02:00
Eugen Rochko
e54b79a777 Adjust privacy policy (#6666)
* Adjust privacy policy to be more specific to Mastodon

Fix #6613

* Change data retention of IP addresses from 5 years to 1 year

* Add even more information

* Remove all (now invalid) translations of the privacy policy

* Add information about archive takeout, remove pointless consent section

* Emphasis on DM privacy

* Improve wording

* Add line about data use for moderation purposes
2018-04-04 21:47:39 +02:00
Eugen Rochko
f6884555d7 Account archive download (#6460)
* Fix #201: Account archive download

* Export actor and private key in the archive

* Optimize BackupService

- Add conversation to cached associations of status, because
  somehow it was forgotten and is source of N+1 queries
- Explicitly call GC between batches of records being fetched
  (Model class allocations are the worst offender)
- Stream media files into the tar in 1MB chunks
  (Do not allocate media file (up to 8MB) as string into memory)
- Use #bytesize instead of #size to calculate file size for JSON
  (Fix FileOverflow error)
- Segment media into subfolders by status ID because apparently
  GIF-to-MP4 media are all named "media.mp4" for some reason

* Keep uniquely generated filename in Paperclip::GifTranscoder

* Ensure dumped files do not overwrite each other by maintaing directory partitions

* Give tar archives a good name

* Add scheduler to remove week-old backups

* Fix code style issue
2018-02-21 23:21:32 +01:00
Eugen Rochko
67ee638c28 Move e-mail digest task to sidekiq, reduce workload, improve hint (#6252) 2018-01-15 04:34:28 +01:00
Eugen Rochko
ff4c28a305 Lists redis clean-up (#5886)
* When list is deleted, remove feed from redis

* Clean up list feeds of inactive users
2017-12-05 23:20:27 +01:00