Commit graph

26 commits

Author SHA1 Message Date
Eugen Rochko
6f8f401ea1 Batched remove status service (#3735)
* Make Pubsubhubbub::DistributionWorker handle both single stream entry
arguments, as well as arrays of stream entries

* Add BatchedRemoveStatusService, make SuspendAccountService use it

* Improve method names

* Add test

* Add more tests

* Use PuSH payloads of 100 to have a clear mapping of
1000 input statuses -> 10 PuSH payloads

It was nice while it lasted
2017-06-14 18:01:35 +02:00
Eugen Rochko
83b444ddab Do not cancel PuSH subscriptions after encountering "permanent" error… (#3046)
* Do not cancel PuSH subscriptions after encountering "permanent" error response

After talking with MMN about it, turns out some servers/php setups do
return 4xx errors while rebooting, so this anti-feature that was meant
to take load off of the hub is doing more harm than good in terms of
breaking subscriptions

* Update delivery_worker.rb
2017-05-14 03:22:48 +02:00
Matt Jankowski
dc59643e73 Spec coverage and refactor for pubsub/delivery worker (#3021)
* Framework for delivery worker spec

* Refactor of pubsub delivery worker
2017-05-12 20:35:36 +02:00
Matt Jankowski
60fa69399c Pubsub confirmation worker spec (#2974)
* Add specs for valid challenge response from pubsub confirmation worker

* Refactor the pubsub confirmation worker
2017-05-11 00:19:37 +02:00
Eugen Rochko
221898cc90 Add spec for Pubsubhubbub::DistributionWorker. PuSH-deliver public items (#2954)
to all subscribers. IDN-normalize callback URLs for subscriptions on insert.
2017-05-10 02:55:43 +02:00
haoyayoi
a5ad348182 add content-type in pubsubhubhub request header (#2943)
* add content-type in pubsubhubhub request header

* fix type
2017-05-09 16:34:47 +02:00
Eugen Rochko
78b890cc81 Fix Scheduler::SubscriptionsScheduler (#2834)
* Fix Scheduler::SubscriptionsScheduler, add worker test for it

* Change production log level of Sidekiq to "warn" instead of "info"
2017-05-06 13:05:03 +02:00
Eugen Rochko
c32e312061 More robust PuSH subscription refreshes (#2799)
* Fix #2473 - Use sidekiq scheduler to refresh PuSH subscriptions instead of cron

Fix an issue where / in domain would raise exception in TagManager#normalize_domain

PuSH subscriptions refresh done in a round-robin way to avoid hammering a single
server's hub in sequence. Correct handling of failures/retries through Sidekiq (see
also #2613). Optimize Account#with_followers scope. Also, since subscriptions
are now delegated to Sidekiq jobs, an uncaught exception will not stop the entire
refreshing operation halfway through

Fix #2702 - Correct user agent header on outgoing http requests

* Add test for SubscribeService

* Extract #expiring_accounts into method

* Make mastodon:push:refresh no-op

* Queues are now defined in sidekiq.yml

* Queues are now in sidekiq.yml
2017-05-05 02:23:01 +02:00
Eugen
87f7a3922c Punycode URI normalization (#2370)
* Fix #2119 - Whenever about to send a HTTP request, normalize the URI

* Add test for IDN request in FetchLinkCardService

* Perform IDN normalization on domains before they are stored in the DB
2017-04-25 02:47:31 +02:00
Eugen
f89e26bdd5 Followers-only post federation (#2111)
* Make private toots get PuSHed to subscription URLs that belong to domains where you have approved followers

* Authorized followers controller, stub for bulk action

* Soft block in the background

* Add simple test for new controller

* Rename Settings::FollowersController to Settings::FollowerDomainsController, paginate results,
rename "private" post setting to "followers-only", fix pagination style, improve post privacy
preferences style, improve warning style

* Extract compose form warnings into own container, show warning when posting to followers-only with unlocked account
2017-04-24 00:38:37 +02:00
Matt Jankowski
d1ebb63c54 Quick best practice cleanup of views/helpers (#1546)
* Remove trailing whitespace

* Use query methods instead of explicit .blank? checks
2017-04-12 18:24:18 +02:00
Eugen
24c77e57b2 Rewrite Atom generation from stream entries to use Ox instead of Nokogiri (#1124)
* Rewrite Atom generation from stream entries to use Ox instead of Nokogiri::Builder

StreamEntry is now limited to only statuses, which allows some optimization. Removed
extra queries on AccountsController#show. AtomSerializer instead of AtomBuilderHelper
used in AccountsController#show, StreamEntriesController#show, StreamEntryRenderer
and PubSubHubbub::DistributionWorker

PubSubHubbub::DistributionWorker moves n+1 DomainBlock query to PubSubHubbub::DeliveryWorker
instead.

All Salmon slaps that aren't based on StreamEntry still use AtomBuilderHelper and Nokogiri

* All Salmon slaps now use Ox instead of Nokogiri. No touch from status on account
2017-04-07 05:56:56 +02:00
Eugen Rochko
d2b7daf6bb Remove PuSH subscriptions when delivery is answered with a 4xx error 2017-04-05 02:34:33 +02:00
Eugen Rochko
94a6dfab3a Disable PuSH for blocked domains 2017-02-14 04:01:37 +01:00
Eugen Rochko
79f3ee31df Take out private toot distribution out of PuSH again 2017-02-12 01:31:04 +01:00
Eugen Rochko
ffd58aad9d Make PuSH only distribute to subscriptions that match follower domains
Allow PuSH to distribute private toots with that condition
2017-02-11 15:41:39 +01:00
Eugen Rochko
a893746e5a Improve background jobs params and error handling 2017-01-05 03:28:21 +01:00
Eugen Rochko
0471207cc3 Fix accidental distribution of hidden stream entries to PuSH subscribers 2017-01-03 00:44:23 +01:00
Eugen Rochko
0f4c9d1124 Adding some rescues to workers, e.g. don't fail removal job when status can't be found in the first place (already removed) 2016-12-19 09:31:12 +01:00
Eugen Rochko
b7c2c5d81d Restoring old async behaviour of thread resolving as it proved to be more robust 2016-12-12 21:12:19 +01:00
Eugen Rochko
a692318382 Track successful PuSH deliveries 2016-11-30 15:24:57 +01:00
Eugen Rochko
3c1c2b0e06 Adding rack timeout of 30sec, PuSH jobs moved to push queue so they
can be processed separately
2016-11-29 02:07:14 +01:00
Eugen Rochko
79075e1303 Fix URLs in inline-rendered XML 2016-11-29 00:26:01 +01:00
Eugen Rochko
02da8fdcbe Fix setting of confirmed=true on successful confirmation 2016-11-28 19:24:49 +01:00
Eugen Rochko
bf7c9e826d More logging to ConfirmationWorker 2016-11-28 19:15:28 +01:00
Eugen Rochko
b5ad0eb4ea Adding embedded PuSH server 2016-11-28 13:36:47 +01:00