Commit graph

42 commits

Author SHA1 Message Date
Eugen Rochko
73c613dcfc Switch to static URIs, new URI format in both protocols for new statuses (#4815)
* Decouple Status#local? from uri being nil

* Replace on-the-fly URI generation with stored URIs

- Generate URI in after_save hook for local statuses
- Use static value in TagManager when available, fallback to tag format
- Make TagManager use ActivityPub::TagManager to understand new format
- Adjust tests

* Use other heuristic for locality of old statuses, do not perform long query

* Exclude tombstone stream entries from Atom feed

* Prevent nil statuses from landing in Pubsubhubbub::DistributionWorker

* Fix URI not being saved (#4818)

* Add more specs for Status

* Save generated uri immediately

and also fix method order to minimize diff.

* Fix alternate HTML URL in Atom

* Fix tests

* Remove not-null constraint from statuses migration to speed it up
2017-09-06 19:01:28 +02:00
abcang
892aeb7ffe Do not execute the job with the same arguments as the retry job (#4814) 2017-09-05 20:56:20 +02:00
Eugen Rochko
3135d20283 Serialize ActivityPub alternate link into OStatus deletes, handle it (#4730)
Requires moving Atom rendering from DistributionWorker (where
`stream_entry.status` is already nil) to inline (where
`stream_entry.status.destroyed?` is true) and distributing that.

Unfortunately, such XML renderings can no longer be easily chained
together into one payload of n items.
2017-08-29 16:11:05 +02:00
Eugen Rochko
cab7fa158a Add configuration to disable private status federation over PuSH (#4582) 2017-08-24 17:51:32 +02:00
Eugen Rochko
1a19358b24 ActivityPub migration procedure (#4617)
* ActivityPub migration procedure

Once one account is detected as going from OStatus to ActivityPub,
invalidate WebFinger cache for other accounts from the same domain

* Unsubscribe from PuSH updates once we receive an ActivityPub payload

* Re-subscribe to PuSH unless already unsubscribed, regardless of protocol
2017-08-21 01:14:40 +02:00
Yamagishi Kazutoshi
f95fca19d4 Change RuboCop rules to loose (#4464) 2017-07-31 15:19:13 +02:00
Eugen Rochko
a8631a9aac When PuSH subscribe attempts are exhausted, unsubscribe (#4422) 2017-07-28 17:21:14 +02:00
Clworld
0c86b36cac Add callback_url/acct information for Sidekiq PuSH workers Exception. (#4281)
* Add destination informations to exception on SubscribeWorker and DeliveryWorker.

* Simplify delivery error message.

* Prevent changing Exception type...

* fix typo.
2017-07-27 00:38:20 +02:00
Eugen Rochko
75ab767791 Fix webfinger retries (#4275)
* Do not raise unretryable exceptions in ResolveRemoteAccountService

* Removed fatal exceptions from ResolveRemoteAccountService

Exceptions that cannot be retried should not be raised. New exception
class for those that can be retried (Mastodon::UnexpectedResponseError)
2017-07-20 01:59:07 +02:00
Eugen Rochko
3852032953 Correct OStatus inflection (Ostatus -> OStatus) (#4255) 2017-07-19 01:37:26 +02:00
Akihiko Odaki
b0f97d9a87 Introduce Ostatus name space (#4164)
* Wrap methods of ProcessFeedService::ProcessEntry in classes

This is a change same with 002ed7dc62, except
that it has the following changes:

* Revert irrelevant change in find_or_create_conversation
* Fix error handling for RemoteActivity

* Introduce Ostatus name space
2017-07-18 16:39:47 +02:00
ThibG
9c1791c546 Optimize uri normalization (#4212)
* Add dependency on idn-ruby to speed up URI normalization

* Use normalized_host instead of normalize.host when applicable

When we are only interested in the normalized host, calling normalized_host
avoids normalizing the other components of the URI as well as creating a
new object
2017-07-15 17:24:35 +02:00
Eugen Rochko
c6e46f7421 Fix #2672 - Connect signed PuSH subscription requests to instance domain (#4205)
* Fix #2672 - Connect signed PuSH subscription requests to instance domain

Resolves #2739

* Fix return of locate_subscription

* Fix tests
2017-07-14 23:01:20 +02:00
Eugen Rochko
c5fa4aba91 HTTP signatures (#4146)
* Add Request class with HTTP signature generator

Spec: https://tools.ietf.org/html/draft-cavage-http-signatures-06

* Add HTTP signature verification concern

* Add test for SignatureVerification concern

* Add basic test for Request class

* Make PuSH subscribe/unsubscribe requests use new Request class

Accidentally fix lease_seconds not being set and sent properly, and
change the new minimum subscription duration to 1 day

* Make all PuSH workers use new Request class

* Make Salmon sender use new Request class

* Make FetchLinkService use new Request class

* Make FetchAtomService use the new Request class

* Make Remotable use the new Request class

* Make ResolveRemoteAccountService use the new Request class

* Add more tests

* Allow +-30 seconds window for signed request to remain valid

* Disable time window validation for signed requests, restore 7 days
as PuSH subscription duration (which was previous default due to a bug)
2017-07-14 20:41:49 +02:00
Eugen Rochko
af4835f176 Limit total subscribe retries to 10, but space them out more (#4142)
Since there is little point in retrying so often when a service is down
or does not exist anymore. Subscriptions are renewed 1 day before they
should expire, so retrying in 30 minutes, then 2 hours, then 12 hours
is fine. If even after that, the remote server does not work, there is
little sense in retrying more often than once a day

Also, uniqueness of the job should ensure that failed retries will
not result in multiple retries for the same endpoint when the next
resubscription cycle comes
2017-07-10 18:04:23 +02:00
Akihiko Odaki (@fn_aki@pawoo.net)
561ca10d86 Some minor change and spec for Account (#3813)
* Introduce domains method to Account relation

Account had followers_domains method, which was excessively specific.
Let relation of Account have domains method instead.

* Move follow_mapping in Account to AccountInteractions

* Introduce shared examples for AccountAvatar inclusion

* Cover Account more
2017-06-22 18:34:27 +02:00
Eugen Rochko
6f8f401ea1 Batched remove status service (#3735)
* Make Pubsubhubbub::DistributionWorker handle both single stream entry
arguments, as well as arrays of stream entries

* Add BatchedRemoveStatusService, make SuspendAccountService use it

* Improve method names

* Add test

* Add more tests

* Use PuSH payloads of 100 to have a clear mapping of
1000 input statuses -> 10 PuSH payloads

It was nice while it lasted
2017-06-14 18:01:35 +02:00
Eugen Rochko
83b444ddab Do not cancel PuSH subscriptions after encountering "permanent" error… (#3046)
* Do not cancel PuSH subscriptions after encountering "permanent" error response

After talking with MMN about it, turns out some servers/php setups do
return 4xx errors while rebooting, so this anti-feature that was meant
to take load off of the hub is doing more harm than good in terms of
breaking subscriptions

* Update delivery_worker.rb
2017-05-14 03:22:48 +02:00
Matt Jankowski
dc59643e73 Spec coverage and refactor for pubsub/delivery worker (#3021)
* Framework for delivery worker spec

* Refactor of pubsub delivery worker
2017-05-12 20:35:36 +02:00
Matt Jankowski
60fa69399c Pubsub confirmation worker spec (#2974)
* Add specs for valid challenge response from pubsub confirmation worker

* Refactor the pubsub confirmation worker
2017-05-11 00:19:37 +02:00
Eugen Rochko
221898cc90 Add spec for Pubsubhubbub::DistributionWorker. PuSH-deliver public items (#2954)
to all subscribers. IDN-normalize callback URLs for subscriptions on insert.
2017-05-10 02:55:43 +02:00
haoyayoi
a5ad348182 add content-type in pubsubhubhub request header (#2943)
* add content-type in pubsubhubhub request header

* fix type
2017-05-09 16:34:47 +02:00
Eugen Rochko
78b890cc81 Fix Scheduler::SubscriptionsScheduler (#2834)
* Fix Scheduler::SubscriptionsScheduler, add worker test for it

* Change production log level of Sidekiq to "warn" instead of "info"
2017-05-06 13:05:03 +02:00
Eugen Rochko
c32e312061 More robust PuSH subscription refreshes (#2799)
* Fix #2473 - Use sidekiq scheduler to refresh PuSH subscriptions instead of cron

Fix an issue where / in domain would raise exception in TagManager#normalize_domain

PuSH subscriptions refresh done in a round-robin way to avoid hammering a single
server's hub in sequence. Correct handling of failures/retries through Sidekiq (see
also #2613). Optimize Account#with_followers scope. Also, since subscriptions
are now delegated to Sidekiq jobs, an uncaught exception will not stop the entire
refreshing operation halfway through

Fix #2702 - Correct user agent header on outgoing http requests

* Add test for SubscribeService

* Extract #expiring_accounts into method

* Make mastodon:push:refresh no-op

* Queues are now defined in sidekiq.yml

* Queues are now in sidekiq.yml
2017-05-05 02:23:01 +02:00
Eugen
87f7a3922c Punycode URI normalization (#2370)
* Fix #2119 - Whenever about to send a HTTP request, normalize the URI

* Add test for IDN request in FetchLinkCardService

* Perform IDN normalization on domains before they are stored in the DB
2017-04-25 02:47:31 +02:00
Eugen
f89e26bdd5 Followers-only post federation (#2111)
* Make private toots get PuSHed to subscription URLs that belong to domains where you have approved followers

* Authorized followers controller, stub for bulk action

* Soft block in the background

* Add simple test for new controller

* Rename Settings::FollowersController to Settings::FollowerDomainsController, paginate results,
rename "private" post setting to "followers-only", fix pagination style, improve post privacy
preferences style, improve warning style

* Extract compose form warnings into own container, show warning when posting to followers-only with unlocked account
2017-04-24 00:38:37 +02:00
Matt Jankowski
d1ebb63c54 Quick best practice cleanup of views/helpers (#1546)
* Remove trailing whitespace

* Use query methods instead of explicit .blank? checks
2017-04-12 18:24:18 +02:00
Eugen
24c77e57b2 Rewrite Atom generation from stream entries to use Ox instead of Nokogiri (#1124)
* Rewrite Atom generation from stream entries to use Ox instead of Nokogiri::Builder

StreamEntry is now limited to only statuses, which allows some optimization. Removed
extra queries on AccountsController#show. AtomSerializer instead of AtomBuilderHelper
used in AccountsController#show, StreamEntriesController#show, StreamEntryRenderer
and PubSubHubbub::DistributionWorker

PubSubHubbub::DistributionWorker moves n+1 DomainBlock query to PubSubHubbub::DeliveryWorker
instead.

All Salmon slaps that aren't based on StreamEntry still use AtomBuilderHelper and Nokogiri

* All Salmon slaps now use Ox instead of Nokogiri. No touch from status on account
2017-04-07 05:56:56 +02:00
Eugen Rochko
d2b7daf6bb Remove PuSH subscriptions when delivery is answered with a 4xx error 2017-04-05 02:34:33 +02:00
Eugen Rochko
94a6dfab3a Disable PuSH for blocked domains 2017-02-14 04:01:37 +01:00
Eugen Rochko
79f3ee31df Take out private toot distribution out of PuSH again 2017-02-12 01:31:04 +01:00
Eugen Rochko
ffd58aad9d Make PuSH only distribute to subscriptions that match follower domains
Allow PuSH to distribute private toots with that condition
2017-02-11 15:41:39 +01:00
Eugen Rochko
a893746e5a Improve background jobs params and error handling 2017-01-05 03:28:21 +01:00
Eugen Rochko
0471207cc3 Fix accidental distribution of hidden stream entries to PuSH subscribers 2017-01-03 00:44:23 +01:00
Eugen Rochko
0f4c9d1124 Adding some rescues to workers, e.g. don't fail removal job when status can't be found in the first place (already removed) 2016-12-19 09:31:12 +01:00
Eugen Rochko
b7c2c5d81d Restoring old async behaviour of thread resolving as it proved to be more robust 2016-12-12 21:12:19 +01:00
Eugen Rochko
a692318382 Track successful PuSH deliveries 2016-11-30 15:24:57 +01:00
Eugen Rochko
3c1c2b0e06 Adding rack timeout of 30sec, PuSH jobs moved to push queue so they
can be processed separately
2016-11-29 02:07:14 +01:00
Eugen Rochko
79075e1303 Fix URLs in inline-rendered XML 2016-11-29 00:26:01 +01:00
Eugen Rochko
02da8fdcbe Fix setting of confirmed=true on successful confirmation 2016-11-28 19:24:49 +01:00
Eugen Rochko
bf7c9e826d More logging to ConfirmationWorker 2016-11-28 19:15:28 +01:00
Eugen Rochko
b5ad0eb4ea Adding embedded PuSH server 2016-11-28 13:36:47 +01:00