* Fix wrong grouping in Twitter valid_url regex
* Add support for xmpp URIs
Fixes#9776
The difficult part is autolinking, because Twitter-text's extractor does
some pretty ad-hoc stuff to find things that “look like” URLs, and XMPP
URIs do not really match the assumptions of that lib, so it doesn't sound
wise to try to shoehorn it into the existing regex.
This is why I used a specific regex (very close, although slightly more
permissive than the RFC), and a specific scan function (a simplified version
of the generalized one from Twitter).
* Remove leading “xmpp:” from auto-linked text
When authenticating via OAuth, the resource owner password grant
strategy is allowed by Mastodon, but (without this PR), it does not
attempt to authenticate against LDAP or PAM. As a result, LDAP or PAM
authenticated users cannot sign in to Mastodon with their
email/password credentials via OAuth (for instance, for native/mobile
app users).
This PR fleshes out the authentication strategy supplied to doorkeeper
in its initializer by looking up the user with LDAP and/or PAM when
devise is configured to use LDAP/PAM backends. It attempts to follow the
same logic as the Auth::SessionsController for handling email/password
credentials.
Note #1: Since this pull request affects an initializer, it's unclear
how to add test automation.
Note #2: The PAM authentication path has not been manually tested. It
was added for completeness sake, and it is hoped that it can be manually
tested before merging.
* Add backend support for bookmarks
Bookmarks behave like favourites, except they aren't shared with other
users and do not have an associated counter.
* Add spec for bookmark endpoints
* Add front-end support for bookmarks
* Introduce OAuth scopes for bookmarks
* Add bookmarks to archive takeout
* Fix migration
* Coding style fixes
* Fix rebase issue
* Update bookmarked_statuses to latest UI changes
* Update bookmark actions to properly reflect status changes in state
* Add bookmarks item to single-column layout
* Make active bookmarks red
Change the behaviour of remotable concern. Previously, it would skip
downloading an attachment if the stored remote URL is identical to
the new one. Now it would not be skipped if the attachment is not
actually currently stored by Paperclip.
The default limit of 10 retries with exponential backoff meant
that if the S3 server was timing out, you would be stuck with it
for much, much longer than the 5 second read timeout we expect.
The uploading happens within a database transaction, which means
a failing S3 server could negatively affect database performance
It's possible that after commit callbacks were not firing when
exceptions occurred in the process. Also, the default Sidekiq
strategy does not push indexing jobs immediately, which is not
necessary and could be part of the issue too.
* Add nodeinfo endpoint
* dont commit stuff from my local dev
* consistant naming since we implimented 2.1 schema
* Add some additional node info stuff
* Add nodeinfo endpoint
* dont commit stuff from my local dev
* consistant naming since we implimented 2.1 schema
* expanding this to include federation info
* codeclimate feedback
* CC feedback
* using activeserializers seems like a good idea...
* get rid of draft 2.1 version
* Reimplement 2.1, also fix metaData -> metadata
* Fix metaData -> metadata here too
* Fix nodeinfo 2.1 tests
* Implement cache for monthly user aggregate
* Useless
* Remove ostatus from the list of supported protocols
* Fix nodeinfo's open_registration reading obsolete setting variable
* Only serialize domain blocks with user-facing limitations
* Do not needlessly list noop severity in nodeinfo
* Only serialize domain blocks info in nodeinfo when they are set to be displayed to everyone
* Enable caching for nodeinfo endpoints
* Fix rendering nodeinfo
* CodeClimate fixes
* Please CodeClimate
* Change InstancePresenter#active_user_count_months for clarity
* Refactor NodeInfoSerializer#metadata
* Remove nodeinfo 2.1 support as the schema doesn't exist
* Clean-up
The instrumentation code was used for StatsD metrics collection
prior to the switch to the nsa gem and should have been removed
at that point as it no longer does anything at all
* Rate limit based on remote address IP, not on potential reverse proxy
* Limit rate of unauthenticated API requests further
* Rate-limit paging requests to one every 3 seconds
Deletions take a lot of resources to execute and cause a lot of
federation traffic, so it makes sense to decrease the number
someone can queue up through the API.
30 per 30 minutes
I also added "public" here, as I can't think of a good reason not to add it. Perhaps it has some marginal benefit in that ISPs (or other proxies) can cache it for all users. The assets are certainly publicly available and the same for all users.
* Add REST API for creating an account
The method is available to apps with a token obtained via the client
credentials grant. It creates a user and account records, as well as
an access token for the app that initiated the request. The user is
unconfirmed, and an e-mail is sent as usual.
The method returns the access token, which the app should save for
later. The REST API is not available to users with unconfirmed
accounts, so the app must be smart to wait for the user to click a
link in their e-mail inbox.
The method is rate-limited by IP to 5 requests per 30 minutes.
* Redirect users back to app from confirmation if they were created with an app
* Add tests
* Return 403 on the method if registrations are not open
* Require agreement param to be true in the API when creating an account
Right now, this includes three endpoints: host-meta, webfinger, and change-password.
host-meta and webfinger are publicly available and do not use any authentication. Nothing bad can be done by accessing them in a user's browser.
change-password being CORS-enabled will only reveal the URL it redirects to (which is /auth/edit) but not anything about the actual /auth/edit page, because it does not have CORS enabled.
The documentation for hosting an instance on a different domain should also be updated to point out that Access-Control-Allow-Origin: * should be set at a minimum for the /.well-known/host-meta redirect to allow browser-based non-proxied instance discovery.
* Verify link ownership with rel="me"
* Add explanation about verification to UI
* Perform link verifications
* Add click-to-copy widget for verification HTML
* Redesign edit profile page
* Redesign forms
* Improve responsive design of settings pages
* Restore landing page sign-up form
* Fix typo
* Support <link> tags, add spec
* Fix links not being verified on first discovery and passive updates
CSFR-prevention is already implemented but adding this doesn't hurt.
A brief introduction to Same-Site cookies (and the difference between strict and
lax) can be found at
https://blog.mozilla.org/security/2018/04/24/same-site-cookies-in-firefox-60/
TLDR: We use lax since we want the cookies to be sent when the user navigates
safely from an external site.
* Fix uncaching worker
* Revert to using Paperclip's filesystem backend instead of fog-local
fog-local has lots of concurrency issues, causing failure to delete files,
dangling file records, and spurious errors UncacheMediaWorker
Adopted from GitLab CE. Generate new migration with:
rails g post_deployment_migration name_of_migration_here
By default they are run together with db:migrate. To not run them,
the env variable SKIP_POST_DEPLOYMENT_MIGRATIONS must be set
Code by Yorick Peterse <yorickpeterse@gmail.com>, see also:
83c8241160
* Add more granular OAuth scopes
* Add human-readable descriptions of the new scopes
* Ensure new scopes look good on the app UI
* Add tests
* Group scopes in screen and color-code dangerous ones
* Fix wrong extra scope
If Mastodon accesses to the hidden service via transparent proxy, it's needed to avoid checking whether it's a private address, since `.onion` is resolved to a private address.
I was previously using the `HIDDEN_SERVICE_VIA_TRANSPARENT_PROXY` to provide that function. However, I realized that using `HIDDEN_SERVICE_VIA_TRANSPARENT_PROXY` is redundant, since this specification is always used with `ALLOW_ACCESS_TO_HIDDEN_SERVICE`. Therefore, I decided to integrate the setting of `HIDDEN_SERVICE_VIA_TRANSPARENT_PROXY` into` ALLOW_ACCESS_TO_HIDDEN_SERVICE`.
- POST /api/v1/push/subscription
- PUT /api/v1/push/subscription
- DELETE /api/v1/push/subscription
- New OAuth scope: "push" (required for the above methods)
The previous rate limit allowed to post media so fast that it is possible
to fill up the disk space even before an administrator notices. The new
rate limit is configured so that it takes 24 hours to eat 10 gigabytes:
10 * 1024 / 8 / (24 * 60 / 30) = 27 (which rounded to 30)
The period is set long so that it does not prevent from attaching several
media to one post, which would happen in a short period. For example,
if the period is 5 minutes, the rate limit would be:
10 * 1024 / 8 / (24 * 60 / 5) = 4
This long period allows to lift the limit up.
* No need to re-require sidekiq plugins, they are required via Gemfile
* Add derailed_benchmarks tool, no need to require TTY gems in Gemfile
* Replace ruby-oembed with FetchOEmbedService
Reduce startup by 45382 allocated objects
* Remove preloaded JSON-LD in favour of caching HTTP responses
Reduce boot RAM by about 6 MiB
* Fix tests
* Fix test suite by stubbing out JSON-LD contexts
* Add support for HTTP client proxy
* Add access control for darknet
Supress error when access to darknet via transparent proxy
* Fix the codes pointed out
* Lint
* Fix an omission + lint
* any? -> include?
* Change detection method to regexp to avoid test fail
* Revert "Bump version to 2.3.2rc1"
This reverts commit 64d5c8a512.
* Revert "Downgrade Dockerfile to Ruby 2.4.3 on Alpine 3.6 (#6806)"
This reverts commit 36734278ba.
* Revert "Handle Mastodon::HostValidationError when pulling remoteable assets (#6782)"
This reverts commit 8f374100ed.
* Revert "Correct the reference to user's password in mastodon:add_user task (#6800)"
This reverts commit 7b247b15f2.
* Revert "Upgrade Paperclip to version 6.0.0 (#6754)"
This reverts commit cd6dee83a1.
In cases where a URL has a trailing hyphen the FetchLinkCardService incorrectly removes the hyphen when it is parsed
The hyphen is not a reserved character in the URI spec https://tools.ietf.org/html/rfc3986#section-2.2
* Add full-text search for authorized statuses
- Search API will return statuses that match the query
- Only for logged in users
- Only if you are author of the status,
- Or you were mentioned in it
- Or you favourited or reblogged it
- Configuration over `ES_ENABLED`, `ES_HOST`, `ES_PORT`, `ES_PREFIX`
- Run `rails chewy:deploy` to create & populate index
Fix#5880Fix#4293Fix#1152
* Add commented out docker-compose configuration for ES container
* Optimize index import, filter search results
* Add basic normalization to the index
* Add better stemming and normalization to the index
* Skip webfinger request if search query includes both @ and a space
* Fix code style
* Visually separate search result sections
* Fix code style issues
CSRF token checking was enabled for API controllers in #6223,
producing "Can't verify CSRF token authenticity" log spam. This
disables logging of failed CSRF checks.
This also changes the protection strategy for
PushSubscriptionsController to use exceptions, making it consistent
with other controllers that use sessions.
Previously each protected path had a separate rate limit. Now they're all in the same bucket, so people are more likely to hit one with register->login. Increasing to 25 per 5 minutes should be fine.
* Add confirmation step for email changes
This adds a confirmation step for email changes of existing users.
Like the initial account confirmation, a confirmation link is sent
to the new address.
Additionally, a notification is sent to the existing address when
the change is initiated. This message includes instruction to reset
the password immediately or to contact the instance admin if the
change was not initiated by the account owner.
Fixes#3871
* Add review fixes
Call to warden.authenticate! in resource_owner_from_credentials would
make the request redirect to sign-in path, which is a bad response for
apps. Now bad credentials just return nil, which leads to HTTP 401
from Doorkeeper. Also, accounts with enabled 2FA cannot be logged into
this way.
* Revert "Enable UniqueRetryJobMiddleware even when called from sidekiq worker (#4836)"
This reverts commit 0080804f44.
* Revert "Do not execute the job with the same arguments as the retry job (#4814)"
This reverts commit 892aeb7ffe.
Keystone V2 is deprecated in favour of V3. This adds the necessary
connection parameters for establishing a V3 connection. Connections
to V2 endpoints are still possible and the configuration should
remain compatible.
This also introduces a SWIFT_REGION variable for multi-region
OpenStack environments and a SWIFT_CACHE_TTL that controls how long
tokens and other meta-data is cached for. Caching tokens avoids
rate-limiting errors that would result in media uploads becoming
unavailable during high load or when using tasks like
media:remove_remote. fog-openstack only supports token caching for
V3 endpoints, so a recommendation for using V3 was added.
* Add handling of Linked Data Signatures in payloads
* Add a way to sign JSON, fix canonicalization of signature options
* Fix signatureValue encoding, send out signed JSON when distributing
* Add missing security context
* Add code for creating/managing apps to settings section
* Add specs for app changes
* Fix controller spec
* Fix view file I pasted over by mistake
* Add locale strings. Add 'my apps' to nav
* Add Client ID/Secret to App page. Add some visual separation
* Fix rubocop warnings
* Fix embarrassing typo
I lost an `end` statement while fixing a merge conflict.
* Add code for creating/managing apps to settings section
- Add specs for app changes
- Add locale strings. Add 'my apps' to nav
- Add Client ID/Secret to App page. Add some visual separation
- Fix some bugs/warnings
* Update to match code standards
* Trigger notification
* Add warning about not sharing API secrets
* Tweak spec a bit
* Cleanup fixture creation by using let!
* Remove unused key
* Add foreign key for application<->user
* Add ActivityPub inbox
* Handle ActivityPub deletes
* Handle ActivityPub creates
* Handle ActivityPub announces
* Stubs for handling all activities that need to be handled
* Add ActivityPub actor resolving
* Handle conversation URI passing in ActivityPub
* Handle content language in ActivityPub
* Send accept header when fetching actor, handle JSON parse errors
* Test for ActivityPub::FetchRemoteAccountService
* Handle public key and icon/image when embedded/as array/as resolvable URI
* Implement ActivityPub::FetchRemoteStatusService
* Add stubs for more interactions
* Undo activities implemented
* Handle out of order activities
* Hook up ActivityPub to ResolveRemoteAccountService, handle
Update Account activities
* Add fragment IDs to all transient activity serializers
* Add tests and fixes
* Add stubs for missing tests
* Add more tests
* Add more tests
* Improve webfinger templates and make tests more flexible
* Clean up AS2 representation of actor
* Refactor outbox
* Create activities representation
* Add representations of followers/following collections, do not redirect /users/:username route if format is empty
* Remove unused translations
* ActivityPub endpoint for single statuses, add ActivityPub::TagManager for better
URL/URI generation
* Add ActivityPub::TagManager#to
* Represent all attachments as Document instead of Image/Video specifically
(Because for remote ones we may not know for sure)
Add mentions and hashtags representation to AP notes
* Add AP-resolvable hashtag URIs
* Use ActiveModelSerializers for ActivityPub
* Clean up unused translations
* Separate route for object and activity
* Adjust cc/to matrices
* Add to/cc to activities, ensure announce activity embeds target status and
not the wrapper status, add "id" to all collections
