Commit graph

415 commits

Author SHA1 Message Date
Eugen Rochko
499b213e8c Remove unused indexes (#7829) 2018-06-17 18:49:35 +02:00
Eugen Rochko
499a9f5f11 Change language opt-out to language opt-in (#7823)
* Switch filtered_languages to chosen_languages

* Adjust interface

* Remove unused translations
2018-06-17 13:54:02 +02:00
Thibaut Girka
3d6c594903 Merge branch 'master' into glitch-soc/merge-upstream
Conflicts:
	app/javascript/mastodon/initial_state.js
	db/schema.rb

Upstream added a new field to initial_state.
Not too sure about what happened with db/schema.rb though…
2018-06-15 20:51:39 +02:00
Eugen Rochko
907694df6a Add autofollow option to invites (#7805)
* Add autofollow option to invites

* Trigger CodeClimate rebuild
2018-06-15 18:00:23 +02:00
David Yip
4692eed7bd Merge branch 'master' into 454-allow-keyword-mutes-to-skip-mentions 2018-06-12 18:13:30 -05:00
Jenkins
695674b2fa Merge remote-tracking branch 'tootsuite/master' into glitchsoc/master 2018-06-12 13:17:21 +00:00
Eugen Rochko
cbec0de81e Migrate old web push subscriptions to ensure deliveries (#7764) 2018-06-09 13:33:04 +02:00
David Yip
2bb5842541 Thread scopes through #matches?. #454.
Also add an apply_to_mentions attribute on Glitch::KeywordMute, which is
used to calculate scope.  Next up: additions to the test suite to
demonstrate how scoping works.
2018-06-03 23:00:50 -05:00
Jenkins
a76f40890e Merge remote-tracking branch 'tootsuite/master' into glitchsoc/master 2018-05-31 01:17:25 +00:00
Eugen Rochko
2ba4e0bf1a Deduplicate accounts and make unique username/domain index case-insensitive (#7658)
Fix #6937
Fix #6837
Fix #6667
2018-05-30 02:51:26 +02:00
Thibaut Girka
efd2441d1f Merge branch 'master' into glitch-soc/merge 2018-05-14 20:51:50 +02:00
Akihiko Odaki
fee0b5f372 Revert index change on statuses for api/v1/accounts account_id statuses (#7484) 2018-05-14 19:56:17 +02:00
Akihiko Odaki
28629d7921 Improve index on statuses for api/v1/accounts account_id statuses (#7476)
Queries with the combination of account_id, id, and visibility can be
categorized in three types:
1. Querying for public and unlisted to enumerate statuses visible to
anyone.
2. Querying for public, unlisted, and private to enumerate statuses
visible to follower.
3. Querying for direct to enumerate own direct statuses.

1 and 2 is covered by the index with condition 'visibility IN (0, 1, 2)'.
It would bring better performance in case that there are many direct
statuses.

The index with condition 'visibility = 3' is just for 3. It would be much
faster to query direct statuses thanks to this query.

The total size of those two indexes are expected to be smaller than the
deleted one because they are partial and does not have to cover all the
table.
2018-05-14 12:50:45 +02:00
Thibaut Girka
d0b753db6b Merge branch 'master' into glitch-soc/merge
Conflicts:
	app/controllers/invites_controller.rb
	app/serializers/initial_state_serializer.rb
	config/locales/ko.yml
2018-05-11 18:12:42 +02:00
Eugen Rochko
e86a4fe36b Add REST API for Web Push Notifications subscriptions (#7445)
- POST /api/v1/push/subscription
- PUT /api/v1/push/subscription
- DELETE /api/v1/push/subscription
- New OAuth scope: "push" (required for the above methods)
2018-05-11 11:49:12 +02:00
Thibaut Girka
af504e62ff Merge branch 'master' into glitch-soc/master
Conflicts:
	app/models/account.rb
	app/views/accounts/_header.html.haml
2018-05-10 00:03:28 +02:00
Eugen Rochko
7ff3c849d1 Bot nameplates (#7391)
* Store actor type in database

* Add bot nameplate to web UI, add setting to preferences, API, AP
Fix #7365

* Fix code style issues
2018-05-07 09:31:07 +02:00
David Yip
087c79669a Merge remote-tracking branch 'origin/master' into gs-master
Conflicts:
 	app/controllers/follower_accounts_controller.rb
 	app/controllers/following_accounts_controller.rb
    	db/schema.rb
2018-05-04 14:16:30 -05:00
Eugen Rochko
154076e8e7 Store URIs of follows, follow requests and blocks for ActivityPub (#7160)
Same URI passed between follow request and follow, since they are
the same thing in ActivityPub. Local URIs are generated during
creation using UUIDs and are passed to serializers.
2018-05-04 21:14:34 +02:00
David Yip
48f5f736a0 db: make schema.rb version subsume bookmarks migration. 2018-04-18 19:08:35 -05:00
David Yip
9697c6d9f0 Merge remote-tracking branch 'origin/master' into gs-master
Conflicts:
 	db/schema.rb
2018-04-14 09:12:50 -05:00
Eugen Rochko
466442b764 Add bio fields (#6645)
* Add bio fields

- Fix #3211
- Fix #232
- Fix #121

* Display bio fields in web UI

* Fix output of links and missing fields

* Federate bio fields over ActivityPub as PropertyValue

* Improve how the fields are stored, add to Edit profile form

* Add rel=me to links in fields

Fix #121
2018-04-14 12:41:08 +02:00
David Yip
26f770cf72 Merge remote-tracking branch 'glitchsoc/master' into gs-master
Conflicts:
 	db/schema.rb
2018-04-14 03:50:31 -05:00
David Yip
fd98bfd108 Merge remote-tracking branch 'origin/master' into gs-master
Conflicts:
 	Gemfile.lock
 	config/application.rb
2018-04-13 16:36:46 -05:00
Yamagishi Kazutoshi
9761b940ac Upgrade Rails to version 5.2.0 (#5898) 2018-04-12 14:45:17 +02:00
Thibaut Girka
12f8473561 Add backend support for bookmarks
Bookmarks behave like favourites, except they aren't shared with other
users and do not have an associated counter.
2018-04-11 20:40:29 +02:00
Jenkins
62ce70f28c Merge remote-tracking branch 'tootsuite/master' into glitchsoc/master 2018-04-03 19:17:20 +00:00
Emelia Smith
5373b138e8 Feature: Report improvements (#6967) (#7000)
* Implement Assignment of Reports (#6967)

* Change translation of admin.report.comment.label to "Report Comment" for clarity

As we'll soon add the ability for reports to have comments on them, this clarification makes sense.

* Implement notes for Reports

This enables moderators to leave comments about a report whilst they work on it

* Fix display of report moderation notes

* Allow reports to be reopened / marked as unresolved

* Redirect to reports listing upon resolution of report

* Implement "resolve with note" functionality

* Add inverse relationship for report notes

* Remove additional database querying when loading report notes

* Fix tests for reports

* Fix localisations for report notes / reports
2018-04-02 22:04:14 +02:00
Jenkins
ae55717f50 Merge remote-tracking branch 'tootsuite/master' into glitchsoc/master 2018-03-25 15:17:21 +00:00
Akihiko Odaki
472fba6771 Change columns in notifications nonnullable (#6764) 2018-03-24 12:51:28 +01:00
Jenkins
54608d0486 Merge remote-tracking branch 'tootsuite/master' into glitchsoc/master 2018-03-09 00:17:17 +00:00
Eugen Rochko
b1f8dfd3cf Federate pinned statuses over ActivityPub (#6610)
* Federate pinned statuses over ActivityPub

* Display pinned toots in web UI

Fix #6117

* Fix migration

* Fix tests

* Update outbox_serializer.rb

* Update remove_serializer.rb

* Update add_serializer.rb

* Update fetch_featured_collection_service.rb
2018-03-04 09:19:11 +01:00
David Yip
c08c971dd3 Merge remote-tracking branch 'origin/master' into merge-upstream
Conflicts:
 	README.md
 	app/controllers/follower_accounts_controller.rb
 	app/controllers/following_accounts_controller.rb
 	app/serializers/rest/instance_serializer.rb
 	app/views/stream_entries/_simple_status.html.haml
 	config/locales/simple_form.ja.yml
2018-03-02 21:46:44 -06:00
Eugen Rochko
0ebd2a1718 Federated reports (#6570)
* Fix #2176: Federated reports

* UI for federated reports

* Add spec for ActivityPub Flag handler

* Add spec for ReportService
2018-02-28 06:54:55 +01:00
imncls
c0aabbec0f Merge branch 'master' of https://github.com/tootsuite/mastodon
# Conflicts:
#	app/controllers/settings/exports_controller.rb
#	app/models/media_attachment.rb
#	app/models/status.rb
#	app/views/about/show.html.haml
#	docker_entrypoint.sh
#	spec/views/about/show.html.haml_spec.rb
2018-02-23 23:28:31 +09:00
Eugen Rochko
f6884555d7 Account archive download (#6460)
* Fix #201: Account archive download

* Export actor and private key in the archive

* Optimize BackupService

- Add conversation to cached associations of status, because
  somehow it was forgotten and is source of N+1 queries
- Explicitly call GC between batches of records being fetched
  (Model class allocations are the worst offender)
- Stream media files into the tar in 1MB chunks
  (Do not allocate media file (up to 8MB) as string into memory)
- Use #bytesize instead of #size to calculate file size for JSON
  (Fix FileOverflow error)
- Segment media into subfolders by status ID because apparently
  GIF-to-MP4 media are all named "media.mp4" for some reason

* Keep uniquely generated filename in Paperclip::GifTranscoder

* Ensure dumped files do not overwrite each other by maintaing directory partitions

* Give tar archives a good name

* Add scheduler to remove week-old backups

* Fix code style issue
2018-02-21 23:21:32 +01:00
Jenkins
777559fa5f Merge remote-tracking branch 'tootsuite/master' into glitchsoc/master 2018-02-08 23:17:14 +00:00
Akihiko Odaki
72e284028f Change user_id column non-nullable (#6435) 2018-02-07 16:35:44 +01:00
David Yip
2b43b1ab6b db/schema.rb: resolve erroneously-unresolved conflict 2018-02-04 17:04:22 -06:00
David Yip
4f8122a98c Merge remote-tracking branch 'origin/master' into merge-upstream
Conflicts:
	.env.production.sample
	app/controllers/auth/confirmations_controller.rb
	db/schema.rb
2018-02-04 16:36:19 -06:00
Eugen Rochko
5322013f25 CAS + SAML authentication feature (#6425)
* Cas authentication feature

* Config

* Remove class_eval + Omniauth initializer

* Codeclimate review

* Codeclimate review 2

* Codeclimate review 3

* Remove uid/email reconciliation

* SAML authentication

* Clean up code

* Improve login form

* Fix code style issues

* Add locales
2018-02-04 05:42:13 +01:00
David Yip
6d1023b2e9 Merge remote-tracking branch 'tootsuite/master' into merge-upstream
Conflicts:
      app/javascript/styles/mastodon/components.scss
2018-02-02 08:39:52 -06:00
Alexander
23ce0c86da pam authentication (#5303)
* add pam support, without extra column

* bugfixes for pam login

* document options

* fix code style

* fix codestyle

* fix tests

* don't call remember_me without password

* fix codestyle

* improve checks for pam usage (should fix tests)

* fix remember_me part 1

* add remember_token column because :rememberable requires either a password or this column.

* migrate db for remember_token

* move pam_authentication to the right place, fix logic bug in edit.html.haml

* fix tests

* fix pam authentication, improve username lookup, add comment

* valid? is sometimes not honored, return nil instead trying to authenticate with pam

* update devise_pam_authenticatable2 and adjust code. Fixes sideeffects observed in tests

* update devise_pam_authenticatable gem, fixes for codeconventions, fix finding user

* codeconvention fixes

* code convention fixes

* fix idention

* update dependency, explicit conflict check

* fix disabled password updates if in pam mode

* fix check password if password is present, fix templates

* block registration if account is maintained by pam

* Revert "block registration if account is maintained by pam"

This reverts commit 8e7a083d650240b6fac414926744b4b90b435f20.

* fix identation error introduced by rebase

* block usernames maintained by pam

* document pam settings better

* fix code style
2018-02-02 10:18:55 +01:00
David Yip
1a7f8eb723 Merge remote-tracking branch 'origin/master' into merge-upstream
Conflicts:
	db/schema.rb
2018-01-09 14:16:45 -06:00
takayamaki
9fb57f8583 add index on statuses for /api/v1/accounts/:account_id/statuses (#6202) 2018-01-09 15:00:19 +01:00
Jenkins
0a2cc2361c Merge remote-tracking branch 'tootsuite/master' into glitchsoc/master 2017-12-26 18:17:16 +00:00
takayamaki
2a2ac6c744 more faster index on notifications table (#6108) 2017-12-26 17:56:31 +01:00
David Yip
d53f727653 Merge branch 'gs-master' into prevent-local-only-federation
Conflicts:
	db/schema.rb
2017-12-15 12:20:56 -06:00
Jenkins
1366e96a02 Merge remote-tracking branch 'tootsuite/master' into glitchsoc/master 2017-12-13 18:17:16 +00:00
abcang
a40aff8dbc Remove duplicate indexes in lists (#5990) 2017-12-12 17:38:42 +01:00
David Yip
a56c9ac5dc Merge remote-tracking branch 'tootsuite/master' into merge-upstream 2017-12-12 02:54:13 -06:00
Akihiko Odaki
a90fa13235 Change account_id non-nullable in lists (#5979) 2017-12-12 04:11:17 +01:00
Erin
5eadff9883 add a local_only column to the statuses table 2017-12-10 16:41:25 -06:00
Jenkins
099159741b Merge remote-tracking branch 'tootsuite/master' into glitchsoc/master 2017-12-07 04:17:14 +00:00
Akihiko Odaki
ce3989fc6a Add embed_url to preview cards (#5775) 2017-12-07 03:37:43 +01:00
Jenkins
a7a59f927d Merge remote-tracking branch 'tootsuite/master' into glitchsoc/master 2017-11-30 03:17:12 +00:00
takayamaki
dfceb51d93 add index on stream_entries table (#5793) 2017-11-30 03:35:54 +01:00
aschmitz
ce341c7cf6 Allow hiding of reblogs from followed users (#5762)
* Allow hiding of reblogs from followed users

This adds a new entry to the account menu to allow users to hide
future reblogs from a user (and then if they've done that, to show
future reblogs instead).

This does not remove or add historical reblogs from/to the user's
timeline; it only affects new statuses.

The API for this operates by sending a "reblogs" key to the follow
endpoint. If this is sent when starting a new follow, it will be
respected from the beginning of the follow relationship (even if
the follow request must be approved by the followee). If this is
sent when a follow relationship already exists, it will simply
update the existing follow relationship. As with the notification
muting, this will now return an object ({reblogs: [true|false]}) or
false for each follow relationship when requesting relationship
information for an account. This should cause few issues due to an
object being truthy in many languages, but some modifications may
need to be made in pickier languages.

Database changes: adds a show_reblogs column (default true,
non-nullable) to the follows and follow_requests tables. Because
these are non-nullable, we use the existing MigrationHelpers to
perform this change without locking those tables, although the
tables are likely to be small anyway.

Tests included.

See also <https://github.com/glitch-soc/mastodon/pull/212>.

* Rubocop fixes

* Code review changes

* Test fixes

This patchset closes #648 and resolves #3271.

* Rubocop fix

* Revert reblogs defaulting in argument, fix tests

It turns out we needed this for the same reason we needed it in muting:
if nil gets passed in somehow (most usually by an API client not passing
any value), we need to detect and handle it.

We could specify a default in the parameter and then also catch nil, but
there's no great reason to duplicate the default value.
2017-11-28 15:00:35 +01:00
Jenkins
36452d6127 Merge remote-tracking branch 'tootsuite/master' into glitchsoc/master 2017-11-27 20:17:12 +00:00
unarist
0d0ad983b7 Merge indexes for reblog on statuses table (#5831)
We added an index for `[account_id, reblog_of_id]`, but we already have a similar index for `reblog_of_id`. Those index will be bigger according to statuses count. For example, `reblog_of_id` index uses 800MB for 10GB statuses table.

So this patch swaps indexed columns like `[reblog_of_id, account_id]`, then it will covers both usage with single index.

Since those index creation may take a while, I've also disabled previous index creation.
2017-11-27 20:22:27 +01:00
David Yip
c9dd67a733 Merge remote-tracking branch 'origin/master' into gs-master 2017-11-27 11:29:04 -06:00
Eugen Rochko
bc8c951083 Add consumable invites (#5814)
* Add consumable invites

* Add UI for generating invite codes

* Add tests

* Display max uses and expiration in invites table, delete invite

* Remove unused column and redundant validator

- Default follows not used, probably bad idea
- InviteCodeValidator is redundant because RegistrationsController
  checks invite code validity

* Add admin setting to disable invites

* Add admin UI for invites, configurable role for invite creation

- Admin UI that lists everyone's invites, always available
- Admin setting min_invite_role to control who can invite people
- Non-admin invite UI only visible if users are allowed to

* Do not remove invites from database, expire them instantly
2017-11-27 16:07:59 +01:00
Jenkins
7fcbe5faee Merge remote-tracking branch 'tootsuite/master' into glitchsoc/master 2017-11-25 05:17:15 +00:00
abcang
17bde15f53 Add index of account and reblog to statuses (#5785) 2017-11-24 14:42:09 +01:00
Jenkins
359ba16721 Merge remote-tracking branch 'tootsuite/master' into glitchsoc/master 2017-11-24 01:17:20 +00:00
Eugen Rochko
efc97d404c Add logging of admin actions (#5757)
* Add logging of admin actions

* Update brakeman whitelist

* Log creates, updates and destroys with history of changes

* i18n: Update Polish translation (#5782)

Signed-off-by: Marcin Mikołajczak <me@m4sk.in>

* Split admin navigation into moderation and administration

* Redesign audit log page

* 🇵🇱 (#5795)

* Add color coding to audit log

* Change dismiss->resolve, log all outcomes of report as resolve

* Update terminology (e-mail blacklist) (#5796)

* Update terminology (e-mail blacklist)

imho looks better

* Update en.yml

* Fix code style issues

* i18n-tasks normalize
2017-11-24 02:05:53 +01:00
David Yip
1dc24ece41 Merge remote-tracking branch 'tootsuite/master' 2017-11-19 01:30:29 -06:00
Eugen Rochko
b89f614bd7 Profile redirect notes (#5746)
* Serialize moved accounts into REST and ActivityPub APIs

* Parse federated moved accounts from ActivityPub

* Add note about moved accounts to public profiles

* Add moved account message to web UI

* Fix code style issues
2017-11-18 19:39:02 +01:00
David Yip
08751360a0 Update annotations on Follow, FollowRequest, and Mute.
Follow and FollowRequest had conflicts in their schema annotations, so I
ran latest migrations and let annotate_models fix them up.
2017-11-17 17:41:15 -06:00
David Yip
f1279d8ff3 Merge remote-tracking branch 'origin/master' into merge-upstream 2017-11-17 17:40:00 -06:00
Eugen Rochko
e149067561 Lists (#5703)
* Add structure for lists

* Add list timeline streaming API

* Add list APIs, bind list-account relation to follow relation

* Add API for adding/removing accounts from lists

* Add pagination to lists API

* Add pagination to list accounts API

* Adjust scopes for new APIs

- Creating and modifying lists merely requires "write" scope
- Fetching information about lists merely requires "read" scope

* Add test for wrong user context on list timeline

* Clean up tests
2017-11-18 00:16:48 +01:00
Surinna Curtis
02ae609d9a Merge tootsuite/master at ab48f2b239 2017-11-16 01:21:16 -06:00
Surinna Curtis
582dd1b7c7 Optional notification muting (#5087)
* Add a hide_notifications column to mutes

* Add muting_notifications? and a notifications argument to mute!

* block notifications in notify_service from hard muted accounts

* Add specs for how mute! interacts with muting_notifications?

* specs testing that hide_notifications in mutes actually hides notifications

* Add support for muting notifications in MuteService

* API support for muting notifications (and specs)

* Less gross passing of notifications flag

* Break out a separate mute modal with a hide-notifications checkbox.

* Convert profile header mute to use mute modal

* Satisfy eslint.

* specs for MuteService notifications params

* add trailing newlines to files for Pork :)

* Put the label for the hide notifications checkbox in a label element.

* Add a /api/v1/mutes/details route that just returns the array of mutes.

* Define a serializer for /api/v1/mutes/details

* Add more specs for the /api/v1/mutes/details endpoint

* Expose whether a mute hides notifications in the api/v1/relationships endpoint

* Show whether muted users' notifications are muted in account lists

* Allow modifying the hide_notifications of a mute with the /api/v1/accounts/:id/mute endpoint

* make the hide/unhide notifications buttons work

* satisfy eslint

* In probably dead code, replace a dispatch of muteAccount that was skipping the modal with launching the mute modal.

* fix a missing import

* add an explanatory comment to AccountInteractions

* Refactor handling of default params for muting to make code cleaner

* minor code style fixes oops

* Fixed a typo that was breaking the account mute API endpoint

* Apply white-space: nowrap to account relationships icons

* Fix code style issues

* Remove superfluous blank line

* Rename /api/v1/mutes/details -> /api/v2/mutes

* Don't serialize "account" in MuteSerializer

Doing so is somewhat unnecessary since it's always the current user's account.

* Fix wrong variable name in api/v2/mutes

* Use Toggle in place of checkbox in the mute modal.

* Make the Toggle in the mute modal look better

* Code style changes in specs and removed an extra space

* Code review suggestions from akihikodaki

Also fixed a syntax error in tests for AccountInteractions.

* Make AddHideNotificationsToMute Concurrent

It's not clear how much this will benefit instances in practice, as the
number of mutes tends to be pretty small, but this should prevent any
blocking migrations nonetheless.

* Fix up migration things

* Remove /api/v2/mutes
2017-11-15 03:56:41 +01:00
Yamagishi Kazutoshi
3a51544e9a Add uniqueness to block email domains (#5692) 2017-11-14 20:37:17 +01:00
Eugen Rochko
97178b6d86 Add moderator role and add pundit policies for admin actions (#5635)
* Add moderator role and add pundit policies for admin actions

* Add rake task for turning user into mod and revoking it again

* Fix handling of unauthorized exception

* Deliver new report e-mails to staff, not just admins

* Add promote/demote to admin UI, hide some actions conditionally

* Fix unused i18n
2017-11-11 20:23:33 +01:00
aschmitz
2aedd8eaad "Show reblogs" per-follower UI/database changes
TODO:

* Tests (particularly for FollowRequests).
* Anything to respect the setting when putting reblogs in timelines.
2017-11-09 08:41:10 -06:00
Eugen Rochko
7f1a0241c1 Add ability to disable login and mark accounts as memorial (#5615)
Fix #5597
2017-11-07 19:06:44 +01:00
David Yip
2aec3c073b Merge branch 'master' into gs-master 2017-10-27 09:45:25 -05:00
nullkal
0a9cc96cdc Feature: Unlisted custom emojis (#5485) 2017-10-27 16:11:30 +02:00
David Yip
74141adca1 Move KeywordMute into Glitch namespace.
There are two motivations for this:

1. It looks like we're going to add other features that require
   server-side storage (e.g. user notes).

2. Namespacing glitchsoc modifications is a good idea anyway: even if we
   do not end up doing (1), if upstream introduces a keyword-mute feature
   that also uses a "KeywordMute" model, we can avoid some merge
   conflicts this way and work on the more interesting task of
   choosing which implementation to use.
2017-10-21 14:54:36 -05:00
David Yip
05ee0aeb8b Allow keywords to match either substrings or whole words.
Word-boundary matching only works as intended in English and languages
that use similar word-breaking characters; it doesn't work so well in
(say) Japanese, Chinese, or Thai.  It's unacceptable to have a feature
that doesn't work as intended for some languages.  (Moreso especially
considering that it's likely that the largest contingent on the Mastodon
bit of the fediverse speaks Japanese.)

There are rules specified in Unicode TR29[1] for word-breaking across
all languages supported by Unicode, but the rules deliberately do not
cover all cases.  In fact, TR29 states

    For example, reliable detection of word boundaries in languages such
    as Thai, Lao, Chinese, or Japanese requires the use of dictionary
    lookup, analogous to English hyphenation.

So we aren't going to be able to make word detection work with regexes
within Mastodon (or glitchsoc).  However, for a first pass (even if it's
kind of punting) we can allow the user to choose whether they want word
or substring detection and warn about the limitations of this
implementation in, say, docs.

[1]: https://unicode.org/reports/tr29/
     https://web.archive.org/web/20171001005125/https://unicode.org/reports/tr29/
2017-10-21 14:54:36 -05:00
David Yip
c123b710ad Add KeywordMute model.
Gist of the proposed keyword mute implementation:

Keyword mutes are represented server-side as one keyword per record.
For each account, there exists a keyword regex that is generated as one
big alternation of all keywords.  This regex is cached (in Redis, I
guess) so we can quickly get it when filtering in FeedManager.
2017-10-21 14:53:41 -05:00
kibigo!
02827345ae Merge upstream 2.0ish #165 2017-10-11 10:43:10 -07:00
nullkal
4a45cc7566 foreign_key, non-nullable, dependent: destroy in account_moderation_notes (#5294)
* Add foreign key constraint to column `account` in `account_moderation_notes`

* Change account_id and target_account_id to non-nullable in account_moderation_notes

* Add dependent: :destroy to account and target_account in account_moderation_notes
2017-10-10 13:12:17 +02:00
nullkal
81489970fb Add moderation note (#5240)
* Add moderation note

* Add frozen_string_literal

* Make rspec pass
2017-10-07 20:26:43 +02:00
Eugen Rochko
60925ce0ae Encode custom emojis as resolveable objects in ActivityPub (#5243)
* Encode custom emojis as resolveable objects in ActivityPub

* Improve code style
2017-10-07 17:43:42 +02:00
Eugen Rochko
b4af50c521 Improve admin UI for custom emojis, add copy/disable/enable (#5231) 2017-10-05 23:42:05 +02:00
utam0k
51f8cbfbeb Implement EmailBlackList (#5109)
* Implement BlacklistedEmailDomain

* Use Faker::Internet.domain_name

* Remove note column

* Add frozen_string_literal comment

* Delete unnecessary codes

* Sort alphabetically

* Change of wording

* Rename BlacklistedEmailDomain to EmailDomainBlock
2017-10-04 15:16:10 +02:00
aschmitz
5b2d855d86 Non-Serial ("Snowflake") IDs (#4801)
* Use non-serial IDs

This change makes a number of nontrivial tweaks to the data model in
Mastodon:

* All IDs are now 8 byte integers (rather than mixed 4- and 8-byte)
* IDs are now assigned as:
  * Top 6 bytes: millisecond-resolution time from epoch
  * Bottom 2 bytes: serial (within the millisecond) sequence number
  * See /lib/tasks/db.rake's `define_timestamp_id` for details, but
    note that the purpose of these changes is to make it difficult to
    determine the number of objects in a table from the ID of any
    object.
* The Redis sorted set used for the feed will have values used to look
  up toots, rather than scores. This is almost always the same as the
  existing behavior, except in the case of boosted toots. This change
  was made because Redis stores scores as double-precision floats,
  which cannot store the new ID format exactly. Note that this doesn't
  cause problems with sorting/pagination, because ZREVRANGEBYSCORE
  sorts lexicographically when scores are tied. (This will still cause
  sorting issues when the ID gains a new significant digit, but that's
  extraordinarily uncommon.)

Note a couple of tradeoffs have been made in this commit:

* lib/tasks/db.rake is used to enforce many/most column constraints,
  because this commit seems likely to take a while to bring upstream.
  Enforcing a post-migrate hook is an easier way to maintain the code
  in the interim.
* Boosted toots will appear in the timeline as many times as they have
  been boosted. This is a tradeoff due to the way the feed is saved in
  Redis at the moment, but will be handled by a future commit.

This would effectively close Mastodon's #1059, as it is a
snowflake-like system of generating IDs. However, given how involved
the changes were simply within Mastodon, it may have unexpected
interactions with some clients, if they store IDs as doubles
(or as 4-byte integers). This was a problem that Twitter ran into with
their "snowflake" transition, particularly in JavaScript clients that
treated IDs as JS integers, rather than strings. It therefore would be
useful to test these changes at least in the web interface and popular
clients before pushing them to all users.

* Fix JavaScript interface with long IDs

Somewhat predictably, the JS interface handled IDs as numbers, which in
JS are IEEE double-precision floats. This loses some precision when
working with numbers as large as those generated by the new ID scheme,
so we instead handle them here as strings. This is relatively simple,
and doesn't appear to have caused any problems, but should definitely
be tested more thoroughly than the built-in tests. Several days of use
appear to support this working properly.

BREAKING CHANGE:

The major(!) change here is that IDs are now returned as strings by the
REST endpoints, rather than as integers. In practice, relatively few
changes were required to make the existing JS UI work with this change,
but it will likely hit API clients pretty hard: it's an entirely
different type to consume. (The one API client I tested, Tusky, handles
this with no problems, however.)

Twitter ran into this issue when introducing Snowflake IDs, and decided
to instead introduce an `id_str` field in JSON responses. I have opted
to *not* do that, and instead force all IDs to 64-bit integers
represented by strings in one go. (I believe Twitter exacerbated their
problem by rolling out the changes three times: once for statuses, once
for DMs, and once for user IDs, as well as by leaving an integer ID
value in JSON. As they said, "If you’re using the `id` field with JSON
in a Javascript-related language, there is a very high likelihood that
the integers will be silently munged by Javascript interpreters. In most
cases, this will result in behavior such as being unable to load or
delete a specific direct message, because the ID you're sending to the
API is different than the actual identifier associated with the
message." [1]) However, given that this is a significant change for API
users, alternatives or a transition time may be appropriate.

1: https://blog.twitter.com/developer/en_us/a/2011/direct-messages-going-snowflake-on-sep-30-2011.html

* Restructure feed pushes/unpushes

This was necessary because the previous behavior used Redis zset scores
to identify statuses, but those are IEEE double-precision floats, so we
can't actually use them to identify all 64-bit IDs. However, it leaves
the code in a much better state for refactoring reblog handling /
coalescing.

Feed-management code has been consolidated in FeedManager, including:

* BatchedRemoveStatusService no longer directly manipulates feed zsets
* RemoveStatusService no longer directly manipulates feed zsets
* PrecomputeFeedService has moved its logic to FeedManager#populate_feed

(PrecomputeFeedService largely made lots of calls to FeedManager, but
didn't follow the normal adding-to-feed process.)

This has the effect of unifying all of the feed push/unpush logic in
FeedManager, making it much more tractable to update it in the future.

Due to some additional checks that must be made during, for example,
batch status removals, some Redis pipelining has been removed. It does
not appear that this should cause significantly increased load, but if
necessary, some optimizations are possible in batch cases. These were
omitted in the pursuit of simplicity, but a batch_push and batch_unpush
would be possible in the future.

Tests were added to verify that pushes happen under expected conditions,
and to verify reblog behavior (both on pushing and unpushing). In the
case of unpushing, this includes testing behavior that currently leads
to confusion such as Mastodon's #2817, but this codifies that the
behavior is currently expected.

* Rubocop fixes

I could swear I made these changes already, but I must have lost them
somewhere along the line.

* Address review comments

This addresses the first two comments from review of this feature:

https://github.com/tootsuite/mastodon/pull/4801#discussion_r139336735
https://github.com/tootsuite/mastodon/pull/4801#discussion_r139336931

This adds an optional argument to FeedManager#key, the subtype of feed
key to generate. It also tests to ensure that FeedManager's settings are
such that reblogs won't be tracked forever.

* Hardcode IdToBigints migration columns

This addresses a comment during review:
https://github.com/tootsuite/mastodon/pull/4801#discussion_r139337452

This means we'll need to make sure that all _id columns going forward
are bigints, but that should happen automatically in most cases.

* Additional fixes for stringified IDs in JSON

These should be the last two. These were identified using eslint to try
to identify any plain casts to JavaScript numbers. (Some such casts are
legitimate, but these were not.)

Adding the following to .eslintrc.yml will identify casts to numbers:

~~~
  no-restricted-syntax:
  - warn
  - selector: UnaryExpression[operator='+'] > :not(Literal)
    message: Avoid the use of unary +
  - selector: CallExpression[callee.name='Number']
    message: Casting with Number() may coerce string IDs to numbers
~~~

The remaining three casts appear legitimate: two casts to array indices,
one in a server to turn an environment variable into a number.

* Only implement timestamp IDs for Status IDs

Per discussion in #4801, this is only being merged in for Status IDs at
this point. We do this in a migration, as there is no longer use for
a post-migration hook. We keep the initialization of the timestamp_id
function as a Rake task, as it is also needed after db:schema:load (as
db/schema.rb doesn't store Postgres functions).

* Change internal streaming payloads to stringified IDs as well

This is equivalent to 591a9af356faf2d5c7e66e3ec715502796c875cd from
#5019, with an extra change for the addition to FeedManager#unpush.

* Ensure we have a status_id_seq sequence

Apparently this is not a given when specifying a custom ID function,
so now we ensure it gets created. This uses the generic version of this
function to more easily support adding additional tables with timestamp
IDs in the future, although it would be possible to cut this down to a
less generic version if necessary. It is only run during db:schema:load
or the relevant migration, so the overhead is extraordinarily minimal.

* Transition reblogs to new Redis format

This provides a one-way migration to transition old Redis reblog entries
into the new format, with a separate tracking entry for reblogs.

It is not invertible because doing so could (if timestamp IDs are used)
require a database query for each status in each users' feed, which is
likely to be a significant toll on major instances.

* Address review comments from @akihikodaki

No functional changes.

* Additional review changes

* Heredoc cleanup

* Run db:schema:load hooks for test in development

This matches the behavior in Rails'
ActiveRecord::Tasks::DatabaseTasks.each_current_configuration, which
would otherwise break `rake db:setup` in development.

It also moves some functionality out to a library, which will be a good
place to put additional related functionality in the near future.
2017-10-04 09:56:37 +02:00
aschmitz
04fde500e6 Make IdsToBigints (mostly!) non-blocking (#5088)
* Make IdsToBigints (mostly!) non-blocking

This pulls in GitLab's MigrationHelpers, which include code to make
column changes in ways that Postgres can do without locking. In general,
this involves creating a new column, adding an index and any foreign
keys as appropriate, adding a trigger to keep it populated alongside
the old column, and then progressively copying data over to the new
column, before removing the old column and replacing it with the new
one.

A few changes to GitLab's MigrationHelpers were necessary:

* Some changes were made to remove dependencies on other GitLab code.
* We explicitly wait for index creation before forging ahead on column
  replacements.
* We use different temporary column names, to avoid running into index
  name length limits.
* We rename the generated indices back to what they "should" be after
  replacing columns.
* We rename the generated foreign keys to use the new column names when
  we had to create them. (This allows the migration to be rolled back
  without incident.)

# Big Scary Warning

There are two things here that may trip up large instances:

1. The change for tables' "id" columns is not concurrent. In
   particular, the stream_entries table may be big, and does not
   concurrently migrate its id column. (On the other hand, x_id type
   columns are all concurrent.)
2. This migration will take a long time to run, *but it should not
   lock tables during that time* (with the exception of the "id"
   columns as described above). That means this should probably be run
   in `screen` or some other session that can be run for a long time.
   Notably, the migration will take *longer* than it would without
   these changes, but the website will still be responsive during that
   time.

These changes were tested on a relatively large statuses table (256k
entries), and the service remained responsive during the migration.
Migrations both forward and backward were tested.

* Rubocop fixes

* MigrationHelpers: Support ID columns in some cases

This doesn't work in cases where the ID column is referred to as a
foreign key by another table.

* MigrationHelpers: support foreign keys for ID cols

Note that this does not yet support foreign keys on non-primary-key
columns, but Mastodon also doesn't yet have any that we've needed to
migrate.

This means we can perform fully "concurrent" migrations to change ID
column types, and the IdsToBigints migration can happen with effectively
no downtime. (A few operations require a transaction, such as renaming
columns or deleting them, but these transactions should not block for
noticeable amounts of time.)

The algorithm for generating foreign key names has changed with this,
and therefore all of those changed in schema.rb.

* Provide status, allow for interruptions

The MigrationHelpers now allow restarting the rename of a column if it
was interrupted, by removing the old "new column" and re-starting the
process.

Along with this, they now provide status updates on the changes which
are happening, as well as indications about when the changes can be
safely interrupted (when there are at least 10 seconds estimated to be
left before copying data is complete).

The IdsToBigints migration now also sorts the columns it migrates by
size, starting with the largest tables. This should provide
administrators a worst-case scenario estimate for the length of
migrations: each successive change will get faster, giving admins a
chance to abort early on if they need to run the migration later. The
idea is that this does not force them to try to time interruptions
between smaller migrations.

* Fix column sorting in IdsToBigints

Not a significant change, but it impacts the order of columns in the
database and db/schema.rb.

* Actually pause before IdsToBigints
2017-10-02 21:28:59 +02:00
Eugen Rochko
795e624d5d Add ability to specify alternative text for media attachments (#5123)
* Fix #117 - Add ability to specify alternative text for media attachments

- POST /api/v1/media accepts `description` straight away
- PUT /api/v1/media/:id to update `description` (only for unattached ones)
- Serialized as `name` of Document object in ActivityPub
- Uploads form adjusted for better performance and description input

* Add tests

* Change undo button blend mode to difference
2017-09-28 15:31:31 +02:00
Ondřej Hruška
0cc795db6b Merge commit '9ab3021562565eeee27820438f32b764414f5ab1' into merging-upstream 2017-09-28 09:18:35 +02:00
Ondřej Hruška
914c0465b1 Merge commit '57c5493d4e7a42ad9e8fec20d71c20bee8674287' into merging-upstream 2017-09-28 09:12:17 +02:00
Eugen Rochko
2e342947ce Forgotten in #5039, change statuses_tags tag_id to bigint (#5070) 2017-09-24 05:58:07 +02:00
aschmitz
f79569ba99 Hardcode IdToBigints migration columns (#5039)
This addresses a comment during review:
https://github.com/tootsuite/mastodon/pull/4801#discussion_r139337452

This means we'll need to make sure that all _id columns going forward
are bigints, but that should happen automatically in most cases.
2017-09-22 13:20:04 +02:00
Eugen Rochko
eb345f8e8b Custom emoji (#4988)
* Custom emoji

- In OStatus: `<link rel="emoji" name="coolcat" href="http://..." />`
- In ActivityPub: `{ type: "Emoji", name: ":coolcat:", href: "http://..." }`
- In REST API: Status object includes `emojis` array (`shortcode`, `url`)
- Domain blocks with reject media stop emojis
- Emoji file up to 50KB
- Web UI handles custom emojis
- Static pages render custom emojis as `<img />` tags

Side effects:

- Undo #4500 optimization, as I needed to modify it to restore
  shortcode handling in emojify()
- Formatter#plaintext should now make sure stripped out line-breaks
  and paragraphs are replaced with newlines

* Fix emoji at the start not being converted
2017-09-19 02:42:40 +02:00
Surinna Curtis
5c5416aa7a fix typos in the migration 2017-09-13 22:35:48 -05:00
Surinna Curtis
0d5ce9d8e6 Add muting_notifications? and a notifications argument to mute! 2017-09-13 21:47:30 -05:00
Eugen Rochko
9ff429aa6d Uploads for admin site settings (#4913)
* Improve OpenGraph tags for about pages

* Add thumbnail admin setting

* Fix error

* Fix up
2017-09-14 00:04:30 +02:00
Eugen Rochko
5ad3153b0b Fix locking migration on statuses table. Nullable column and NO default value (#4825) 2017-09-06 20:57:52 +02:00
Eugen Rochko
73c613dcfc Switch to static URIs, new URI format in both protocols for new statuses (#4815)
* Decouple Status#local? from uri being nil

* Replace on-the-fly URI generation with stored URIs

- Generate URI in after_save hook for local statuses
- Use static value in TagManager when available, fallback to tag format
- Make TagManager use ActivityPub::TagManager to understand new format
- Adjust tests

* Use other heuristic for locality of old statuses, do not perform long query

* Exclude tombstone stream entries from Atom feed

* Prevent nil statuses from landing in Pubsubhubbub::DistributionWorker

* Fix URI not being saved (#4818)

* Add more specs for Status

* Save generated uri immediately

and also fix method order to minimize diff.

* Fix alternate HTML URL in Atom

* Fix tests

* Remove not-null constraint from statuses migration to speed it up
2017-09-06 19:01:28 +02:00
takayamaki
1ecae86d4f add index_notifications_on_id_and_account_id_and_activity_type on notifications table (#4750) 2017-09-05 12:30:01 +02:00
Eugen Rochko
e9e271878e Make PreviewCard records reuseable between statuses (#4642)
* Make PreviewCard records reuseable between statuses

**Warning!** Migration truncates preview_cards tablec

* Allow a wider thumbnail for link preview, display it in horizontal layout (#4648)

* Delete preview cards files before truncating

* Rename old table instead of truncating it

* Add mastodon:maintenance:remove_deprecated_preview_cards

* Ignore deprecated_preview_cards in schema definition

* Fix null behaviour
2017-09-01 16:20:16 +02:00
Eugen Rochko
86ffdf6083 Remove unneccesary indices (#4738)
We only look up status_pins by account_id, or account_id and status_id,
never by status_id
2017-08-30 05:04:20 +02:00
nullkal
f9ebff4b13 Allow multiple pinned statuses to be shown and make them be ordered b… (#4690)
* Allow multiple pinned statuses to be shown and make them be ordered by pinned date

* Set timestamps NOT NULL

* Make single-line pinned_statuses

* Spec for pinned_statuses

* Remove redundant empty line
2017-08-25 18:50:52 +02:00
Eugen Rochko
3e20cb7025 Pinned statuses (#4675)
* Pinned statuses

* yarn manage:translations
2017-08-25 01:41:18 +02:00
Eugen Rochko
b13501268f Fix up the applications area (#4664)
- Section it into "Development" area
- Improve UI of application form, index, and details
2017-08-23 00:59:35 +02:00
Colin Mitchell
63b98318c5 Application prefs section (#2758)
* Add code for creating/managing apps to settings section

* Add specs for app changes

* Fix controller spec

* Fix view file I pasted over by mistake

* Add locale strings. Add 'my apps' to nav

* Add Client ID/Secret to App page. Add some visual separation

* Fix rubocop warnings

* Fix embarrassing typo

I lost an `end` statement while fixing a merge conflict.

* Add code for creating/managing apps to settings section

- Add specs for app changes
- Add locale strings. Add 'my apps' to nav
- Add Client ID/Secret to App page. Add some visual separation
- Fix some bugs/warnings

* Update to match code standards

* Trigger notification

* Add warning about not sharing API secrets

* Tweak spec a bit

* Cleanup fixture creation by using let!

* Remove unused key

* Add foreign key for application<->user
2017-08-22 18:33:57 +02:00
Akihiko Odaki
fd4c92f105 Add index favourites on account_id and id (#4360) 2017-07-26 03:35:25 +02:00
Eugen Rochko
81ebd4025a Add ActivityPub attributes to accounts (#4273) 2017-07-19 17:06:46 +02:00
Eugen Rochko
c6e46f7421 Fix #2672 - Connect signed PuSH subscription requests to instance domain (#4205)
* Fix #2672 - Connect signed PuSH subscription requests to instance domain

Resolves #2739

* Fix return of locate_subscription

* Fix tests
2017-07-14 23:01:20 +02:00
Sorin Davidoi
ecab38fd66 Web Push Notifications (#3243)
* feat: Register push subscription

* feat: Notify when mentioned

* feat: Boost, favourite, reply, follow, follow request

* feat: Notification interaction

* feat: Handle change of public key

* feat: Unsubscribe if things go wrong

* feat: Do not send normal notifications if push is enabled

* feat: Focus client if open

* refactor: Move push logic to WebPushSubscription

* feat: Better title and body

* feat: Localize messages

* chore: Fix lint errors

* feat: Settings

* refactor: Lazy load

* fix: Check if push settings exist

* feat: Device-based preferences

* refactor: Simplify logic

* refactor: Pull request feedback

* refactor: Pull request feedback

* refactor: Create /api/web/push_subscriptions endpoint

* feat: Spec PushSubscriptionController

* refactor: WebPushSubscription => Web::PushSubscription

* feat: Spec Web::PushSubscription

* feat: Display first media attachment

* feat: Support direction

* fix: Stuff broken while rebasing

* refactor: Integration with session activations

* refactor: Cleanup

* refactor: Simplify implementation

* feat: Set VAPID keys via environment

* chore: Comments

* fix: Crash when no alerts

* fix: Set VAPID keys in testing environment

* fix: Follow link

* feat: Notification actions

* fix: Delete previous subscription

* chore: Temporary logs

* refactor: Move migration to a later date

* fix: Fetch the correct session activation and misc bugs

* refactor: Move migration to a later date

* fix: Remove follow request (no notifications)

* feat: Send administrator contact to push service

* feat: Set time-to-live

* fix: Do not show sensitive images

* fix: Reducer crash in error handling

* feat: Add badge

* chore: Fix lint error

* fix: Checkbox label overlap

* fix: Check for payload support

* fix: Rename action "type" (crash in latest Chrome)

* feat: Action to expand notification

* fix: Lint errors

* fix: Unescape notification body

* fix: Do not allow boosting if the status is hidden

* feat: Add VAPID keys to the production sample environment

* fix: Strip HTML tags from status

* refactor: Better error messages

* refactor: Handle browser not implementing the VAPID protocol (Samsung Internet)

* fix: Error when target_status is nil

* fix: Handle lack of image

* fix: Delete reference to invalid subscriptions

* feat: Better error handling

* fix: Unescape HTML characters after tags are striped

* refactor: Simpify code

* fix: Modify to work with #4091

* Sort strings alphabetically

* i18n: Updated Polish translation

it annoys me that it's not fully localized :P

* refactor: Use current_session in PushSubscriptionController

* fix: Rebase mistake

* fix: Set cacheName to mastodon

* refactor: Pull request feedback

* refactor: Remove logging statements

* chore(yarn): Fix conflicts with master

* chore(yarn): Copy latest from master

* chore(yarn): Readd offline-plugin

* refactor: Use save! and update!

* refactor: Send notifications async

* fix: Allow retry when push fails

* fix: Save track for failed pushes

* fix: Minify sw.js

* fix: Remove account_id from fabricator
2017-07-13 22:15:32 +02:00
unarist
ae26d7b557 Make tag search case insensitive again (#4184) 2017-07-13 19:31:33 +02:00
Eugen Rochko
4e7288d18a Fix boolean columns sometimes having a null value (#4162)
* Fix boolean columns sometimes having a null value

* Fix wrong value being set instead of null
2017-07-13 03:12:25 +02:00
Eugen Rochko
ad531f90b1 Bind web UI access tokens to sessions (#3940)
* Add overview of active sessions

* Better display of browser/platform name

* Improve how browser information is stored and displayed for sessions overview

* Fix test

* Fix #2347 - Bind web UI access token to session

When you logout, session also destroys the access token, so it's no longer
valid. If access token is destroyed some other way, the session is also
destroyed, requiring a re-login.

Fix #1681 - Add scheduler to remove revoked access tokens and grants

* Fix test
2017-06-25 23:51:32 +02:00
Eugen Rochko
c465c5b3a8 Add overview of active sessions (#3929)
* Add overview of active sessions

* Better display of browser/platform name

* Improve how browser information is stored and displayed for sessions overview

* Fix test
2017-06-25 16:54:30 +02:00
Sorin Davidoi
1280559503 Revocable sessions (#3616)
* feat: Revocable sessions

* fix: Tests using sign_in

* feat: Configuration entry for the maximum number of session activations
2017-06-23 18:50:53 +02:00
Akihiko Odaki (@fn_aki@pawoo.net)
794afdca8d Add index statuses on account_id and id (#3895) 2017-06-23 17:46:00 +02:00
Matt Jankowski
b439018342 Language detection defaults to nil (#3666)
* Default to nil for statuses.language

* Language detection defaults to nil instead of instance UI default
2017-06-09 18:09:37 +02:00
unarist
117d333a84 Fix tag search order and not to use tsvector (#3611)
* Sort results by the name
* Switch search method to simple `LIKE` matching instead of tsvector/tsquery

Previously we used scores from ts_rank_cd() to sort results, but it didn't work
because the function returns same score for all results. It's not for calculate
similarity of single words. Sometimes this bug even push out exact matching tag
from results.

Additionally, PostgreSQL supports prefix searching with standard btree index.
Using it offers simpler code, but also less index size and some speed.
2017-06-06 16:07:06 +02:00
Eugen Rochko
d88c50a7ea Fix #3550 - Add all missing foreign keys (#3562)
* Fix #3550 - Add all missing foreign keys

* Add missing foreign keys
2017-06-05 13:24:00 +02:00
Matt Jankowski
a7ec1e71c5 Add index to media_attachments.account_id (#3510) 2017-06-02 16:18:54 +02:00
Matt Jankowski
1122249e51 Filter languages with opt out (#3175)
* Remove allowed_languages and add filtered_languages

* Use filtered_languages instead of allowed_languages
2017-05-20 17:32:44 +02:00
Eugen Rochko
0cafe62561 Account domain blocks (#2381)
* Add <ostatus:conversation /> tag to Atom input/output

Only uses ref attribute (not href) because href would be
the alternate link that's always included also.

Creates new conversation for every non-reply status. Carries
over conversation for every reply. Keeps remote URIs verbatim,
generates local URIs on the fly like the rest of them.

* Conversation muting - prevents notifications that reference a conversation
(including replies, favourites, reblogs) from being created. API endpoints
/api/v1/statuses/:id/mute and /api/v1/statuses/:id/unmute

Currently no way to tell when a status/conversation is muted, so the web UI
only has a "disable notifications" button, doesn't work as a toggle

* Display "Dismiss notifications" on all statuses in notifications column, not just own

* Add "muted" as a boolean attribute on statuses JSON

For now always false on contained reblogs, since it's only relevant for
statuses returned from the notifications endpoint, which are not nested

Remove "Disable notifications" from detailed status view, since it's
only relevant in the notifications column

* Up max class length

* Remove pending test for conversation mute

* Add tests, clean up

* Rename to "mute conversation" and "unmute conversation"

* Raise validation error when trying to mute/unmute status without conversation

* Adding account domain blocks that filter notifications and public timelines

* Add tests for domain blocks in notifications, public timelines
Filter reblogs of blocked domains from home

* Add API for listing and creating account domain blocks

* API for creating/deleting domain blocks, tests for Status#ancestors
and Status#descendants, filter domain blocks from them

* Filter domains in streaming API

* Update account_domain_block_spec.rb
2017-05-19 01:14:30 +02:00
Shunsuke Michii
37c8d4ce7e Make faster ProcessFeedService (#3080)
* Add index accounts on uri.

* Remove a blank line.
2017-05-16 12:12:29 +02:00
Eugen Rochko
5039bc93d5 Feature conversations muting (#3017)
* Add <ostatus:conversation /> tag to Atom input/output

Only uses ref attribute (not href) because href would be
the alternate link that's always included also.

Creates new conversation for every non-reply status. Carries
over conversation for every reply. Keeps remote URIs verbatim,
generates local URIs on the fly like the rest of them.

* Conversation muting - prevents notifications that reference a conversation
(including replies, favourites, reblogs) from being created. API endpoints
/api/v1/statuses/:id/mute and /api/v1/statuses/:id/unmute

Currently no way to tell when a status/conversation is muted, so the web UI
only has a "disable notifications" button, doesn't work as a toggle

* Display "Dismiss notifications" on all statuses in notifications column, not just own

* Add "muted" as a boolean attribute on statuses JSON

For now always false on contained reblogs, since it's only relevant for
statuses returned from the notifications endpoint, which are not nested

Remove "Disable notifications" from detailed status view, since it's
only relevant in the notifications column

* Up max class length

* Remove pending test for conversation mute

* Add tests, clean up

* Rename to "mute conversation" and "unmute conversation"

* Raise validation error when trying to mute/unmute status without conversation
2017-05-15 03:04:13 +02:00
Matt Jankowski
5f5d1d823d Remove reference to non-existent table in schema rb (#3019) 2017-05-12 19:22:33 +02:00
Eugen Rochko
12ef99556e Add conversation model, <ostatus:conversation /> (#3016)
* Add <ostatus:conversation /> tag to Atom input/output

Only uses ref attribute (not href) because href would be
the alternate link that's always included also.

Creates new conversation for every non-reply status. Carries
over conversation for every reply. Keeps remote URIs verbatim,
generates local URIs on the fly like the rest of them.

* Fix conversation migration

* More spec coverage for status before_create

* Prevent n+1 query when generating Atom with the new conversations

* Improve code style

* Remove redundant local variable
2017-05-12 19:09:21 +02:00
alpaca-tc
25e2322ca1 Add effective index to subscriptions (#2885) 2017-05-07 19:00:20 +02:00
Matt Jankowski
c44ef23e50 Filter on allowed user language preferences (#2361)
* Naive approached to timeline filtering

* Convert allowed_languages into a db column

* Allow users to choose languages to see statuses in

* Style list items as two columns

* Add a hint to explain language filtering preference
2017-05-01 17:42:13 +02:00
Akihiko Odaki
a28863bf95 Revert "Use PostgreSQL inheritance for blocks and mutes (#2520)" (#2634)
This reverts commit 7cd8e61b5a.
2017-04-30 08:49:24 -04:00
Akihiko Odaki
7cd8e61b5a Use PostgreSQL inheritance for blocks and mutes (#2520) 2017-04-30 00:27:31 +02:00
Eugen Rochko
4a7dc4fadc OEmbed support for PreviewCard (#2337)
* OEmbed support for PreviewCard

* Improve ProviderDiscovery code failure treatment

* Do not crawl links if there is a content warning, since those
don't display a link card anyway

* Reset db schema

* Fresh migrate

* Fix rubocop style issues
Fix #1681 - return existing access token when applicable instead of creating new

* Fix test

* Extract http client to helper

* Improve oembed controller
2017-04-27 14:42:22 +02:00
Francis Chong
395608442d Add media dimensions (#2448)
* Fixes #1985

- add migration AddMediaAttachmentMeta, which add meta field to media_attachments
- before saving attachment, set file meta if needed
- add meta in api

* add spec

* align the “size” format for image and video

* fix code climate

* fixes media_attachment_spec.rb
2017-04-26 03:48:12 +02:00
alpaca-tc
ec17577e47 Add index to statuses_tags#status_id (#2394) 2017-04-24 15:10:14 +02:00
Keiji, Yoshimi
532877fe9a it may be mentions_status_id_index is duplicated of index_mentions_on_status_id index. (#2224) 2017-04-20 17:13:06 +02:00
Matt Jankowski
4b6be67e85 Admin reports with accounts (#2092)
* Add a ReportFilter class

* Add reports and targeted_reports relationships to Account

* Use ReportFilter from admin/reports controller

* Link to admin/reports filtered views from admin account show view

* Add indexes to reports.account_id and reports.target_account_id
2017-04-18 19:36:18 +02:00
Eugen
b9b9edd4c8 Add language detection (#1772)
* Add language detection via WhatLanguage and (de)serialization of it through Atom

* Fix default language in ProcessFeedService

* Re-add newline before 'react-rails' Gem to fix groupings

Fixes Code Climate issue
2017-04-16 20:32:17 +02:00
Patrick Figel
15b393201e Add recovery code support for two-factor auth (#1773)
* Add recovery code support for two-factor auth

When users enable two-factor auth, the app now generates ten
single-use recovery codes. Users are encouraged to print the codes
and store them in a safe place.

The two-factor prompt during login now accepts both OTP codes and
recovery codes.

The two-factor settings UI allows users to regenerated lost
recovery codes. Users who have set up two-factor auth prior to
this feature being added can use it to generate recovery codes
for the first time.

Fixes #563 and fixes #987

* Set OTP_SECRET in test enviroment

* add missing .html to view file names
2017-04-15 13:26:03 +02:00
ThibG
626e7d1548 Refresh webfinger (#1323)
* Refresh local info for remote accounts when webfinger returns new values

It only refreshes account info if one of the URLs or the public-key changes,
in which cases it refreshes the full info, re-downloading the feeds from that
user.

Some special handling should probably be done when the public key changes,
but I have been unable to find any use for it in Mastodon yet.

* Re-fetch remote users we aren't subscribed to.

This might induce performance issues, we might want to only do that for users
we explicitly attempted to subscribe but failed to.

* Refactor changes

* Do not refresh existing remote account details more than once a day

* Avoid re-fetching webfinger info in tests unless otherwise specified
2017-04-15 03:16:05 +02:00
Eugen Rochko
b184c13102 Low-hanging fruit of query optimization, these indices were missing 2017-04-07 00:04:38 +02:00
Eugen Rochko
6687c4a114 Add index on mentions status_id 2017-04-05 13:32:57 +02:00
Eugen Rochko
c9ffa7ab1d Add basic logging of who resolved report 2017-04-03 19:35:00 +02:00
Eugen Rochko
244da78105 Import feature for following/blocking lists (addresses #62, #177, #201, #454) 2017-03-30 19:42:33 +02:00
Eugen Rochko
f1a3855f30 Fix up null values on latest migration, add notes 2017-03-30 16:06:27 +02:00
Eugen Rochko
96e58cf289 Add counter caches for a large performance increase on API requests 2017-03-30 15:06:59 +02:00
Eugen Rochko
1b4583ca91 Fix full-text search query quotation, improve tag search performance with an index,
add ability to open status by URL from search (fix #53)
2017-03-22 17:36:34 +01:00
Eugen Rochko
52f940fba9 Upgrade status IDs to bigint 2017-03-22 15:46:58 +01:00
Eugen Rochko
afd53467eb More efficient single account retrieval (0.9ms vs 50ms before) 2017-03-22 03:21:38 +01:00
Eugen Rochko
a19062b726 Federate header images, fix open-uri http->https redirection error 2017-03-18 22:51:20 +01:00
Eugen Rochko
64be072702 Make account search blazing fast and rank followers/followees higher in the results 2017-03-17 20:48:14 +01:00
Eugen Rochko
4b93cf88e5 Fix #431 - convert gif to webm during upload. Web UI treats them like it did
before. In the API, attachments now can be either image, video or gifv. Gifv
is to be treated like images in terms of behaviour, but are videos by file
type.
2017-03-04 23:02:24 +01:00
Eugen Rochko
0cb5a2a0a7 Add digest e-mails 2017-03-04 00:00:48 +01:00
Kit Redgrave
4554ccd5d0 Mute button progress so far. WIP, doesn't entirely work correctly. 2017-03-01 22:31:21 -06:00
Eugen Rochko
60cc2a32c2 Add foreign key to prevent reblogs of non-existent (after race conditions) statuses from happening
Fix issue with detailed status view not supporting unreblogging/unfavouriting
2017-02-17 02:33:10 +01:00
Eugen Rochko
2cc31b3194 Adding POST /api/v1/reports API, and a UI for submitting reports 2017-02-14 20:59:26 +01:00
Eugen Rochko
81be690f95 Fix #614 - extra reply-boolean on statuses to account for cases when replied-to
status is not in the system at time of distribution; fix #607 - reset privacy
settings to defaults when cancelling replies
2017-02-09 20:25:39 +01:00
Eugen Rochko
d384845469 Fix #61 - Add list of blocked users to the UI; clean up failed push notifications API
Try to fix Travis CI setup
2017-02-05 19:18:11 +01:00
Eugen Rochko
9327d05bf7 API for apps to register for push notifications 2017-01-29 01:30:32 +01:00
Eugen Rochko
9b8670c939 Added optional two-factor authentication 2017-01-27 20:35:16 +01:00
Eugen Rochko
f751e16e2a Fix spoiler_text not having "not null" constraint 2017-01-25 15:24:19 +01:00
Eugen Rochko
30abab790e Fix order of migrations 2017-01-25 15:01:00 +01:00
Eugen Rochko
959e064186 Instead of using spoiler boolean and spoiler_text, simply check for non-blank spoiler_text
Federate spoiler_text using warning attribute on <content /> instead of a <category term="spoiler" />
Clean up schema file from accidental development migrations
2017-01-25 01:29:16 +01:00
blackle
e25fc71c2c Implement a click-to-view spoiler system 2017-01-23 21:07:40 -05:00
Eugen Rochko
3710d18889 Optional domain block attribute that prevents media attachments from being downloaded 2017-01-23 21:36:08 +01:00
Eugen Rochko
77a76d5171 Domain blocks now have varying severity - auto-suspend vs auto-silence 2017-01-23 17:38:38 +01:00
Eugen Rochko
f748a91ec7 Fix #463 - Fetch and display previews of URLs using OpenGraph tags 2017-01-20 01:00:14 +01:00
Effy Elden
8e0c1914fb Add tracking of OAuth app that posted a status, extend OAuth apps to have optional website field, add application details to API, show application name and website on detailed status views. Resolves #11 2017-01-15 08:58:50 +11:00
Eugen Rochko
f2667139ad Adding about/more page with extended information that can be set up by an admin 2017-01-13 20:16:38 +01:00
Eugen Rochko
6d98465db2 Extend rails-settings-cached to merge db-saved hash values with defaults 2017-01-13 02:42:22 +01:00
Eugen Rochko
7e7c2bbb0f Migrate from ledermann/rails-settings to rails-settings-cached which allows global settings
with YAML-defined defaults. Add admin page for editing global settings. Add "site_description"
setting that would show as a paragraph on the frontpage
2017-01-12 20:46:24 +01:00
Eugen Rochko
4293e132d1 Persist UI settings, add missing localizations for German 2017-01-09 14:00:55 +01:00
Eugen Rochko
0df070596e Fix #416 - Generate random unique 14-byte (19 characters) shortcodes
for local attachments, use them in URLs. Check status privacy
before redirecting to actual file.
2017-01-06 00:29:12 +01:00
Eugen Rochko
bef225e5da Add illustration to getting started column 2016-12-27 23:30:11 +01:00
Eugen Rochko
238233440f Follow call on locked account creates follow request instead
Reflect "requested" relationship in API and UI
Reflect inability of private posts to be reblogged in the UI
Disable Webfinger for locked accounts
2016-12-22 23:03:57 +01:00
Eugen Rochko
09a477c782 Add "locked" flag to accounts, prevent blocked users from following, force-unfollow blocked users 2016-12-22 21:34:19 +01:00
Eugen Rochko
d417da7d3a Private visibility on statuses prevents non-followers from seeing those
Filters out hidden stream entries from Atom feed
Blocks now generate hidden stream entries, can be used to federate blocks
Private statuses cannot be reblogged (generates generic 422 error for now)
POST /api/v1/statuses now takes visibility=(public|unlisted|private) param instead of unlisted boolean
Statuses JSON now contains visibility=(public|unlisted|private) field
2016-12-21 20:04:13 +01:00
Eugen Rochko
ca0757a6cf Add account suspension 2016-12-05 22:59:30 +01:00
Eugen Rochko
047bdf6fe0 Improve notification model 2016-12-03 20:04:19 +01:00
Eugen Rochko
2d4ce8a867 Fix #248 - Reload all accounts when fetching from cache 2016-12-03 18:21:26 +01:00
Eugen Rochko
23a4f93345 Fix #323 - self-replies to appear in public timelines again 2016-12-02 14:33:20 +01:00
Eugen Rochko
8d4ef0b6c3 Per-status control for unlisted mode, also federation for unlisted mode
Fix #233, fix #268
2016-11-30 21:34:59 +01:00
Eugen Rochko
a692318382 Track successful PuSH deliveries 2016-11-30 15:24:57 +01:00
Eugen Rochko
b5ad0eb4ea Adding embedded PuSH server 2016-11-28 13:36:47 +01:00
Eugen Rochko
65d6191147 Adding sensitive marker to statuses in API 2016-11-23 10:46:48 +01:00
Eugen Rochko
f07b0dc82f Remove unneeded indices, improve error handling in background workers, don't needlessly reload reblogged status, send Devise e-mails asynchronously 2016-11-22 17:32:51 +01:00
Eugen Rochko
951843e646 Adding notifications column 2016-11-20 19:39:58 +01:00
Eugen Rochko
38025dfea3 Adding unified streamable notifications 2016-11-20 19:39:58 +01:00
Eugen Rochko
b8e6ca45e5 Add user locale setting 2016-11-16 17:56:31 +01:00
Eugen Rochko
cb22dce970 Adding hashtags 2016-11-05 17:13:14 +01:00
Eugen Rochko
082e57fc13 Adding hashtag model 2016-11-04 19:12:59 +01:00
Eugen Rochko
1eac265f35 Adding public timeline silencing 2016-10-27 19:33:04 +02:00
Eugen Rochko
9bf5a73968 Adding domain blocks 2016-10-09 14:48:59 +02:00
Eugen Rochko
d51efdd1dc Adding user settings (model and mailer), no form yet 2016-10-07 13:17:56 +02:00
Eugen Rochko
9d59d7b463 Adding a block model and filter mentions from blocked users (fix #60) 2016-10-03 17:12:13 +02:00
Eugen Rochko
fc198a8b4c Adding e-mail confirmations 2016-10-03 16:51:00 +02:00
Eugen Rochko
7e14eefc81 Replace logo, fix #57 - delete/unreblog/unfavourite API, fix #45 - app
registration API
2016-09-26 23:56:53 +02:00
Eugen Rochko
608a2bfffc Upgrade to PubSubHubbub 0.4 (removing verify_token) 2016-09-20 02:43:20 +02:00
Eugen Rochko
059ebbf48d Separate PuSH subscriptions from following, add mastodon:push:refresh task,
respect hub.lease_seconds (fix #46)
2016-09-20 00:43:36 +02:00
Eugen Rochko
ae1fac0062 Add API to upload media attachments 2016-09-05 17:46:36 +02:00
Eugen Rochko
92afd29650 The frontend will now be an OAuth app, auto-authorized. The frontend will use an access token for API requests
Adding better errors for the API controllers, posting a simple status works from the frontend now
2016-08-26 19:12:19 +02:00
Eugen Rochko
10ba09f546 Upgrade to Rails 5.0.0.1 2016-08-17 17:58:00 +02:00
Eugen Rochko
e24bfbde1a Fixing FanOutOnWriteService, fixing Sidekiq not having enough DB connections
in the pool, adding a throttle of 60rpm per IP, adding mini profiler, adding
admin status to users
2016-03-25 14:12:24 +01:00
Eugen Rochko
02e4fb2e06 Only re-download avatar if URL changed (fix #19) 2016-03-22 21:05:23 +01:00
Eugen Rochko
8767a98fbb Adding CSS for form errors, adding missing indices 2016-03-16 11:46:25 +01:00
Eugen Rochko
6fec8afc3f Bind oauth applications to users 2016-03-14 17:49:13 +01:00
Eugen Rochko
3441361568 Adding simple_form, adding profile settings, header image 2016-03-12 20:47:22 +01:00
Eugen Rochko
b919f39b31 Customizing doorkeeper views for authorizing app 2016-03-11 01:58:55 +01:00
Eugen Rochko
ab6696e855 Adding doorkeeper, adding a REST API
POST /api/statuses                  Params: status (text contents), in_reply_to_id (optional)
GET  /api/statuses/:id
POST /api/statuses/:id/reblog

GET  /api/accounts/:id
GET  /api/accounts/:id/following
GET  /api/accounts/:id/followers
POST /api/accounts/:id/follow
POST /api/accounts/:id/unfollow

POST /api/follows                  Params: uri (e.g. user@domain)

OAuth authentication is currently disabled, but the API can be used with HTTP Auth.
2016-03-07 12:42:33 +01:00
Eugen Rochko
7e93da3f8d Removing grape and adding devise 2016-03-05 13:12:24 +01:00
Eugen Rochko
2c70f0ecaa Adding paperclip for avatars, fixing design of the public pages 2016-02-28 00:02:59 +01:00
Eugen Rochko
71fe24096c Adding a Mention model, test stubs 2016-02-25 00:17:01 +01:00
Eugen Rochko
fa33750105 Adding reblogs, favourites, improving atom generation 2016-02-23 19:17:37 +01:00
Eugen Rochko
709c6685a9 Made some progress 2016-02-22 16:00:20 +01:00
Eugen Rochko
9c4856bdb1 Initial commit 2016-02-20 22:53:20 +01:00