Commit graph

1756 commits

Author SHA1 Message Date
Claire
e06a615357 Merge branch 'main' into glitch-soc/main 2022-12-09 16:48:55 +01:00
Claire
145a3506b3 Fix failure when “Require a reason to join” is set with open registrations (#22127) 2022-12-07 16:39:58 +01:00
Claire
1e6e6426ea Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `app/models/concerns/domain_materializable.rb`:
  Fixed a code style issue upstream in a PR that got merged in glitch-soc
  earlier.
  Changed the code to match upstream's.
2022-12-07 09:03:42 +01:00
Claire
d4e4d272e1 Fix 500 error when trying to migrate to an invalid address (#21462)
* Fix 500 error when trying to migrate to an invalid address

* Add tests
2022-12-07 02:35:39 +01:00
Claire
aec7de494f Fix unbounded recursion in account discovery (#22025)
* Fix trying to fetch posts from other users when fetching featured posts

* Rate-limit discovery of new subdomains

* Put a limit on recursively discovering new accounts
2022-12-07 00:15:24 +01:00
Claire
b97260426b Fix irreversible and whole_word parameters handling in /api/v1/filters (#21988)
Fixes #21965
2022-12-07 00:10:53 +01:00
Claire
afa828e3d4 Fix unbounded recursion in account discovery (#1994)
* Fix trying to fetch posts from other users when fetching featured posts

* Rate-limit discovery of new subdomains

* Put a limit on recursively discovering new accounts
2022-12-04 21:23:19 +01:00
Claire
02992b262a Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `README.md`:
  Our README is completely different.
  Discarded upstream changes.
2022-11-28 11:33:34 +01:00
Claire
851c6d647f Fix spaces not being stripped in admin account search (#21324)
Fixes #21058

Regression from #18641
2022-11-27 20:47:29 +01:00
Kaspar V
f2ec356cb6 refactor(vacuum statuses): reduce amount of db queries and load for each query - improve performance (#21487)
* refactor(statuses_vacuum): remove dead code - unused

Method is not called inside class and private.
Clean up dead code.

* refactor(statuses_vacuum): make retention_period present test explicit

This private method only hides functionality.
It is best practice to be as explicit as possible.

* refactor(statuses_vacuum): improve query performance

- fix statuses_scope having sub-select for Account.remote scope by
  `joins(:account).merge(Account.remote)`
- fix statuses_scope unnecessary use of `Status.arel_table[:id].lt`
  because it is inexplicit, bad practice and even slower than normal
  `.where('statuses.id < ?'`
- fix statuses_scope remove select(:id, :visibility) for having reusable
  active record query batches (no re queries)
- fix vacuum_statuses! to use in_batches instead of find_in_batches,
  because in_batches delivers a full blown active record query result,
  in stead of an array - no requeries necessary
- send(:unlink_from_conversations) not to perform another db query, but
  reuse the in_batches result instead.
- remove now obsolete remove_from_account_conversations method
- remove_from_search_index uses array of ids, instead of mapping
  the ids from an array - this should be more efficient
- use the in_batches scope to call delete_all, instead of running
  another db query for this - because it is again more efficient
- add TODO comment for calling models private method with send

* refactor(status): simplify unlink_from_conversations

- add `has_many through:` relation mentioned_accounts
- use model scope local instead of method call `Status#local?`
- more readable add account to inbox_owners when account.local?

* refactor(status): searchable_by way less sub selects

These queries all included a sub-select. Doing the same with a joins
should be more efficient.
Since this method does 5 such queries, this should be significant,
since it technically halves the query count.

This is how it was:

```ruby
[3] pry(main)> Status.first.mentions.where(account: Account.local, silent: false).explain
  Status Load (1.6ms)  SELECT "statuses".* FROM "statuses" WHERE "statuses"."deleted_at" IS NULL ORDER BY "statuses"."id" DESC LIMIT $1  [["LIMIT", 1]]
  Mention Load (1.5ms)  SELECT "mentions".* FROM "mentions" WHERE "mentions"."status_id" = $1 AND "mentions"."account_id" IN (SELECT "accounts"."id" FROM "accounts" WHERE "accounts"."domain" IS NULL) AND "mentions"."silent" = $2  [["status_id", 109382923142288414], ["silent", false]]
=> EXPLAIN for: SELECT "mentions".* FROM "mentions" WHERE "mentions"."status_id" = $1 AND "mentions"."account_id" IN (SELECT "accounts"."id" FROM "accounts" WHERE "accounts"."domain" IS NULL) AND "mentions"."silent" = $2 [["status_id", 109382923142288414], ["silent", false]]
                                                    QUERY PLAN
------------------------------------------------------------------------------------------------------------------
 Nested Loop  (cost=0.15..23.08 rows=1 width=41)
   ->  Seq Scan on accounts  (cost=0.00..10.90 rows=1 width=8)
         Filter: (domain IS NULL)
   ->  Index Scan using index_mentions_on_account_id_and_status_id on mentions  (cost=0.15..8.17 rows=1 width=41)
         Index Cond: ((account_id = accounts.id) AND (status_id = '109382923142288414'::bigint))
         Filter: (NOT silent)
(6 rows)
```

This is how it is with this change:

```ruby
[4] pry(main)> Status.first.mentions.joins(:account).merge(Account.local).active.explain
  Status Load (1.7ms)  SELECT "statuses".* FROM "statuses" WHERE "statuses"."deleted_at" IS NULL ORDER BY "statuses"."id" DESC LIMIT $1  [["LIMIT", 1]]
  Mention Load (0.7ms)  SELECT "mentions".* FROM "mentions" INNER JOIN "accounts" ON "accounts"."id" = "mentions"."account_id" WHERE "mentions"."status_id" = $1 AND "accounts"."domain" IS NULL AND "mentions"."silent" = $2  [["status_id", 109382923142288414], ["silent", false]]
=> EXPLAIN for: SELECT "mentions".* FROM "mentions" INNER JOIN "accounts" ON "accounts"."id" = "mentions"."account_id" WHERE "mentions"."status_id" = $1 AND "accounts"."domain" IS NULL AND "mentions"."silent" = $2 [["status_id", 109382923142288414], ["silent", false]]
                                                    QUERY PLAN
------------------------------------------------------------------------------------------------------------------
 Nested Loop  (cost=0.15..23.08 rows=1 width=41)
   ->  Seq Scan on accounts  (cost=0.00..10.90 rows=1 width=8)
         Filter: (domain IS NULL)
   ->  Index Scan using index_mentions_on_account_id_and_status_id on mentions  (cost=0.15..8.17 rows=1 width=41)
         Index Cond: ((account_id = accounts.id) AND (status_id = '109382923142288414'::bigint))
         Filter: (NOT silent)
(6 rows)
```
2022-11-27 20:41:18 +01:00
afontenot
05b1b5790f Clear voter count when poll is reset (#21700)
When a poll is edited, we reset the poll and remove all previous
votes. However, prior to this commit, the voter count on the poll
was not reset. This leads to incorrect percentages being shown in
poll results.

Fixes #21696
2022-11-26 23:08:25 +01:00
Skyler Hawthorne
ce10b313bb fix media uploads with ffmpeg 5 (#21191) 2022-11-25 16:20:47 +01:00
Claire
cfdcfa999e Merge branch 'main' into glitch-soc/merge-upstream 2022-11-22 10:28:43 +01:00
David Leadbeater
72a60150de Don't allow URLs that contain non-normalized paths to be verified (#20999)
* Don't allow URLs that contain non-normalized paths to be verified

This stops things like https://example.com/otheruser/../realuser where
"/otheruser" appears to be the verified URL, but the actual URL being
verified is "/realuser" due to the "/../".

Also fix a test to use 'https', so it is testing the right thing, now
that since #20304 https is required.

* missing do
2022-11-20 19:28:13 +01:00
Claire
3e4b8636a0 Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `.github/workflows/build-image.yml`:
  Upstream changed how docker images were built, including how
  they were cached.
  I don't know much about it, so applied upstream's changes.
- `app/controllers/admin/domain_blocks_controller.rb`:
  The feature, that was in glitch-soc, got backported upstream.
  It also had a few fixes upstream, so those have been ported!
- `app/javascript/packs/admin.js`:
  Glitch-soc changes have been backported upstream. As a result,
  some code from `app/javascript/core/admin.js` got added upstream.
  Kept our version since our shared Javascript already has that feature.
- `app/models/user.rb`:
  Upstream added something to distinguish unusable and unusable-because-moved
  accounts, while glitch-soc considers moved accounts usable.
  Took upstream's code for `functional_or_moved?` and made `functional?`
  call it.
- `app/views/statuses/_simple_status.html.haml`:
  Upstream cleaned up code style a bit, on a line that we had custom changes
  for.
  Applied upstream's change while keeping our change.
- `config/initializers/content_security_policy.rb`:
  Upstream adopted one CSP directive we already had.
  The conflict is because of our files being structurally different, but the
  change itself was already part of glitch-soc.
  Kept our version.
2022-11-17 12:01:37 +01:00
Claire
0297acfe70 Fix various issues with domain block import (#1944)
- stop using Paperclip for processing domain allow/block imports
- stop leaving temporary files
- better error handling
- assume CSV files are UTF-8-encoded
2022-11-17 11:42:20 +01:00
lenore gilbert
498e9478c7 Support for import/export of instance-level domain blocks/allows for 4.x w/ additional fixes (#20597)
* Allow import/export of instance-level domain blocks/allows (#1754)

* Allow import/export of instance-level domain blocks/allows.
Fixes #15095

* Pacify circleci

* Address simple code review feedback

* Add headers to exported CSV

* Extract common import/export functionality to
AdminExportControllerConcern

* Add additional fields to instance-blocked domain export

* Address review feedback

* Split instance domain block/allow import/export into separate pages/controllers

* Address code review feedback

* Pacify DeepSource

* Work around Paperclip::HasAttachmentFile for Rails 6

* Fix deprecated API warning in export tests

* Remove after_commit workaround

(cherry picked from commit b8319d0578)

* Add confirmation page when importing blocked domains (#1773)

* Move glitch-soc-specific strings to glitch-soc-specific locale files

* Add confirmation page when importing blocked domains

(cherry picked from commit 73a2e3326f)

* Fix authorization check in domain blocks controller

(cherry picked from commit 655bac2c3a)

* Fix error strings for domain blocks and email-domain blocks

Corrected issue with non-error message used for Mastodon:NotPermittedError in Domain Blocks
Corrected issue Domain Blocks using the Email Domain Blocks message on ActionContoller::ParameterMissing
Corrected issue with Email Domain Blocks using the not_permitted string from "custom emojii's"

* Ran i18n-tasks normalize to address test failure

* Removed unused admin.export_domain_blocks.not_permitted string

Removing unused string as indicated by Check i18n

* Fix tests

(cherry picked from commit 1ba41b34c1)

* Fix domain block export not exporting blocks with only media rejection

(cherry picked from commit 1686374b2f)

* Fix various issues with domain block import

- stop using Paperclip for processing domain allow/block imports
- stop leaving temporary files
- better error handling
- assume CSV files are UTF-8-encoded

(cherry picked from commit cad824d8f501b95377e4f0a957e5a00d517a1902)

Co-authored-by: Levi Bard <taktaktaktaktaktaktaktaktaktak@gmail.com>
Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2022-11-17 11:05:09 +01:00
Claire
c7accdb6ac Change automatic post deletion configuration to be accessible to redirected users (#20774)
Fixes #20550
2022-11-17 10:55:23 +01:00
Claire
3e40b7ef8d Change batch account suspension to create a strike (#20897) 2022-11-17 10:52:51 +01:00
Claire
997553a99e Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `app/views/admin/announcements/edit.html.haml`:
  Upstream change too close to theming-related glitch-soc change.
  Ported upstream changes.
- `app/views/admin/announcements/new.html.haml`
  Upstream change too close to theming-related glitch-soc change.
  Ported upstream changes.
2022-11-14 08:35:55 +01:00
Eugen Rochko
2b3b063852 Fix error when invalid domain name is submitted (#19474)
Fix #19175
2022-11-14 08:07:14 +01:00
Eugen Rochko
8b98bebc17 Fix error when passing unknown filter param in REST API (#20626)
Fix #19156
2022-11-14 08:06:06 +01:00
Eugen Rochko
2018fe879c Fix image type not being set after conversion for convertible image types (#20624) 2022-11-14 07:13:14 +01:00
Eugen Rochko
12b819c78c Fix rules with same priority being sorted non-deterministically (#20623) 2022-11-14 06:28:19 +01:00
Claire
81d24609e5 Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `app/models/account.rb`:
  Conflict because we (glitch-soc) have disabled trending of posts without
  review.
  Discarded that upstream change.
- `app/views/admin/settings/discovery/show.html.haml`:
  Just an extra setting in glitch-soc.
  Kept that extra setting.
2022-11-12 13:28:03 +01:00
Jeong Arm
f9e6d2bb38 Fix trendable status without review (#20214) 2022-11-11 21:24:10 +01:00
Emily Strickland
32798102ad Guard against error extracting body from URL (#20428)
If `Nokogiri::HTML(value).at_xpath('//body')` fails to find the `body` element, it will return `nil`. We can guard against that with an early return. Avoids calling `children` on `Nilclass` in those cases.
2022-11-11 21:22:28 +01:00
Eugen Rochko
361d6793e8 Fix metadata scrubbing removing color profile from images (#20389) 2022-11-11 09:20:10 +01:00
Claire
2fe47bc7db Remove dead code and refactor status threading code (#20357)
* Remove dead code

* Remove unneeded/broken parameters and refactor descendant computation
2022-11-10 22:30:00 +01:00
Eugen Rochko
faaa47c421 Change verification to only work for https links (#20304)
Fix #20242
2022-11-10 21:09:03 +01:00
Claire
09f987bce1 Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `app/models/custom_emoji.rb`:
  Not a real conflict, just upstream changing a line too close to
  a glitch-soc-specific validation.
  Applied upstream changes.
- `app/models/public_feed.rb`:
  Not a real conflict, just upstream changing a line too close to
  a glitch-soc-specific parameter documentation.
  Applied upstream changes.
2022-11-10 09:36:47 +01:00
Eugen Rochko
ea7cc10811 Change link verification to ignore IDN domains (#20295)
Fix #3833
2022-11-10 06:27:45 +01:00
James Tucker
c4ff2640c1 Improve performance by avoiding regex construction (#20215)
```ruby
10.times { p /#{FOO}/.object_id }
10.times { p FOO_RE.object_id }
```
2022-11-10 05:49:30 +01:00
Eugen Rochko
17881148e4 Revert filtering public timelines by locale by default (#20294) 2022-11-10 05:34:42 +01:00
Eugen Rochko
3c80a6e445 Fix being able to spoof link verification (#20217)
- Change verification to happen in `default` queue
- Change verification worker to only be queued if there's something to do
- Add `link` tags from metadata fields to page header of profiles
2022-11-09 08:24:21 +01:00
Claire
0f9c70d5a2 Merge branch 'main' into glitch-soc/merge-upstream 2022-11-08 15:44:57 +01:00
Postmodern
97ee940b1a Micro-optimization: use if/else instead of Array#compact and Array#min (#19906)
* Technically `if`/`else` is faster than using `[value1, value2].compact.min` to find the lesser of two values, one of which may be `nil`.
2022-11-08 03:50:47 +01:00
Claire
5778ba1028 Fix validation error in SynchronizeFeaturedTagsCollectionWorker (#20018)
* Fix followers count not being updated when migrating follows

Fixes #19900

* Fix validation error in SynchronizeFeaturedTagsCollectionWorker

Also saves remote user's chosen case for hashtags

* Limit remote featured tags before validation
2022-11-07 22:35:53 +01:00
Claire
99cdfec05e Merge branch 'main' into glitch-soc/merge-upstream 2022-11-07 19:53:30 +01:00
Postmodern
41d43a1e1f Micro-optimization: only split acct into two Strings (#19901)
* Since `acct` is split by `@` and assigned to `username` and `domain`, we only need to split `acct` into two Strings.
2022-11-07 16:17:55 +01:00
Claire
495243d9ae Merge branch 'main' into glitch-soc/merge-upstream 2022-11-07 07:50:47 +01:00
nightpool
149a8246d2 Skip Webfinger cache during migrations as well (#19883) 2022-11-07 03:31:38 +01:00
Claire
20fb459dee Merge branch 'main' into glitch-soc/merge-upstream
Conflicts:
- `app/javascript/mastodon/features/compose/components/poll_form.js`:
  glitch-soc change because of having changed the default number of
  available poll options.
  Applied upstream's changes while keeping glitch-soc's default number of
  poll options.
- `public/oops.png`:
  We had a minor graphics change, probably not worth diverging from upstream.
  Took upstream version.
2022-11-06 09:50:41 +01:00
Eugen Rochko
b2cf46ee21 Fix not using GIN index for account search queries (#19830) 2022-11-06 06:16:34 +01:00
Claire
4b0e72b2b7 Fix reblogs being discarded after the reblogged status (#19731) 2022-11-04 16:31:44 +01:00
Eugen Rochko
6b2ae9e5a5 Fix featured tags not saving preferred casing (#19732) 2022-11-04 16:08:29 +01:00
Claire
a5e1f56347 Fix inaccurate admin log entry for re-sending confirmation e-mails (#19674)
Fixes #19593
2022-11-02 18:50:21 +01:00
txt-file
a2703b8852 Add support for AVIF uploads (#19647) 2022-11-01 22:08:41 +01:00
Eugen Rochko
ca1dec3831 Add support for HEIC uploads (#19618) 2022-11-01 16:26:25 +01:00
Eugen Rochko
529fe25add Change post-processing to be deferred only for large media types (#19617) 2022-11-01 15:27:58 +01:00