Merge remote-tracking branch 'treehouse/main' into th-downstream

* Fix insufficient Content-Type checking of fetched ActivityStreams objects

* Allow JSON-LD documents with multiple profiles

.dockerignore

@@ -19,6 +19,7 @@
 .github
 .gitignore
 .woodpecker.yml
+/*.md
 build
 chart
 coverage

CHANGELOG.md

@@ -2,6 +2,101 @@
 
 All notable changes to this project will be documented in this file.
 
+## [4.2.7] - 2024-02-16
+
+### Fixed
+
+- Fix OmniAuth tests and edge cases in error handling ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/29201), [ClearlyClaire](https://github.com/mastodon/mastodon/pull/29207))
+- Fix new installs by upgrading to the latest release of the `nsa` gem, instead of a no longer existing commit ([mjankowski](https://github.com/mastodon/mastodon/pull/29065))
+
+### Security
+
+- Fix insufficient checking of remote posts ([GHSA-jhrq-qvrm-qr36](https://github.com/mastodon/mastodon/security/advisories/GHSA-jhrq-qvrm-qr36))
+
+## [4.2.6] - 2024-02-14
+
+### Security
+
+- Update the `sidekiq-unique-jobs` dependency (see [GHSA-cmh9-rx85-xj38](https://github.com/mhenrixon/sidekiq-unique-jobs/security/advisories/GHSA-cmh9-rx85-xj38))
+  In addition, we have disabled the web interface for `sidekiq-unique-jobs` out of caution.
+  If you need it, you can re-enable it by setting `ENABLE_SIDEKIQ_UNIQUE_JOBS_UI=true`.
+  If you only need to clear all locks, you can now use `bundle exec rake sidekiq_unique_jobs:delete_all_locks`.
+- Update the `nokogiri` dependency (see [GHSA-xc9x-jj77-9p9j](https://github.com/sparklemotion/nokogiri/security/advisories/GHSA-xc9x-jj77-9p9j))
+- Disable administrative Doorkeeper routes ([ThisIsMissEm](https://github.com/mastodon/mastodon/pull/29187))
+- Fix ongoing streaming sessions not being invalidated when applications get deleted in some cases ([GHSA-7w3c-p9j8-mq3x](https://github.com/mastodon/mastodon/security/advisories/GHSA-7w3c-p9j8-mq3x))
+  In some rare cases, the streaming server was not notified of access tokens revocation on application deletion.
+- Change external authentication behavior to never reattach a new identity to an existing user by default ([GHSA-vm39-j3vx-pch3](https://github.com/mastodon/mastodon/security/advisories/GHSA-vm39-j3vx-pch3))
+  Up until now, Mastodon has allowed new identities from external authentication providers to attach to an existing local user based on their verified e-mail address.
+  This allowed upgrading users from a database-stored password to an external authentication provider, or move from one authentication provider to another.
+  However, this behavior may be unexpected, and means that when multiple authentication providers are configured, the overall security would be that of the least secure authentication provider.
+  For these reasons, this behavior is now locked under the `ALLOW_UNSAFE_AUTH_PROVIDER_REATTACH` environment variable.
+  In addition, regardless of this environment variable, Mastodon will refuse to attach two identities from the same authentication provider to the same account.
+
+## [4.2.5] - 2024-02-01
+
+### Security
+
+- Fix insufficient origin validation (CVE-2024-23832, [GHSA-3fjr-858r-92rw](https://github.com/mastodon/mastodon/security/advisories/GHSA-3fjr-858r-92rw))
+
+## [4.2.4] - 2024-01-24
+
+### Fixed
+
+- Fix error when processing remote files with unusually long names ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/28823))
+- Fix processing of compacted single-item JSON-LD collections ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/28816))
+- Retry 401 errors on replies fetching ([ShadowJonathan](https://github.com/mastodon/mastodon/pull/28788))
+- Fix `RecordNotUnique` errors in LinkCrawlWorker ([tribela](https://github.com/mastodon/mastodon/pull/28748))
+- Fix Mastodon not correctly processing HTTP Signatures with query strings ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/28443), [ClearlyClaire](https://github.com/mastodon/mastodon/pull/28476))
+- Fix potential redirection loop of streaming endpoint ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/28665))
+- Fix streaming API redirection ignoring the port of `streaming_api_base_url` ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/28558))
+- Fix error when processing link preview with an array as `inLanguage` ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/28252))
+- Fix unsupported time zone or locale preventing sign-up ([Gargron](https://github.com/mastodon/mastodon/pull/28035))
+- Fix "Hide these posts from home" list setting not refreshing when switching lists ([brianholley](https://github.com/mastodon/mastodon/pull/27763))
+- Fix missing background behind dismissable banner in web UI ([Gargron](https://github.com/mastodon/mastodon/pull/27479))
+- Fix line wrapping of language selection button with long locale codes ([gunchleoc](https://github.com/mastodon/mastodon/pull/27100), [ClearlyClaire](https://github.com/mastodon/mastodon/pull/27127))
+- Fix `Undo Announce` activity not being sent to non-follower authors ([MitarashiDango](https://github.com/mastodon/mastodon/pull/18482))
+- Fix N+1s because of association preloaders not actually getting called ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/28339))
+- Fix empty column explainer getting cropped under certain conditions ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/28337))
+- Fix `LinkCrawlWorker` error when encountering empty OEmbed response ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/28268))
+- Fix call to inefficient `delete_matched` cache method in domain blocks ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/28367))
+
+### Security
+
+- Add rate-limit of TOTP authentication attempts at controller level ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/28801))
+
+## [4.2.3] - 2023-12-05
+
+### Fixed
+
+- Fix dependency on `json-canonicalization` version that has been made unavailable since last release
+
+## [4.2.2] - 2023-12-04
+
+### Changed
+
+- Change dismissed banners to be stored server-side ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/27055))
+- Change GIF max matrix size error to explicitly mention GIF files ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/27927))
+- Change `Follow` activities delivery to bypass availability check ([ShadowJonathan](https://github.com/mastodon/mastodon/pull/27586))
+- Change single-column navigation notice to be displayed outside of the logo container ([renchap](https://github.com/mastodon/mastodon/pull/27462), [renchap](https://github.com/mastodon/mastodon/pull/27476))
+- Change Content-Security-Policy to be tighter on media paths ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/26889))
+- Change post language code to include country code when relevant ([gunchleoc](https://github.com/mastodon/mastodon/pull/27099), [ClearlyClaire](https://github.com/mastodon/mastodon/pull/27207))
+
+### Fixed
+
+- Fix upper border radius of onboarding columns ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/27890))
+- Fix incoming status creation date not being restricted to standard ISO8601 ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/27655), [ClearlyClaire](https://github.com/mastodon/mastodon/pull/28081))
+- Fix some posts from threads received out-of-order sometimes not being inserted into timelines ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/27653))
+- Fix posts from force-sensitized accounts being able to trend ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/27620))
+- Fix error when trying to delete already-deleted file with OpenStack Swift ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/27569))
+- Fix batch attachment deletion when using OpenStack Swift ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/27554))
+- Fix processing LDSigned activities from actors with unknown public keys ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/27474))
+- Fix error and incorrect URLs in `/api/v1/accounts/:id/featured_tags` for remote accounts ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/27459))
+- Fix report processing notice not mentioning the report number when performing a custom action ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/27442))
+- Fix handling of `inLanguage` attribute in preview card processing ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/27423))
+- Fix own posts being removed from home timeline when unfollowing a used hashtag ([kmycode](https://github.com/mastodon/mastodon/pull/27391))
+- Fix some link anchors being recognized as hashtags ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/27271), [ClearlyClaire](https://github.com/mastodon/mastodon/pull/27584))
+- Fix format-dependent redirects being cached regardless of requested format ([ClearlyClaire](https://github.com/mastodon/mastodon/pull/27634))
+
 ## [4.2.1] - 2023-10-10
 
 ### Added

DIVERGENCES.md

@@ -0,0 +1,14 @@
+# Divergences
+
+## Major Features
+
+- quote posting
+- Treehouse::Automod (experimental feature flagged)
+
+## Other Changes
+
+- various build system changes
+  - a better dockerfile
+  - yarn v2 (a mistake, tbh)
+  - various dev env changes
+- various css/style changes

Gemfile.lock

@@ -506,7 +506,7 @@ GEM
     parslet (2.0.0)
     pastel (0.8.0)
       tty-color (~> 0.5)
-    pg (1.5.4)
+    pg (1.5.5)
     pghero (3.4.1)
       activerecord (>= 6)
     posix-spawn (0.3.15)

app/controllers/auth/omniauth_callbacks_controller.rb

@@ -17,6 +17,9 @@ class Auth::OmniauthCallbacksController < Devise::OmniauthCallbacksController
         session["devise.#{provider}_data"] = request.env['omniauth.auth']
         redirect_to new_user_registration_url
       end
+    rescue ActiveRecord::RecordInvalid
+      flash[:alert] = I18n.t('devise.failure.omniauth_user_creation_failure') if is_navigational_format?
+      redirect_to new_user_session_url
     end
   end
 

app/helpers/jsonld_helper.rb

@@ -174,7 +174,19 @@ module JsonLdHelper
     build_request(uri, on_behalf_of, options: request_options).perform do |response|
       raise Mastodon::UnexpectedResponseError, response unless response_successful?(response) || response_error_unsalvageable?(response) || !raise_on_temporary_error
 
-      body_to_json(response.body_with_limit) if response.code == 200
+      body_to_json(response.body_with_limit) if response.code == 200 && valid_activitypub_content_type?(response)
+    end
+  end
+
+  def valid_activitypub_content_type?(response)
+    return true if response.mime_type == 'application/activity+json'
+
+    # When the mime type is `application/ld+json`, we need to check the profile,
+    # but `http.rb` does not parse it for us.
+    return false unless response.mime_type == 'application/ld+json'
+
+    response.headers[HTTP::Headers::CONTENT_TYPE]&.split(';')&.map(&:strip)&.any? do |str|
+      str.start_with?('profile="') && str[9...-1].split.include?('https://www.w3.org/ns/activitystreams')
     end
   end
 

app/lib/activitypub/activity/create.rb

@@ -89,6 +89,7 @@ class ActivityPub::Activity::Create < ActivityPub::Activity
 
     resolve_thread(@status)
     fetch_replies(@status)
+    return if Treehouse::Automod.process_status!(@status)
     distribute
     forward_for_reply
   end

app/services/activitypub/process_account_service.rb

@@ -1,5 +1,7 @@
 # frozen_string_literal: true
 
+require 'digest'
+
 class ActivityPub::ProcessAccountService < BaseService
   include JsonLdHelper
   include DomainControlHelper
@@ -90,6 +92,9 @@ class ActivityPub::ProcessAccountService < BaseService
     set_immediate_protocol_attributes!
     set_fetchable_key! unless @account.suspended? && @account.suspension_origin_local?
     set_immediate_attributes! unless @account.suspended?
+
+    Treehouse::Automod.process_account!(@account)
+
     set_fetchable_attributes! unless @options[:only_key] || @account.suspended?
 
     @account.save_with_optional_media!

app/services/fetch_resource_service.rb

@@ -44,7 +44,7 @@ class FetchResourceService < BaseService
     @response_code = response.code
     return nil if response.code != 200
 
-    if ['application/activity+json', 'application/ld+json'].include?(response.mime_type)
+    if valid_activitypub_content_type?(response)
       body = response.body_with_limit
       json = body_to_json(body)
 

app/services/report_service.rb

@@ -40,6 +40,7 @@ class ReportService < BaseService
   end
 
   def notify_staff!
+    return if @options[:th_skip_notify_staff]
     return if @report.unresolved_siblings?
 
     User.those_who_can(:manage_reports).includes(:account).find_each do |u|
@@ -65,6 +66,7 @@ class ReportService < BaseService
   end
 
   def forward?
+    return false if @options[:th_skip_forward]
     !@target_account.local? && ActiveModel::Type::Boolean.new.cast(@options[:forward])
   end
 

config/application.rb

@@ -52,6 +52,8 @@ require_relative '../lib/active_record/batches'
 require_relative '../lib/simple_navigation/item_extensions'
 require_relative '../lib/http_extensions'
 
+require_relative '../lib/treehouse/automod'
+
 Dotenv::Railtie.load
 
 Bundler.require(:pam_authentication) if ENV['PAM_ENABLED'] == 'true'
@@ -107,5 +109,9 @@ module Mastodon
       Devise::FailureApp.include AbstractController::Callbacks
       Devise::FailureApp.include Localized
     end
+
+    config.x.th_automod.automod_account_username = ENV['TH_STAFF_ACCOUNT']
+    config.x.th_automod.account_service_heuristic_auto_suspend_active = ENV.fetch('TH_ACCOUNT_SERVICE_HEURISTIC_AUTO_SUSPEND', '') == 'that-one-spammer'
+    config.x.th_automod.mention_spam_heuristic_auto_limit_active = ENV.fetch('TH_MENTION_SPAM_HEURISTIC_AUTO_LIMIT_ACTIVE', '') == 'can-spam'
   end
 end

config/locales/devise.en.yml

@@ -12,6 +12,7 @@ en:
       last_attempt: You have one more attempt before your account is locked.
       locked: Your account is locked.
       not_found_in_database: Invalid %{authentication_keys} or password.
+      omniauth_user_creation_failure: Error creating an account for this identity.
       pending: Your account is still under review.
       timeout: Your session expired. Please login again to continue.
       unauthenticated: You need to login or sign up before continuing.

docker-compose.yml

@@ -56,7 +56,7 @@ services:
 
   web:
     build: .
-    image: ghcr.io/mastodon/mastodon:v4.2.0
+    image: ghcr.io/mastodon/mastodon:v4.2.7
     restart: always
     env_file: .env.production
     command: bundle exec puma -C config/puma.rb
@@ -77,7 +77,7 @@ services:
 
   streaming:
     build: .
-    image: ghcr.io/mastodon/mastodon:v4.2.0
+    image: ghcr.io/mastodon/mastodon:v4.2.7
     restart: always
     env_file: .env.production
     command: node ./streaming
@@ -95,7 +95,7 @@ services:
 
   sidekiq:
     build: .
-    image: ghcr.io/mastodon/mastodon:v4.2.0
+    image: ghcr.io/mastodon/mastodon:v4.2.7
     restart: always
     env_file: .env.production
     command: bundle exec sidekiq

lib/mastodon/version.rb

@@ -17,7 +17,7 @@ module Mastodon
     end
 
     def default_prerelease
-      'alpha.2'
+      'alpha.3'
     end
 
     def prerelease

lib/treehouse/automod.rb

@@ -0,0 +1,128 @@
+module Treehouse
+  module Automod
+    COMMENT_HEADER = <<~EOS
+      Tracking Report - automatically created by TreehouseAutomod
+    EOS
+
+    WARNING_TEXT = <<~EOS
+      Tracking Infraction - automatically created by TreehouseAutomod
+    EOS
+
+    def self.suspend_with_tracking_report!(account, status_ids: [], explanation: "")
+      account.save!
+
+      self.file_tracking_report!(account, status_ids: status_ids) unless account.suspension_origin == "local"
+
+      account.suspend! unless account.suspension_origin == "local"
+    end
+
+    def self.file_tracking_report!(account, status_ids: [], explanation: "")
+      reporter = self.staff_account
+      return if reporter.nil?
+
+      report = ReportService.new.call(
+        reporter,
+        account,
+        {
+          status_ids: status_ids,
+          comment: explanation.blank? ? COMMENT_HEADER : "#{COMMENT_HEADER}\n\n#{EXPLANATION}",
+          th_skip_notify_staff: true,
+          th_skip_forward: true,
+        }
+      )
+      report.spam!
+      report.assign_to_self!(reporter)
+
+      account_action = Admin::AccountAction.new(
+        type: "suspend",
+        report_id: report.id,
+        target_account: account,
+        current_account: reporter,
+        send_email_notification: false,
+        text: WARNING_TEXT,
+      )
+      account_action.save!
+
+      report.resolve!(reporter)
+    end
+
+    def self.staff_account
+      username = Rails.configuration.x.th_automod.automod_account_username
+      Account.find_local(username) unless username.blank?
+    end
+
+    def self.process_status!(status)
+      ActivityPubActivityCreateExt.process!(status)
+    end
+
+    def self.process_account!(account)
+      AccountServiceExt.process!(account)
+    end
+
+    module ActivityPubActivityCreateExt
+      EXPLANATION = <<~EOS
+        This account was automatically suspended by TreehouseAutomod, an unsupported feature.
+
+        Currently, the account-only heuristic should only automatically suspend accounts with one specific username and display name.
+
+        If this action is unexpected, please unset TH_MENTION_SPAM_HEURISTIC_AUTO_LIMIT_ACTIVE.
+      EOS
+
+      # check if the status should be considered spam
+      # @return true if the status was reported and the account was infracted
+      def process!(status)
+        return false unless Rails.configuration.x.th_automod.mention_spam_heuristic_auto_limit_active
+        account = status.account
+        minimal_effort = account.note.blank? && account.avatar_remote_url.blank? && account.header_remote_url.blank?
+        return false if (account.local? ||
+                         account.local_followers_account > 0 ||
+                         !minimal_effort)
+
+        # minimal effort account, check mentions and account-known age
+        status.mentions.size > 8 && account.created_at > (Time.now - 1.day)
+      end
+    end
+
+    module AccountServiceExt
+      # hardcoded for now
+      # md5 because they don't deserve more mentions
+      HEURISTIC_NAMES = {
+        "0116a9deace3289b7092e945ef5ca0a5" => Set["57d3d0b932cc9cd01be6b2f4e82c1a4a"],
+      }
+      # probably mathematically impossible to collide, but just in case...
+      HEURISTIC_MAX_LEN = 16
+
+      EXPLANATION = <<~EOS
+        This account was automatically suspended by TreehouseAutomod, an unsupported feature.
+
+        Currently, the account-only heuristic should only automatically suspend accounts with one specific username and display name.
+
+        If this action is unexpected, please unset TH_HEURISTIC_AUTO_SUSPEND.
+      EOS
+
+      # @return true if the account was infracted
+      def self.process!(account)
+        return false unless heuristic_auto_suspend?(account)
+
+        Automod.suspend_with_tracking_report!(account, explanation: EXPLANATION) unless account.suspension_origin == "local"
+
+        true
+      end
+
+      def self.matches_evil_hash?(account)
+        username_md5 = Digest::MD5.hexdigest(account.username)
+        display_name_md5 = Digest::MD5.hexdigest(account.display_name)
+
+        HEURISTIC_NAMES[username_md5].include?(display_name_md5)
+      end
+
+      def self.heuristic_auto_suspend?(account)
+        return false unless Rails.configuration.x.th_automod.account_service_heuristic_auto_suspend_active
+
+        return unless account.username.length < HEURISTIC_MAX_LEN && account.display_name.length < HEURISTIC_MAX_LEN
+
+        self.matches_evil_hash?(account)
+      end
+    end
+  end
+end

spec/helpers/json_ld_helper_spec.rb

@@ -56,15 +56,15 @@ describe JsonLdHelper do
   describe '#fetch_resource' do
     context 'when the second argument is false' do
       it 'returns resource even if the retrieved ID and the given URI does not match' do
-        stub_request(:get, 'https://bob.test/').to_return body: '{"id": "https://alice.test/"}'
+        stub_request(:get, 'https://bob.test/').to_return(body: '{"id": "https://alice.test/"}', headers: { 'Content-Type': 'application/activity+json' })
-        stub_request(:get, 'https://alice.test/').to_return body: '{"id": "https://alice.test/"}'
+        stub_request(:get, 'https://alice.test/').to_return(body: '{"id": "https://alice.test/"}', headers: { 'Content-Type': 'application/activity+json' })
 
         expect(fetch_resource('https://bob.test/', false)).to eq({ 'id' => 'https://alice.test/' })
       end
 
       it 'returns nil if the object identified by the given URI and the object identified by the retrieved ID does not match' do
-        stub_request(:get, 'https://mallory.test/').to_return body: '{"id": "https://marvin.test/"}'
+        stub_request(:get, 'https://mallory.test/').to_return(body: '{"id": "https://marvin.test/"}', headers: { 'Content-Type': 'application/activity+json' })
-        stub_request(:get, 'https://marvin.test/').to_return body: '{"id": "https://alice.test/"}'
+        stub_request(:get, 'https://marvin.test/').to_return(body: '{"id": "https://alice.test/"}', headers: { 'Content-Type': 'application/activity+json' })
 
         expect(fetch_resource('https://mallory.test/', false)).to be_nil
       end
@@ -72,7 +72,7 @@ describe JsonLdHelper do
 
     context 'when the second argument is true' do
       it 'returns nil if the retrieved ID and the given URI does not match' do
-        stub_request(:get, 'https://mallory.test/').to_return body: '{"id": "https://alice.test/"}'
+        stub_request(:get, 'https://mallory.test/').to_return(body: '{"id": "https://alice.test/"}', headers: { 'Content-Type': 'application/activity+json' })
         expect(fetch_resource('https://mallory.test/', true)).to be_nil
       end
     end
@@ -80,12 +80,12 @@ describe JsonLdHelper do
 
   describe '#fetch_resource_without_id_validation' do
     it 'returns nil if the status code is not 200' do
-      stub_request(:get, 'https://host.test/').to_return status: 400, body: '{}'
+      stub_request(:get, 'https://host.test/').to_return(status: 400, body: '{}', headers: { 'Content-Type': 'application/activity+json' })
       expect(fetch_resource_without_id_validation('https://host.test/')).to be_nil
     end
 
     it 'returns hash' do
-      stub_request(:get, 'https://host.test/').to_return status: 200, body: '{}'
+      stub_request(:get, 'https://host.test/').to_return(status: 200, body: '{}', headers: { 'Content-Type': 'application/activity+json' })
       expect(fetch_resource_without_id_validation('https://host.test/')).to eq({})
     end
   end

spec/lib/activitypub/activity/announce_spec.rb

@@ -35,7 +35,7 @@ RSpec.describe ActivityPub::Activity::Announce do
     context 'when sender is followed by a local account' do
       before do
         Fabricate(:account).follow!(sender)
-        stub_request(:get, 'https://example.com/actor/hello-world').to_return(body: Oj.dump(unknown_object_json))
+        stub_request(:get, 'https://example.com/actor/hello-world').to_return(body: Oj.dump(unknown_object_json), headers: { 'Content-Type': 'application/activity+json' })
         subject.perform
       end
 
@@ -120,7 +120,7 @@ RSpec.describe ActivityPub::Activity::Announce do
       let(:object_json) { 'https://example.com/actor/hello-world' }
 
       before do
-        stub_request(:get, 'https://example.com/actor/hello-world').to_return(body: Oj.dump(unknown_object_json))
+        stub_request(:get, 'https://example.com/actor/hello-world').to_return(body: Oj.dump(unknown_object_json), headers: { 'Content-Type': 'application/activity+json' })
       end
 
       context 'when the relay is enabled' do

spec/requests/omniauth_callbacks_spec.rb

@@ -60,11 +60,13 @@ describe 'OmniAuth callbacks' do
         end
 
         context 'when ALLOW_UNSAFE_AUTH_PROVIDER_REATTACH is not set to true' do
-          it 'does not match the existing user or create an identity' do
+          it 'does not match the existing user or create an identity, and redirects to login page' do
             expect { subject }
               .to not_change(User, :count)
               .and not_change(Identity, :count)
               .and not_change(LoginActivity, :count)
+
+            expect(response).to redirect_to(new_user_session_url)
           end
         end
       end

spec/services/activitypub/fetch_featured_collection_service_spec.rb

@@ -72,11 +72,11 @@ RSpec.describe ActivityPub::FetchFeaturedCollectionService, type: :service do
 
   shared_examples 'sets pinned posts' do
     before do
-      stub_request(:get, 'https://example.com/account/pinned/known').to_return(status: 200, body: Oj.dump(status_json_pinned_known))
+      stub_request(:get, 'https://example.com/account/pinned/known').to_return(status: 200, body: Oj.dump(status_json_pinned_known), headers: { 'Content-Type': 'application/activity+json' })
-      stub_request(:get, 'https://example.com/account/pinned/unknown-inlined').to_return(status: 200, body: Oj.dump(status_json_pinned_unknown_inlined))
+      stub_request(:get, 'https://example.com/account/pinned/unknown-inlined').to_return(status: 200, body: Oj.dump(status_json_pinned_unknown_inlined), headers: { 'Content-Type': 'application/activity+json' })
       stub_request(:get, 'https://example.com/account/pinned/unknown-unreachable').to_return(status: 404)
-      stub_request(:get, 'https://example.com/account/pinned/unknown-reachable').to_return(status: 200, body: Oj.dump(status_json_pinned_unknown_reachable))
+      stub_request(:get, 'https://example.com/account/pinned/unknown-reachable').to_return(status: 200, body: Oj.dump(status_json_pinned_unknown_reachable), headers: { 'Content-Type': 'application/activity+json' })
-      stub_request(:get, 'https://example.com/account/collections/featured').to_return(status: 200, body: Oj.dump(featured_with_null))
+      stub_request(:get, 'https://example.com/account/collections/featured').to_return(status: 200, body: Oj.dump(featured_with_null), headers: { 'Content-Type': 'application/activity+json' })
 
       subject.call(actor, note: true, hashtag: false)
     end
@@ -94,7 +94,7 @@ RSpec.describe ActivityPub::FetchFeaturedCollectionService, type: :service do
   describe '#call' do
     context 'when the endpoint is a Collection' do
       before do
-        stub_request(:get, actor.featured_collection_url).to_return(status: 200, body: Oj.dump(payload))
+        stub_request(:get, actor.featured_collection_url).to_return(status: 200, body: Oj.dump(payload), headers: { 'Content-Type': 'application/activity+json' })
       end
 
       it_behaves_like 'sets pinned posts'
@@ -111,7 +111,7 @@ RSpec.describe ActivityPub::FetchFeaturedCollectionService, type: :service do
       end
 
       before do
-        stub_request(:get, actor.featured_collection_url).to_return(status: 200, body: Oj.dump(payload))
+        stub_request(:get, actor.featured_collection_url).to_return(status: 200, body: Oj.dump(payload), headers: { 'Content-Type': 'application/activity+json' })
       end
 
       it_behaves_like 'sets pinned posts'
@@ -120,7 +120,7 @@ RSpec.describe ActivityPub::FetchFeaturedCollectionService, type: :service do
         let(:items) { 'https://example.com/account/pinned/unknown-reachable' }
 
         before do
-          stub_request(:get, 'https://example.com/account/pinned/unknown-reachable').to_return(status: 200, body: Oj.dump(status_json_pinned_unknown_reachable))
+          stub_request(:get, 'https://example.com/account/pinned/unknown-reachable').to_return(status: 200, body: Oj.dump(status_json_pinned_unknown_reachable), headers: { 'Content-Type': 'application/activity+json' })
           subject.call(actor, note: true, hashtag: false)
         end
 
@@ -147,7 +147,7 @@ RSpec.describe ActivityPub::FetchFeaturedCollectionService, type: :service do
       end
 
       before do
-        stub_request(:get, actor.featured_collection_url).to_return(status: 200, body: Oj.dump(payload))
+        stub_request(:get, actor.featured_collection_url).to_return(status: 200, body: Oj.dump(payload), headers: { 'Content-Type': 'application/activity+json' })
       end
 
       it_behaves_like 'sets pinned posts'
@@ -156,7 +156,7 @@ RSpec.describe ActivityPub::FetchFeaturedCollectionService, type: :service do
         let(:items) { 'https://example.com/account/pinned/unknown-reachable' }
 
         before do
-          stub_request(:get, 'https://example.com/account/pinned/unknown-reachable').to_return(status: 200, body: Oj.dump(status_json_pinned_unknown_reachable))
+          stub_request(:get, 'https://example.com/account/pinned/unknown-reachable').to_return(status: 200, body: Oj.dump(status_json_pinned_unknown_reachable), headers: { 'Content-Type': 'application/activity+json' })
           subject.call(actor, note: true, hashtag: false)
         end
 

spec/services/activitypub/fetch_featured_tags_collection_service_spec.rb

@@ -38,7 +38,7 @@ RSpec.describe ActivityPub::FetchFeaturedTagsCollectionService, type: :service d
   describe '#call' do
     context 'when the endpoint is a Collection' do
       before do
-        stub_request(:get, collection_url).to_return(status: 200, body: Oj.dump(payload))
+        stub_request(:get, collection_url).to_return(status: 200, body: Oj.dump(payload), headers: { 'Content-Type': 'application/activity+json' })
       end
 
       it_behaves_like 'sets featured tags'
@@ -46,7 +46,7 @@ RSpec.describe ActivityPub::FetchFeaturedTagsCollectionService, type: :service d
 
     context 'when the account already has featured tags' do
       before do
-        stub_request(:get, collection_url).to_return(status: 200, body: Oj.dump(payload))
+        stub_request(:get, collection_url).to_return(status: 200, body: Oj.dump(payload), headers: { 'Content-Type': 'application/activity+json' })
 
         actor.featured_tags.create!(name: 'FoO')
         actor.featured_tags.create!(name: 'baz')
@@ -67,7 +67,7 @@ RSpec.describe ActivityPub::FetchFeaturedTagsCollectionService, type: :service d
       end
 
       before do
-        stub_request(:get, collection_url).to_return(status: 200, body: Oj.dump(payload))
+        stub_request(:get, collection_url).to_return(status: 200, body: Oj.dump(payload), headers: { 'Content-Type': 'application/activity+json' })
       end
 
       it_behaves_like 'sets featured tags'
@@ -88,7 +88,7 @@ RSpec.describe ActivityPub::FetchFeaturedTagsCollectionService, type: :service d
       end
 
       before do
-        stub_request(:get, collection_url).to_return(status: 200, body: Oj.dump(payload))
+        stub_request(:get, collection_url).to_return(status: 200, body: Oj.dump(payload), headers: { 'Content-Type': 'application/activity+json' })
       end
 
       it_behaves_like 'sets featured tags'

spec/services/activitypub/fetch_remote_account_service_spec.rb

@@ -44,7 +44,7 @@ RSpec.describe ActivityPub::FetchRemoteAccountService, type: :service do
       before do
         actor[:inbox] = nil
 
-        stub_request(:get, 'https://example.com/alice').to_return(body: Oj.dump(actor))
+        stub_request(:get, 'https://example.com/alice').to_return(body: Oj.dump(actor), headers: { 'Content-Type': 'application/activity+json' })
         stub_request(:get, 'https://example.com/.well-known/webfinger?resource=acct:alice@example.com').to_return(body: Oj.dump(webfinger), headers: { 'Content-Type': 'application/jrd+json' })
       end
 
@@ -67,7 +67,7 @@ RSpec.describe ActivityPub::FetchRemoteAccountService, type: :service do
       let!(:webfinger) { { subject: 'acct:alice@example.com', links: [{ rel: 'self', href: 'https://example.com/alice' }] } }
 
       before do
-        stub_request(:get, 'https://example.com/alice').to_return(body: Oj.dump(actor))
+        stub_request(:get, 'https://example.com/alice').to_return(body: Oj.dump(actor), headers: { 'Content-Type': 'application/activity+json' })
         stub_request(:get, 'https://example.com/.well-known/webfinger?resource=acct:alice@example.com').to_return(body: Oj.dump(webfinger), headers: { 'Content-Type': 'application/jrd+json' })
       end
 
@@ -93,7 +93,7 @@ RSpec.describe ActivityPub::FetchRemoteAccountService, type: :service do
       let!(:webfinger) { { subject: 'acct:alice@iscool.af', links: [{ rel: 'self', href: 'https://example.com/alice' }] } }
 
       before do
-        stub_request(:get, 'https://example.com/alice').to_return(body: Oj.dump(actor))
+        stub_request(:get, 'https://example.com/alice').to_return(body: Oj.dump(actor), headers: { 'Content-Type': 'application/activity+json' })
         stub_request(:get, 'https://example.com/.well-known/webfinger?resource=acct:alice@example.com').to_return(body: Oj.dump(webfinger), headers: { 'Content-Type': 'application/jrd+json' })
         stub_request(:get, 'https://iscool.af/.well-known/webfinger?resource=acct:alice@iscool.af').to_return(body: Oj.dump(webfinger), headers: { 'Content-Type': 'application/jrd+json' })
       end
@@ -125,7 +125,7 @@ RSpec.describe ActivityPub::FetchRemoteAccountService, type: :service do
       let!(:webfinger) { { subject: 'acct:alice@example.com', links: [{ rel: 'self', href: 'https://example.com/bob' }] } }
 
       before do
-        stub_request(:get, 'https://example.com/alice').to_return(body: Oj.dump(actor))
+        stub_request(:get, 'https://example.com/alice').to_return(body: Oj.dump(actor), headers: { 'Content-Type': 'application/activity+json' })
         stub_request(:get, 'https://example.com/.well-known/webfinger?resource=acct:alice@example.com').to_return(body: Oj.dump(webfinger), headers: { 'Content-Type': 'application/jrd+json' })
       end
 
@@ -148,7 +148,7 @@ RSpec.describe ActivityPub::FetchRemoteAccountService, type: :service do
       let!(:webfinger) { { subject: 'acct:alice@iscool.af', links: [{ rel: 'self', href: 'https://example.com/bob' }] } }
 
       before do
-        stub_request(:get, 'https://example.com/alice').to_return(body: Oj.dump(actor))
+        stub_request(:get, 'https://example.com/alice').to_return(body: Oj.dump(actor), headers: { 'Content-Type': 'application/activity+json' })
         stub_request(:get, 'https://example.com/.well-known/webfinger?resource=acct:alice@example.com').to_return(body: Oj.dump(webfinger), headers: { 'Content-Type': 'application/jrd+json' })
         stub_request(:get, 'https://iscool.af/.well-known/webfinger?resource=acct:alice@iscool.af').to_return(body: Oj.dump(webfinger), headers: { 'Content-Type': 'application/jrd+json' })
       end

spec/services/activitypub/fetch_remote_actor_service_spec.rb

@@ -44,7 +44,7 @@ RSpec.describe ActivityPub::FetchRemoteActorService, type: :service do
       before do
         actor[:inbox] = nil
 
-        stub_request(:get, 'https://example.com/alice').to_return(body: Oj.dump(actor))
+        stub_request(:get, 'https://example.com/alice').to_return(body: Oj.dump(actor), headers: { 'Content-Type': 'application/activity+json' })
         stub_request(:get, 'https://example.com/.well-known/webfinger?resource=acct:alice@example.com').to_return(body: Oj.dump(webfinger), headers: { 'Content-Type': 'application/jrd+json' })
       end
 
@@ -67,7 +67,7 @@ RSpec.describe ActivityPub::FetchRemoteActorService, type: :service do
       let!(:webfinger) { { subject: 'acct:alice@example.com', links: [{ rel: 'self', href: 'https://example.com/alice' }] } }
 
       before do
-        stub_request(:get, 'https://example.com/alice').to_return(body: Oj.dump(actor))
+        stub_request(:get, 'https://example.com/alice').to_return(body: Oj.dump(actor), headers: { 'Content-Type': 'application/activity+json' })
         stub_request(:get, 'https://example.com/.well-known/webfinger?resource=acct:alice@example.com').to_return(body: Oj.dump(webfinger), headers: { 'Content-Type': 'application/jrd+json' })
       end
 
@@ -93,7 +93,7 @@ RSpec.describe ActivityPub::FetchRemoteActorService, type: :service do
       let!(:webfinger) { { subject: 'acct:alice@iscool.af', links: [{ rel: 'self', href: 'https://example.com/alice' }] } }
 
       before do
-        stub_request(:get, 'https://example.com/alice').to_return(body: Oj.dump(actor))
+        stub_request(:get, 'https://example.com/alice').to_return(body: Oj.dump(actor), headers: { 'Content-Type': 'application/activity+json' })
         stub_request(:get, 'https://example.com/.well-known/webfinger?resource=acct:alice@example.com').to_return(body: Oj.dump(webfinger), headers: { 'Content-Type': 'application/jrd+json' })
         stub_request(:get, 'https://iscool.af/.well-known/webfinger?resource=acct:alice@iscool.af').to_return(body: Oj.dump(webfinger), headers: { 'Content-Type': 'application/jrd+json' })
       end
@@ -125,7 +125,7 @@ RSpec.describe ActivityPub::FetchRemoteActorService, type: :service do
       let!(:webfinger) { { subject: 'acct:alice@example.com', links: [{ rel: 'self', href: 'https://example.com/bob' }] } }
 
       before do
-        stub_request(:get, 'https://example.com/alice').to_return(body: Oj.dump(actor))
+        stub_request(:get, 'https://example.com/alice').to_return(body: Oj.dump(actor), headers: { 'Content-Type': 'application/activity+json' })
         stub_request(:get, 'https://example.com/.well-known/webfinger?resource=acct:alice@example.com').to_return(body: Oj.dump(webfinger), headers: { 'Content-Type': 'application/jrd+json' })
       end
 
@@ -148,7 +148,7 @@ RSpec.describe ActivityPub::FetchRemoteActorService, type: :service do
       let!(:webfinger) { { subject: 'acct:alice@iscool.af', links: [{ rel: 'self', href: 'https://example.com/bob' }] } }
 
       before do
-        stub_request(:get, 'https://example.com/alice').to_return(body: Oj.dump(actor))
+        stub_request(:get, 'https://example.com/alice').to_return(body: Oj.dump(actor), headers: { 'Content-Type': 'application/activity+json' })
         stub_request(:get, 'https://example.com/.well-known/webfinger?resource=acct:alice@example.com').to_return(body: Oj.dump(webfinger), headers: { 'Content-Type': 'application/jrd+json' })
         stub_request(:get, 'https://iscool.af/.well-known/webfinger?resource=acct:alice@iscool.af').to_return(body: Oj.dump(webfinger), headers: { 'Content-Type': 'application/jrd+json' })
       end

spec/services/activitypub/fetch_remote_key_service_spec.rb

@@ -50,7 +50,7 @@ RSpec.describe ActivityPub::FetchRemoteKeyService, type: :service do
   end
 
   before do
-    stub_request(:get, 'https://example.com/alice').to_return(body: Oj.dump(actor))
+    stub_request(:get, 'https://example.com/alice').to_return(body: Oj.dump(actor), headers: { 'Content-Type': 'application/activity+json' })
     stub_request(:get, 'https://example.com/.well-known/webfinger?resource=acct:alice@example.com').to_return(body: Oj.dump(webfinger), headers: { 'Content-Type': 'application/jrd+json' })
   end
 
@@ -59,7 +59,7 @@ RSpec.describe ActivityPub::FetchRemoteKeyService, type: :service do
 
     context 'when the key is a sub-object from the actor' do
       before do
-        stub_request(:get, public_key_id).to_return(body: Oj.dump(actor))
+        stub_request(:get, public_key_id).to_return(body: Oj.dump(actor), headers: { 'Content-Type': 'application/activity+json' })
       end
 
       it 'returns the expected account' do
@@ -71,7 +71,7 @@ RSpec.describe ActivityPub::FetchRemoteKeyService, type: :service do
       let(:public_key_id) { 'https://example.com/alice-public-key.json' }
 
       before do
-        stub_request(:get, public_key_id).to_return(body: Oj.dump(key_json.merge({ '@context': ['https://www.w3.org/ns/activitystreams', 'https://w3id.org/security/v1'] })))
+        stub_request(:get, public_key_id).to_return(body: Oj.dump(key_json.merge({ '@context': ['https://www.w3.org/ns/activitystreams', 'https://w3id.org/security/v1'] })), headers: { 'Content-Type': 'application/activity+json' })
       end
 
       it 'returns the expected account' do
@@ -84,7 +84,7 @@ RSpec.describe ActivityPub::FetchRemoteKeyService, type: :service do
       let(:actor_public_key) { 'https://example.com/alice-public-key.json' }
 
       before do
-        stub_request(:get, public_key_id).to_return(body: Oj.dump(key_json.merge({ '@context': ['https://www.w3.org/ns/activitystreams', 'https://w3id.org/security/v1'] })))
+        stub_request(:get, public_key_id).to_return(body: Oj.dump(key_json.merge({ '@context': ['https://www.w3.org/ns/activitystreams', 'https://w3id.org/security/v1'] })), headers: { 'Content-Type': 'application/activity+json' })
       end
 
       it 'returns the nil' do

spec/services/activitypub/fetch_replies_service_spec.rb

@@ -58,7 +58,7 @@ RSpec.describe ActivityPub::FetchRepliesService, type: :service do
 
       context 'when passing the URL to the collection' do
         before do
-          stub_request(:get, collection_uri).to_return(status: 200, body: Oj.dump(payload))
+          stub_request(:get, collection_uri).to_return(status: 200, body: Oj.dump(payload), headers: { 'Content-Type': 'application/activity+json' })
         end
 
         it 'spawns workers for up to 5 replies on the same server' do
@@ -93,7 +93,7 @@ RSpec.describe ActivityPub::FetchRepliesService, type: :service do
 
       context 'when passing the URL to the collection' do
         before do
-          stub_request(:get, collection_uri).to_return(status: 200, body: Oj.dump(payload))
+          stub_request(:get, collection_uri).to_return(status: 200, body: Oj.dump(payload), headers: { 'Content-Type': 'application/activity+json' })
         end
 
         it 'spawns workers for up to 5 replies on the same server' do
@@ -132,7 +132,7 @@ RSpec.describe ActivityPub::FetchRepliesService, type: :service do
 
       context 'when passing the URL to the collection' do
         before do
-          stub_request(:get, collection_uri).to_return(status: 200, body: Oj.dump(payload))
+          stub_request(:get, collection_uri).to_return(status: 200, body: Oj.dump(payload), headers: { 'Content-Type': 'application/activity+json' })
         end
 
         it 'spawns workers for up to 5 replies on the same server' do

spec/services/activitypub/process_account_service_spec.rb

@@ -200,6 +200,114 @@ RSpec.describe ActivityPub::ProcessAccountService, type: :service do
     end
   end
 
+  context 'treehouse automod' do
+    subject { described_class.new.call(account_username, 'foo.test', payload) }
+    let(:account_username) { 'evil' }
+    let(:account_display_name) { 'evil display name' }
+    let(:account_payload_suspended) { false }
+
+    let(:automod_account_username) { nil }
+
+    let(:payload) do
+      {
+        id: 'https://foo.test',
+        type: 'Actor',
+        inbox: 'https://foo.test/inbox',
+        suspended: account_payload_suspended,
+        name: account_display_name,
+      }.with_indifferent_access
+    end
+
+    let(:name_hash_hash) do
+      {
+        # 'evil' => 'evil display name'
+        '4034a346ccee15292d823416f7510a2f' => Set['225e44a7c4a792ee22a4ada2032da7cd']
+      }
+    end
+
+    before do
+      allow(Rails.configuration.x.th_automod).to receive(:account_service_heuristic_auto_suspend_active).and_return(true)
+      allow(Rails.configuration.x.th_automod).to receive(:automod_account_username).and_return(automod_account_username)
+
+      stub_const('::Treehouse::Automod::AccountServiceExt::HEURISTIC_NAMES', name_hash_hash)
+      stub_const('::Treehouse::Automod::AccountServiceExt::HEURISTIC_MAX_LEN', 20)
+    end
+
+    context 'new account' do
+      context 'heuristic matching' do
+        it 'suspends the user locally' do
+          expect(subject.suspended?).to be true
+          expect(subject.suspension_origin_local?).to be true
+        end
+      end
+
+      context 'heuristic not matching' do
+        let(:account_display_name) { '' }
+        it 'does nothing' do
+          expect(subject.suspended?).to be false
+        end
+      end
+    end
+
+    context 'existing account' do
+      let!(:account) { Fabricate(:account, username: account_username, domain: 'foo.test', display_name: account_display_name) }
+
+      before do
+        allow(Admin::SuspensionWorker).to receive(:perform_async)
+      end
+
+      context 'heuristic matching' do
+        it 'suspends the user locally' do
+          expect(subject.suspended?).to be true
+          expect(subject.suspension_origin_local?).to be true
+        end
+      end
+
+      context 'heuristic not matching' do
+        let(:account_display_name) { 'not evil display name' }
+
+        it 'does nothing' do
+          expect(subject.suspended?).to be false
+        end
+
+        context 'suspended locally' do
+          before do
+            account.suspend!(origin: :local)
+          end
+
+          it 'does nothing' do
+            expect(subject.suspended?).to be true
+          end
+        end
+      end
+    end
+
+    context 'tracking report' do
+      let(:automod_account_username) { 'automod_test' }
+
+      let!(:automod_user_role) { Fabricate(:user_role, name: 'Automod', permissions: UserRole::FLAGS[:administrator]) }
+
+      let!(:automod_account) do
+        account = Fabricate(:account, username: automod_account_username)
+        account.user.role_id = automod_user_role.id
+        account.user.save!
+        account
+      end
+
+      it 'creates report' do
+        expect(subject.targeted_reports.empty?).to be_falsy
+
+        report = Report.find_by(target_account_id: subject.id, account_id: automod_account.id, assigned_account_id: automod_account.id)
+        expect(report.comment.starts_with?('Tracking Report - automatically created by TreehouseAutomod')).to be_truthy
+      end
+
+      it 'creates account action' do
+        subject
+        expect(Admin::ActionLog.find_by(account_id: automod_account.id, target_id: subject.id)).not_to be nil
+      end
+    end
+  end
+
   private
 
   def create_some_remote_accounts
@@ -209,4 +317,5 @@ RSpec.describe ActivityPub::ProcessAccountService, type: :service do
   def create_fewer_than_rate_limit_accounts
     change(Account.remote, :count).by_at_most(5)
   end
+
 end

spec/services/activitypub/synchronize_followers_service_spec.rb

@@ -60,7 +60,7 @@ RSpec.describe ActivityPub::SynchronizeFollowersService, type: :service do
   describe '#call' do
     context 'when the endpoint is a Collection of actor URIs' do
       before do
-        stub_request(:get, collection_uri).to_return(status: 200, body: Oj.dump(payload))
+        stub_request(:get, collection_uri).to_return(status: 200, body: Oj.dump(payload), headers: { 'Content-Type': 'application/activity+json' })
       end
 
       it_behaves_like 'synchronizes followers'
@@ -77,7 +77,7 @@ RSpec.describe ActivityPub::SynchronizeFollowersService, type: :service do
       end
 
       before do
-        stub_request(:get, collection_uri).to_return(status: 200, body: Oj.dump(payload))
+        stub_request(:get, collection_uri).to_return(status: 200, body: Oj.dump(payload), headers: { 'Content-Type': 'application/activity+json' })
       end
 
       it_behaves_like 'synchronizes followers'
@@ -98,7 +98,7 @@ RSpec.describe ActivityPub::SynchronizeFollowersService, type: :service do
       end
 
       before do
-        stub_request(:get, collection_uri).to_return(status: 200, body: Oj.dump(payload))
+        stub_request(:get, collection_uri).to_return(status: 200, body: Oj.dump(payload), headers: { 'Content-Type': 'application/activity+json' })
       end
 
       it_behaves_like 'synchronizes followers'

spec/workers/activitypub/fetch_replies_worker_spec.rb

@@ -21,7 +21,7 @@ describe ActivityPub::FetchRepliesWorker do
 
   describe 'perform' do
     it 'performs a request if the collection URI is from the same host' do
-      stub_request(:get, 'https://example.com/statuses_replies/1').to_return(status: 200, body: json)
+      stub_request(:get, 'https://example.com/statuses_replies/1').to_return(status: 200, body: json, headers: { 'Content-Type': 'application/activity+json' })
       subject.perform(status.id, 'https://example.com/statuses_replies/1')
       expect(a_request(:get, 'https://example.com/statuses_replies/1')).to have_been_made.once
     end