Commit graph

422 commits

Author SHA1 Message Date
Eugen Rochko
c32e312061 More robust PuSH subscription refreshes (#2799)
* Fix #2473 - Use sidekiq scheduler to refresh PuSH subscriptions instead of cron

Fix an issue where / in domain would raise exception in TagManager#normalize_domain

PuSH subscriptions refresh done in a round-robin way to avoid hammering a single
server's hub in sequence. Correct handling of failures/retries through Sidekiq (see
also #2613). Optimize Account#with_followers scope. Also, since subscriptions
are now delegated to Sidekiq jobs, an uncaught exception will not stop the entire
refreshing operation halfway through

Fix #2702 - Correct user agent header on outgoing http requests

* Add test for SubscribeService

* Extract #expiring_accounts into method

* Make mastodon:push:refresh no-op

* Queues are now defined in sidekiq.yml

* Queues are now in sidekiq.yml
2017-05-05 02:23:01 +02:00
Matt Jankowski
9db85085a9 Language improvements, replace whatlanguage with CLD (#2753)
* add failing en specs

* add cld2 gem

* Replace WhatLanguage with CLD
2017-05-03 16:59:31 +02:00
Yamagishi Kazutoshi
4a582aceed Add target=_blank to user note (#2622)
* Add target=_blank to user note

Open new window when click link from user profile in remote instance.

* fix rubocop
2017-04-30 00:28:41 +02:00
Eugen Rochko
9841dcaca7 Hotfix remote status formatting (#2543) 2017-04-27 23:37:22 +02:00
Yamagishi Kazutoshi
cc6d730bdc Fix broken oEmbed provider (#2537) 2017-04-27 19:29:11 +02:00
Eugen Rochko
4a7dc4fadc OEmbed support for PreviewCard (#2337)
* OEmbed support for PreviewCard

* Improve ProviderDiscovery code failure treatment

* Do not crawl links if there is a content warning, since those
don't display a link card anyway

* Reset db schema

* Fresh migrate

* Fix rubocop style issues
Fix #1681 - return existing access token when applicable instead of creating new

* Fix test

* Extract http client to helper

* Improve oembed controller
2017-04-27 14:42:22 +02:00
Yamagishi Kazutoshi
7f02d09887 Decodes URL containing IDN (#2436) 2017-04-25 15:03:51 +02:00
Eugen
87f7a3922c Punycode URI normalization (#2370)
* Fix #2119 - Whenever about to send a HTTP request, normalize the URI

* Add test for IDN request in FetchLinkCardService

* Perform IDN normalization on domains before they are stored in the DB
2017-04-25 02:47:31 +02:00
178inaba
a1bc7b0fdc Add simple_format to simplified_format (#2198) 2017-04-23 14:23:27 +02:00
ThibG
e9bb5e8e8a Allow webfinger controller to reply to user@WEB_DOMAIN in addition to user@LOCAL_DOMAIN (#2351)
This provides a hotfix for outbound salmon requests to other Mastodon instances
as they currently will try to resovle user@WEB_DOMAIN instead of user@LOCAL_DOMAIN
(see #2012 and #20312).

Furthermore, this should ease transition from users switching from
LOCAL_DOMAIN = WEB_DOMAIN to another LOCAL_DOMAIN when WEB_DOMAIN does not change.
2017-04-23 14:19:54 +02:00
Matt Jankowski
1a0960ba88 [WIP] Html lang on statuses (#2297)
* Add html lang attributes around statuses

* Remove urls from language detection
2017-04-22 04:26:25 +02:00
Eugen
1ccf2c8e52 Improve bio compatibility (#2278)
* Fix #1057 (close #1819) - Move HTML-formatted bio from <poco:note /> to <summary type="html" />

* Ensure <poco:note /> is plaintext for remote accounts, also, by stripping out HTML
2017-04-22 03:33:24 +02:00
Mingye Wang
e3fbd2db30 Fix mangling of ##tag matches (#2194) (#2247)
This commit fixes hashtag_html so it correctly handles matches with multiple hash-signs.

Bug located by @over9001, initial fix suggested by @nightpool.
2017-04-21 18:18:58 +02:00
Matt Jankowski
be1ecde300 Clean up settings/preferences controller (#2237)
* Add missing fields group on preferences page

* Clean up settings/preferences controller

* Extract a UserSettingsDecorator
2017-04-21 03:26:52 +02:00
abcang
815462feff Fix html escape characters in the URL (#2138)
* fix character escaping in URL

* add tests

* put a comma after the last item

* add HTML escape test
2017-04-19 14:52:18 +02:00
Matt Jankowski
4591c7dbdb Language detection refactor (#2099)
* Extract detect_language to separate class

* Use default locale, not just en

* Add spec to confirm that whatlanguage cant identify empty string

* Allow account locale to override default in language detector

* PostStatusService supplies an account to detect language
2017-04-18 22:20:12 +02:00
Eugen
7e230fc786 Fix #1972, fix #1870 - Fix special characters in XML, add tests (#1988)
Also improve efficiency of the mastodon:maintenance:add_static_avatars task
2017-04-17 19:58:54 +02:00
Eugen
4921bf4b20 Fix #1870 - Strip control characters out of strings in AtomSerializer (#1876)
* Fix #1870 - Strip control characters out of strings in AtomSerializer

* Adjust according to comment by @alpaca-tc
2017-04-16 20:32:27 +02:00
Eugen
b9b9edd4c8 Add language detection (#1772)
* Add language detection via WhatLanguage and (de)serialization of it through Atom

* Fix default language in ProcessFeedService

* Re-add newline before 'react-rails' Gem to fix groupings

Fixes Code Climate issue
2017-04-16 20:32:17 +02:00
Eugen
86e2b25355 Fix #1852 - Ensure feeds have valid <title> tags (#1875) 2017-04-16 02:40:36 +02:00
ThibG
c45c67c2ac Allow running mastodon on a different domain as the one used for identifying users (#1267)
* Allow running mastodon on a different domain as the one used for identifying users

* Alter documentation of WEB_DOMAIN to make clear it shouldn't be used unless the admin knows what they are doing

* Compare to web_domain instead of local_domain when dealing with feeds/API

* Correctly identify mentions to local accounts

Mentions URLs point to the person's web profile, i.e., the user page served on WEB_DOMAIN.
2017-04-15 02:15:46 +02:00
Effy Elden
089c5a76ba Change usage of gsub to delete, as per Code Climate/Rubocop recommendation (#1753) 2017-04-14 12:50:00 +02:00
Eugen
5c414c98e2 Fix #1220, fix #1671 - Hook up comment box to the Redux comment value (#1699)
Fix username styling regression introduced in #1063
Fix report screen background regression introduced in #1415
2017-04-13 19:36:41 +02:00
Eugen
5f9cd9b246 Fix #1609, fix #1628 - Revert #1397 (#1700)
When transmitting data in a HTML-encoded element like <content type="html" />,
relying on newlines being preserved is not wise, since HTML by itself
does not care for newlines - it cares for <p> and <br>

Additional fix: reset NSFW toggle after sending toot
2017-04-13 19:23:36 +02:00
Matt Jankowski
49c3000d62 Organize coverage dirs (#1695)
* Add `Presenters` group to SimpleCov configuration

* Move validators to app/validators, add to simplecov config
2017-04-13 16:00:31 +02:00
Ben Roberts
462e956164 significant improvement in microformats markup (#1063)
* significant improvement in microformats markup

This is a huge improvement and I believe will close #965.

Had these microformats reviewed by others in the community to help
ensure they are at least correct, if not complete.

I did not want to change the structure of the page, and so there it does
not fully mark up the entire ancestry chain, or reply chain, only the
direct decendants and direct ancestors are correctly associated, but
this is likely fine as the most important bit is to have access to the
urls for those toots which are now correctly fetchable.

* improve code climate

* trying to pass code climate tests

* code climate

* fix p-summary for content warning posts

* fix error introduced when merging via github
2017-04-12 20:04:33 +02:00
Matt Jankowski
d1ebb63c54 Quick best practice cleanup of views/helpers (#1546)
* Remove trailing whitespace

* Use query methods instead of explicit .blank? checks
2017-04-12 18:24:18 +02:00
Matt Jankowski
9d1d8797fd Webfinger resource to extract username from resource string (#1607)
* Add WebfingerResource class to extract usernames

* Use WebfingerResource in xrd#webfinger
2017-04-12 18:22:38 +02:00
Matt Jankowski
43143fb22e Remove order prior to .find_in_batches (#1470)
The `Status` class has a default order on it, so when this query gets built and
gets all the way to `find_in_batches` there is an order already there.

When `find_in_batches` is run it discards any existing order on the query, and
emits a warning to the logs if there is one there.

This change removes the order prior calling `find_in_batches`, which will stop
the logged warning from occurring as well.
2017-04-11 00:38:34 +02:00
Matt Jankowski
874a240a58 Clean up generation of account webfinger string (#1477)
* Consolidate webfinger string creation under Account#to_webfinger_s

* Introduce Account#local_username_and_domain for consolidation
2017-04-10 22:58:06 +02:00
Rachel H
b07990c94f Keep newlines in xml (#1397) 2017-04-10 21:32:45 +02:00
Eugen
d30a82a22f Fix #1339 - better Atom titles (#1343) 2017-04-09 20:55:54 +02:00
Joël Quenneville
8a67fb5eca DRY up reblog vs original status check
Checking reblog vs original status was happening in multiple places
across the app. For views, this logic was encapsulated in a helper
method named `proper_status` but in the other layers of the app, the
logic was duplicated.

Because the logic is used at all layers of the app, we extracted it into
a `Status#proper` method on the model and changed all uses of the logic
to use this method. There is now a single source of truth for this
condition.

We added test coverage to untested methods that got refactored.
2017-04-07 14:18:30 -04:00
Eugen Rochko
eaa0b8e669 Fix nil#object_type error 2017-04-07 13:05:34 +02:00
Eugen Rochko
a1c63cb01d Re-add forgotten <author> element on standalone <entry> 2017-04-07 12:21:00 +02:00
Eugen
1781fcaea4 Force UTF8 encoding on generated XML (#1140) 2017-04-07 11:09:14 +02:00
Eugen
24c77e57b2 Rewrite Atom generation from stream entries to use Ox instead of Nokogiri (#1124)
* Rewrite Atom generation from stream entries to use Ox instead of Nokogiri::Builder

StreamEntry is now limited to only statuses, which allows some optimization. Removed
extra queries on AccountsController#show. AtomSerializer instead of AtomBuilderHelper
used in AccountsController#show, StreamEntriesController#show, StreamEntryRenderer
and PubSubHubbub::DistributionWorker

PubSubHubbub::DistributionWorker moves n+1 DomainBlock query to PubSubHubbub::DeliveryWorker
instead.

All Salmon slaps that aren't based on StreamEntry still use AtomBuilderHelper and Nokogiri

* All Salmon slaps now use Ox instead of Nokogiri. No touch from status on account
2017-04-07 05:56:56 +02:00
Eugen Rochko
5189ce9cd1 Replace calls to FeedManager#inline_render and #broadcast 2017-04-05 19:45:18 +02:00
Kurtis Rainbolt-Greene
9b8f587c8d Moved to the worker 2017-04-04 20:51:44 -07:00
Kurtis Rainbolt-Greene
9e58c2b0e3 This method isn't used anymore 2017-04-04 20:38:07 -07:00
Kurtis Rainbolt-Greene
de1bb4bd56 By pushing this into a worker we can reduce the amount of time the feed manager using workers eat up a connection 2017-04-04 20:23:40 -07:00
Eugen
4751381a24 Merge pull request #852 from peterkeen/email-whitelist-817
[#817] Add email whitelist
2017-04-05 03:04:58 +02:00
Eugen
6a8bb89f80 Merge pull request #858 from krainboltgreene/patch-6
Use active record shorthand
2017-04-05 02:53:39 +02:00
Eugen Rochko
fa2a1af90a Spawn FeedInsertWorker to deliver status into personal feed 2017-04-04 19:21:37 +02:00
Kurtis Rainbolt-Greene
8dace45447 Use active record shorthand 2017-04-04 09:04:07 -07:00
Pete Keen
f28fcf9080 [#817] Add email whitelist
This adds the ability to filter user signup with a whitelist
instead of or in addition to a blacklist.

Fixes #817
2017-04-04 11:20:15 -04:00
Eugen Rochko
e60c473917 Reduce number of items in feeds, optimize regeneration worker slightly,
make regeneration worker unique, (only schedule/execute once at a time)
2017-04-04 13:58:34 +02:00
Eugen Rochko
9123aca6df Optimize FeedManager#unmerge, and slightly optimize FeedManager#merge 2017-04-04 13:43:36 +02:00
Eugen Rochko
fbf64105a5 Optimize filter methods in FeedManager a bit, use redis pipelining on merge/unmerge feed methods,
do not re-create a dynamic class on each feed push call, make sure redis-rb uses hiredis
2017-04-04 13:01:14 +02:00
Eugen Rochko
8796f7a125 Add check for visibility.nil? even though it can't ever be, to check for race conditions 2017-04-03 22:54:46 +02:00
Eugen Rochko
4db2e2f4eb Fix issue with feed merge-in code as well 2017-04-02 15:58:25 +02:00
Eugen Rochko
4ef48bcc42 Fix #408 - link @ names in bios 2017-03-28 14:16:08 +02:00
Eugen Rochko
b3cb765a94 Prettier account and stream entry URLs 2017-03-22 19:55:14 +01:00
Eugen Rochko
f1b9644cfd Add "direct" visibility level in the backend. Web UI is not yet
adjusted to allow choosing it, yet
2017-03-15 22:55:45 +01:00
Eugen Rochko
0cb5a2a0a7 Add digest e-mails 2017-03-04 00:00:48 +01:00
Eugen Rochko
9989890e2e Fix #231 - Muting 2017-03-02 18:49:32 +01:00
Kit Redgrave
4554ccd5d0 Mute button progress so far. WIP, doesn't entirely work correctly. 2017-03-01 22:31:21 -06:00
Eugen Rochko
cb17a23483 If a status is within 40 statuses from the top of a home feed, do not
reinsert it when someone boosts it
2017-03-01 01:07:11 +01:00
Eugen Rochko
ff8a080d40 Add validation of media attachments, clean up mastodon-own exception classes 2017-02-26 23:23:06 +01:00
Eugen Rochko
75d5dc1df8 Improve glow 2017-02-23 02:14:35 +01:00
Eugen Rochko
7413dcb28a Fix #555 - Use a better URL parser 2017-02-22 19:35:11 +01:00
Eugen Rochko
ce193afa4f Mentions in private statuses allow mentioned people to see them 2017-02-11 15:10:22 +01:00
Eugen Rochko
59c8c2b28a Make follow requests federate 2017-02-11 02:58:00 +01:00
Eugen Rochko
81be690f95 Fix #614 - extra reply-boolean on statuses to account for cases when replied-to
status is not in the system at time of distribution; fix #607 - reset privacy
settings to defaults when cancelling replies
2017-02-09 20:25:39 +01:00
Eugen Rochko
9c736975bd Fix showing ellipsis even when link hasn't been cut off 2017-02-05 15:29:16 +01:00
Eugen Rochko
728817fba2 Add tracking of delay to streaming API 2017-02-05 03:19:04 +01:00
Eugen Rochko
727d236fcc Cleaning up format of broadcast real-time messages, removing
redis-backed "mentions" timeline as redundant (given notifications)
2017-02-02 00:03:31 +01:00
Eugen Rochko
27ad4ce9e8 Do not run FetchLinkCardService on local URLs, increase file size limit to 8MB,
fix ProcessFeedService pushing status into distribution if called a second time
while the first is still running (i.e. when a PuSH comes after a Salmon slap),
fix not running escape on spoiler text before emojify
2017-01-27 16:57:23 +01:00
Eugen Rochko
959e064186 Instead of using spoiler boolean and spoiler_text, simply check for non-blank spoiler_text
Federate spoiler_text using warning attribute on <content /> instead of a <category term="spoiler" />
Clean up schema file from accidental development migrations
2017-01-25 01:29:16 +01:00
Eugen
c7778752e3 Merge branch 'master' into master 2017-01-24 21:56:06 +01:00
Eugen Rochko
cb33960e0e Fix #204, fix #515 - URL truncating is now a style so copypasting is not
affected, replaced onClick handler with onMouseUp/Down to detect text
selection not trigger onClick handler then
2017-01-24 17:05:44 +01:00
blackle
e25fc71c2c Implement a click-to-view spoiler system 2017-01-23 21:07:40 -05:00
Eugen Rochko
98660a76d9 Move merging/unmerging of timelines into background. Move blocking into
background as well since it's a computationally expensive
2017-01-23 21:29:34 +01:00
Eugen Rochko
4d39cc7bf9 Add /api/v1/notifications/clear, non-existing link cards for statuses will
now return empty hash instead of throwing a 404 error. When following,
merge into timeline will filter statuses
2017-01-23 21:09:27 +01:00
Eugen Rochko
28e65027b2 Fix #365, 1/2 of #408 - replace rails_autolink with URI.regexp, run link_hashtags on simplified_format 2017-01-23 14:45:09 +01:00
Eugen Rochko
f748a91ec7 Fix #463 - Fetch and display previews of URLs using OpenGraph tags 2017-01-20 01:00:14 +01:00
Eugen Rochko
86264a950c Add optional StatsD performance tracking 2017-01-18 23:44:29 +01:00
Eugen Rochko
5d88ef90c9 Fix tests, add applications to eager loading/cache for statuses, fix
application website validation, don't link to app website if website isn't set,
also comment out animated boost icon from #464 until it's consistent with non-animated version
2017-01-15 14:01:33 +01:00
Eugen Rochko
533448be42 Add extended about page stub 2017-01-13 03:24:41 +01:00
Eugen Rochko
6d98465db2 Extend rails-settings-cached to merge db-saved hash values with defaults 2017-01-13 02:42:22 +01:00
Eugen Rochko
7e7c2bbb0f Migrate from ledermann/rails-settings to rails-settings-cached which allows global settings
with YAML-defined defaults. Add admin page for editing global settings. Add "site_description"
setting that would show as a paragraph on the frontpage
2017-01-12 20:46:24 +01:00
Eugen Rochko
043d9114bf Federate blocks with the http://mastodon.social/schema/1.0 verb namespace 2017-01-02 12:17:51 +01:00
Eugen Rochko
d7bc0d8a33 Set in_reply_to_account on statuses to non-self value when possible, thus
resolving the confusion from self-chain replies ultimately linking to a
non-self status. Adjust filters
2016-12-31 14:36:25 +01:00
Eugen Rochko
337c3d0ed8 Private posts mentioning non-followers should not notify them, neither locally nor via Salmon 2016-12-22 23:14:24 +01:00
Eugen Rochko
238233440f Follow call on locked account creates follow request instead
Reflect "requested" relationship in API and UI
Reflect inability of private posts to be reblogged in the UI
Disable Webfinger for locked accounts
2016-12-22 23:03:57 +01:00
Eugen Rochko
09a477c782 Add "locked" flag to accounts, prevent blocked users from following, force-unfollow blocked users 2016-12-22 21:34:19 +01:00
Eugen Rochko
47921a626c Fixed AS delete verb 2016-12-14 15:13:39 +01:00
Eugen Rochko
a2fee40e8f Fix whitespace preservation in status text 2016-12-11 23:08:46 +01:00
Eugen Rochko
e5e702a976 Adding configurable e-mail blacklist 2016-12-04 19:07:02 +01:00
Eugen Rochko
42b065d3d7 Automatically block outgoing notifications from silenced accounts to users
who don't follow them
2016-12-04 16:51:49 +01:00
Eugen Rochko
8d4ef0b6c3 Per-status control for unlisted mode, also federation for unlisted mode
Fix #233, fix #268
2016-11-30 21:34:59 +01:00
Eugen Rochko
ea0846645a Fix #65 - Options to block notifications from people you don't follow/who don't follow you 2016-11-25 13:13:16 +01:00
Eugen Rochko
3b3bf2dd59 Filter statuses that mention blocked users 2016-11-25 12:35:52 +01:00
Eugen Rochko
38025dfea3 Adding unified streamable notifications 2016-11-20 19:39:58 +01:00
Eugen Rochko
e71b152d89 Fix rubocop issues, introduce usage of frozen literal to improve performance 2016-11-15 16:56:29 +01:00
Eugen Rochko
7a55e3c409 Fix #63 - Filter mentions that reply to blocked users 2016-11-13 21:11:45 +01:00
Eugen Rochko
ebe726f774 Fix #122 - Add microformats-2 classes to mention links 2016-11-10 01:19:33 +01:00
Eugen Rochko
6d9f8ee11e Improve filtering of public/hashtag timelines, both in backlog and real-time 2016-11-10 00:03:33 +01:00
Eugen Rochko
f1d2c46ded Improve feed regeneration 2016-11-08 02:08:32 +01:00
Eugen Rochko
8f0869876b Improved configuration from ENV, cleaned up timeline filter methods
to be more readable, add extra logging to process feed service
2016-11-07 23:20:52 +01:00
Eugen Rochko
6d948a809e Account notes (bios) can now contain links. Character limit upped to 160 2016-11-07 01:14:12 +01:00
Eugen Rochko
cb22dce970 Adding hashtags 2016-11-05 17:13:14 +01:00
Eugen Rochko
082e57fc13 Adding hashtag model 2016-11-04 19:12:59 +01:00
Eugen Rochko
72565e9e61 Fix subtle bugs, new icon button 2016-11-02 22:29:19 +01:00
Eugen Rochko
84495615b4 Shorten rendered links (strip protocol and www, truncate to 30 chars), redirect
to sign in page after sign up instead of root path which redirects to /about
2016-10-13 16:51:34 +02:00
Eugen Rochko
12a6eb95be Make account domains case-insensitive, downcase before checking against local 2016-10-10 03:34:15 +02:00
Eugen Rochko
c3f5dfeabb Adding public timeline 2016-10-07 16:00:11 +02:00
Eugen Rochko
b23c4b488c Better comparison of "local" domain 2016-10-06 16:36:16 +02:00
Eugen Rochko
9d59d7b463 Adding a block model and filter mentions from blocked users (fix #60) 2016-10-03 17:12:13 +02:00
Eugen Rochko
be86d4e0a3 Preparing feeds for better filtering 2016-10-02 15:28:47 +02:00
Eugen Rochko
a4f7eca5fa Fix #25 - Only latest reblog of status kept on feed as zset value is set to "true" status ID 2016-09-29 21:40:37 +02:00
Eugen Rochko
927333f4f8 Improve code style 2016-09-29 21:28:21 +02:00
Eugen Rochko
4f9b7432dd Fix #52 - Add API versioning (v1) 2016-09-27 16:59:08 +02:00
Eugen
3f75f52285 Fix subtle home feed filtering bug 2016-09-27 10:52:37 +02:00
Eugen Rochko
ce29624c6d Fixing image upload limits, allowing webm, merge/unmerge events trigger
timeline reload in UI, other small fixes
2016-09-12 18:22:43 +02:00
Eugen Rochko
3d566279cb Fix formatter return block 2016-09-10 19:18:17 +02:00
Eugen Rochko
0077fc26df Merge and unmerge timelines on follow/unfollow, solves #21, #22 2016-09-10 18:36:48 +02:00
Eugen Rochko
73615d7a79 Newlines and paragraphs in local statuses properly formatted 2016-09-10 10:07:56 +02:00
Eugen Rochko
33ad2e5423 Shorten mentions to only usernames 2016-09-10 09:57:41 +02:00
Eugen Rochko
3cc47beb6e Refactored generation of unique tags, URIs and object URLs into own classes,
as well as formatting of content
2016-09-09 20:04:34 +02:00
Eugen Rochko
e24bfbde1a Fixing FanOutOnWriteService, fixing Sidekiq not having enough DB connections
in the pool, adding a throttle of 60rpm per IP, adding mini profiler, adding
admin status to users
2016-03-25 14:12:24 +01:00
Eugen Rochko
a08e724476 Fix subscriptions:clear task, refactor feeds, refactor streamable activites
and atom feed generation to some extent, as well as the way mentions are
stored
2016-03-25 02:13:30 +01:00