Commit graph

71 commits

Author SHA1 Message Date
unarist
4acb1c73dd Improve error handling on LinkCrawlWorker (#5250)
* Improve error handling on LinkCrawlWorker

* Ignore TimeoutError and InvalidURIError too
* Record errors to debug log
* Enable dead job queue on LinkCrawlWorker

Since most of acceptable errors were already ignored, only our side issue should go to dead job queue.

* Ignore all http gem errors
2017-10-06 20:39:08 +02:00
ふぁぼ原
c71727ca55 Enable to recognize most kinds of characters as URL paths (#4941) 2017-09-14 18:03:20 +02:00
Eugen Rochko
41d6ada41f Support OpenGraph video embeds (#4897)
* Support OpenGraph video embeds

It's not really OpenGraph, it's twitter:player property, but it's
not OEmbed so that fits. For example, this allows Twitch clips to
be displayed as embeds.

Also, fixes glitch-soc/mastodon#135

* Fix invalid OpenGraph cards being saved through attaching and
revisit URLs after 14 days
2017-09-14 04:11:36 +02:00
Eugen Rochko
e9e271878e Make PreviewCard records reuseable between statuses (#4642)
* Make PreviewCard records reuseable between statuses

**Warning!** Migration truncates preview_cards tablec

* Allow a wider thumbnail for link preview, display it in horizontal layout (#4648)

* Delete preview cards files before truncating

* Rename old table instead of truncating it

* Add mastodon:maintenance:remove_deprecated_preview_cards

* Ignore deprecated_preview_cards in schema definition

* Fix null behaviour
2017-09-01 16:20:16 +02:00
Eugen Rochko
c5fa4aba91 HTTP signatures (#4146)
* Add Request class with HTTP signature generator

Spec: https://tools.ietf.org/html/draft-cavage-http-signatures-06

* Add HTTP signature verification concern

* Add test for SignatureVerification concern

* Add basic test for Request class

* Make PuSH subscribe/unsubscribe requests use new Request class

Accidentally fix lease_seconds not being set and sent properly, and
change the new minimum subscription duration to 1 day

* Make all PuSH workers use new Request class

* Make Salmon sender use new Request class

* Make FetchLinkService use new Request class

* Make FetchAtomService use the new Request class

* Make Remotable use the new Request class

* Make ResolveRemoteAccountService use the new Request class

* Add more tests

* Allow +-30 seconds window for signed request to remain valid

* Disable time window validation for signed requests, restore 7 days
as PuSH subscription duration (which was previous default due to a bug)
2017-07-14 20:41:49 +02:00
nullkal
07024f56df Use charlock_holmes instead of nkf at FetchLinkCardService (#4080)
* Specs for language detection

* Use CharlockHolmes instead of NKF

* Correct mistakes

* Correct style

* Set hint_enc instead of falling back and strip_tags

* Improve specs

* Add dependencies
2017-07-08 22:44:31 +02:00
abcang
8041c97d52 Fix Nokogiri::HTML at FetchLinkCardService (#4072) 2017-07-05 14:54:21 +02:00
abcang
43d97dea48 Rescue exceptions caused by FetchLinkCardService (#4045) 2017-07-03 11:03:34 +02:00
ThibG
3af5774a71 Fix an error when TagManager.local_url? is called with a bad URI (#3701)
TagManager.local_url? was sometimes called with an URI with a nil host,
leading to a crash in TagManager.local_url?. This fixes moves the
already-existing uri.host.blank? check in front to avoid this case.
2017-06-11 22:53:12 +02:00
Yamagishi Kazutoshi
bd1f7d0b9c Fetch remote image using http.rb (#3114) 2017-05-18 15:43:10 +02:00
Eugen Rochko
93e9f8b3ed Fix #2572 - Resolve preview cards for remote statuses as well as local ones (#3088) 2017-05-17 00:41:15 +02:00
Eugen Rochko
ee7719d540 Fix #2955 - Send HEAD request ahead of GET when fetching URL previews (#2972) 2017-05-10 23:30:07 +02:00
Yamagishi Kazutoshi
93ec05f976 Decode IDNA in PreviewCard (#2781) 2017-05-04 15:52:08 +02:00
Eugen Rochko
4a7dc4fadc OEmbed support for PreviewCard (#2337)
* OEmbed support for PreviewCard

* Improve ProviderDiscovery code failure treatment

* Do not crawl links if there is a content warning, since those
don't display a link card anyway

* Reset db schema

* Fresh migrate

* Fix rubocop style issues
Fix #1681 - return existing access token when applicable instead of creating new

* Fix test

* Extract http client to helper

* Improve oembed controller
2017-04-27 14:42:22 +02:00
Eugen
87f7a3922c Punycode URI normalization (#2370)
* Fix #2119 - Whenever about to send a HTTP request, normalize the URI

* Add test for IDN request in FetchLinkCardService

* Perform IDN normalization on domains before they are stored in the DB
2017-04-25 02:47:31 +02:00
Yamagishi Kazutoshi
45bee6a000 Show multibyte URI preview card (#2363) 2017-04-23 19:08:59 +02:00
ik-fib
3d7666f63c Add Mastodon version to user agent (#2253) 2017-04-21 17:26:23 +02:00
ik-fib
1b8613bc7a Add "Mastodon" to user agent (#2073)
Remove non-const version string

Freeze option is unnecessary
2017-04-18 16:04:13 +02:00
Eugen Rochko
27ad4ce9e8 Do not run FetchLinkCardService on local URLs, increase file size limit to 8MB,
fix ProcessFeedService pushing status into distribution if called a second time
while the first is still running (i.e. when a PuSH comes after a Salmon slap),
fix not running escape on spoiler text before emojify
2017-01-27 16:57:23 +01:00
Eugen Rochko
959e064186 Instead of using spoiler boolean and spoiler_text, simply check for non-blank spoiler_text
Federate spoiler_text using warning attribute on <content /> instead of a <category term="spoiler" />
Clean up schema file from accidental development migrations
2017-01-25 01:29:16 +01:00
Eugen Rochko
f748a91ec7 Fix #463 - Fetch and display previews of URLs using OpenGraph tags 2017-01-20 01:00:14 +01:00