1
0
Fork 1
mirror of https://github.com/mastodon/mastodon.git synced 2024-12-14 22:30:20 +00:00
Commit graph

1028 commits

Author SHA1 Message Date
Claire b9197486e5
Fix UnmergeWorker being exceptionally slow in some conditions () 2022-04-26 21:48:36 +02:00
Claire 0360135d4d
Fix PeerTube videos appearing with an erroneous “Edited at” marker ()
* Fix PeerTube videos appearing with an erroneous “Edited at” marker

PeerTube videos have an `updated` field equal to `published`.
When processing an incoming activity that has the same value for `updated` and
`published`, assume this doesn't represent an actual edit.

* Please CodeClimate
2022-04-26 21:25:26 +02:00
dogelover911 4c9949560d
FeedManager: skip account when target_account's last status is too old ()
Co-authored-by: dogelover911 <dogelover911@localhost>
2022-04-11 21:17:03 +02:00
0x2019 012537452a
Fix error resposes for from search prefix ()
* Fix error responses in `from` search prefix (addresses )

Using unsupported prefixes now reports a 422; searching for posts from an
account the instance is not aware of reports a 404. TODO: The UI for this
on the front end is abysmal.

Searching `from:username@domain` now succeeds when `domain` is the local
domain; searching `from:@username(@domain)?` now works as expected.

* Remove unused methods on new Error classes as they are not being used

Currently when `raise`d there are error messages being supplied, but
this is not actually being used. The associated `raise`s have been
edited accordingly.

* Remove needless comments

* Satisfy rubocop

* Try fixing tests being unable to find AccountFindingConcern methods

* Satisfy rubocop

* Simplify `from` prefix logic

This incorporates @ClearlyClaire's suggestion (see
https://github.com/mastodon/mastodon/pull/17963#pullrequestreview-933986737).

Accepctable account strings in `from:` clauses are more lenient than
before this commit; for example, `from:@user@example.org@asnteo +cat`
will not error, and return posts by @user@example.org containing the
word "cat". This is more consistent with how Mastodon matches mentions
in statuses. In addition, `from` clauses will not be checked for
syntatically invalid usernames or domain names, simply 404ing when
`Account.find_remote!` raises ActiveRecord::NotFound.

New code for this PR that is no longer used has been removed.
2022-04-08 21:21:49 +02:00
Eugen Rochko 8e20e16cf0
Change e-mail notifications to only be sent when recipient is offline ()
* Change e-mail notifications to only be sent when recipient is offline

Change the default for follow and mention notifications back on

* Add preference to always send e-mail notifications

* Change wording
2022-04-08 18:03:31 +02:00
Eugen Rochko 6221b36b27
Remove sign-in token authentication, instead send e-mail about new sign-in () 2022-04-06 20:58:12 +02:00
Claire ef196c913c
Fix error MethodError in Chewy::Strategy::Sidekiq::Worker ()
Also refactor a bit to reduce code duplication.
2022-03-31 00:49:24 +02:00
Eugen Rochko bbc7afa2a2
Fix being able to post URLs longer than 4096 characters () 2022-03-30 14:46:03 +02:00
Claire 2c45859ca9
Refactor account and status formatting ()
* Refactor status formatting

* Add account formatting helpers

* Remove StatusFormatter

* Fixup

* Fix copied typo
2022-03-28 01:17:17 +02:00
Eugen Rochko cefa526c6d
Refactor formatter ()
* Refactor formatter

* Move custom emoji pre-rendering logic to view helpers

* Move more methods out of Formatter

* Fix code style issues

* Remove Formatter

* Add inline poll options to RSS feeds

* Remove unused helper method

* Fix code style issues

* Various fixes and improvements

* Fix test
2022-03-26 02:53:34 +01:00
Eugen Rochko 71f2b95106
Fix edits with no actual changes being allowed ()
* Fix edits with no actual changes being allowed locally

* Fix edits with no actual changes being allowed through ActivityPub

* Fix false positive changes caused by description processing in model

* Fix not recording poll expiration update

* Fix test

* Revert changes to ProcessStatusUpdateService

* Various fixes and improvements

* Fix code style issues

* Various changes and improvements

* Add guard clause
2022-03-26 00:38:44 +01:00
Claire f65eaa5aae
Add admin dashboard checks for Elasticsearch version () 2022-03-25 21:00:59 +01:00
Eugen Rochko b58db8f12e
Add workaround for YouTube Shorts links ()
* Add workaround for YouTube Shorts links

* Update link_details_extractor_spec.rb
2022-03-25 19:31:35 +01:00
Claire c9b3cf8510
Fix searching already-known converted activities by URL ()
Fixes 
2022-03-17 00:46:49 +01:00
Eugen Rochko ddbe906c25
Fix not updating a status when newer version is fetched manually () 2022-03-12 09:11:36 +01:00
Eugen Rochko d17fb70131
Change how changes to media attachments are stored for edits ()
* Change how changes to media attachments are stored for edits

Fix not being able to re-order media attachments

* Fix not broadcasting updates when polls/media is changed through ActivityPub

* Various fixes and improvements

* Update app/models/report.rb

Co-authored-by: Claire <claire.github-309c@sitedethib.com>

* Add tracking of media attachment description changes

* Change poll in status edit to have a structure closer to the real one

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2022-03-09 09:06:17 +01:00
Eugen Rochko bd53dd5210
Change design of federation pages in admin UI ()
* Change design of federation pages in admin UI

* Fix query performance in instance media attachments measure

* Fix reblogs being included in instance languages dimension
2022-03-09 08:52:32 +01:00
Eugen Rochko edf09ec747
Add /api/v1/accounts/familiar_followers to REST API ()
* Add `/api/v1/accounts/familiar_followers` to REST API

* Change hide network preference to be stored consistently for local and remote accounts

* Add dummy classes to migration

* Apply suggestions from code review

Co-authored-by: Claire <claire.github-309c@sitedethib.com>

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2022-03-07 09:36:47 +01:00
Eugen Rochko 2ea754b861
Fix duplicate notifications being possible after poll expiration () 2022-03-04 01:06:33 +01:00
Eugen Rochko 04c3ac896a
Fix infinite re-forwarding of update activities temporarily () 2022-03-04 01:00:13 +01:00
Eugen Rochko 085f286750
Fix returning empty <p> tag for blank account note in REST API () 2022-03-03 06:45:46 +01:00
Eugen Rochko 50ea54b3ed
Change authorized applications page ()
* Change authorized applications page

* Hide revoke button for superapps and suspended accounts

* Clean up db/schema.rb
2022-03-01 16:48:58 +01:00
Eugen Rochko 0dc57ab6ed
Fix status updates not being forwarded like deletes through ActivityPub ()
Fix 
2022-02-26 17:51:59 +01:00
Eugen Rochko 27965ce5ed
Add trending statuses ()
* Add trending statuses

* Fix dangling items with stale scores in localized sets

* Various fixes and improvements

- Change approve_all/reject_all to approve_accounts/reject_accounts
- Change Trends::Query methods to not mutate the original query
- Change Trends::Query#skip to offset
- Change follow recommendations to be refreshed in a transaction

* Add tests for trending statuses filtering behaviour

* Fix not applying filtering scope in controller
2022-02-25 00:34:14 +01:00
luzpaz 73f5e4a1d9
Fix various typos ()
Found via `codespell -q 3 -S ./CHANGELOG.md,./AUTHORS.md,./config/locales,./app/javascript/mastodon/locales -L ba,keypair,medias,ro`
2022-02-22 20:14:17 +01:00
Claire 166f6e4b50
Fix some media attachments being converted with too high framerates ()
Video files with variable framerates are converted to constant framerate videos
and the output framerate picked by ffmpeg is based on the original file's
container framerate (which can be different from the average framerate).

This means that an input video with variable framerate with about 30 frames per
second on average, but a maximum of 120 fps will be converted to a constant 120
fps file, which won't be processed by other Mastodon servers.

This commit changes it so that input files with VFR and a maximum framerate
above the framerate threshold are converted to VFR files with the maximum frame
rate enforced.
2022-02-22 17:11:22 +01:00
Eugen Rochko b377022cf9
Add caching layer to metrics () 2022-02-22 15:27:08 +01:00
Claire 8f537a1168
Change relays handling to not record boosts ()
* Change relays handling to not record boosts

* Update tests
2022-02-16 14:36:44 +01:00
Claire 73fce8d311
Fix performance of server-side filtering ()
Fixes 
2022-02-16 14:28:45 +01:00
Jeong Arm 9a015e43ef
Add from: query operator to search syntax ()
* Add 'by:userhandle' parameter to search api

* Use search syntax for "by" prefix

* Codeclimate

* Use 'from' instead of 'by'
2022-02-14 00:17:09 +01:00
Eugen Rochko 3aebe711fd
Change languages to be listed under standard instead of native name in admin UI () 2022-02-09 04:15:38 +01:00
Eugen Rochko fd3a45e348
Add edit history to web UI ()
* Add edit history to web UI

* Change history reducer to store items per status

* Fix missing loading prop
2022-02-09 01:17:07 +01:00
Eugen Rochko b6d7726ecb
Remove language detection through cld3 ()
* Remove language detection through cld3

* Update app/helpers/languages_helper.rb

Co-authored-by: Yamagishi Kazutoshi <ykzts@desire.sh>

Co-authored-by: Yamagishi Kazutoshi <ykzts@desire.sh>
2022-02-08 02:41:17 +01:00
Eugen Rochko f1f6ddd536
Fix structured data parsing from links choking on bad data ()
* Fix structured data parsing from links choking on bad data

- Fix og:url meta tag being prioritized over canonical link tag
- Fix structured data parsing choking on commented-out CDATA declarations
- Fix HTML entities in title, description, provider_name, author_name
- Change structured data parsing to attempt every JSON-LD script tag

* Remove unnecessary slash escapes from CDATA regex pattern
2022-02-07 18:16:31 +01:00
Claire d1ecc323e7
Compact JSON-LD signed incoming activities ()
Co-authored-by: Puck Meerburg <puck@puck.moe>
2022-02-03 14:07:29 +01:00
Claire 03d59340da
Fix Sidekiq warnings about JSON serialization ()
* Fix Sidekiq warnings about JSON serialization

This occurs on every symbol argument we pass, and every symbol key in hashes,
because Sidekiq expects strings instead.

See https://github.com/mperham/sidekiq/pull/5071

We do not need to change how workers parse their arguments because this has
not changed and we were already converting to symbols adequately or using
`with_indifferent_access`.

* Set Sidekiq to raise on unsafe arguments in test mode

In order to more easily catch issues that would produce warnings in production
code.
2022-01-28 00:43:56 +01:00
Claire a63495230a
Change percent to rate in retention metrics API () 2022-01-23 16:01:25 +01:00
Eugen Rochko 1060666c58
Add support for editing for published statuses ()
* Add support for editing for published statuses

* Fix references to stripped-out code

* Various fixes and improvements

* Further fixes and improvements

* Fix updates being potentially sent to unauthorized recipients

* Various fixes and improvements

* Fix wrong words in test

* Fix notifying accounts that were tagged but were not in the audience

* Fix mistake
2022-01-19 22:37:27 +01:00
Eugen Rochko 14f436c457
Add notifications for statuses deleted by moderators () 2022-01-17 09:41:33 +01:00
Claire d5c9feb7b7
Add support for private pinned posts ()
* Add support for private pinned toots

* Allow local user to pin private toots

* Change wording to avoid "direct message"
2022-01-17 00:49:55 +01:00
Eugen Rochko fe71548844
Fix warnings on Rails boot () 2021-12-27 00:47:20 +01:00
Jeong Arm 4a109ec1ba
Fix server graph on admin/tags/:id () 2021-11-28 23:13:07 +01:00
Eugen Rochko 7de0ee7aba
Remove Keybase integration () 2021-11-26 05:58:18 +01:00
Eugen Rochko 6e50134a42
Add trending links ()
* Add trending links

* Add overriding specific links trendability

* Add link type to preview cards and only trend articles

Change trends review notifications from being sent every 5 minutes to being sent every 2 hours

Change threshold from 5 unique accounts to 15 unique accounts

* Fix tests
2021-11-25 13:07:38 +01:00
Claire 3517867b76
Fix confusing error when webfinger request returns empty document ()
For some reason, some misconfigured servers return an empty document when
queried over webfinger. Since an empty document does not lead to a parse
error, the error is not caught properly and triggers uncaught exceptions
later on.

This PR fixes that by immediately erroring out with `Webfinger::Error` on
getting an empty response.
2021-11-14 21:55:40 +01:00
Eugen Rochko 5d7c852283
Fix no link previews being generated for pages with invalid structured data ()
Fix 
2021-11-13 23:07:13 +01:00
Claire ef8deb0878
Fix OpenGraph/LinkedData embeds having incorrectly-generated iframes () 2021-11-13 03:30:27 +01:00
Eugen Rochko 39cdf61ab7
Add support for structured data and more OpenGraph tags to link cards ()
Save preview cards under their canonical URL

Increase max redirects to follow from 2 to 3
2021-11-05 23:23:05 +01:00
Claire a184c860ca
Fix retention rate computation () 2021-10-14 22:20:37 +02:00
Eugen Rochko 07341e7aa6
Add graphs and retention metrics to admin dashboard () 2021-10-14 20:44:59 +02:00
Claire fc3ae1343d
Switch from unmaintained paperclip to kt-paperclip ()
* Switch from unmaintained paperclip to kt-paperclip

* Drop some compatibility monkey-patches not required by kt-paperclip

* Drop media spoof check monkey-patching

It's broken with kt-paperclip and hopefully it won't be needed anymore

* Fix regression introduced by paperclip 6.1.0

* Do not rely on pathname to call FastImage

* Add test for ogg vorbis file with cover art

* Add audio/vorbis to the accepted content-types

This seems erroneous as this would be the content-type for a vorbis stream
without an ogg container, but that's what the `marcel` gem outputs, so…

* Restore missing for_as_default method

* Refactor Attachmentable concern and delay Paperclip's content-type spoof check

Check for content-type spoofing *after* setting the extension ourselves, this
fixes a regression with kt-paperclip's validations being more strict than
paperclip 6.0.0 and rejecting some Pleroma uploads because of unknown
extensions.

* Please CodeClimate

* Add audio/vorbis to the unreliable set

It doesn't correspond to a file format and thus has no extension associated.
2021-09-29 23:52:36 +02:00
Eugen Rochko 52e5c07948
Change routing paths to use usernames in web UI () 2021-09-26 05:46:13 +02:00
Claire 12cd097e7c
Fix addressing of remote groups' followers ()
Fixes 
2021-09-08 23:33:23 +02:00
Takeshi Umeda 709876bd6c
Fix invalid blurhash handling in Create activity () 2021-08-09 13:33:19 +02:00
Claire d8629e7b86
Add logging of S3-related errors () 2021-07-21 18:34:39 +02:00
Claire 211d5c3c30
Fix inefficiencies in auto-linking code ()
The auto-linking code basically rewrote the whole string escaping non-ascii
characters in an inefficient way, and building a full character offset map
between the unescaped and escaped texts before sending the contents to
TwitterText's extractor.

Instead of doing that, this commit changes the TwitterText regexps to include
valid IRI characters in addition to valid URI characters.
2021-07-15 15:56:58 +02:00
Claire 9a19227f17
Fix some RedisLocks auto-releasing too fast ()
* Fix Delete and Create-related locks expiring too fast

Fixes 

By default, RedisLock expires after 10 seconds, which may not be enough to
process statuses, especially when those have attached media files.

This commit extends those 10 seconds to 15 minutes, which should be plenty
enough to handle any status, while being short enough to not waste many
sidekiq job retries in the exceedingly rare case in which a sidekiq process
would crash when processing a `Create` or `Delete`.

* Fix other RedisLock autorelease durations

Fixes 

- things that only perform a few simple database queries (e.g. finding and
  saving a record) have been left unchanged, so they'll still use the default
  10s duration
- things that perform significantly more complex database queries have been
  changed to a 5 minutes timeout
- things that perform multiple HTTP queries have been changed to a 15 minutes
  timeout
2021-05-19 23:52:08 +02:00
abcang b5ad787ebf
Fix rubocop warning () 2021-05-11 14:19:01 +02:00
Claire afb7882189
Fix blocking someone not clearing up list feeds () 2021-05-10 17:31:55 +02:00
Eugen Rochko 74081433d0
Change trending hashtags to be affected be reblogs ()
If a status with a hashtag becomes very popular, it stands to
reason that the hashtag should have a chance at trending

Fix no stats being recorded for hashtags that are not allowed
to trend, and stop ignoring bots

Remove references to hashtags in profile directory from the code
and the admin UI
2021-05-07 14:33:43 +02:00
Takeshi Umeda 7cb34b32f8
Add management of delivery availability in Federation settings ()
* Add management of delivery availavility in Federation settings

* fix translate

* Remove useless object creation

* Fix DeepSource issue

* Add shortcut for all

* Fix DeepSource(skipcq)

* Change 'remove' to 'clear'

* Fix style

* Change class method name (exhausted_deliveries_key_by)
2021-05-05 23:39:02 +02:00
Eugen Rochko 036556d350
Fix media processing getting stuck on too much stdin/stderr ()
* Fix media processing getting stuck on too much stdin/stderr

See 

* Remove dependency on paperclip-av-transcoder gem

* Remove dependency on streamio-ffmpeg gem

* Disable stdin on ffmpeg process
2021-05-05 19:44:01 +02:00
Eugen Rochko daccc07dc1
Change auto-following admin-selected accounts, show in recommendations () 2021-04-24 17:01:43 +02:00
Eugen Rochko f4b7c6b619
Fix nil error when removing status caused by race condition () 2021-04-24 13:35:39 +02:00
Eugen Rochko 2eb17360df
Fix delete of local reply to local parent not being forwarded () 2021-04-22 14:26:11 +02:00
Claire 0b36e3419d
Fix processing of remote Delete activities ()
* Add tests

* Ensure deleted statuses are marked as such

* Save some redis memory by not storing URIs in delete_upon_arrival values

* Avoid possible race condition when processing incoming Deletes

* Avoid potential duplicate Delete forwards

* Lower lock durations to reduce issues in case of hard crash of the Rails process

* Check for `lock.aquired?` and improve comment

* Refactor RedisLock usage in app/lib/activitypub

* Fix using incorrect or non-existent sender for relaying Deletes
2021-04-21 04:46:09 +02:00
Eugen Rochko ca3bc1b09f
Refactor StatusReachFinder to handle followers and relays as well () 2021-04-17 15:41:57 +02:00
Eugen Rochko 6d6000f61f
Fix remote reporters not receiving suspend/unsuspend activities () 2021-04-17 14:55:46 +02:00
Eugen Rochko dde8739020
Fix reports of already suspended accounts being recorded () 2021-04-16 22:01:05 +02:00
Eugen Rochko 3b8d085436
Fix app name, website and redirect URIs not having a maximum length ()
Fix app scopes not being validated
2021-04-15 16:28:43 +02:00
Takeshi Umeda c968d22ee9
Fix an error with 'multiple mentions with same username' () 2021-04-14 15:48:49 +02:00
Eugen Rochko f7117646af
Add cold-start follow recommendations () 2021-04-12 12:37:14 +02:00
Eugen Rochko 619fad6cf8
Remove spam check and dependency on nilsimsa gem () 2021-04-11 11:22:50 +02:00
Eugen Rochko 7183d9a113
Change multiple mentions with same username to render with domain ()
Fix 
2021-04-10 11:51:02 +02:00
Claire c5fe0864d1
Fix SidekiqProcessCheck checking for a queue name that isn't used in Mastodon () 2021-04-05 23:41:37 +02:00
Eugen Rochko 487e37d6d4
Add system checks to dashboard in admin UI () 2021-04-03 14:12:30 +02:00
Claire cbd0ee1d07
Update Mastodon to Rails 6.1 ()
* Update devise-two-factor to unreleased fork for Rails 6 support

Update tests to match new `rotp` version.

* Update nsa gem to unreleased fork for Rails 6 support

* Update rails to 6.1.3 and rails-i18n to 6.0

* Update to unreleased fork of pluck_each for Ruby 6 support

* Run "rails app:update"

* Add missing ActiveStorage config file

* Use config.ssl_options instead of removed ApplicationController#force_ssl

Disabled force_ssl-related tests as they do not seem to be easily testable
anymore.

* Fix nonce directives by removing Rails 5 specific monkey-patching

* Fix fixture_file_upload deprecation warning

* Fix yield-based test failing with Rails 6

* Use Rails 6's index_with when possible

* Use ActiveRecord::Cache::Store#delete_multi from Rails 6

This will yield better performances when deleting an account

* Disable Rails 6.1's automatic preload link headers

Since Rails 6.1, ActionView adds preload links for javascript files
in the Links header per default.

In our case, that will bloat headers too much and potentially cause
issues with reverse proxies. Furhermore, we don't need those links,
as we already output them as HTML link tags.

* Switch to Rails 6.0 default config

* Switch to Rails 6.1 default config

* Do not include autoload paths in the load path
2021-03-24 10:44:31 +01:00
Claire 1c4dee4554
Fix Mastodon not understanding as:Public and Public ()
Fixes 
2021-03-24 10:19:40 +01:00
Claire a4dcaef53b
Prepare Mastodon for zeitwerk autoloader ()
* Prepare Mastodon for zeitwerk autoloader (Rails 6)

Add inflections and rename/move a few classes.

In particular, app/lib/exceptions.rb and app/lib/sanitize_config.rb
were manually loaded while still in autoload paths.

* Add inflection for Url → URL
2021-03-19 02:42:43 +01:00
Claire 5027abecd1
Fix cache_collection crashing when given an empty collection ()
* Fix cache_collection crashing when given an empty collection

* Add tests
2021-03-18 00:41:32 +01:00
Claire 43eff898a0
Prepare Mastodon for Rails 6 ()
* Fix misuse of foreign_type

* Fix use of removed "add_template_helper"

* Use response.media_type instead of response.content_type in tests

* Fix CSV export controller test on Rails 6

Rails 6 sets a "filename*" field in the Content-Disposition header to
explicitly encode the filename as UTF-8.

This changes checks the first part of the Content-Disposition header so
it matches in both Rails 5 and Rails 6.

* Fix emoji formatting with Rails 6

* Make emoji output more idiomatic and robust

* Switch from redis-rails gem to built-in Rails redis cache storage
2021-03-17 10:09:55 +01:00
Claire 5614e6724e
Fix URL scanning in note length validator and preview card fetching ()
* Add tests

* Fix URL scanning in note length validator and preview card fetching
2021-03-04 00:12:26 +01:00
Claire 65db262550
Update twitter-text from 1.14 to 3.1.0 and fix toot character counting ()
* Update twitter-text from 1.14 to 3.1.0

* Disable emoji parsing

* Properly depend on twitter-text for url detection

* Fix some URLs being wrongly detected client-side

* Add test for server-side validation of non-autolinkable URLs

* Fix server-side status length counting
2021-03-02 12:02:56 +01:00
Eugen Rochko 9aa37b32c3
Add details to error response for POST /api/v1/accounts in REST API () 2021-03-01 04:59:13 +01:00
Justin Tracey c9e8e1739c
replace all instances of "ends_with?" with "end_with?" ()
The "ends_with?" method is just a Rails alias of Ruby's "end_with?" method.
Using the latter makes the code less brittle.
2021-02-19 09:56:14 +01:00
Cecylia Bocovich e79f8dd85c
Onion service related changes to HTTPS handling ()
* Enable secure cookie flag for https only

* Disable force_ssl for .onion hosts only

Co-authored-by: Aiden McClelland <me@drbonez.dev>
2021-02-11 04:40:13 +01:00
Claire be3b9f8151
Fix URI of repeat follow requests not being recorded ()
* Fix URI of repeat follow requests not being recorded

In case we receive a “repeat” or “duplicate” follow request, we automatically
fast-forward the accept with the latest received Activity `id`, but we don't
record it.

In general, a “repeat” or “duplicate” follow request may happen if for some
reason (e.g. inconsistent handling of Block or Undo Accept activities, an
instance being brought back up from the dead, etc.) the local instance thought
the remote actor were following them while the remote actor thought otherwise.

In those cases, the remote instance does not know about the older Follow
activity `id`, so keeping that record serves no purpose, but knowing the most
recent one is useful if the remote implementation at some point refers to it
by `id` without inlining it.

* Add tests
2021-02-11 01:53:44 +01:00
ThibG a044ddac5b
Fix race conditions on account migration creation ()
* Atomically check for processing lock in Move handler

* Prevent race condition when creating account migrations

Fixes 

* Add tests

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2021-02-02 14:49:57 +01:00
luigi 7ea9588520
Use Enumerable#filter_map in more places () 2021-01-22 16:28:15 +01:00
luigi eb51e43fb4
Optimize some regex matching ()
* Use Regex#match?

* Replace =~ too

* Avoid to call match? from Nil

* Keep value of Regexp.last_match
2021-01-22 10:09:08 +01:00
ThibG 54d4e5252b
Use Rails' index_by where it makes sense ()
* Use Rails' index_by where it makes sense

* Fix tests

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2021-01-12 09:27:38 +01:00
ThibG 7bed25f3ea
Fix processing of incoming Block activities ()
Unlike locally-issued blocks, they weren't clearing follow
relationships in both directions, follow requests or notifications.

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2021-01-12 09:25:01 +01:00
luigi 087ed84367
Optimize map { ... }.compact calls ()
* Optimize map { ... }.compact

using Enumerable#filter_map, supported since Ruby 2.7

* Add poyfill for Enumerable#filter_map
2021-01-10 00:32:01 +01:00
ThibG 3249d35bdc
Improve account deletion performances further ()
* Delete status records by batches of 50

* Do not precompute values that are only used once

* Do not generate redis events for removal of public toots older than two weeks

* Filter reported toots a priori for polls and status deletion

* Do not process reblogs when cleaning up public timelines

As in Mastodon proper, reblogs don't appear in public TLs

* Clean the deleted account's own feed in one go

* Refactor Account#clean_feed_manager and List#clean_feed_manager

* Delete instead of destroy a few more associations

* Fix preloading

Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2020-12-22 23:57:46 +01:00
Eugen Rochko 1045549f85
Add stoplight for object storage failures, return HTTP 503 () 2020-12-15 12:55:29 +01:00
trwnh 127c543a6e
rename replies_policy enumerables () 2020-12-09 04:34:17 +01:00
Eugen Rochko e7e099d1a0
Fix deletes not reaching every server that interacted with status ()
Extract logic for determining ActivityPub inboxes to send deletes
to to its own class and explicitly include the person the status
replied to (even if not mentioned), people who favourited it, and
people who replied to it (though that one is still not recursive)
2020-11-27 15:48:31 +01:00
ThibG 2f6831f318
Fix sending spurious Rejects when processing remote account deletion ()
* Fix sending spurious Rejects when processing remote account deletion

* Make skip_side_effects imply skip_activitypub
2020-11-19 17:39:47 +01:00
Eugen Rochko df1653174b
Add cache buster feature for media files ()
Nginx can be configured to bypass proxy cache when a special header
is in the request. If the response is cacheable, it will replace
the cache for that request. Proxy caching of media files is
desirable when using object storage as a way of minimizing bandwidth
costs, but has the drawback of leaving deleted media files for
a configured amount of cache time. A cache buster can make those
media files immediately unavailable. This especially makes sense
when suspending and unsuspending an account.
2020-11-19 17:38:06 +01:00
Eugen Rochko aa10200e58
Fix streaming API allowing connections to persist after access token invalidation ()
Fix 
2020-11-12 23:05:24 +01:00
ThibG b0893291ef
Fix Move handler not being triggered when failing to fetch target ()
When failing to fetch the target account, the ProcessingWorker fails
as expected, but since it hasn't cleared the `move_in_progress` flag,
the next attempt at processing skips the `Move` activity altogether.

This commit changes it to clear the flag when encountering any
unexpected error on fetching the target account. This is likely to
occur because, of, e.g., a timeout, when many instances query the
same actor at the same time.
2020-11-09 01:41:52 +01:00
Eugen Rochko 3134691948
Add support for reversible suspensions through ActivityPub () 2020-11-08 00:28:39 +01:00
Takeshi Umeda d6fe0c94ca
Add account sensitized ()
* Add account sensitized

* Fix i18n normalize

* Fix description and spec

* Fix spec

* Fix wording
2020-11-04 20:45:01 +01:00
ThibG ca56527140
Add follower synchronization mechanism ()
* Add support for followers synchronization on the receiving end

Check the `collectionSynchronization` attribute on `Create` and `Announce`
activities and synchronize followers from provided collection if possible.

* Add tests for followers synchronization on the receiving end

* Add support for follower synchronization on the sender's end

* Add tests for the sending end

* Switch from AS attributes to HTTP header

Replace the custom `collectionSynchronization` ActivityStreams attribute by
an HTTP header (`X-AS-Collection-Synchronization`) with the same syntax as
the `Signature` header and the following fields:
- `collectionId` to specify which collection to synchronize
- `digest` for the SHA256 hex-digest of the list of followers known on the
   receiving instance (where “receiving instance” is determined by accounts
   sharing the same host name for their ActivityPub actor `id`)
- `url` of a collection that should be fetched by the instance actor

Internally, move away from the webfinger-based `domain` attribute and use
account `uri` prefix to group accounts.

* Add environment variable to disable followers synchronization

Since the whole mechanism relies on some new preconditions that, in some
extremely rare cases, might not be met, add an environment variable
(DISABLE_FOLLOWERS_SYNCHRONIZATION) to disable the mechanism altogether and
avoid followers being incorrectly removed.

The current conditions are:
1. all managed accounts' actor `id` and inbox URL have the same URI scheme and
   netloc.
2. all accounts whose actor `id` or inbox URL share the same URI scheme and
   netloc as a managed account must be managed by the same Mastodon instance
   as well.

As far as Mastodon is concerned, breaking those preconditions require extensive
configuration changes in the reverse proxy and might also cause other issues.

Therefore, this environment variable provides a way out for people with highly
unusual configurations, and can be safely ignored for the overwhelming majority
of Mastodon administrators.

* Only set follower synchronization header on non-public statuses

This is to avoid unnecessary computations and allow Follow-related
activities to be handled by the usual codepath instead of going through
the synchronization mechanism (otherwise, any Follow/Undo/Accept activity
would trigger the synchronization mechanism even if processing the activity
itself would be enough to re-introduce synchronization)

* Change how ActivityPub::SynchronizeFollowersService handles follow requests

If the remote lists a local follower which we only know has sent a follow
request, consider the follow request as accepted instead of sending an Undo.

* Integrate review feeback

- rename X-AS-Collection-Synchronization to Collection-Synchronization
- various minor refactoring and code style changes

* Only select required fields when computing followers_hash

* Use actor URI rather than webfinger domain in synchronization endpoint

* Change hash computation to be a XOR of individual hashes

Makes it much easier to be memory-efficient, and avoid sorting discrepancy issues.

* Marginally improve followers_hash computation speed

* Further improve hash computation performances by using pluck_each
2020-10-21 18:04:09 +02:00
abcang 9649ca0fbe
Removed disabling comments for Style/MethodMissingSuper ()
* Removed disabling comments for Style/MethodMissingSuper

* Update rubocop for codeclimate
2020-10-20 09:13:39 +02:00
Josh Leeb-du Toit 0c24f4dce2
Add support for Gemini urls ()
This PR updates the `valid_url` regex and sanitizer allowlist to provide
support for Gemini urls.

Closes 
2020-10-19 17:02:13 +02:00
Takeshi Umeda b4c4af18dc
Fix a bear check when the activity object is nil () 2020-10-13 22:41:00 +02:00
Eugen Rochko 5e1364c448
Add IP-based rules () 2020-10-12 16:33:49 +02:00
Eugen Rochko 7d985f2aac
Remove dependency on goldfinger gem ()
There are edge cases where requests to certain hosts timeout when
using the vanilla HTTP.rb gem, which the goldfinger gem uses. Now
that we no longer need to support OStatus servers, webfinger logic
is so simple that there is no point encapsulating it in a gem, so
we can just use our own Request class. With that, we benefit from
more robust timeout code and IPv4/IPv6 resolution.

Fix 
2020-10-08 00:34:57 +02:00
ThibG 78e45a5285
Add option to disable swiping motions across the WebUI ()
Fixes 
2020-09-30 19:31:03 +02:00
ThibG d60290044e
Add environment variable to allow requests to some private addresses () 2020-09-26 20:57:39 +02:00
Eugen Rochko 974b1b79ce
Add option to be notified when a followed user posts ()
* Add bell button

Fix 

* Remove duplicate type from post-deployment migration

* Fix legacy class type mappings

* Improve query performance with better index

* Fix validation

* Remove redundant index from notifications
2020-09-18 17:26:45 +02:00
Eugen Rochko ed099d8bdc
Change account suspensions to be reversible by default () 2020-09-15 14:37:58 +02:00
Eugen Rochko 65760f59df
Refactor feed manager () 2020-09-08 03:41:16 +02:00
ThibG 517af45e32
Fix multiple boosts of a same toot erroneously appearing in TL ()
* Check for and record reblog info atomically

Instead of using ZREVRANK to determine whether a reblog is a new reblog or not,
use ZADD's NX option to perform the check/addition option atomically.

* Replace ZREVRANK call with ZSCORE key which is more efficient

* Make tests a bit stricter

* Fix off-by-one
2020-09-07 18:00:15 +02:00
Eugen Rochko 17340365bb
Add featured hashtags as an ActivityPub collection () 2020-09-02 02:11:12 +02:00
ThibG 79305428a7
Add configuration option to filter replies in lists ()
* Add database support for list show-reply preferences

* Add backend support to read and update list-specific show_replies settings

* Add basic UI to set list replies setting

* Add specs for list replies policy

* Switch "cycling" reply policy link to a set of radio inputs

* Capitalize replies_policy strings

* Change radio button design to be consistent with that of the directory explorer
2020-09-01 13:31:28 +02:00
Eugen Rochko 1c308af84c
Change own direct-visibility statuses to be in the home feed again ()
And remove highlighting in web UI

Full circle from 
2020-09-01 13:30:42 +02:00
dependabot[bot] 8972e5f7f6
Bump rubocop from 0.86.0 to 0.88.0 ()
* Bump rubocop from 0.86.0 to 0.88.0

Bumps [rubocop](https://github.com/rubocop-hq/rubocop) from 0.86.0 to 0.88.0.
- [Release notes](https://github.com/rubocop-hq/rubocop/releases)
- [Changelog](https://github.com/rubocop-hq/rubocop/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rubocop-hq/rubocop/compare/v0.86.0...v0.88.0)

Signed-off-by: dependabot[bot] <support@github.com>

* Fix for latest RuboCop

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Yamagishi Kazutoshi <ykzts@desire.sh>
2020-09-01 03:04:00 +02:00
Eugen Rochko 52157fdcba
Add support for dereferencing objects through bearcaps () 2020-08-30 12:34:20 +02:00
ThibG 92319d7340
Fix dereferencing remote statuses not using the correct account ()
Follow-up to 

In the case of limited toots, the receiver may not be explicitly part of the
audience. If a specific user's inbox URI was specified, it makes sense to
dereference the toot from the corresponding user, instead of trying to find
someone in the explicit audience.
2020-08-24 16:56:21 +02:00
ThibG 720214feb0
Add support for inlined objects in activity audience ()
* Add support for inlined objects in activity audience

* Add tests
2020-08-24 14:11:47 +02:00
ThibG bfd5aea206
Fix handling of Reject Follow when a matching follow relationship exists ()
* Add tests

* Fix handling of Reject Follow when a matching follow relationship exists

Regression from 
2020-08-01 18:20:37 +02:00
ThibG 5d9acc0ce4
Fix not handling Undo on some activity types when they aren't inlined ()
* Fix not handling Undo on some activity types when they aren't inlined

When receiving an Undo for a non-inlined activity, try looking it up in
database using the URI. The queries are ad-hoc because we don't have a global
index of object URIs, and not all activity types are stored in database with
an index on their URI.

Announces are just statuses, and have an index on URIs, so this check can
be done efficiently.

Accepts cannot be handled at all because we don't record their URI at any
point.

Follows don't have an index on URI, but they have an index on the issuing
account, which should make such queries largely manageable.

Likes don't have an index on URI, they have an index on the issuing account,
but the number of favs per account may be very high, so I decided not to
handle that.

Blocks don't have an index on URI, but they have an index on the issuing
account, which should make such queries largely manageable.

In all cases, if an Undo could not be handled properly, we call `delete_later!`
because that does not require us to know more than the URI of the undone
property.

* Add tests

* Make newer blocks overwrite older ones

Allows re-synchronizing block info by re-blocking and un-blocking again
when the original Undo Block has been lost.
2020-07-22 11:45:35 +02:00
ThibG bcf85b5208
Dereference object URIs in Create and Update messages ()
* Dereference object URIs in Create and Update messages

Fixes 

Signed-off-by: Thibaut Girka <thib@sitedethib.com>

* Refactor, and perform origin check *before* attempting to fetch object

Co-authored-by: Fire Demon <firedemon@creature.cafe>
2020-07-22 11:43:17 +02:00
Takeshi Umeda fcb3f259e5
Fix to add RedisLock to handle Announce activity () 2020-07-20 11:25:26 +02:00
Takeshi Umeda 7540e235a2
Fix movie width and frame_rate returning nil ()
* Fix movie width and frame_rate returning nil

* Add StreamValidationError and raise

* Fix code style
2020-07-19 22:28:27 +02:00
ThibG 322d74fc2a
Fix boosted toots from blocked account not being retroactively removed from TL ()
* Fix boosted toots from blocked account not being retroactively removed from TL

Fixes 

* Add test for clear_from_timeline
2020-07-17 07:07:54 +02:00
Takeshi Umeda 5c23ec2fa6
Improve group processing ()
* Fix the local group's followers collection

* Fix to accept followed relayed_through_account

* Add local delivery to the group's followers

* Fix code style

* Revert "Add local delivery to the group's followers"

This reverts commit 3237effc19.
2020-07-14 19:05:34 +02:00
abcang 6a96af4d20
Fix rubocop warning ()
* Fix rubocop warning

* use limit variable

* use ContextCreatingMethods option
2020-07-14 19:05:07 +02:00
Eugen Rochko 6d23d40420
Change Redis#exists calls to Redis#exists? to avoid deprecation warning () 2020-07-01 19:05:21 +02:00
Eugen Rochko 7aaf2b44ec
Fix remote files not using Content-Type header, streaming () 2020-06-30 23:58:02 +02:00
Eugen Rochko 64aac30733
Add customizable thumbnails for audio and video attachments ()
- Change audio files to not be stripped of metadata
- Automatically extract cover art from audio if it exists
- Add `thumbnail` parameter to `POST /api/v1/media`, `POST /api/v2/media` and `PUT /api/v1/media/:id`
- Add `icon` to represent it in attachments in ActivityPub
- Fix `preview_url` containing URL of missing missing image when there is no thumbnail instead of null
- Fix duration of audio not being displayed on public pages until the file is loaded
2020-06-29 13:56:55 +02:00
Sasha Sorokin f5cea4d2ea
Fix Thai being skipped from language detection ()
Thai does not separate words by spaces, so I figured out it should be
in 'reliable characters regexp' that denotes languages that do the same.

Related .
2020-06-25 22:45:01 +02:00
ThibG 01a99f7ec7
Fix crash in MergeWorker ()
Similarly to , the code is passing an Account object where an id
is expected.
2020-06-23 16:40:01 +02:00
Ben Lubar c66403b257
FIX: filters ignore media descriptions ()
* FIX: filters ignore media descriptions

* remove parentheses to make codeclimate happy

* combine the text and run the regular expression only once.

https://github.com/tootsuite/mastodon/pull/13837#discussion_r431752581

* Fix use of “filter” instead of “compact”, fix coding style issues

Co-authored-by: Thibaut Girka <thib@sitedethib.com>
2020-06-09 00:11:42 +02:00
Eugen Rochko 5d8398c8b8
Add E2EE API () 2020-06-02 19:24:53 +02:00
ThibG a319c1e60f
Add support for summary field for media description () 2020-05-15 17:08:59 +02:00
ThibG 71fce71c94
Fix webfinger returning wrong status code on malformed or missing param ()
Fixes 
2020-05-14 23:28:06 +02:00
Takeshi Umeda 5daae6dc61
Fix keybase config serializer () 2020-05-12 15:28:29 +02:00
Eugen Rochko e9ecbca70d
Fix error within error when limiting backtrace to 3 lines ()
Fix , close 
2020-05-10 10:30:27 +02:00
ThibG a4240fd027
Improve RSS entries for statuses ()
* Improve RSS entries for statuses

- Render polls in both accounts and tags serializers
- Refactor RSS serializers
- Change title preview to include ellipsis when truncated
- Change title preview to show CW instead of toot text
- Add tests

* Remove title from OEmbed serialization

Twitter doesn't serialize title either, and tihs allows us to move the
title formatting code to the RSS serializers.
2020-05-10 09:50:54 +02:00
Eugen Rochko 5edff32733
Change delivery failure tracking to work with hostnames instead of URLs () 2020-04-15 20:33:24 +02:00
Eugen Rochko c9efb400b4
Add rate limit for reporting () 2020-04-05 14:40:08 +02:00
ThibG 89e28c7674
Fix PostgreSQL load when linking in announcements ()
* Fix PostgreSQL load when linking in announcements

Fixes  by caching status lookups

Since statuses are supposed to be known already and we only
need their URLs and a few other things, caching them should
be fine.

Since it's only used by announcements so far, there won't
be much statuses to cache.

* Perform status lookup when saving announcements, not when rendering them

* Change EntityCache#status to fetch URLs instead of looking into the database

* Move announcement link lookup to publishing worker

* Address issues pointed out during review
2020-04-05 12:51:22 +02:00
Eugen Rochko 0c8945e5ff
Change tootctl media remove-orphans to work for all classes ()
Change `tootctl media lookup` to not use an interactive prompt
2020-03-26 01:56:41 +01:00
dependabot-preview[bot] 7088633ae1
Bump cld3 from 3.2.6 to 3.3.0 ()
* Bump cld3 from 3.2.6 to 3.3.0

Bumps [cld3](https://github.com/akihikodaki/cld3-ruby) from 3.2.6 to 3.3.0.
- [Release notes](https://github.com/akihikodaki/cld3-ruby/releases)
- [Commits](https://github.com/akihikodaki/cld3-ruby/compare/v3.2.6...v3.3.0)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* Fix compatibility with cld3 3.3.0

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
Co-authored-by: Eugen Rochko <eugen@zeonfederated.com>
2020-03-09 00:12:52 +01:00
Eugen Rochko 339ce1c4e9
Add specific rate limits for posting and following () 2020-03-08 15:17:39 +01:00
Eugen Rochko b1349342d2
Fix rendering <a> without href when scheme unsupported ()
- Disallow links with relative paths
- Disallow iframes with non-http protocols and relative paths

Close 
2020-02-08 21:22:38 +01:00
Eugen Rochko a64973aecf
Fix malformed HTML causing uncaught error ()
Fix OEmbed preview API leaking existence of private statuses (see )
2020-02-07 15:24:22 +01:00
Eugen Rochko b9d74d4076
Add streaming API updates for announcements being modified or deleted ()
Change `all_day` to be a visual client-side cue only

Publish immediately if `scheduled_at` is in the past

Add `published_at` and `updated_at` to announcements JSON
2020-01-26 20:07:26 +01:00
Eugen Rochko f52c988e12
Add announcements ()
* Add announcements

Fix 

* Add reactions to announcements

* Add admin UI for announcements

* Add unit tests

* Fix issues

- Add `with_dismissed` param to announcements API
- Fix end date not being formatted when time range is given
- Fix announcement delete causing reactions to send streaming updates
- Fix announcements container growing too wide and mascot too small
- Fix `all_day` being settable when no time range is given
- Change text "Update" to "Announcement"

* Fix scheduler unpublishing announcements before they are due

* Fix filter params not being passed to announcements filter
2020-01-23 22:00:13 +01:00
ThibG a8e46cf7a1 Add support for magnet: URIs () 2020-01-23 21:27:26 +01:00
ThibG 345bba4812 Exempt staff members from spam check ()
Consider admins and moderators as trusted, for the purpose of the
spam checker.

Fixes 
2020-01-17 07:50:30 +01:00
ThibG 24cd2126c6 Fix Ruby 2.7 support ()
- update http gem to avoid errors
- update blurhash gem to avoid shared object loading error
- update goldfinger gem so the http gem could be updated
- update json gem to avoid warnings
2020-01-12 15:49:30 +01:00
ThibG ea436b355b Add support for linking XMPP URIs in toots ()
* Fix wrong grouping in Twitter valid_url regex

* Add support for xmpp URIs

Fixes 

The difficult part is autolinking, because Twitter-text's extractor does
some pretty ad-hoc stuff to find things that “look like” URLs, and XMPP
URIs do not really match the assumptions of that lib, so it doesn't sound
wise to try to shoehorn it into the existing regex.

This is why I used a specific regex (very close, although slightly more
permissive than the RFC), and a specific scan function (a simplified version
of the generalized one from Twitter).

* Remove leading “xmpp:” from auto-linked text
2020-01-11 02:15:25 +01:00
Daigo 3 Dango e9ea09d173 Suppress backtrace when delivering toots ()
This is to suppress irrelevant backtrace from errors raised when
delivering toots to remote servers. The errors are usually out of
control by the local server and backtraces don't provide much
information.

This is similar to https://github.com/tootsuite/mastodon/pull/5174
and shortens backtraces like below:

```
WARN: Mastodon::UnexpectedResponseError: https://example.com/inbox returned code 523
WARN: app/workers/activitypub/delivery_worker.rb:48:in `block (3 levels) in perform_request'
app/lib/request.rb:75:in `perform'
app/workers/activitypub/delivery_worker.rb:47:in `block (2 levels) in perform_request'
app/lib/request_pool.rb:53:in `use'
app/lib/request_pool.rb:108:in `block (2 levels) in with'
vendor/bundle/ruby/2.7.0/gems/activesupport-5.2.4.1/lib/active_support/notifications.rb:170:in `instrument'
app/lib/request_pool.rb:107:in `block in with'
app/lib/connection_pool/shared_connection_pool.rb:21:in `block (2 levels) in with'
app/lib/connection_pool/shared_connection_pool.rb:20:in `handle_interrupt'
app/lib/connection_pool/shared_connection_pool.rb:20:in `block in with'
app/lib/connection_pool/shared_connection_pool.rb:16:in `handle_interrupt'
app/lib/connection_pool/shared_connection_pool.rb:16:in `with'
app/lib/request_pool.rb:106:in `with'
app/workers/activitypub/delivery_worker.rb:46:in `block in perform_request'
vendor/bundle/ruby/2.7.0/gems/stoplight-2.2.0/lib/stoplight/light/runnable.rb:51:in `run_code'
vendor/bundle/ruby/2.7.0/gems/stoplight-2.2.0/lib/stoplight/light/runnable.rb:42:in `run_yellow'
vendor/bundle/ruby/2.7.0/gems/stoplight-2.2.0/lib/stoplight/light/runnable.rb:24:in `run'
app/workers/activitypub/delivery_worker.rb:57:in `perform_request'
app/workers/activitypub/delivery_worker.rb:25:in `perform'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:192:in `execute_job'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:165:in `block (2 levels) in process'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'
vendor/bundle/ruby/2.7.0/gems/nsa-0.2.7/lib/nsa/collectors/sidekiq.rb:31:in `block in call'
vendor/bundle/ruby/2.7.0/gems/nsa-0.2.7/lib/nsa/statsd/publisher.rb:27:in `statsd_time'
vendor/bundle/ruby/2.7.0/gems/nsa-0.2.7/lib/nsa/collectors/sidekiq.rb:30:in `call'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
app/lib/sidekiq_error_handler.rb:5:in `call'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
vendor/bundle/ruby/2.7.0/gems/scout_apm-2.3.0.pre3/lib/scout_apm/background_job_integrations/sidekiq.rb:69:in `call'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
vendor/bundle/ruby/2.7.0/gems/sidekiq-unique-jobs-6.0.18/lib/sidekiq_unique_jobs/server/middleware.rb:29:in `call'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:133:in `invoke'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:164:in `block in process'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:137:in `block (6 levels) in dispatch'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:109:in `local'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:136:in `block (5 levels) in dispatch'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/rails.rb:43:in `block in call'
vendor/bundle/ruby/2.7.0/gems/activesupport-5.2.4.1/lib/active_support/execution_wrapper.rb:87:in `wrap'
vendor/bundle/ruby/2.7.0/gems/activesupport-5.2.4.1/lib/active_support/reloader.rb:73:in `block in wrap'
vendor/bundle/ruby/2.7.0/gems/activesupport-5.2.4.1/lib/active_support/execution_wrapper.rb:87:in `wrap'
vendor/bundle/ruby/2.7.0/gems/activesupport-5.2.4.1/lib/active_support/reloader.rb:72:in `wrap'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/rails.rb:42:in `call'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:132:in `block (4 levels) in dispatch'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:250:in `stats'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:127:in `block (3 levels) in dispatch'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/job_logger.rb:8:in `call'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:126:in `block (2 levels) in dispatch'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:74:in `global'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:125:in `block in dispatch'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:48:in `with_context'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:42:in `with_job_hash_context'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:124:in `dispatch'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:163:in `process'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:83:in `process_one'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:71:in `run'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:16:in `watchdog'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:25:in `block in safe_thread'
```

```
WARN: Stoplight::Error::RedLight: https://example.com/inbox
WARN: vendor/bundle/ruby/2.7.0/gems/stoplight-2.2.0/lib/stoplight/light/runnable.rb:46:in `run_red'
vendor/bundle/ruby/2.7.0/gems/stoplight-2.2.0/lib/stoplight/light/runnable.rb:25:in `run'
app/workers/activitypub/delivery_worker.rb:57:in `perform_request'
app/workers/activitypub/delivery_worker.rb:25:in `perform'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:192:in `execute_job'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:165:in `block (2 levels) in process'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'
vendor/bundle/ruby/2.7.0/gems/nsa-0.2.7/lib/nsa/collectors/sidekiq.rb:31:in `block in call'
vendor/bundle/ruby/2.7.0/gems/nsa-0.2.7/lib/nsa/statsd/publisher.rb:27:in `statsd_time'
vendor/bundle/ruby/2.7.0/gems/nsa-0.2.7/lib/nsa/collectors/sidekiq.rb:30:in `call'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
app/lib/sidekiq_error_handler.rb:5:in `call'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
vendor/bundle/ruby/2.7.0/gems/scout_apm-2.3.0.pre3/lib/scout_apm/background_job_integrations/sidekiq.rb:69:in `call'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
vendor/bundle/ruby/2.7.0/gems/sidekiq-unique-jobs-6.0.18/lib/sidekiq_unique_jobs/server/middleware.rb:29:in `call'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:133:in `invoke'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:164:in `block in process'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:137:in `block (6 levels) in dispatch'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:109:in `local'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:136:in `block (5 levels) in dispatch'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/rails.rb:43:in `block in call'
vendor/bundle/ruby/2.7.0/gems/activesupport-5.2.4.1/lib/active_support/execution_wrapper.rb:87:in `wrap'
vendor/bundle/ruby/2.7.0/gems/activesupport-5.2.4.1/lib/active_support/reloader.rb:73:in `block in wrap'
vendor/bundle/ruby/2.7.0/gems/activesupport-5.2.4.1/lib/active_support/execution_wrapper.rb:87:in `wrap'
vendor/bundle/ruby/2.7.0/gems/activesupport-5.2.4.1/lib/active_support/reloader.rb:72:in `wrap'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/rails.rb:42:in `call'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:132:in `block (4 levels) in dispatch'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:250:in `stats'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:127:in `block (3 levels) in dispatch'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/job_logger.rb:8:in `call'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:126:in `block (2 levels) in dispatch'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:74:in `global'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:125:in `block in dispatch'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:48:in `with_context'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:42:in `with_job_hash_context'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:124:in `dispatch'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:163:in `process'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:83:in `process_one'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:71:in `run'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:16:in `watchdog'
vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:25:in `block in safe_thread'
```
2020-01-11 02:15:03 +01:00
Daigo 3 Dango 206dfd7dad Clarify keyword arguments with ** ()
This change is to suppress the warning below on on ruby-2.7.0:

- warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call

https://www.ruby-lang.org/en/news/2019/12/12/separation-of-positional-and-keyword-arguments-in-ruby-3-0/
2020-01-11 05:57:05 +09:00
Eugen Rochko bb27ca14ef
Fix error when searching for empty phrase ()
Fix 
2019-12-30 18:30:24 +01:00
ThibG 2ee5a9d9c3 Clean up OStatus-related codepaths ()
* Remove “protocol” argument and return value, as only ActivityPub is supported

* Remove FetchRemoteAccountService, only use ActivityPub::FetchRemoteAccountService

* Fix tests
2019-12-17 13:32:57 +01:00
ThibG da2143b308 Fixes featured hashtag setting page erroring out instead of rejecting invalid tags ()
* Revert "Fix ignoring whole status because of one invalid hashtag ()"

This reverts commit dff46b260b.

* Fix statuses being rejected because of invalid hashtag names

* Add spec for invalid hashtag names in statuses

* Add test for featured tags controller
2019-12-17 13:31:56 +01:00
Thomas Citharel 8094955461 Add Event activity-type support ()
This adds support for Event AP type in Mastodon. Events are converted
into toots by taking their title (AS name) and their URL (AP ID). Event
picture is also brought in if available.

Testable by fetching event content from https://test.mobilizon.org

Signed-off-by: Thomas Citharel <tcit@tcit.fr>
2019-12-16 23:55:28 +01:00
Takeshi Umeda f43f1e0184 Add basic support for group actors ()
* Show badge on group actor in WebUI

* Do not notify in case of  by following group actor

* If you mention group actor, also mention group actor followers

* Relax characters that can be used in username (same as Application)

* Revert "Relax characters that can be used in username (same as Application)"

This reverts commit 7e10a137b8.

* Delete display_name method
2019-12-04 20:36:33 +01:00
Yamagishi Kazutoshi 6be16d02cb Update ESLint and RuboCop in Code Climate () 2019-12-02 18:25:43 +01:00
ThibG f05b0463db Fallback to Create audience when object has no defined audience ()
Fixes 
2019-11-30 19:58:00 +01:00
Darius Kazemi 0092096328 Fix type mismatch ()
This was [causing an issue with feed regeneartion in tootctl](https://github.com/hometown-fork/hometown/issues/24), and @davefp fixed the issue.
2019-11-07 21:51:48 +02:00
ThibG e37358b3bc Fix code referencing wrong class ()
For some reason, I have seen this only triggered here:
https://circleci.com/gh/tootsuite/mastodon/98324?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-checks-link

But Follow.find_by referenced the ActivityPub::Activity::Follow class
instead of the model class.
2019-11-04 13:02:27 +01:00
Faye Duxovni 48f75b86ae Add setting for whether to crop images in unexpanded toots () 2019-10-24 22:51:41 +02:00
puckipedia d2919f7e94 Allow Accept/Reject with a non-embedded object ()
Some ActivityPub servers refuse to embed remote objects into their own
output. This is because they are not the authoritative source for these
objects, and as such embedding them is always a waste of space. The
follow request and follow models contain a URI, so this can be used to
match them.
2019-10-24 22:45:43 +02:00
BSKY fccf83e1f2 Add noopener and/or noreferrer () 2019-10-24 22:44:42 +02:00
Eugen Rochko aa509a3d8a
Fix auto-report string saying the account has been auto-silenced () 2019-10-10 18:47:24 +02:00
Eugen Rochko b5f7e12817
Remove auto-silence behaviour from spam check ()
Fix 
2019-10-09 07:11:23 +02:00
Eugen Rochko f665901e3c
Fix performance of home feed regeneration ()
Fetching statuses from all followed accounts at once takes too long
within Postgres. Fetching them one by one and merging in Ruby
could be a lot less resource-intensive

Because the query for dynamically fetching the home timeline is so
heavy, we can no longer offer it when the home timeline is missing
2019-10-06 22:11:17 +02:00
ThibG 3babf8464b Add voters count support ()
* Add voters count to polls

* Add ActivityPub serialization and parsing of voters count

* Add support for voters count in WebUI

* Move incrementation of voters count out of redis lock

* Reword “voters” to “people”
2019-09-29 22:58:01 +02:00
Eugen Rochko 5f69eb89e2
Add a nodeinfo endpoint ()
* Add nodeinfo endpoint

* dont commit stuff from my local dev

* consistant naming since we implimented 2.1 schema

* Add some additional node info stuff

* Add nodeinfo endpoint

* dont commit stuff from my local dev

* consistant naming since we implimented 2.1 schema

* expanding this to include federation info

* codeclimate feedback

* CC feedback

* using activeserializers seems like a good idea...

* get rid of draft 2.1 version

* Reimplement 2.1, also fix metaData -> metadata

* Fix metaData -> metadata here too

* Fix nodeinfo 2.1 tests

* Implement cache for monthly user aggregate

* Useless

* Remove ostatus from the list of supported protocols

* Fix nodeinfo's open_registration reading obsolete setting variable

* Only serialize domain blocks with user-facing limitations

* Do not needlessly list noop severity in nodeinfo

* Only serialize domain blocks info in nodeinfo when they are set to be displayed to everyone

* Enable caching for nodeinfo endpoints

* Fix rendering nodeinfo

* CodeClimate fixes

* Please CodeClimate

* Change InstancePresenter#active_user_count_months for clarity

* Refactor NodeInfoSerializer#metadata

* Remove nodeinfo 2.1 support as the schema doesn't exist

* Clean-up
2019-09-29 21:31:51 +02:00
Eugen Rochko 368a87755b
Fix account migration not affecting followers on origin server () 2019-09-29 16:23:13 +02:00
ThibG 18b451c0e6 Change silences to always require approval on follow ()
* Change silenced accounts to require approval on follow

* Also require approval for follows by people explicitly muted by target accounts

* Do not auto-accept silenced or muted accounts when switching from locked to unlocked

* Add `follow_requests_count` to verify_credentials

* Show “Follow requests” menu item if needed even if account is locked

* Add tests

* Correctly reflect that follow requests weren't auto-accepted when local account is silenced

* Accept follow requests from user-muted accounts to avoid leaking mutes
2019-09-27 21:13:51 +02:00
Yamagishi Kazutoshi 67bef15e53 Add fallback section ID with ToC () 2019-09-23 17:25:10 +02:00
Yamagishi Kazutoshi 73a5ef03b2 Respect original ID with ToC () 2019-09-20 17:13:44 +02:00
Eugen Rochko d930eb88b6
Add table of contents to about page ()
Move public domain blocks information to about page
2019-09-19 11:09:05 +02:00
Eugen Rochko 4f6af87906
Change spam check to apply to local accounts and add a threshold ()
Instead of detecting spam on first duplicate message, add a
threshold of 5 such messages to reduce false positives
2019-09-18 12:53:13 +02:00
Eugen Rochko 38dc51b2d6
Fix Move handler queuing jobs that will fail if account is suspended ()
Don't put Move handler on cooldown if it didn't run. Skip unmerging
from timelines to save unnecessary work.
2019-09-17 08:44:45 +02:00
Eugen Rochko c5d37f18cb
Change deletes to preserve soft-deleted statuses in unresolved reports ()
Change all account actions except "none" to resolve all unresolved reports

Refactor `SuspendAccountService` to be more readable
2019-09-11 16:32:44 +02:00
Tao Bror Bojlén 4fe127664b add admin setting for default search engine indexing (fix ) () 2019-09-11 08:44:58 +02:00
Eugen Rochko 031ca25014
Add retry for failed media downloads and tootctl media refresh () 2019-09-10 15:29:12 +02:00
ThibG ec2a439a22 Add account bio to account admin view ()
* Add account bio to account admin view

* Change styling to make bio fields / content more readable
2019-09-09 14:15:52 +02:00
ThibG 1653b58777 Attempt to concurrently connect to remote IP addresses ()
* Attempt to concurrently connect to remote IP addresses

* Reduce code length to please CodeClimate 🤷
2019-09-05 05:32:53 +02:00
ThibG 692c5b439a Fix ActivityPub context not being dynamically computed ()
* Fix contexts not being dynamically included

Fixes 

* Refactor Note context in serializer

* Refactor Actor serializer
2019-09-03 22:52:32 +02:00
Eugen Rochko 70ddef2654
Change trending hashtags to not disappear instantly after midnight () 2019-09-02 18:11:13 +02:00
Eugen Rochko cb447b28c4
Add profile directory to web UI ()
* Add profile directory to web UI

* Add a line of bio to the directory
2019-08-30 00:14:36 +02:00
Eugen Rochko 97192d9a77
Fix remote and staff-removed statuses leaving media behind for a day ()
The reason for unattaching media instead of removing it is to support
delete & redraft functionality, but remote or staff-removed statuses
will never be redrafted, so the media should be deleted immediately
2019-08-22 04:17:12 +02:00
Eugen Rochko 70da6d6630
Fix accounts search by full/partial display name and others ()
- Restrict followers counts to local users to minimize local advantage
- Fix emoji shortcodes causing error in search
- Fix search syntax parse errors not being caught
2019-08-16 13:00:30 +02:00
Eugen Rochko 5e35aa8280
Fix non-lowercase hashtags not being picked up by the streaming API ()
Regression from f371b32

Fix hashtag links always being lowercase
2019-08-07 10:01:19 +02:00
Eugen Rochko 9072fe5ab6
Add trends UI with admin and user settings () 2019-08-06 17:57:52 +02:00
Eugen Rochko e46e9c9a8e
Fix delete regression ()
Regression from ff789a751a
2019-07-31 09:23:30 +02:00
ThibG ff789a751a Fix boosting & unboosting preventing a boost from appearing in the TL ()
* Fix boosting & unboosting preventing a boost from appearing in the TL

* Add tests

* Avoids side effects when aggregate_reblogs isn't true
2019-07-30 13:18:23 +02:00
Eugen Rochko b9b0313c78
Revert "Remove conversation URI ()" ()
This reverts commit 75f7f9930e.
2019-07-28 17:47:37 +02:00
Eugen Rochko 75f7f9930e
Remove conversation URI ()
It is not part of ActivityPub and will free up a lot of space
2019-07-28 17:30:12 +02:00
Eugen Rochko f371b32137
Change hashtags to preserve first-used casing () 2019-07-28 05:59:51 +02:00
Eugen Rochko b9fbcbfe4e
Add search syntax for operators and phrases () 2019-07-27 04:42:08 +02:00
ThibG 7de8c51873 Play animated custom emoji on hover ()
* Play animated custom emoji on hover in status

* Play animated custom emoji on hover in display names

* Play animated custom emoji on hover in bios/bio fields

* Add support for animation on hover on public pages emojis too

* Fix tests

* Code style cleanup
2019-07-21 18:10:40 +02:00
ThibG fda437a020 Fix sanitizing lists contents ()
* Add test

* Fix code for sanitizing nested lists stripping all tags
2019-07-19 01:44:58 +02:00
ThibG 730c4053d6 Add ActivityPub actor representing the entire server ()
* Add support for an instance actor

* Skip username validation for local Application accounts

* Add migration script to create instance actor

* Make Codeclimate happy

* Switch to id -99 for instance actor

* Remove unused `icon` and `image` attributes from instance actor

* Use if/elsif/else instead of return + ternary operator

* Add instance actor to fresh installs

* Use instance actor as instance representative

Use instance actor for forwarding reports, relay operations, and spam
auto-reporting.

* Seed database in test environment

* Fix single-user mode

* Fix tests

* Fix specs to accomodate for an extra `Account`

* Auto-reject follows on instance actor

Following an instance actor might make sense, but we are not handling that
right now, so auto-reject.

* Fix webfinger lookup and serialization for instance actor

* Rename instance actor

* Make it clear in the HTML view that the instance actor should not be blocked

* Raise cache time for instance actor as there's no dynamic content

* Re-use /about/more with a flash message for instance actor profile
2019-07-19 01:44:42 +02:00
Eugen Rochko 5bfe1e1f05
Change language detection to include hashtags as words () 2019-07-18 03:02:15 +02:00
ThibG 7e2b6da57f Add setting to disable the anti-spam ()
* Add environment variable to disable the anti-spam

* Move antispam setting to admin settings

* Fix typo

* antispam → spam_check
2019-07-17 21:09:15 +02:00
Eugen Rochko 9b1d3e4acb
Add option to disable real-time updates in web UI ()
Fix 
Fix 
2019-07-16 06:30:47 +02:00
Eugen Rochko 6ff67be0f6
Add a spam check ()
* Add a spam check

* Use Nilsimsa to generate locality-sensitive hashes and compare using Levenshtein distance

* Add more tests

* Add exemption when the message is a reply to something that mentions the sender

* Use Nilsimsa Compare Value instead of Levenshtein distance

* Use MD5 for messages shorter than 10 characters

* Add message to automated report, do not add non-public statuses to
automated report, add trust level to accounts and make unsilencing
raise the trust level to prevent repeated spam checks on that account

* Expire spam check data after 3 months

* Add support for local statuses, reduce expiration to 1 week, always create a report

* Add content warnings to the spam check and exempt empty statuses

* Change Nilsimsa threshold to 95 and make sure removed statuses are removed from the spam check

* Add all matched statuses into automatic report
2019-07-13 16:45:50 +02:00
Eugen Rochko 5bf67ca913
Add ActivityPub secure mode ()
* Add HTTP signature requirement for served ActivityPub resources

* Change `SECURE_MODE` to `AUTHORIZED_FETCH`

* Add 'Signature' to 'Vary' header and improve code style

* Improve code style by adding `public_fetch_mode?` method
2019-07-11 20:11:09 +02:00
Eugen Rochko 4e8dcc5dbb
Add HTTP signatures to all outgoing ActivityPub GET requests () 2019-07-11 14:49:55 +02:00
Eugen Rochko 5d3feed191
Refactor fetching of remote resources () 2019-07-10 18:59:28 +02:00
Eugen Rochko 4e92183227
Refactor domain block checks () 2019-07-09 03:27:35 +02:00
Eugen Rochko 63c7fe8e48
Refactor controllers for statuses, accounts, and more () 2019-07-08 12:03:45 +02:00
Eugen Rochko b851456139
Remove Atom feeds and old URLs in the form of GET /:username/updates/:id () 2019-07-07 16:16:51 +02:00
Eugen Rochko 406b46395d
Fix URLs appearing twice in errors of ActivityPub::DeliveryWorker () 2019-07-07 03:37:01 +02:00
ThibG 58276715be Fix support for HTTP proxies ()
* Disable incorrect check for hidden services in Socket

Hidden services can only be accessed with an HTTP proxy, in which
case the host seen by the Socket class will be the proxy, not the
target host.

Hidden services are already filtered in `Request#initialize`.

* Use our Socket class to connect to HTTP proxies

Avoid the timeout logic being bypassed

* Add support for IP addresses in Request::Socket

* Refactor a bit, no need to keep the DNS resolver around
2019-07-07 02:05:38 +02:00
Eugen Rochko 23aeef52cc
Remove Salmon and PubSubHubbub ()
* Remove Salmon and PubSubHubbub endpoints

* Add error when trying to follow OStatus accounts

* Fix new accounts not being created in ResolveAccountService
2019-07-06 23:26:16 +02:00
ThibG ed544f3166 Fix HTTP requests to IPv6 hosts () 2019-07-05 22:03:00 +02:00
Eugen Rochko 1a747f7020
Fix statsd UDP sockets not being cleaned up in Sidekiq () 2019-07-02 01:01:17 +02:00
Eugen Rochko 0d9ffe56fb
Add request pool to improve delivery performance ()
* Add request pool to improve delivery performance

Fix 

* Ensure connection is closed when exception interrupts execution

* Remove Timeout#timeout from socket connection

* Fix infinite retrial loop on HTTP::ConnectionError

* Close sockets on failure, reduce idle time to 90 seconds

* Add MAX_REQUEST_POOL_SIZE option to limit concurrent connections to the same server

* Use a shared pool size, 512 by default, to stay below open file limit

* Add some tests

* Add more tests

* Reduce MAX_IDLE_TIME from 90 to 30 seconds, reap every 30 seconds

* Use a shared pool that returns preferred connection but re-purposes other ones when needed

* Fix wrong connection being returned on subsequent calls within the same thread

* Reduce mutex calls on flushes from 2 to 1 and add test for reaping
2019-07-02 00:34:38 +02:00
ThibG 3086c645fd Add option to disable blurhash previews ()
* Add option to disable blurhash previews

* Update option text

* Change options order
2019-06-26 19:33:04 +02:00
ThibG 915c619394 Add support for Audio activities ()
Fixes 
2019-06-26 19:32:36 +02:00
ThibG 47ef4a6c7a Apply filters to poll options ()
* Apply filters to poll options in WebUI

Fixes 

* Apply filters to poll options server-side

* Add poll options to searchable text
2019-06-25 14:45:14 +02:00
Eugen Rochko aa9b37822b
Fix audio not being downloaded from remote servers () 2019-06-22 02:50:36 +02:00
Eugen Rochko 707ddf7808
Change domain blocks to automatically support subdomains ()
* Change domain blocks to automatically support subdomains

If a more authoritative domain is blocked (example.com), then the
same block will be applied to a subdomain (foo.example.com)

* Match subdomains of existing accounts when blocking/unblocking domains

* Improve code style
2019-06-22 00:13:10 +02:00
Eugen Rochko 103a9f4466
Fix sanitizer making block level elements unreadable ()
Fix 
2019-06-16 21:46:36 +02:00
Eugen Rochko 8746f4d17b
Change priority of delete activity forwards for replies and reblogs ()
Fix 
2019-06-09 12:47:33 +02:00
ThibG 6c464cd424 Do not misattribute inlined boosts if attributedTo isn't present ()
* Do not misattribute inlined boosts if `attributedTo` isn't present

Fixes 

* Fix tests
2019-06-04 23:24:31 +02:00
Eugen Rochko f2b743e715
Refactor all ActivityPub deliveries to be serialized and signed through one concern () 2019-06-04 23:11:18 +02:00
ThibG 3a84bacf86 Handle blank poll options more gracefully ()
Pleroma currently allows (erroneously imho) empty poll options, that is,
options with an empty (but existing) `name`.
2019-06-03 20:04:00 +02:00
Eugen Rochko 1e5532e693
Add responsive panels to the single-column layout ()
* Add responsive panels to the single-column layout

* Fixes

* Fix not being able to save the preference

* Fix code style issues

* Set max-height on the compose textarea and add a link to relationship manager
2019-05-25 21:27:00 +02:00
ThibG 9a5561a5b8 Fix possible race condition when processing statuses () 2019-05-23 20:00:39 +02:00
ThibG c1c8c7b2c8 Minor performance improvements and cleanup in formatter () 2019-05-15 06:54:06 +02:00
ThibG 62f5235b6f Prevent silenced local users from notifying remote users not following them ()
* Prevent silenced local users from notifying remote users not following them

This is an attempt to extend the local restrictions of silenced users to the
federation.

* Add tests

* Add tests for making sure private status don't get sent over OStatus
2019-05-09 22:05:43 +02:00
Eugen Rochko fba96c808d
Add blurhash ()
* Add blurhash

* Use fallback color for spoiler when blurhash missing

* Federate the blurhash and accept it as long as it's at most 5x5

* Display unknown media attachments as blurhash placeholders

* Improve style of embed actions and spoiler button

* Change blurhash resolution from 3x3 to 4x4

* Improve dependency definitions

* Fix code style issues
2019-04-27 03:24:09 +02:00
Ben Lubar be8692b938 Default to the web domain (eg. mastodon.lubar.me) instead of the local domain (eg. lubar.me) for keybase proofs () 2019-04-21 04:53:24 +02:00
Eugen Rochko a9f130b8d8
Fix Keybase verification using wrong domain for remote accounts () 2019-04-10 20:28:43 +02:00
Takeshi Umeda a4d5e15327 Fix config serializer regex () 2019-04-08 07:55:38 +02:00
Eugen Rochko 1714ea5978
Add ActivityPub representation for identity proofs ()
* Add ActivityPub representation for identity proofs

* Add tests
2019-03-30 02:12:06 +01:00
Alex Gessner 69141dca26 squashed identity proof updates () 2019-03-28 18:01:09 +01:00
Eugen Rochko f1bc90ab50
Rename :poll to :preloadable_poll and :owned_poll to :poll on Status ()
Also, fix some n+1 queries

Resolve 
2019-03-28 04:44:59 +01:00
Eugen Rochko e86663b1da
Fix alternative relay support regression ()
Fix 
2019-03-27 19:58:24 +01:00
Eugen Rochko 11fe293e1b
Remove unused ActivityPub @context values depending on response ()
Fix 
2019-03-27 15:55:23 +01:00
Eugen Rochko e6cfa7ab89
Change language detector threshold from 140 characters to 4 words ()
Add `lang` attribute to statuses in web UI
2019-03-26 01:23:59 +01:00
Eugen Rochko 3411fbef19
Improve config serializer for Keybase ()
- Regex must no longer be surrounded by `/`
- Description must be short and cannot contain HTML tags
2019-03-21 23:33:28 +01:00
ThibG 80f0910e21 Add support for custom emojis in poll options ()
* Backend changes for custom emoji support in poll options

* Serialize poll emojis in REST API

* Render custom emojis in poll options

* Render custom emoji in poll options on public pages
2019-03-20 17:29:12 +01:00
Eugen Rochko 9c4cbdbafb
Add Keybase integration ()
* create account_identity_proofs table

* add endpoint for keybase to check local proofs

* add async task to update validity and liveness of proofs from keybase

* first pass keybase proof CRUD

* second pass keybase proof creation

* clean up proof list and add badges

* add avatar url to keybase api

* Always highlight the “Identity Proofs” navigation item when interacting with proofs.

* Update translations.

* Add profile URL.

* Reorder proofs.

* Add proofs to bio.

* Update settings/identity_proofs front-end.

* Use `link_to`.

* Only encode query params if they exist.

URLs without params had a trailing `?`.

* Only show live proofs.

* change valid to active in proof list and update liveness before displaying

* minor fixes

* add keybase config at well-known path

* extremely naive feature flagging off the identity proof UI

* fixes for rubocop

* make identity proofs page resilient to potential keybase issues

* normalize i18n

* tweaks for brakeman

* remove two unused translations

* cleanup and add more localizations

* make keybase_contacts an admin setting

* fix ExternalProofService my_domain

* use Addressable::URI in identity proofs

* use active model serializer for keybase proof config

* more cleanup of keybase proof config

* rename proof is_valid and is_live to proof_valid and proof_live

* cleanup

* assorted tweaks for more robust communication with keybase

* Clean up

* Small fixes

* Display verified identity identically to verified links

* Clean up unused CSS

* Add caching for Keybase avatar URLs

* Remove keybase_contacts setting
2019-03-18 21:00:55 +01:00
ThibG a20354a20b Set and store report URIs ()
Fixes 
2019-03-17 15:34:56 +01:00
Eugen Rochko 1b167707c2
Fix language detection of non-latin alphabets even at few characters () 2019-03-15 05:07:09 +01:00
Eugen Rochko 9f5b55ad4f
Fix poll update handler calling method was that was not available ()
* Fix poll update handler calling method was that was not available

Fix regression from 

* Refactor VoteService

* Refactor ActivityPub::DistributePollUpdateWorker and optimize it

* Fix typo

* Fix typo
2019-03-12 22:58:59 +01:00
ThibG 79b00d294d Increase DNS timeouts () 2019-03-11 13:27:57 +01:00
ThibG 3a92885a86 Support pushing and receiving updates to poll tallies ()
* Process incoming poll tallies update

* Send Update on poll vote

* Do not send Updates for a poll more often than once every 3 minutes

* Include voters in people to notify of results update

* Schedule closing poll worker on poll creation

* Add new notification type for ending polls

* Add front-end support for ended poll notifications

* Fix UpdatePollSerializer

* Fix Updates not being triggered by local votes

* Fix tests failure

* Fix web push notifications for closing polls

* Minor cleanup

* Notify voters of both remote and local polls when those close

* Fix delivery of poll updates to mentioned accounts and voters
2019-03-11 00:49:31 +01:00
ThibG 3aaac4f134 Do not allow adding votes to expired polls ()
* Do not allow adding votes to expired polls

* Only validate expires_at on create
2019-03-08 00:54:50 +01:00
ThibG 34f261e6af Avoid unnecessarily fetching the replies collection when it is empty () 2019-03-07 01:50:37 +01:00
ThibG efd0fb8088 Fix newlines in OStatus and RSS serializations () 2019-03-05 23:58:58 +01:00
ThibG 636db1f54f When serializing polls over OStatus, serialize poll options to text ()
* When serializing polls over OStatus, serialize poll options to text

* Do the same for RSS feeds

* Use “[ ] ” as a prefix for poll options instead of “- ”
2019-03-05 21:09:18 +01:00
ThibG 833ffce2df Store remote votes URI ()
* Store remote votes URI

* Add spec for accepting remote votes

* Make poll vote id generation work the same way as follows
2019-03-04 22:51:23 +01:00
Eugen Rochko 0e6998da3c
Add tests for ActivityPub poll processing () 2019-03-04 01:13:42 +01:00
Eugen Rochko ae1b9cf70a
Fix remote poll expiration time () 2019-03-04 00:44:34 +01:00
Eugen Rochko 230a012f00
Add polls ()
* Add polls

Fix 

* Add tests

* Fixes

* Change API for creating polls

* Use name instead of content for votes

* Remove poll validation for remote polls

* Add polls to public pages

* When updating the poll, update options just in case they were changed

* Fix public pages showing both poll and other media
2019-03-03 22:18:23 +01:00
ThibG d8498b3983 Give the replies collection an identifier and enable pagination () 2019-02-28 18:16:34 +01:00
ThibG 9d3c6f1849 Improved remote thread fetching ()
* Fetch up to 5 replies when discovering a new remote status

This is used for resolving threads downwards. The originating
server must add a “replies” attributes with such replies for it to
be useful.

* Add some tests for ActivityPub::FetchRepliesWorker

* Add specs for ActivityPub::FetchRepliesService

* Serialize up to 5 public self-replies for ActivityPub notes

* Add specs for ActivityPub::NoteSerializer

* Move exponential backoff logic to a worker concern

* Fetch first page of paginated collections when fetching thread replies

* Add specs for paginated collections in replies

* Move Note replies serialization to a first CollectionPage

The collection isn't actually paginable yet as it has no id nor
a `next` field. This may come in another PR.

* Use pluck(:uri) instead of map(&:uri) to improve performances

* Fix fetching replies when they are in a CollectionPage
2019-02-28 15:22:21 +01:00
ThibG 9edab46368 Fix mention processing for unknwon accounts on incoming ActivityPub Notes ()
`::FetchRemoteAccountService` is not `ActivityPub::FetchRemoteAccountService`,
its second argument is the pre-fetched body. Passing `id: false` actually passed
a `Hash` as the prefetched body, instead of properly resolving unknown remote
accounts.
2019-02-27 14:57:14 +01:00
Eugen Rochko 1a1b8170bb
Fix Announce activities of unknown statuses not fetching those statuses ()
Regression from 
2019-02-17 15:16:36 +01:00
Eugen Rochko 147b4c2c3a
Add logging for rejected ActivityPub payloads and add tests () 2019-02-17 03:38:25 +01:00
Eugen Rochko c417e8c198
Filter incoming Announce activities by relation to local activity ()
* Filter incoming Announce activities by relation to local activity

Reject if announcer is not followed by local accounts, and is not
from an enabled relay, and the object is not a local status

Follow-up to 

* Fix tests
2019-02-15 18:19:45 +01:00
Eugen Rochko dad339da6d
Filter incoming Create activities by relation to local activity ()
Reject those from accounts with no local followers, from relays
that are not enabled, which do not address local accounts and are
not replies to accounts that do have local followers
2019-02-13 18:42:47 +01:00
ThibG 6a5307a573 Alternative handling of private self-boosts ()
* When self-boosting, embed original toot into Announce serialization

* Process unknown self-boosts from Announce object if it is more than an URI

* Add some self-boost specs

* Only serialize private toots in self-Announces
2019-02-13 18:36:23 +01:00
Eugen Rochko 016ad37bc8
Fix URL linkifier grabbing full-width spaces and quotations ()
Fix 
Fix 
2019-02-09 20:13:11 +01:00
Hinaloe 157d3af46c Only URLs extract with pre-escaped text ()
* [test] add japanese hashtag testcase

* Only URLs extract with pre-escaped text

( https://github.com/tootsuite/mastodon/issues/9989 )
2019-02-09 03:39:38 +01:00
ThibG ed30110618 Make displaying application used to toot opt-in ()
* Make storing and displaying application used to toot opt-in

* Revert to storing application info, and display it to the author via API
2019-02-02 19:18:15 +01:00
ysksn bcfff65195 Create Redisable#redis ()
* Create Redisable

* Use #redis instead of Redis.current
2019-02-02 19:11:38 +01:00
Jakub Mendyk 6a5e3da6b0 Allow most kinds of characters in URL query (fixes ) ()
* Allow unicode characters in URL query strings

Fixes 

* Alternative approach to unicode support in urls

Adds PoC/idea to approch this problem.
2019-02-02 19:01:18 +01:00
ThibG 75b1488cf4 Add tombstones for remote statuses ()
* Add Tombstone model to remember object deletion

* Do not recreate a status if it has been deleted

* Record Tombstone for remote deleted items

Also, only record deleted items from same-host actors

* Clear an user's tombstones when their key change
2019-01-18 15:56:55 +01:00
Eugen Rochko 31f396b57d
Add support for non-public reblogs from ActivityPub ()
Fix 
2019-01-18 15:56:21 +01:00
Eugen Rochko 4ab42287c0
Use summary as summary for converted ActivityPub objects ()
Fix 
2019-01-16 18:36:17 +01:00
ThibG a4f07bad95 Reduce chances of race conditions when processing deleted toots ()
* Reduce chances of race conditions when processing deleted toots

* Prevent race condition when processing deleted toots
2019-01-16 15:42:00 +01:00
ThibG 70be301d69 Ensure blocked user unfollows blocker if Block/Undo Block are processed out of order ()
* Ensure blocked user unfollows blocker if Block/Undo Block are processed out of order

* Add specs for Block causing unfollow and for out-of-order Block + Undo
2019-01-02 01:12:02 +01:00
ThibG b2f4114550 Do not ignore federated reports targetting already-reported accounts () 2018-12-30 18:58:51 +01:00
ThibG 290932602b Reduce usage of LD signatures ()
* Do not LDS-sign Follow, Accept, Reject, Undo, Block

* Do not use LDS for Create activities of private toots

* Minor cleanup

* Ignore unsigned activities instead of misattributing them

* Use status.distributable? instead of querying visibility directly
2018-12-30 09:48:59 +01:00
Eugen Rochko 0f938ff29c
Add handler for Move activity () 2018-12-29 02:24:36 +01:00
Eugen Rochko aa9a20cde0
Fix ThreadResolveWorker getting queued with invalid URLs () 2018-12-26 19:15:53 +01:00
ThibG 81bda7d67c Add setting to not aggregate reblogs ()
* Add setting to not aggregate reblogs

Fixes 

* Handle cases where user is nil in add_to_home and add_to_list

* Add hint for setting_aggregate_reblogs option

* Reword setting_aggregate_reblogs label
2018-12-09 13:03:01 +01:00
Eugen Rochko c39d7e7b2b
Fix TLS handshake timeout not being enforced ()
Follow-up to 
2018-11-27 19:46:05 +01:00
Eugen Rochko 43c311b3a1
Fix nil error when no DNS addresses are found for host () 2018-11-27 18:13:36 +01:00
Eugen Rochko fd8145d232
Fix connect timeout not being enforced ()
* Fix connect timeout not being enforced

The loop was catching the timeout exception that should stop execution, so the next IP would no longer be within a timed block, which led to requests taking much longer than 10 seconds.

* Use timeout on each IP attempt, but limit to 2 attempts

* Fix code style issue

* Do not break Request#perform if no block given

* Update method stub in spec for Request

* Move timeout inside the begin/rescue block

* Use Resolv::DNS with timeout of 1 to get IP addresses

* Update Request spec to stub Resolv::DNS instead of Addrinfo

* Fix Resolve::DNS stubs in Request spec
2018-11-22 20:12:04 +01:00
ThibG 466e3d710c Include replies to list owner and replies to list members in list statuses () 2018-11-21 17:02:58 +01:00
Eugen Rochko 384e953b75
Revert connect timeout from 1s to 10s ()
The failure rate in Sidekiq is too high
2018-11-21 17:00:56 +01:00
Eugen Rochko 9311430ed7
Prevent multiple handlers for Delete of Actor from running () 2018-11-16 19:46:23 +01:00
Eugen Rochko 8069fd636b
Remove intermediary arrays when creating hash maps from results () 2018-11-16 15:02:18 +01:00
ThibG 7f5a4be580 Fix emoji update date processing () 2018-11-10 23:59:51 +01:00
Eugen Rochko 4615512285
Reduce connect timeout limit and limit signature failures by source IP ()
* Reduce connect timeout from 10s to 1s

* Limit failing signature verifications per source IP
2018-11-08 21:35:58 +01:00
Eugen Rochko be202f9377
Accept the same payload in multiple inboxes and deliver () 2018-10-30 15:03:55 +01:00
m.b 2b18f5f85d Add Page AP type support () 2018-10-29 13:23:29 +01:00
ThibG e53cc673e7 Ignore invalid hashtags on remote statuses instead of rejecting them ()
Fixes 
2018-10-26 22:48:35 +02:00
ThibG 82e7988afc Fix missing mention argument when processing incoming Create activities ()
* Fix missing `mention` argument when processing incoming Create activities

* Fix typo (param → params)
2018-10-26 12:59:59 +02:00
Eugen Rochko d4cf963749
Allow inbox owner to view implicitly targeted ActivityPub payload ()
Fix 
2018-10-25 18:12:22 +02:00
Eugen Rochko fd5285658f
Add option to block reports from domain () 2018-10-20 08:02:44 +02:00
Eugen Rochko ddd30f331c
Improve support for aspects/circles ()
* Add silent column to mentions

* Save silent mentions in ActivityPub Create handler and optimize it

Move networking calls out of the database transaction

* Add "limited" visibility level masked as "private" in the API

Unlike DMs, limited statuses are pushed into home feeds. The access
control rules between direct and limited statuses is almost the same,
except for counter and conversation logic

* Ensure silent column is non-null, add spec

* Ensure filters don't check silent mentions for blocks/mutes

As those are "this person is also allowed to see" rather than "this
person is involved", therefore does not warrant filtering

* Clean up code

* Use Status#active_mentions to limit returned mentions

* Fix code style issues

* Use Status#active_mentions in Notification

And remove stream_entry eager-loading from Notification
2018-10-17 17:13:04 +02:00
Eugen Rochko 21ad21cb50
Improve signature verification safeguards ()
* Downcase signed_headers string before building the signed string

The HTTP Signatures draft does not mandate the “headers” field to be downcased,
but mandates the header field names to be downcased in the signed string, which
means that prior to this patch, Mastodon could fail to process signatures from
some compliant clients. It also means that it would not actually check the
Digest of non-compliant clients that wouldn't use a lowercased Digest field
name.

Thankfully, I don't know of any such client.

* Revert "Remove dead code ()"

This reverts commit a00ce8c92c.

* Restore time window checking, change it to 12 hours

By checking the Date header, we can prevent replaying old vulnerable
signatures. The focus is to prevent replaying old vulnerable requests
from software that has been fixed in the meantime, so a somewhat long
window should be fine and accounts for timezone misconfiguration.

* Escape users' URLs when formatting them

Fixes possible HTML injection

* Escape all string interpolations in Formatter class

Slightly improve performance by reducing class allocations
from repeated Formatter#encode calls

* Fix code style issues
2018-10-12 00:15:55 +02:00
Eugen Rochko 61d44dd11f
Fix typo in ActivityPub Create handler ()
Regression from 
2018-10-11 02:10:15 +02:00
Eugen Rochko 790d3bc637
Move network calls out of transaction in ActivityPub handler ()
Mention and emoji code may perform network calls, but does not need
to do that inside the database transaction. This may improve availability
of database connections when using pgBouncer in transaction mode.
2018-10-11 00:50:18 +02:00
Eugen Rochko 774ac47373
Add conversations API ()
* Add conversations API

* Add web UI for conversations

* Add test for conversations API

* Add tests for ConversationAccount

* Improve web UI

* Rename ConversationAccount to AccountConversation

* Remove conversations on block and mute

* Change last_status_id to be a denormalization of status_ids

* Add optimistic locking
2018-10-07 23:44:58 +02:00
Jeong Arm 144d73730d Leave unknown language as nil if account is remote ()
* Force use language detector if account is remote

* Set unknown remote toot's language as nil
2018-10-05 19:17:46 +02:00
ThibG ba444797d2 Fix handling of ActivityPub activities lacking some attributes () 2018-10-03 23:44:13 +02:00
Eugen Rochko ef69c655cc
Fix class autoloading issue in ActivityPub::Activity::Create () 2018-09-28 17:02:53 +02:00
cbayerlein 4b78546135 Exclude replies from list timelines ()
* Changed list behaviour

I added the following line to the FeedManager (app/lib/feed_manager.rb) in the push_to_list function:

`return false if status.reply?`

Now all posts that are replies are filtered out, so that now only "genuine" posts are displayed in the list.

This is a first approach to solve issue 

* Update feed_manager.rb

As suggested by @Gargron
2018-09-28 00:37:21 +02:00
ふぁぼ原 f7a6f9489d Add a new preference to always hide all media () 2018-09-25 05:09:35 +02:00
Matt Sweetman 1889526e23 Add user preference to always expand toots marked with content warnings () 2018-09-24 05:44:01 +02:00
Eugen Rochko f4d549d300
Redesign forms, verify link ownership with rel="me" ()
* Verify link ownership with rel="me"

* Add explanation about verification to UI

* Perform link verifications

* Add click-to-copy widget for verification HTML

* Redesign edit profile page

* Redesign forms

* Improve responsive design of settings pages

* Restore landing page sign-up form

* Fix typo

* Support <link> tags, add spec

* Fix links not being verified on first discovery and passive updates
2018-09-18 16:45:58 +02:00
Renato "Lond" Cerqueira fe56d26f7b Fix autoplay issue with spoiler tag ()
Add tests to avoid similar issues in the future
2018-08-31 15:16:59 +02:00
Eugen Rochko e3764bdb52
Do not sign useless User-Agent or Accept-Encoding headers ()
Fix 
2018-08-31 04:22:52 +02:00
Renato "Lond" Cerqueira 11658d8653 Add animate custom emoji param to embed pages ()
* Add animate custom emoji param to embed pages

* Rename param, use it for avatars and gifs

* Fix issues pointed by codeclimate and breaking test

* Ignore brakeman warning
2018-08-30 23:14:01 +02:00
Eugen Rochko cabdbb7f9c
Add CLI task for rotating keys ()
* If an Update is signed with known key, skip re-following procedure

Because it means the remote actor did *not* lose their database

* Add CLI method for rotating keys

    bin/tootctl accounts rotate [USERNAME]

Generates a new RSA key per account and sends out an Update activity
signed with the old key.

* Key rotation: Space out Update fan-outs every 5 minutes per 1000 accounts

* Skip suspended accounts in key rotation
2018-08-26 20:21:03 +02:00
Quint Guvernator da13fa5021 Fix low-hanging rubocop gripes ()
* rubocop: quit being so picky

* rubocop: miscellany

* rubocop: prefer present to blank
2018-08-26 19:22:46 +02:00
Eugen Rochko b4ba4b1b5d
Spread out crawling randomly to avoid DDoSing the link ()
* Spread out crawling randomly to avoid DDoSing the link

Fix 

* Remove trailing whitespace
2018-08-26 00:33:57 +02:00
M Somerville 95bd0d4528 Support ActivityStreams’ summaryMap. ()
In the same way as contentMap and nameMap.
2018-08-25 13:27:34 +02:00
Jakub Mendyk 6cb3514d64 Add ability to change an instance default theme from the administration panel () ()
* Add default_settings class method to ScopedSettings

ScopedSettings was extended to use value of unscoped setting instead of
only using defaults set in config/settings.yml for selected settings.
This adds possibility for admins to set default values of users' settings,
for example default theme (as requested in ).

* Add ability to change an instance default theme

Closes 
2018-08-23 14:17:35 +02:00
ThibG 42573b76f1 Do not crash if remote custom emoji does not define updated date (fixes ) () 2018-08-23 00:27:58 +02:00
Eugen Rochko 802cf6a4c5
Improve federated ID validation ()
* Fix URI not being sufficiently validated with prefetched JSON

* Add additional id validation to OStatus documents, when possible
2018-08-22 20:55:14 +02:00
ThibG 59f7f4c923 Implement Undo { Accept { Follow } } (fixes ) ()
* Add Follow#revoke_request!

* Implement Undo { Accept { Follow } } (fixes )
2018-08-17 16:24:56 +02:00
ThibG 1ee675d68b Use correct activity id in Accept when receiving duplicate Follow (fixes ) () 2018-08-17 14:08:17 +02:00
Eugen Rochko 39e361a56d
Expect relays to answer with accept/reject () 2018-08-12 18:16:26 +02:00
ThibG c52bcc0331 Serialize text-less statuses as '.' over OStatus (fixes ) () 2018-08-05 12:22:23 +02:00
Eugen Rochko 07b799468d
Fix incorrect context definition for the "featured" keyword ()
* Fix incorrect context definition for the "featured" keyword

Fix 

* Adjust context definition for movedTo
2018-07-30 19:33:37 +02:00
Eugen Rochko a23ac107e4
Fix activity:interactions counter to count all interactions () 2018-07-16 18:35:43 +02:00
Eugen Rochko 8f64b17d98 Whitelist dat/ipfs/gopher links in sanitizer ()
Fix 
2018-07-16 22:17:43 +09:00
Eugen Rochko 2354b10eb5
Add admin dashboard () 2018-07-16 01:11:53 +02:00
Eugen Rochko 38e9662d78
Disable language detection for texts shorter than 140 characters ()
If the input text is blank after preparation (only mention, or
only URL, or empty as in a media post), then use nil as language,
since it's OK to show to everyone.

Otherwise, always fall back to the server's default locale
2018-07-14 04:05:36 +02:00
Eugen Rochko 47c59591e0 Do not count self in potential friendships () 2018-07-14 10:44:23 +09:00
Eugen Rochko 20fefdb714
Make whole-word filter regex consistent between Ruby and JS () 2018-07-10 03:01:50 +02:00
ThibG 1ca4e51eb3 Add option to not consider word boundaries when processing keyword filtering ()
* Add option to not consider word boundaries when filtering phrases

* Add a few tests for keyword/phrase filtering
2018-07-09 02:22:09 +02:00
Eugen Rochko 404c7702ec
In keyword filter, account for reblogs, HTML and whole-words ()
* In keyword filter, account for reblogs, HTML and whole-words

* Match whole words in JS filter, too

* Fix typo
2018-07-06 02:15:44 +02:00
Eugen Rochko 4b198b172d
Check reblogged status for blocked/muted mentions () 2018-07-05 18:43:37 +02:00
Eugen Rochko da8fe8079e
Re-add follow recommendations API ()
* Re-add follow recommendations API

    GET /api/v1/suggestions

Removed in 8efa081f21 due to Neo4J
dependency. The algorithm uses triadic closures, takes into account
suspensions, blocks, mutes, domain blocks, excludes locked and moved
accounts, and prefers more recently updated accounts.

* Track interactions with people you don't follow

Replying to, favouriting and reblogging someone you're not following
will make them show up in follow recommendations. The interactions
have different weights:

- Replying is 1
- Favouriting is 10 (decidedly positive interaction, but private)
- Reblogging is 20

Following them, muting or blocking will remove them from the list,
obviously.

* Remove triadic closures, ensure potential friendships are trimmed
2018-07-03 01:47:56 +02:00
MIYAGI Hikaru ddd0bb69e1 Merge HIDDEN_SERVICE_VIA_TRANSPARENT_PROXY into ALLOW_ACCESS_TO_HIDDEN_SERVICE ()
If Mastodon accesses to the hidden service via transparent proxy, it's needed to avoid checking whether it's a private address, since `.onion` is resolved to a private address.
I was previously using the `HIDDEN_SERVICE_VIA_TRANSPARENT_PROXY` to provide that function. However, I realized that using `HIDDEN_SERVICE_VIA_TRANSPARENT_PROXY` is redundant, since this specification is always used with `ALLOW_ACCESS_TO_HIDDEN_SERVICE`. Therefore, I decided to integrate the setting of `HIDDEN_SERVICE_VIA_TRANSPARENT_PROXY` into` ALLOW_ACCESS_TO_HIDDEN_SERVICE`.
2018-06-29 15:36:02 +02:00
Eugen Rochko cdb101340a
Keyword/phrase filtering ()
* Add keyword filtering

    GET|POST       /api/v1/filters
    GET|PUT|DELETE /api/v1/filters/:id

- Irreversible filters can drop toots from home or notifications
- Other filters can hide toots through the client app
- Filters use a phrase valid in particular contexts, expiration

* Make sure expired filters don't get applied client-side

* Add missing API methods

* Remove "regex filter" from column settings

* Add tests

* Add test for FeedManager

* Add CustomFilter test

* Add UI for managing filters

* Add streaming API event to allow syncing filters

* Fix tests
2018-06-29 15:34:36 +02:00
Eugen Rochko a58ec29631
Allow selecting default posting language instead of auto-detect ()
* Allow selecting default posting language instead of auto-detect

* Enable default language setting in credentials API

* Fix form saving
2018-06-17 18:57:31 +02:00
Eugen Rochko 6ba7d9d0d9
Do not accept ActivityPub follow requests from blocked user ()
* Do not accept ActivityPub follow requests from blocked user

Fix 

* Deliver auto-rejection immediately when follow-requested by blocked account

* Fix trailing whitespace
2018-06-08 00:14:06 +02:00
Eugen Rochko 1e938b966e
Exclude unlisted, private and direct toots from affecting trends () 2018-05-31 16:47:28 +02:00
abcang 90908fc24b Fix N+1 on AtomSerializer () 2018-05-29 13:34:02 +02:00
Eugen Rochko ab36e0ef72 Record trending tags from ActivityPub, too () 2018-05-28 12:21:04 +09:00
ThibG cdbdf7f98b Ignore multiple occurrences of a hashtag within a status (fixes ) () 2018-05-25 14:26:45 +02:00
Eugen Rochko 57b503d4ef
Resolve unknown status from Add activity, skip Remove if unknown ()
Fix 
2018-05-18 11:33:56 +02:00
Eugen Rochko 1e02dc8715
Add preference to hide following/followers lists ()
* Add preference to hide following/followers lists

- Public pages
- ActivityPub collections (does not return pages but does give total)
- REST API (unless it's your own) (does not federate)

Fix 

* Add preference

* Add delegation

* Fix issue

* Fix issue
2018-05-18 02:26:51 +02:00
MIYAGI Hikaru 919eef3098 User agent for WebFinger ()
* User agent for WebFinger

* local_domain → web_domain

* 'http' is away accidentally...
2018-05-18 01:47:22 +02:00
Akihiko Odaki 55fd55714a Raise Mastodon::RaceConditionError if Redis lock failed ()
An explicit error allows user agents to know the error and Sidekiq to
retry.
2018-05-16 12:29:45 +02:00
ThibG 7467361d70 Fetch boosted statuses on behalf of a follower (fixes ) ()
When an ActivityPub Announce is processed and the boosted toot is not known,
fetch it on behalf of one of the booster's followers. This is to allow
fetching self-boosts of previously-unknown private toots.

If fetching on behalf of a user fails, try fetching it anonymously: the
selected follower of a boosting user may be banned by the boosted toot's
author.
2018-05-12 16:48:32 +02:00
Yamagishi Kazutoshi 6588f6a0a9 Disable inflate gzip on head method () 2018-05-10 14:36:12 +02:00
Yamagishi Kazutoshi edf882320a Support gzip encoding on Request () 2018-05-09 16:59:39 +02:00
Eugen Rochko 6208ea5a53
If an OStatus message contains nsfw hashtag, mark it as sensitive ()
* If an OStatus message contains nsfw hashtag, mark it as sensitive

Undo parts of 

* Put nsfw hashtag on OStatus messages if they have any media

* Fix code style issues
2018-05-07 14:49:13 +02:00
Eugen Rochko 61a9018607
Enable custom emojis in profiles (notes, field values, display names) ()
Follow-up to 
2018-05-06 11:48:51 +02:00
Eugen Rochko c947e2e4c5
Fix handling of malformed ActivityPub payloads when URIs are nil ()
* Fix handling of malformed ActivityPub payloads when URIs are nil

* Gracefully handle JSON-LD canonicalization failures
2018-05-05 18:22:34 +02:00
Eugen Rochko 6793bec4c6
Store URIs of follows, follow requests and blocks for ActivityPub ()
Same URI passed between follow request and follow, since they are
the same thing in ActivityPub. Local URIs are generated during
creation using UUIDs and are passed to serializers.
2018-05-04 21:14:34 +02:00
ThibG a24605961a Fixes/do not override timestamps ()
* Revert "Fixes/do not override timestamps ()"

This reverts commit 581a5c9d29.

* Document Snowflake ID corner-case a bit more

Snowflake IDs are used for two purposes: making object identifiers harder to
guess and ensuring they are in chronological order. For this reason, they
are based on the `created_at` attribute of the object.

Unfortunately, inserting items with older snowflakes IDs will break the
assumption of consumers of the paging APIs that new items will always have
a greater identifier than the last seen one.

* Add `override_timestamps` virtual attribute to not correlate snowflake ID with created_at
2018-05-03 23:02:46 +02:00
ThibG 581a5c9d29 Fixes/do not override timestamps ()
* Do not override timestamps for incoming toots

* Remove every reference to override_timestamps

Statuses are now created with the announced publishing date
and are only pushed to timelines if that date is at most
6 hours earlier than the time at which it is processed.
2018-05-03 13:33:08 +02:00
Eugen Rochko a5293fdf61
Fix n+1 queries in StatusThreadingConcern () 2018-05-03 10:41:58 +02:00
Eugen Rochko cb5b5cb5f7
Slightly reduce RAM usage ()
* No need to re-require sidekiq plugins, they are required via Gemfile

* Add derailed_benchmarks tool, no need to require TTY gems in Gemfile

* Replace ruby-oembed with FetchOEmbedService

Reduce startup by 45382 allocated objects

* Remove preloaded JSON-LD in favour of caching HTTP responses

Reduce boot RAM by about 6 MiB

* Fix tests

* Fix test suite by stubbing out JSON-LD contexts
2018-05-02 18:58:48 +02:00
ThibG d0cdd5cf94 Accept actor object updates from all supported actor types () 2018-05-02 16:08:16 +02:00
Eugen Rochko f62539ce5c
Remove most behaviour disparities between blocks and mutes ()
* Remove most behaviour disparities between blocks and mutes

The only differences between block and mute should be:

- Mutes can optionally NOT affect notifications
- Mutes should not be visible to the muted

Fix 
Fix 

* Do not allow boosting someone you blocked

Fix 

* Do not allow favouriting someone you blocked

* Fix nil error in StatusPolicy
2018-05-02 15:50:20 +02:00
Eugen Rochko 965345316f
Guard against nil URLs in Request class ()
Fix 
2018-05-02 15:44:22 +02:00
Surinna Curtis dc786c0cf4 Support Actors/Statuses with multiple types ()
* Add equals_or_includes_any? helper in JsonLdHelper

* Support arrays in JSON-LD type fields for actors/tags/objects.

* Spec for resolving accounts with extension types

* Style tweaks for codeclimate
2018-05-02 12:40:24 +02:00
Eugen Rochko 705f1d7bf1
Fix missing updated_at attribute on emoji EntityCache ()
Just don't try to save space by only selecting few attributes. If
anyone is wondering, this is needed because the emoji entity cache
is not really only used for entities, it's accessed again to
generate Emoji tags in ActivityPub/OStatus, so a lot more properties
are used than what is needed in HTML alone...
2018-04-30 22:49:33 +02:00
Eugen Rochko 295e3ef02b
Fix missing domain attribute in EntityCache for emoji () 2018-04-30 09:12:55 +02:00
Eugen Rochko a872392cd9
Add entity cache ()
* Add entity cache

Use a caching layer for mentions and custom emojis that are
dynamically extracted from text.

Reduce duplicate text extractions

* Fix code style issue
2018-04-27 01:38:10 +02:00
MIYAGI Hikaru f58dcbc981 HTTP proxy support for outgoing request, manage access to hidden service ()
* Add support for HTTP client proxy

* Add access control for darknet

Supress error when access to darknet via transparent proxy

* Fix the codes pointed out

* Lint

* Fix an omission + lint

* any? -> include?

* Change detection method to regexp to avoid test fail
2018-04-25 02:14:49 +02:00
Eugen Rochko 9d4710ed00
Add RSS feeds for end-users ()
* Add RSS feed for accounts

* Add RSS feeds for hashtags

* Fix code style issues

* Fix code style issues
2018-04-25 02:10:02 +02:00
Eugen Rochko 7db7d68136
Detect and prevent image bombs, max. processable dimension 4096^2 () 2018-04-23 09:16:38 +02:00
Eugen Rochko 75c4ab9d12
Remove "nsfw" category for sensitive statuses in OStatus serializer ()
Fix 
2018-04-22 22:09:03 +02:00
Eugen Rochko 78ed4ab75f
Add bio fields ()
* Add bio fields

- Fix 
- Fix 
- Fix 

* Display bio fields in web UI

* Fix output of links and missing fields

* Federate bio fields over ActivityPub as PropertyValue

* Improve how the fields are stored, add to Edit profile form

* Add rel=me to links in fields

Fix 
2018-04-14 12:41:08 +02:00
Levi Bard cd0eaa349c Enable updating additional account information from user preferences via rest api ()
* Enable updating additional account information from user preferences via rest api
Resolves 

* Pacify rubocop

* Decoerce incoming settings in UserSettingsDecorator

* Create user preferences hash directly from incoming credentials instead of going through ActionController::Parameters

* Clean up user preferences update

* Use ActiveModel::Type::Boolean instead of manually checking stringified number equivalence
2018-04-08 13:43:10 +02:00
Eugen Rochko b5726def55
Forward deletes on the same path as reply forwarding ()
* Forward deletes on the same path as reply forwarding

* Remove trailing whitespace
2018-04-07 18:54:46 +02:00
David Underwood 123a343d11 [WIP] Enable custom emoji on account pages and in the sidebar ()
Federate custom emojis with accounts
2018-04-01 23:55:42 +02:00
ThibG e573bb0990 Fix compatibility with PeerTube ()
* Support fetching objects of convertible types by URL (fixes )

* Ignore invalid hashtags
2018-03-30 15:44:54 +02:00
ThibG 9ed5eebd7c Do not ignore unknown media attachments, only skip them ()
That way, they are displayed in a list below the corresponding toot.
2018-03-29 00:52:24 +02:00
Akihiko Odaki 40e5d2303b Validate HTTP response length while receiving ()
to_s method of HTTP::Response keeps blocking while it receives the whole
content, no matter how it is big. This means it may waste time to receive
unacceptably large files. It may also consume memory and disk in the
process. This solves the inefficency by checking response length while
receiving.
2018-03-26 14:02:10 +02:00
Akihiko Odaki 54b273bf99 Close http connection in perform method of Request class ()
HTTP connections must be explicitly closed in many cases, and letting
perform method close connections makes its callers less redundant and
prevent them from forgetting to close connections.
2018-03-24 12:49:54 +01:00
Rey Tucker 36b5703796 request: in the event of failure, try other IPs () ()
* request: in the event of failure, try other IPs ()

In the case where a name has multiple A/AAAA records, we should
try subsequent records instead of immediately failing when we have a
failure on the first IP address.

This significantly improves delivery success when there are network
connectivity problems affecting only IPv4 or IPv6.

* fix method call style

* request_spec: adjust test case to use Addrinfo

* request: Request/open: move private addr check to within begin/rescue

* request_spec: add case to test failover, fix exception check

* Double Addrinfo.foreach so that it correctly yields instances
2018-03-20 09:06:08 +01:00
ThibG ff6b8a6443 Serialize mentions in the order they are added ()
Up until now, the order seemed to be in the *opposite* order,
which caused the WebUI to populate mentions in reversed order
when replying to toots local to one's instance.
2018-03-19 20:19:35 +01:00
Eugen Rochko f02411da40
Ignore media validation when attaching to status during processing ()
Fix 
2018-03-19 01:51:19 +01:00
MitarashiDango 6dcf96271e fix validation error (media only status) ()
* fix validation error (media only status)

* Incorporating review suggestions

* Reflect similar fix to OStatus side

* Fix not to include media in transaction

* Restore the limit of the number of media

* Fix not to return nil
2018-03-08 01:22:47 +01:00
Eugen Rochko cfa9b6e13a
Remove text requirement when media attached from statuses () 2018-03-07 08:28:52 +01:00
Eugen Rochko e6520c0270
Fix - Use target instead of origin in Remove activity () 2018-03-07 03:54:46 +01:00
Eugen Rochko 9110db41c5
Federate pinned statuses over ActivityPub ()
* Federate pinned statuses over ActivityPub

* Display pinned toots in web UI

Fix 

* Fix migration

* Fix tests

* Update outbox_serializer.rb

* Update remove_serializer.rb

* Update add_serializer.rb

* Update fetch_featured_collection_service.rb
2018-03-04 09:19:11 +01:00
Eugen Rochko e852872846
Fix : Reject->Follow will remove the follow if it exists () 2018-02-28 06:55:06 +01:00
Eugen Rochko 41a01bec23
Federated reports ()
* Fix : Federated reports

* UI for federated reports

* Add spec for ActivityPub Flag handler

* Add spec for ReportService
2018-02-28 06:54:55 +01:00
Akihiko Odaki 2e8a492e88 Raise Mastodon::HostValidationError when host for HTTP request is private () 2018-02-24 19:16:11 +01:00
Akihiko Odaki f8f0572ee0 Do not push status to feed if its reblog is already inserted ()
A complemental change for precompute_feed_service_spec.rb also fixes its
random failure which is caused by the Snowlake randomization of the order
of an original status and its reblog.
2018-02-24 05:40:18 +01:00
Eugen Rochko 90f12f2e5a
Focal points ()
* Add focus param to media API, center thumbnails on focus point

* Add UI for setting a focal point

* Improve focal point icon on upload item

* Use focal point in upload preview

* Add focalPoint property to ActivityPub

* Don't show focal point button for non-image attachments
2018-02-22 00:35:46 +01:00
Eugen Rochko a7171af0a3
Fix avatar and header issues by using custom geometry detector ()
* Fix avatar and header issues by using custom geometry detector

Revert a part of . The file passed to dynamic styles method
was not actually a file, but an instance of Paperclip::Attachment,
which broke all styles by always returning {} from the method.

One problem with GIF avatars was that Paperclip::GeometryDetector
reported wrong dimensions for them, e.g. 120x120 GIF avatar would
for some reason be detected as 120x53. By writing our own geometry
parser, we can use FastImage, which also happens to be faster than
ImageMagick, to detect image dimensions, which are also correct.

Unfortunately, this PR does not implement skipping a `convert`
entirely if the dimensions are already correct, as I found no easy
way to write that behaviour into Paperclip without rewriting the
Paperclip::Thumbnail class.

* Only invoke convert if dimension or format needs to be changed
2018-02-21 03:40:12 +01:00
Eugen Rochko a71af98401
Push discovered status through streaming API within a time window ()
Time window of 6 hours
2018-02-17 14:28:48 +01:00
Eugen Rochko 3ebc0ad4d3
Full-text search for authorized statuses ()
* Add full-text search for authorized statuses

- Search API will return statuses that match the query
- Only for logged in users
- Only if you are author of the status,
- Or you were mentioned in it
- Or you favourited or reblogged it
- Configuration over `ES_ENABLED`, `ES_HOST`, `ES_PORT`, `ES_PREFIX`
- Run `rails chewy:deploy` to create & populate index

Fix 
Fix 
Fix 

* Add commented out docker-compose configuration for ES container

* Optimize index import, filter search results

* Add basic normalization to the index

* Add better stemming and normalization to the index

* Skip webfinger request if search query includes both @ and a space

* Fix code style

* Visually separate search result sections

* Fix code style issues
2018-02-09 23:04:47 +01:00
Eugen Rochko 76f3d5d16b
Add preference to always display sensitive media () 2018-02-09 00:26:57 +01:00
Akihiko Odaki 0be9a1e321 Accept ActivityPub announce from the author of the original note () 2018-02-02 10:22:15 +01:00
ThibG b1e03197fa Process mentions and reblogs even from resolved threads ()
This may lead to out-of-order notifications, but this is better than not having
notifications at all.
2018-01-19 19:11:35 +01:00
Eugen Rochko 35b84985a8
Skip ActivityPub Announces of non-public objects ()
* Skip ActivityPub Announces of non-public objects

* Skip OStatus reblogs of non-public statuses
2018-01-09 19:35:10 +01:00
puckipedia 628358aeea Add the author of a status to cc if reblogged ()
This makes slightly more sense, and ensures that the author of a post is always referenced in the audience (which some servers might rely on). And the announce is POSTed to the author's inbox anyways.
2018-01-09 00:47:43 +01:00
Eugen Rochko dbda87c31f
Revert () 2018-01-08 10:57:52 +01:00
Eugen Rochko e4a241abef
Fix bad URL schemes being accepted ()
* Fix actors accepting invalid URI schemes or different host between URI and URL

* Fix statuses accepting invalid URI scheme or different host to actor

* Adjust tests to new requirements

* Improve readability of mismatching_origin?/invalid_origin? methods
2018-01-08 05:00:23 +01:00
puckipedia 0eff42d688 Move Article from supported to converted types () 2018-01-08 00:21:14 +01:00
ThibG a8b51124ba Don't normalize URLs in toots ()
* Don't normalize URLs in toots

URL normalization is ill-defined and may cause certain links to break.

* Change specs since we are not normalizing user-provided URLs
2018-01-03 20:51:33 +01:00