* Add account statuses cleanup policy model
* Record last inspected toot to delete to speed up successive calls to statuses_to_delete
* Add service to cleanup a given account's statuses within a budget
* Add worker to go through account policies and delete old toots
* Fix last inspected status id logic
All existing statuses older or equal to last inspected status id must be
kept by the current policy. This is an invariant that must be kept so that
resuming deletion from the last inspected status remains sound.
* Add tests
* Refactor scheduler and add tests
* Add user interface
* Add support for discriminating based on boosts/favs
* Add UI support for min_reblogs and min_favs, rework UI
* Address first round of review comments
* Replace Snowflake#id_at_start with with_random parameter
* Add tests
* Add tests for StatusesCleanupController
* Rework settings page
* Adjust load-avoiding mechanisms
* Please CodeClimate
* Add account_notes relationship
* Add tests
* Fix owned account notes not being deleted when an account is deleted
* Add post-migration to clean up orphaned account notes
* Fix account deletion sometimes failing because of optimistic locks
In some rare occasions[1], deleting accounts would fail with a
`StaleObjectError` exception.
Indeed, account deletion manually sets the `AccountStat` values without
handling cases where the optimistic locking on `AccountStat` would fail.
To my knowledge, with the rewrite of account counters in #15913, the
`DeleteAccountService` is now the only place that changes the counters in
a way that is not atomic.
Since in this specific case, we do not care about the previous values of the
account counters, it appears we don't need locking at all for this table
anymore.
[1]: https://discourse.joinmastodon.org/t/account-cant-be-deleted/3602
* Bump MAX_SUPPORTED_VERSION in maintenance script
* Update devise-two-factor to unreleased fork for Rails 6 support
Update tests to match new `rotp` version.
* Update nsa gem to unreleased fork for Rails 6 support
* Update rails to 6.1.3 and rails-i18n to 6.0
* Update to unreleased fork of pluck_each for Ruby 6 support
* Run "rails app:update"
* Add missing ActiveStorage config file
* Use config.ssl_options instead of removed ApplicationController#force_ssl
Disabled force_ssl-related tests as they do not seem to be easily testable
anymore.
* Fix nonce directives by removing Rails 5 specific monkey-patching
* Fix fixture_file_upload deprecation warning
* Fix yield-based test failing with Rails 6
* Use Rails 6's index_with when possible
* Use ActiveRecord::Cache::Store#delete_multi from Rails 6
This will yield better performances when deleting an account
* Disable Rails 6.1's automatic preload link headers
Since Rails 6.1, ActionView adds preload links for javascript files
in the Links header per default.
In our case, that will bloat headers too much and potentially cause
issues with reverse proxies. Furhermore, we don't need those links,
as we already output them as HTML link tags.
* Switch to Rails 6.0 default config
* Switch to Rails 6.1 default config
* Do not include autoload paths in the load path
* Improve account counters handling
* Use ActiveRecord::Base::sanitize_sql to pass values instead of interpolating them
Keep using string interpolation for `key` as it is safe and using
“ActiveRecord::Base::sanitize_sql_hash_for_assignment” would require stitching
bits of SQL in a way that is not more easily checked for safety.
* Add migration hook to catch PostgreSQL versions earlier than 9.5
An uncommon but somewhat difficult to digagnose issue is dealing with
improperly-seeded databases. In such cases, instance-signed fetches will
fail with a ActiveRecord::RecordNotFound error, usually caught and handled
as generic 404, leading people to think the remote resource itself has not
been found, while it's the local instance actor that does not exist.
This commit changes the code so that failure to find the instance actor
automatically creates a new one, so that improperly-seeded databases do
not cause any issue.
* Fix being able to import more than allowed number of follows
Without this commit, if someone tries importing a second list of accounts to
follow before the first one has been processed, this will queue imports for
the two whole lists, even if they exceed the account's allowed number of
outgoing follows.
This commit changes it so the individual queued imports aren't exempt from
the follow limit check (they remain exempt from the rate-limiting check
though).
* Catch validation errors to not re-queue failed follows
Co-authored-by: Claire <claire.github-309c@sitedethib.com>
* Add support for followers synchronization on the receiving end
Check the `collectionSynchronization` attribute on `Create` and `Announce`
activities and synchronize followers from provided collection if possible.
* Add tests for followers synchronization on the receiving end
* Add support for follower synchronization on the sender's end
* Add tests for the sending end
* Switch from AS attributes to HTTP header
Replace the custom `collectionSynchronization` ActivityStreams attribute by
an HTTP header (`X-AS-Collection-Synchronization`) with the same syntax as
the `Signature` header and the following fields:
- `collectionId` to specify which collection to synchronize
- `digest` for the SHA256 hex-digest of the list of followers known on the
receiving instance (where “receiving instance” is determined by accounts
sharing the same host name for their ActivityPub actor `id`)
- `url` of a collection that should be fetched by the instance actor
Internally, move away from the webfinger-based `domain` attribute and use
account `uri` prefix to group accounts.
* Add environment variable to disable followers synchronization
Since the whole mechanism relies on some new preconditions that, in some
extremely rare cases, might not be met, add an environment variable
(DISABLE_FOLLOWERS_SYNCHRONIZATION) to disable the mechanism altogether and
avoid followers being incorrectly removed.
The current conditions are:
1. all managed accounts' actor `id` and inbox URL have the same URI scheme and
netloc.
2. all accounts whose actor `id` or inbox URL share the same URI scheme and
netloc as a managed account must be managed by the same Mastodon instance
as well.
As far as Mastodon is concerned, breaking those preconditions require extensive
configuration changes in the reverse proxy and might also cause other issues.
Therefore, this environment variable provides a way out for people with highly
unusual configurations, and can be safely ignored for the overwhelming majority
of Mastodon administrators.
* Only set follower synchronization header on non-public statuses
This is to avoid unnecessary computations and allow Follow-related
activities to be handled by the usual codepath instead of going through
the synchronization mechanism (otherwise, any Follow/Undo/Accept activity
would trigger the synchronization mechanism even if processing the activity
itself would be enough to re-introduce synchronization)
* Change how ActivityPub::SynchronizeFollowersService handles follow requests
If the remote lists a local follower which we only know has sent a follow
request, consider the follow request as accepted instead of sending an Undo.
* Integrate review feeback
- rename X-AS-Collection-Synchronization to Collection-Synchronization
- various minor refactoring and code style changes
* Only select required fields when computing followers_hash
* Use actor URI rather than webfinger domain in synchronization endpoint
* Change hash computation to be a XOR of individual hashes
Makes it much easier to be memory-efficient, and avoid sorting discrepancy issues.
* Marginally improve followers_hash computation speed
* Further improve hash computation performances by using pluck_each
* Add bell button
Fix#4890
* Remove duplicate type from post-deployment migration
* Fix legacy class type mappings
* Improve query performance with better index
* Fix validation
* Remove redundant index from notifications
* Make Array-creation behavior of Paginable more predictable
Paginable.paginate_by_id usually returns ActiveRecord::Relation, but it
returns an Array if min_id option is present. The behavior caused problems
fixed with the following commits:
- 552e886b64
- b63ede5005
- 64ef37b89d
To prevent from recurring similar problems, this commit introduces two
changes:
- The scope now always returns an Array whether min_id option is present
or not.
- The scope is renamed to to_a_paginated_by_id to clarify it returns an
Array.
* Transform Paginable.to_a_paginated_by_id from a scope to a class method
https://api.rubyonrails.org/classes/ActiveRecord/Scoping/Named/ClassMethods.html#method-i-scope
> The method is intended to return an ActiveRecord::Relation object, which
> is composable with other scopes.
Paginable.to_a_paginated_by_id returns an Array and is not appropriate
as a scope.
* Add UserNote model
* Add UI for user notes
* Put comment in relationships entity
* Add API to create user notes
* Copy user notes to new account when receiving a Move activity
* Address some of the review remarks
* Replace modal by inline edition
* Please CodeClimate
* Button design changes
* Change design again
* Cancel note edition when pressing Escape
* Fixes
* Tweak design again
* Move “Add note” item, and allow users to add notes to themselves
* Rename UserNote into AccountNote, rename “comment” Relationship attribute to “note”
- Change audio files to not be stripped of metadata
- Automatically extract cover art from audio if it exists
- Add `thumbnail` parameter to `POST /api/v1/media`, `POST /api/v2/media` and `PUT /api/v1/media/:id`
- Add `icon` to represent it in attachments in ActivityPub
- Fix `preview_url` containing URL of missing missing image when there is no thumbnail instead of null
- Fix duration of audio not being displayed on public pages until the file is loaded
* Fix admin page crashing when trying to block an invalid domain name
Fixes#13880
* Fix trailing and leading spaces not being properly stripped for domain blocks
When deduplicating account usernames for OAuthable users, the routine did check if any account was known with that username, including remote accounts. This caused some unnecessary deduplication, and usernames ending with unexpected trailing _1.
This fixes#13580
* Add announcements
Fix#11006
* Add reactions to announcements
* Add admin UI for announcements
* Add unit tests
* Fix issues
- Add `with_dismissed` param to announcements API
- Fix end date not being formatted when time range is given
- Fix announcement delete causing reactions to send streaming updates
- Fix announcements container growing too wide and mascot too small
- Fix `all_day` being settable when no time range is given
- Change text "Update" to "Announcement"
* Fix scheduler unpublishing announcements before they are due
* Fix filter params not being passed to announcements filter
* Add backend support for bookmarks
Bookmarks behave like favourites, except they aren't shared with other
users and do not have an associated counter.
* Add spec for bookmark endpoints
* Add front-end support for bookmarks
* Introduce OAuth scopes for bookmarks
* Add bookmarks to archive takeout
* Fix migration
* Coding style fixes
* Fix rebase issue
* Update bookmarked_statuses to latest UI changes
* Update bookmark actions to properly reflect status changes in state
* Add bookmarks item to single-column layout
* Make active bookmarks red
Change the behaviour of remotable concern. Previously, it would skip
downloading an attachment if the stored remote URL is identical to
the new one. Now it would not be skipped if the attachment is not
actually currently stored by Paperclip.
* Change animated GIF detection to not shell out to ImageMagick
Signed-off-by: Eugen Rochko <eugen@zeonfederated.com>
* Change video encoding parameters to limit to 10800 video frames
Signed-off-by: Eugen Rochko <eugen@zeonfederated.com>
* Limit GIF image size further
Signed-off-by: Eugen Rochko <eugen@zeonfederated.com>
* Always strip metadata from video files
* Fix code style issues
Allow access to account settings, 2FA, authorized applications, and
account deletions to unconfirmed and pending users, as well as
users who had their accounts disabled. Suspended users cannot update
their e-mail or password or delete their account.
Display account status on account settings page, for example, when
an account is frozen, limited, unconfirmed or pending review.
After sign up, login users straight away and show a simple page that
tells them the status of their account with links to account settings
and logout, to reduce onboarding friction and allow users to correct
wrongly typed e-mail addresses.
Move the final sign-up step of SSO integrations to be the same
as above to reduce code duplication.
* Add support for an instance actor
* Skip username validation for local Application accounts
* Add migration script to create instance actor
* Make Codeclimate happy
* Switch to id -99 for instance actor
* Remove unused `icon` and `image` attributes from instance actor
* Use if/elsif/else instead of return + ternary operator
* Add instance actor to fresh installs
* Use instance actor as instance representative
Use instance actor for forwarding reports, relay operations, and spam
auto-reporting.
* Seed database in test environment
* Fix single-user mode
* Fix tests
* Fix specs to accomodate for an extra `Account`
* Auto-reject follows on instance actor
Following an instance actor might make sense, but we are not handling that
right now, so auto-reject.
* Fix webfinger lookup and serialization for instance actor
* Rename instance actor
* Make it clear in the HTML view that the instance actor should not be blocked
* Raise cache time for instance actor as there's no dynamic content
* Re-use /about/more with a flash message for instance actor profile
* Record account suspend/silence time and keep track of domain blocks
* Also unblock users who were suspended/silenced before dates were recorded
* Add tests
* Keep track of suspending date for users suspended through the CLI
* Show accurate number of accounts that would be affected by unsuspending an instance
* Change migration to set silenced_at and suspended_at
* Revert "Also unblock users who were suspended/silenced before dates were recorded"
This reverts commit a015c65d2d1e28c7b7cfab8b3f8cd5fb48b8b71c.
* Switch from using suspended and silenced to suspended_at and silenced_at
* Add post-deployment migration script to remove `suspended` and `silenced` columns
* Use Account#silence! and Account#suspend! instead of updating the underlying property
* Add silenced_at and suspended_at migration to post-migration
* Change account fabricator to translate suspended and silenced attributes
* Minor fixes
* Make unblocking domains always retroactive
* create account_identity_proofs table
* add endpoint for keybase to check local proofs
* add async task to update validity and liveness of proofs from keybase
* first pass keybase proof CRUD
* second pass keybase proof creation
* clean up proof list and add badges
* add avatar url to keybase api
* Always highlight the “Identity Proofs” navigation item when interacting with proofs.
* Update translations.
* Add profile URL.
* Reorder proofs.
* Add proofs to bio.
* Update settings/identity_proofs front-end.
* Use `link_to`.
* Only encode query params if they exist.
URLs without params had a trailing `?`.
* Only show live proofs.
* change valid to active in proof list and update liveness before displaying
* minor fixes
* add keybase config at well-known path
* extremely naive feature flagging off the identity proof UI
* fixes for rubocop
* make identity proofs page resilient to potential keybase issues
* normalize i18n
* tweaks for brakeman
* remove two unused translations
* cleanup and add more localizations
* make keybase_contacts an admin setting
* fix ExternalProofService my_domain
* use Addressable::URI in identity proofs
* use active model serializer for keybase proof config
* more cleanup of keybase proof config
* rename proof is_valid and is_live to proof_valid and proof_live
* cleanup
* assorted tweaks for more robust communication with keybase
* Clean up
* Small fixes
* Display verified identity identically to verified links
* Clean up unused CSS
* Add caching for Keybase avatar URLs
* Remove keybase_contacts setting
* Fix poll update handler calling method was that was not available
Fix regression from #10209
* Refactor VoteService
* Refactor ActivityPub::DistributePollUpdateWorker and optimize it
* Fix typo
* Fix typo
* Add polls
Fix#1629
* Add tests
* Fixes
* Change API for creating polls
* Use name instead of content for votes
* Remove poll validation for remote polls
* Add polls to public pages
* When updating the poll, update options just in case they were changed
* Fix public pages showing both poll and other media
* Fetch up to 5 replies when discovering a new remote status
This is used for resolving threads downwards. The originating
server must add a “replies” attributes with such replies for it to
be useful.
* Add some tests for ActivityPub::FetchRepliesWorker
* Add specs for ActivityPub::FetchRepliesService
* Serialize up to 5 public self-replies for ActivityPub notes
* Add specs for ActivityPub::NoteSerializer
* Move exponential backoff logic to a worker concern
* Fetch first page of paginated collections when fetching thread replies
* Add specs for paginated collections in replies
* Move Note replies serialization to a first CollectionPage
The collection isn't actually paginable yet as it has no id nor
a `next` field. This may come in another PR.
* Use pluck(:uri) instead of map(&:uri) to improve performances
* Fix fetching replies when they are in a CollectionPage
* Add hashtag filter to profiles
GET /@:username/tagged/:hashtag
GET /api/v1/accounts/:id/statuses?tagged=:hashtag
* Display featured hashtags on public profile
* Use separate model for featured tags
* Update featured hashtag counters on-write
* Limit featured tags to 10
* Use Contact User as Relay, Report, Subscribe.
* Use Account.representative to fetch contact user.
* Use find_local.
* No reason to use Account.representative in subscribe_service.
* Don't required representative!
* Fallback is included in Account.representative method.
* Add moderation warnings
Replace individual routes for disabling, silencing, and suspending
a user, as well as the report update route, with a unified account
action controller that allows you to select an action (none,
disable, silence, suspend) as well as whether it should generate an
e-mail notification with optional custom text. That notification,
with the optional custom text, is saved as a warning.
Additionally, there are warning presets you can configure to save
time when performing the above.
* Use Account#local_username_and_domain
- Some associations were missing from the clean-up
- Some attributes were not reset on suspension
- Skip federation and streaming deletes when purging a dead domain
- Move account association definitions to concern
* Allow min_id pagination in Feed#get
* Add min_id pagination to home and list timeline APIs
* Add min_id pagination to account statuses, public and tag APIs
* Remove unused stub in reports API
* Use min_id pagination in notifications, favourites, and fix order
* Fix HomeFeed#from_database not using paginate_by_id
DELETE /api/v1/suggestions/:account_id
When blocking, remove suggestion from both sides. Muting not affected,
since muting is supposed to be invisible to the target.
This replace calls of String#match? with rails Regex#match?
This follows the same idea used to keep Rails 5.2 compatible with Ruby
2.2.2 in https://github.com/rails/rails/pull/32973
* Re-add follow recommendations API
GET /api/v1/suggestions
Removed in 8efa081f21 due to Neo4J
dependency. The algorithm uses triadic closures, takes into account
suspensions, blocks, mutes, domain blocks, excludes locked and moved
accounts, and prefers more recently updated accounts.
* Track interactions with people you don't follow
Replying to, favouriting and reblogging someone you're not following
will make them show up in follow recommendations. The interactions
have different weights:
- Replying is 1
- Favouriting is 10 (decidedly positive interaction, but private)
- Reblogging is 20
Following them, muting or blocking will remove them from the list,
obviously.
* Remove triadic closures, ensure potential friendships are trimmed
* Add keyword filtering
GET|POST /api/v1/filters
GET|PUT|DELETE /api/v1/filters/:id
- Irreversible filters can drop toots from home or notifications
- Other filters can hide toots through the client app
- Filters use a phrase valid in particular contexts, expiration
* Make sure expired filters don't get applied client-side
* Add missing API methods
* Remove "regex filter" from column settings
* Add tests
* Add test for FeedManager
* Add CustomFilter test
* Add UI for managing filters
* Add streaming API event to allow syncing filters
* Fix tests
