* Validate internal actor
* Use “internal.actor” by default for the server actor username
* Fix instance actor username on the fly if it includes ':'
* Change actor name from internal.actor to mastodon.internal
* Do not offer to mark statuses as sensitive if there is no undeleted status with media attachments
* Fix crash when marking statuses as sensitive while some statuses are deleted
Fixes#21910
* Fix multiple strikes being created for a single report when selecting “Mark as sensitive”
* Add tests
* fix(status): remove send usage for private unlink_from_conversations
- make unlink_from_conversations public method
- rename unlink_from_conversations to unlink_from_conversations!
- fix send call on private method in statuses_vacuum and batched_remove_status_service
* fix(feeds_vacuum): replace find_in_batches with in_batches
because active record query results should be a little more efficient than
itterating with map and each. Postgres can grasp such lists of ids much quicker
than ruby can.
Will probably make allmost no difference, but cannot hurt either.
* Make autosuggest for mentions return followed accounts first
This makes it so that (when elasticsearch is disabled) when a user types '@foo' in the compose box, they are first going to get accounts they follow ordered by the ranking algorithm, and then second they will get accounts they do not follow, also ordered by the ranking algorithm.
This makes behavior more consistent with user expectation and also with results when elasticsearch is enabled.
* Fix ranking order to correct direction
* One more fixup per @gargron suggestion
* Tweak to ranking to no longer include following modifier
* Clear sessions on password change
* Rename User::clear_sessions to revoke_access for a clearer meaning
* Add reset paassword controller test
* Use User.find instead of User.find_for_authentication for reset password test
* Use redirect and render for better test meaning in reset password
Co-authored-by: Effy Elden <effy@effy.space>
* Fix trying to fetch posts from other users when fetching featured posts
* Rate-limit discovery of new subdomains
* Put a limit on recursively discovering new accounts
* refactor(statuses_vacuum): remove dead code - unused
Method is not called inside class and private.
Clean up dead code.
* refactor(statuses_vacuum): make retention_period present test explicit
This private method only hides functionality.
It is best practice to be as explicit as possible.
* refactor(statuses_vacuum): improve query performance
- fix statuses_scope having sub-select for Account.remote scope by
`joins(:account).merge(Account.remote)`
- fix statuses_scope unnecessary use of `Status.arel_table[:id].lt`
because it is inexplicit, bad practice and even slower than normal
`.where('statuses.id < ?'`
- fix statuses_scope remove select(:id, :visibility) for having reusable
active record query batches (no re queries)
- fix vacuum_statuses! to use in_batches instead of find_in_batches,
because in_batches delivers a full blown active record query result,
in stead of an array - no requeries necessary
- send(:unlink_from_conversations) not to perform another db query, but
reuse the in_batches result instead.
- remove now obsolete remove_from_account_conversations method
- remove_from_search_index uses array of ids, instead of mapping
the ids from an array - this should be more efficient
- use the in_batches scope to call delete_all, instead of running
another db query for this - because it is again more efficient
- add TODO comment for calling models private method with send
* refactor(status): simplify unlink_from_conversations
- add `has_many through:` relation mentioned_accounts
- use model scope local instead of method call `Status#local?`
- more readable add account to inbox_owners when account.local?
* refactor(status): searchable_by way less sub selects
These queries all included a sub-select. Doing the same with a joins
should be more efficient.
Since this method does 5 such queries, this should be significant,
since it technically halves the query count.
This is how it was:
```ruby
[3] pry(main)> Status.first.mentions.where(account: Account.local, silent: false).explain
Status Load (1.6ms) SELECT "statuses".* FROM "statuses" WHERE "statuses"."deleted_at" IS NULL ORDER BY "statuses"."id" DESC LIMIT $1 [["LIMIT", 1]]
Mention Load (1.5ms) SELECT "mentions".* FROM "mentions" WHERE "mentions"."status_id" = $1 AND "mentions"."account_id" IN (SELECT "accounts"."id" FROM "accounts" WHERE "accounts"."domain" IS NULL) AND "mentions"."silent" = $2 [["status_id", 109382923142288414], ["silent", false]]
=> EXPLAIN for: SELECT "mentions".* FROM "mentions" WHERE "mentions"."status_id" = $1 AND "mentions"."account_id" IN (SELECT "accounts"."id" FROM "accounts" WHERE "accounts"."domain" IS NULL) AND "mentions"."silent" = $2 [["status_id", 109382923142288414], ["silent", false]]
QUERY PLAN
------------------------------------------------------------------------------------------------------------------
Nested Loop (cost=0.15..23.08 rows=1 width=41)
-> Seq Scan on accounts (cost=0.00..10.90 rows=1 width=8)
Filter: (domain IS NULL)
-> Index Scan using index_mentions_on_account_id_and_status_id on mentions (cost=0.15..8.17 rows=1 width=41)
Index Cond: ((account_id = accounts.id) AND (status_id = '109382923142288414'::bigint))
Filter: (NOT silent)
(6 rows)
```
This is how it is with this change:
```ruby
[4] pry(main)> Status.first.mentions.joins(:account).merge(Account.local).active.explain
Status Load (1.7ms) SELECT "statuses".* FROM "statuses" WHERE "statuses"."deleted_at" IS NULL ORDER BY "statuses"."id" DESC LIMIT $1 [["LIMIT", 1]]
Mention Load (0.7ms) SELECT "mentions".* FROM "mentions" INNER JOIN "accounts" ON "accounts"."id" = "mentions"."account_id" WHERE "mentions"."status_id" = $1 AND "accounts"."domain" IS NULL AND "mentions"."silent" = $2 [["status_id", 109382923142288414], ["silent", false]]
=> EXPLAIN for: SELECT "mentions".* FROM "mentions" INNER JOIN "accounts" ON "accounts"."id" = "mentions"."account_id" WHERE "mentions"."status_id" = $1 AND "accounts"."domain" IS NULL AND "mentions"."silent" = $2 [["status_id", 109382923142288414], ["silent", false]]
QUERY PLAN
------------------------------------------------------------------------------------------------------------------
Nested Loop (cost=0.15..23.08 rows=1 width=41)
-> Seq Scan on accounts (cost=0.00..10.90 rows=1 width=8)
Filter: (domain IS NULL)
-> Index Scan using index_mentions_on_account_id_and_status_id on mentions (cost=0.15..8.17 rows=1 width=41)
Index Cond: ((account_id = accounts.id) AND (status_id = '109382923142288414'::bigint))
Filter: (NOT silent)
(6 rows)
```
When a poll is edited, we reset the poll and remove all previous
votes. However, prior to this commit, the voter count on the poll
was not reset. This leads to incorrect percentages being shown in
poll results.
Fixes#21696
* Don't allow URLs that contain non-normalized paths to be verified
This stops things like https://example.com/otheruser/../realuser where
"/otheruser" appears to be the verified URL, but the actual URL being
verified is "/realuser" due to the "/../".
Also fix a test to use 'https', so it is testing the right thing, now
that since #20304 https is required.
* missing do
If `Nokogiri::HTML(value).at_xpath('//body')` fails to find the `body` element, it will return `nil`. We can guard against that with an early return. Avoids calling `children` on `Nilclass` in those cases.
- Change verification to happen in `default` queue
- Change verification worker to only be queued if there's something to do
- Add `link` tags from metadata fields to page header of profiles
* Fix followers count not being updated when migrating follows
Fixes#19900
* Fix validation error in SynchronizeFeaturedTagsCollectionWorker
Also saves remote user's chosen case for hashtags
* Limit remote featured tags before validation
* Change admin announcement edition interface to use datetime-local
* Dynamically set announcement stop date as required if start date is set, set minimum date for stop date
* Change `all_day` to not be bound to presence of time-range
* Add pattern and placeholder as minimal fallback for browsers not supporting datetime-local
* Display datetime-local inputs as local time
Co-authored-by: Eugen Rochko <eugen@zeonfederated.com>
* Change featured tag updates to add/remove activity
* Fix to check for the existence of feature tag
* Rename service and worker
* Merge AddHashtagSerializer with AddSerializer
* Undo removal of sidekiq_options
* Change public accounts pages to mount the web UI
* Fix handling of remote usernames in routes
- When logged in, serve web app
- When logged out, redirect to permalink
- Fix `app-body` class not being set sometimes due to name conflict
* Fix missing `multiColumn` prop
* Fix failing test
* Use `discoverable` attribute to control indexing directives
* Fix `<ColumnLoading />` not using `multiColumn`
* Add `noindex` to accounts in REST API
* Change noindex directive to not be rendered by default before a route is mounted
* Add loading indicator for detailed status in web UI
* Fix missing indicator appearing while account is loading in web UI
* Add LIMIT of featured tag to instance API response
* Add featured_tags_collection_url to Account
* Add synchronization of remote featured tags
* Deliver update activity when updating featured tag
* Remove featured_tags_collection_url
* Revert "Add featured_tags_collection_url to Account"
This reverts commit cff349fc27.
* Add hashtag sync from featured collections
* Fix tag name normalize
* Add target option to fetch featured collection
* Refactor fetch_featured_tags_collection_service
* Add LIMIT of featured tag to v1/instance API response
In the absence of an opt-in to multiple specific languages in the
preferences, it makes more sense to filter by the user's presumed
language only (interface language or `lang` override)
When user has no locale preference saved (such as never changing it
from the default), the preferred posting language is nil, and
the dropdown is not visible
* Add database table for status-specific filters
* Add REST endpoints, entities and attributes
* Show status filters in /filters interface
* Perform server-side filtering for individual posts filters
* Fix filtering on context mismatch
* Refactor `toServerSideType` by moving it to its own module
* Move loupe and delete icons to their own module
* Add ability to filter individual posts from WebUI
* Replace keyword list by warnings (expired, context mismatch)
* Refactor server-side filtering code
* Add tests
* Add model for custom filter keywords
* Use CustomFilterKeyword internally
Does not change the API
* Fix /filters/edit and /filters/new
* Add migration tests
* Remove whole_word column from custom_filters (covered by custom_filter_keywords)
* Redesign /filters
Instead of a list, present a card that displays more information and handles
multiple keywords per filter.
* Redesign /filters/new and /filters/edit to add and remove keywords
This adds a new gem dependency: cocoon, as well as a npm dependency:
cocoon-js-vanilla. Those are used to easily populate and remove form fields
from the user interface when manipulating multiple keyword filters at once.
* Add /api/v2/filters to edit filter with multiple keywords
Entities:
- `Filter`: `id`, `title`, `filter_action` (either `hide` or `warn`), `context`
`keywords`
- `FilterKeyword`: `id`, `keyword`, `whole_word`
API endpoits:
- `GET /api/v2/filters` to list filters (including keywords)
- `POST /api/v2/filters` to create a new filter
`keywords_attributes` can also be passed to create keywords in one request
- `GET /api/v2/filters/:id` to read a particular filter
- `PUT /api/v2/filters/:id` to update a new filter
`keywords_attributes` can also be passed to edit, delete or add keywords in
one request
- `DELETE /api/v2/filters/:id` to delete a particular filter
- `GET /api/v2/filters/:id/keywords` to list keywords for a filter
- `POST /api/v2/filters/:filter_id/keywords/:id` to add a new keyword to a
filter
- `GET /api/v2/filter_keywords/:id` to read a particular keyword
- `PUT /api/v2/filter_keywords/:id` to edit a particular keyword
- `DELETE /api/v2/filter_keywords/:id` to delete a particular keyword
* Change from `irreversible` boolean to `action` enum
* Remove irrelevent `irreversible_must_be_within_context` check
* Fix /filters/new and /filters/edit with update for filter_action
* Fix Rubocop/Codeclimate complaining about task names
* Refactor FeedManager#phrase_filtered?
This moves regexp building and filter caching to the `CustomFilter` class.
This does not change the functional behavior yet, but this changes how the
cache is built, doing per-custom_filter regexps so that filters can be matched
independently, while still offering caching.
* Perform server-side filtering and output result in REST API
* Fix numerous filters_changed events being sent when editing multiple keywords at once
* Add some tests
* Use the new API in the WebUI
- use client-side logic for filters we have fetched rules for.
This is so that filter changes can be retroactively applied without
reloading the UI.
- use server-side logic for filters we haven't fetched rules for yet
(e.g. network error, or initial timeline loading)
* Minor optimizations and refactoring
* Perform server-side filtering on the streaming server
* Change the wording of filter action labels
* Fix issues pointed out by linter
* Change design of “Show anyway” link in accordence to review comments
* Drop “irreversible” filtering behavior
* Move /api/v2/filter_keywords to /api/v1/filters/keywords
* Rename `filter_results` attribute to `filtered`
* Rename REST::LegacyFilterSerializer to REST::V1::FilterSerializer
* Fix systemChannelId value in streaming server
* Simplify code by removing client-side filtering code
The simplifcation comes at a cost though: filters aren't retroactively
applied anymore.
- `GET /api/v1/admin/domain_allows` lists allowed domains
- `GET /api/v1/admin/domain_allows/:id` shows one by ID
- `DELETE /api/v1/admin/domain_allows/:id` deletes a given domain from the list
of allowed domains
- `POST /api/v1/admin/domain_allows` to allow a new domain:
if that domain is already allowed, the existing DomainAllow will be returned
* Add /api/v1/admin/domain_blocks
Fixes#18140
- `GET /api/v1/admin/domain_blocks` lists domain blocks
- `GET /api/v1/admin/domain_blocks/:id` shows one by ID
- `DELETE /api/v1/admin/domain_blocks/:id` deletes a given domain block
- `POST /api/v1/admin/domain_blocks` to create a new domain block:
if it conflicts with an existing one, returns an error with
an attribute `existing_domain_block` with the rendered domain block
* Simplify conflict handling as suggested in review
* Change unapproved and unconfirmed account to not be accessible in the REST API
* Change Account#searchable? to reject unconfirmed and unapproved users
* Disable search for unapproved and unconfirmed users in Account.search_for
* Disable search for unapproved and unconfirmed users in Account.advanced_search_for
* Remove unconfirmed and unapproved accounts from Account.searchable scope
* Prevent mentions to unapproved/unconfirmed accounts
* Fix some old tests for Account.advanced_search_for
* Add some Account.advanced_search_for tests for existing behaviors
* Add some tests for Account.search_for
* Add Account.advanced_search_for tests unconfirmed and unapproved accounts
* Add Account.searchable tests
* Fix Account.without_unapproved scope potentially messing with previously-applied scopes
* Allow lookup of unconfirmed/unapproved accounts through /api/v1/accounts/lookup
This is so that the API can still be used to check whether an username is free
to use.
* Fix warning an account outside of a report closing all reports for that account
* Make it clear what actions solve other reports
* Revert "Make it clear what actions solve other reports"
This reverts commit ad006de821.
Calculate trends in temporary sets to avoid having to manage items
that go below the decay threshold while not having any moments
where a half-processed set is accessible to end-users
* Change e-mail notifications to only be sent when recipient is offline
Change the default for follow and mention notifications back on
* Add preference to always send e-mail notifications
* Change wording
* Fix /api/v1/admin/accounts
Compatibility was broken since #17009 which changed the underlying filter class
without changing the controller.
This commits restore support for the old parameters.
* Add /api/v2/admin/accounts with the new parameters
* Add tests
* Add missing filter for `silenced` status
Co-authored-by: Eugen Rochko <eugen@zeonfederated.com>
Co-authored-by: Eugen Rochko <eugen@zeonfederated.com>
* Fix edits with no actual changes being allowed locally
* Fix edits with no actual changes being allowed through ActivityPub
* Fix false positive changes caused by description processing in model
* Fix not recording poll expiration update
* Fix test
* Revert changes to ProcessStatusUpdateService
* Various fixes and improvements
* Fix code style issues
* Various changes and improvements
* Add guard clause
* Update fix-duplicates task to 2022_02_10_153119
Also add support for Appeal to AccountMerging#merge_with!
* Update fix-duplicates task to 2022_03_07_094650
* Update fix-duplicates task to 2022_03_09_213005
* Update fix-duplicates task to 2022_03_07_083603
* Update fix-duplicates task to 2022_03_10_060626
* Update fix-duplicates script to 2022_03_07_083603
* Update fix-duplicates task to 2022_03_10_060706
* Update fix-duplicates task to 2022_03_10_060959
* Silence CodeClimate
* added OpenID Connect as an SSO option
* minor fixes
* added comments, removed an option that shouldn't be set
* fixed Gemfile.lock
* added newline to end of Gemfile.lock
* removed tab from Gemfile.lock
* remove chomp
* codeclimate changes and small name change to make function's purpose clearer
* codeclimate fix
* added SSO buttons to /about page
* minor refactor
* minor style change
* removed spurious change
* removed unecessary conditional from ensure_valid_username and added support for auth.info.name in user_params_from_auth
* minor changes
* Change how changes to media attachments are stored for edits
Fix not being able to re-order media attachments
* Fix not broadcasting updates when polls/media is changed through ActivityPub
* Various fixes and improvements
* Update app/models/report.rb
Co-authored-by: Claire <claire.github-309c@sitedethib.com>
* Add tracking of media attachment description changes
* Change poll in status edit to have a structure closer to the real one
Co-authored-by: Claire <claire.github-309c@sitedethib.com>
* Change design of federation pages in admin UI
* Fix query performance in instance media attachments measure
* Fix reblogs being included in instance languages dimension
* Fix performance of account timelines
* Various fixes and improvements
* Fix duplicate results being returned
Co-authored-by: Claire <claire.github-309c@sitedethib.com>
* Fix grouping for pinned statuses scope
Co-authored-by: Claire <claire.github-309c@sitedethib.com>
* Add `/api/v1/accounts/familiar_followers` to REST API
* Change hide network preference to be stored consistently for local and remote accounts
* Add dummy classes to migration
* Apply suggestions from code review
Co-authored-by: Claire <claire.github-309c@sitedethib.com>
Co-authored-by: Claire <claire.github-309c@sitedethib.com>
* Change old moderation strikes to be displayed in a separate page
Fixes#17552
This changes the moderation strikes displayed on `/auth/edit` to be those from
the past 3 months, and make all moderation strikes targeting the current user
available in `/disputes`.
* Add short description of what the strikes page is for
* Move link to list of strikes to “Account status” instead of navigation item
* Normalize i18n file
* Fix layout and styling of strikes link
* Revert highlights_on regexp
* Reintroduce account status summary
- this way, “Account status” is never empty
- account status is not necessarily bound to strikes, or recent strikes
* Add trending statuses
* Fix dangling items with stale scores in localized sets
* Various fixes and improvements
- Change approve_all/reject_all to approve_accounts/reject_accounts
- Change Trends::Query methods to not mutate the original query
- Change Trends::Query#skip to offset
- Change follow recommendations to be refreshed in a transaction
* Add tests for trending statuses filtering behaviour
* Fix not applying filtering scope in controller
Video files with variable framerates are converted to constant framerate videos
and the output framerate picked by ffmpeg is based on the original file's
container framerate (which can be different from the average framerate).
This means that an input video with variable framerate with about 30 frames per
second on average, but a maximum of 120 fps will be converted to a constant 120
fps file, which won't be processed by other Mastodon servers.
This commit changes it so that input files with VFR and a maximum framerate
above the framerate threshold are converted to VFR files with the maximum frame
rate enforced.
* Display an error when an appeal could not be submitted
* Do not offer users to appeal old strikes
* Fix 500 error when trying to appeal a strike that is too old
* Avoid using an extra translatable string
* Add appeals
* Add ability to reject appeals and ability to browse pending appeals in admin UI
* Add strikes to account page in settings
* Various fixes and improvements
- Add separate notification setting for appeals, separate from reports
- Fix style of links in report/strike header
- Change approving an appeal to not restore statuses (due to federation complexities)
- Change style of successfully appealed strikes on account settings page
- Change account settings page to only show unappealed or recently appealed strikes
* Change appealed_at to overruled_at
* Fix missing method error
* Add notifications when a reblogged status has been updated
* Change wording to say "edit" instead of "update" and add missing controls
* Replace previous update notifications with the most up-to-date one
* Add editing for published statuses
* Fix change of multiple-choice boolean in poll not resetting votes
* Remove the ability to update existing media attachments for now
* Fix Sidekiq warnings about JSON serialization
This occurs on every symbol argument we pass, and every symbol key in hashes,
because Sidekiq expects strings instead.
See https://github.com/mperham/sidekiq/pull/5071
We do not need to change how workers parse their arguments because this has
not changed and we were already converting to symbols adequately or using
`with_indifferent_access`.
* Set Sidekiq to raise on unsafe arguments in test mode
In order to more easily catch issues that would produce warnings in production
code.
* Fix error-prone SQL queries in Account search
While this code seems to not present an actual vulnerability, one could
easily be introduced by mistake due to how the query is built.
This PR parameterises the `to_tsquery` input to make the query more robust.
* Harden code for Status#tagged_with_all and Status#tagged_with_none
Those two scopes aren't used in a way that could be vulnerable to an SQL
injection, but keeping them unchanged might be a hazard.
* Remove unneeded spaces surrounding tsquery term
* Please CodeClimate
* Move advanced_search_for SQL template to its own function
This avoids one level of indentation while making clearer that the SQL template
isn't build from all the dynamic parameters of advanced_search_for.
* Add tests covering tagged_with, tagged_with_all and tagged_with_none
* Rewrite tagged_with_none to avoid multiple joins and make it more robust
* Remove obsolete brakeman warnings
* Revert "Remove unneeded spaces surrounding tsquery term"
The two queries are not strictly equivalent.
This reverts commit 86f16c537e.
* Add support for editing for published statuses
* Fix references to stripped-out code
* Various fixes and improvements
* Further fixes and improvements
* Fix updates being potentially sent to unauthorized recipients
* Various fixes and improvements
* Fix wrong words in test
* Fix notifying accounts that were tagged but were not in the audience
* Fix mistake
* Add admin option to remove canonical email blocks from a deleted account
* Add tootctl canonical_email_blocks to inspect and remove canonical email blocks
* Add trending links
* Add overriding specific links trendability
* Add link type to preview cards and only trend articles
Change trends review notifications from being sent every 5 minutes to being sent every 2 hours
Change threshold from 5 unique accounts to 15 unique accounts
* Fix tests
* Fix error when suspending user with an already-existing canonical email block
Fixes#17033
While attempting to create a `CanonicalEmailBlock` with an existing hash would
raise an `ActiveRecord::RecordNotUnique` error, this being done within a
transaction would cancel the whole transaction. For this reason, checking for
uniqueness in Rails would query the database within the transaction and avoid
invalidating the whole transaction for this reason.
A race condition is still possible, where multiple accounts sharing a canonical
email would be blocked in concurrent transactions, in which only one would
succeed, but that is way less likely to happen that the current issue, and can
always be retried after the first failure, unlike the current situation.
* Add tests
Up until now, we have used Devise's Rememberable mechanism to re-log users
after the end of their browser sessions. This mechanism relies on a signed
cookie containing a token. That token was stored on the user's record,
meaning it was shared across all logged in browsers, meaning truly revoking
a browser's ability to auto-log-in involves revoking the token itself, and
revoking access from *all* logged-in browsers.
We had a session mechanism that dynamically checks whether a user's session
has been disabled, and would log out the user if so. However, this would only
clear a session being actively used, and a new one could be respawned with
the `remember_user_token` cookie.
In practice, this caused two issues:
- sessions could be revived after being closed from /auth/edit (security issue)
- auto-log-in would be disabled for *all* browsers after logging out from one
of them
This PR removes the `remember_token` mechanism and treats the `_session_id`
cookie/token as a browser-specific `remember_token`, fixing both issues.
* Add followed_by? to account_interactions
* Add RemoveFromFollowersService
* Fix AccountBatch to use RemoveFromFollowersService
* Add remove from followers API
Currently, there is no way for an admin to delete an account record. However,
should that happen in the future, or should an admin mistype an account id,
`/admin/action_logs?target_account_id=<non-existent-account-id>` currently
returns a 404 instead of an empty list.
With this commit, the aforementioned page returns a correct list instead
(potentially non-empty if the account has been manually deleted)
* Fix media attachment size validation not correctly accounting for file type
Fixes a regression introduced in #16724 caused by the fact that kt-paperclip
now correctly runs validations before processing, meaning that file size
verification could not rely on our before_post_processing hook.
Moved the `before_post_processing` hooks to `before_validate` to make sure
the media attachment type is set correctly before the file gets validated.
* Add tests
* Switch from unmaintained paperclip to kt-paperclip
* Drop some compatibility monkey-patches not required by kt-paperclip
* Drop media spoof check monkey-patching
It's broken with kt-paperclip and hopefully it won't be needed anymore
* Fix regression introduced by paperclip 6.1.0
* Do not rely on pathname to call FastImage
* Add test for ogg vorbis file with cover art
* Add audio/vorbis to the accepted content-types
This seems erroneous as this would be the content-type for a vorbis stream
without an ogg container, but that's what the `marcel` gem outputs, so…
* Restore missing for_as_default method
* Refactor Attachmentable concern and delay Paperclip's content-type spoof check
Check for content-type spoofing *after* setting the extension ourselves, this
fixes a regression with kt-paperclip's validations being more strict than
paperclip 6.0.0 and rejecting some Pleroma uploads because of unknown
extensions.
* Please CodeClimate
* Add audio/vorbis to the unreliable set
It doesn't correspond to a file format and thus has no extension associated.
* Stop setting a shortcode to newly-created media attachments
The WebUI has stopped using the “short media URL” in ages. This isn't used
anywhere except for mail notifications.
Deprecating it would allow us to eventually get rid of at least a database
column and corruption-prone index, as well as a controller.
* Fix tests
* Fix followers synchronization mechanism not working when URI has empty path
To my knowledge, there is no current implementation on the fediverse
that can use bare domains (e.g., actor is at https://example.org instead of
something like https://example.org/actor) that also plans to support the
followers synchronization mechanism. However, Mastodon's current implementation
would exclude such accounts from followers list.
Also adds tests and rename them to reflect the proper method names.
* Move url prefix regexp to its own constant
* Add account statuses cleanup policy model
* Record last inspected toot to delete to speed up successive calls to statuses_to_delete
* Add service to cleanup a given account's statuses within a budget
* Add worker to go through account policies and delete old toots
* Fix last inspected status id logic
All existing statuses older or equal to last inspected status id must be
kept by the current policy. This is an invariant that must be kept so that
resuming deletion from the last inspected status remains sound.
* Add tests
* Refactor scheduler and add tests
* Add user interface
* Add support for discriminating based on boosts/favs
* Add UI support for min_reblogs and min_favs, rework UI
* Address first round of review comments
* Replace Snowflake#id_at_start with with_random parameter
* Add tests
* Add tests for StatusesCleanupController
* Rework settings page
* Adjust load-avoiding mechanisms
* Please CodeClimate
* Add account_notes relationship
* Add tests
* Fix owned account notes not being deleted when an account is deleted
* Add post-migration to clean up orphaned account notes
3.4.1 dropped account_stats.lock_version, but in a way breaking the usual
upgrade path by requiring services to be reloaded after the post-migrations.
Indeed, `self.locking_column = nil` was not enough for Rails to ignore the
`lock_version` column when preparing statements on application load, resulting
in some ActiveRecord queries (typically those involving
`includes(:account_stat)`) erroring out with:
ActiveRecord::StatementInvalid (PG::UndefinedColumn: ERROR: column account_stats.lock_version does not exist
* Fix account deletion sometimes failing because of optimistic locks
In some rare occasions[1], deleting accounts would fail with a
`StaleObjectError` exception.
Indeed, account deletion manually sets the `AccountStat` values without
handling cases where the optimistic locking on `AccountStat` would fail.
To my knowledge, with the rewrite of account counters in #15913, the
`DeleteAccountService` is now the only place that changes the counters in
a way that is not atomic.
Since in this specific case, we do not care about the previous values of the
account counters, it appears we don't need locking at all for this table
anymore.
[1]: https://discourse.joinmastodon.org/t/account-cant-be-deleted/3602
* Bump MAX_SUPPORTED_VERSION in maintenance script
Since #16173, `account_summaries` and `follow_recommendations` are not
populated at creation time, which causes concurrent refresh to fail.
As we currently only use those materialized views right after explicitly
refreshing them, this commit changes refreshes to not be performed
concurrently. This will fix the issue and ensure the refresh completes
faster while using less resources.
If a status with a hashtag becomes very popular, it stands to
reason that the hashtag should have a chance at trending
Fix no stats being recorded for hashtags that are not allowed
to trend, and stop ignoring bots
Remove references to hashtags in profile directory from the code
and the admin UI
* Fix issues with POSIX::Spawn, Terrapin and Ruby 3.0
Also improve the Terrapin monkey-patch for the stderr/stdout issue.
* Fix keyword argument handling throughout the codebase
* Monkey-patch Paperclip to fix keyword arguments handling in validators
* Change validation_extensions to please CodeClimate
* Bump microformats from 4.2.1 to 4.3.1
* Allow Ruby 3.0
* Add Ruby 3.0 test target to CircleCI
* Add test for admin dashboard warnings
* Fix admin dashboard warnings on Ruby 3.0
Express follow_recommendations in terms of account_summaries rather than
accounts, integrate filters that are unconditionally used, and materialize
the resulting view.
This should result in the bulk of the computation being performed only once
instead of **once per recommendation language**.
* Fix media processing getting stuck on too much stdin/stderr
See thoughtbot/terrapin#5
* Remove dependency on paperclip-av-transcoder gem
* Remove dependency on streamio-ffmpeg gem
* Disable stdin on ffmpeg process
Prevent new accounts from being created using the same underlying
e-mail as a suspended account using extensions and period
permutations. Stores e-mails as a SHA256 hash
* Update devise-two-factor to unreleased fork for Rails 6 support
Update tests to match new `rotp` version.
* Update nsa gem to unreleased fork for Rails 6 support
* Update rails to 6.1.3 and rails-i18n to 6.0
* Update to unreleased fork of pluck_each for Ruby 6 support
* Run "rails app:update"
* Add missing ActiveStorage config file
* Use config.ssl_options instead of removed ApplicationController#force_ssl
Disabled force_ssl-related tests as they do not seem to be easily testable
anymore.
* Fix nonce directives by removing Rails 5 specific monkey-patching
* Fix fixture_file_upload deprecation warning
* Fix yield-based test failing with Rails 6
* Use Rails 6's index_with when possible
* Use ActiveRecord::Cache::Store#delete_multi from Rails 6
This will yield better performances when deleting an account
* Disable Rails 6.1's automatic preload link headers
Since Rails 6.1, ActionView adds preload links for javascript files
in the Links header per default.
In our case, that will bloat headers too much and potentially cause
issues with reverse proxies. Furhermore, we don't need those links,
as we already output them as HTML link tags.
* Switch to Rails 6.0 default config
* Switch to Rails 6.1 default config
* Do not include autoload paths in the load path
* Improve account counters handling
* Use ActiveRecord::Base::sanitize_sql to pass values instead of interpolating them
Keep using string interpolation for `key` as it is safe and using
“ActiveRecord::Base::sanitize_sql_hash_for_assignment” would require stitching
bits of SQL in a way that is not more easily checked for safety.
* Add migration hook to catch PostgreSQL versions earlier than 9.5
* Fix misuse of foreign_type
* Fix use of removed "add_template_helper"
* Use response.media_type instead of response.content_type in tests
* Fix CSV export controller test on Rails 6
Rails 6 sets a "filename*" field in the Content-Disposition header to
explicitly encode the filename as UTF-8.
This changes checks the first part of the Content-Disposition header so
it matches in both Rails 5 and Rails 6.
* Fix emoji formatting with Rails 6
* Make emoji output more idiomatic and robust
* Switch from redis-rails gem to built-in Rails redis cache storage
An uncommon but somewhat difficult to digagnose issue is dealing with
improperly-seeded databases. In such cases, instance-signed fetches will
fail with a ActiveRecord::RecordNotFound error, usually caught and handled
as generic 404, leading people to think the remote resource itself has not
been found, while it's the local instance actor that does not exist.
This commit changes the code so that failure to find the instance actor
automatically creates a new one, so that improperly-seeded databases do
not cause any issue.
* disable NewCops
* update TargetRubyVersion
* Fix Lint/MissingSuper for ActiveModelSerializers::Model
* Fix Lint/MissingSuper for feed
* Fix Lint/FloatComparison
* Do not use instance variables
* Fix being able to import more than allowed number of follows
Without this commit, if someone tries importing a second list of accounts to
follow before the first one has been processed, this will queue imports for
the two whole lists, even if they exceed the account's allowed number of
outgoing follows.
This commit changes it so the individual queued imports aren't exempt from
the follow limit check (they remain exempt from the rate-limiting check
though).
* Catch validation errors to not re-queue failed follows
Co-authored-by: Claire <claire.github-309c@sitedethib.com>
* Delete status records by batches of 50
* Do not precompute values that are only used once
* Do not generate redis events for removal of public toots older than two weeks
* Filter reported toots a priori for polls and status deletion
* Do not process reblogs when cleaning up public timelines
As in Mastodon proper, reblogs don't appear in public TLs
* Clean the deleted account's own feed in one go
* Refactor Account#clean_feed_manager and List#clean_feed_manager
* Delete instead of destroy a few more associations
* Fix preloading
Co-authored-by: Claire <claire.github-309c@sitedethib.com>
* Fix external user creation failing when invite request text is required
Also fixes tootctl-based user creation.
* Add test about invites when invite request text is otherwise required
Co-authored-by: Claire <claire.github-309c@sitedethib.com>
* Fix deleting polls not deleting notifications
* Fix fav notification deletion when deleting a toot
* Refactor DeleteAccountService spec
* Add DeleteAccountService tests for other associations and notifications
* Add favourite handling spec in status removal
Co-authored-by: Claire <claire.github-309c@sitedethib.com>
* Fix account deletion workers being queued multiple times for a single account
* Fix poll votes being unnecessarily instantiated on poll deletion
* Fix favourites being unnecessarily instantiated on status deletion
* Remove inaccurate comments
* Delete polls instead of destroying them
Co-authored-by: Claire <claire.github-309c@sitedethib.com>
* Fix admin being able to suspend their own instance account
* Add text about the instance's own actor in admin view
* Change instance actor notice from flash message to template
* Do not list local instance actor in account moderation list
* Add indication to admin UI of whether a report has been forwarded
* Rework how forwarded status is displayed
Co-authored-by: Claire <claire.github-309c@sitedethib.com>
* Add honeypot fields to limit non-specialized spam
Add two honeypot fields: a fake website input and a fake password confirmation
one. The label/placeholder/aria-label tells not to fill them, and they are
hidden in CSS, so legitimate users should not fall into these.
This should cut down on some non-Mastodon-specific spambots.
* Require a 3 seconds delay before submitting the registration form
* Fix tests
* Move registration form time check to model validation
* Give people a chance to clear the honeypot fields
* Refactor honeypot translation strings
Co-authored-by: Claire <claire.github-309c@sitedethib.com>
If someone tries logging in to an account and is prompted for a 2FA
code or sign-in token, even if the account's password or e-mail is
updated in the meantime, the session will show the prompt and allow
the login process to complete with a valid 2FA code or sign-in token
Do not downcase the queried tag before passing it to postgres when searching:
- tags are not downcased on creation
- `arel_table[:name].lower.matches(pattern)` generates an ILIKE anyway
- if Postgres and Rails happen to use different case-folding rules,
downcasing before query but not before insertion may mean that some
tags with some casings are not searchable
* Add support for followers synchronization on the receiving end
Check the `collectionSynchronization` attribute on `Create` and `Announce`
activities and synchronize followers from provided collection if possible.
* Add tests for followers synchronization on the receiving end
* Add support for follower synchronization on the sender's end
* Add tests for the sending end
* Switch from AS attributes to HTTP header
Replace the custom `collectionSynchronization` ActivityStreams attribute by
an HTTP header (`X-AS-Collection-Synchronization`) with the same syntax as
the `Signature` header and the following fields:
- `collectionId` to specify which collection to synchronize
- `digest` for the SHA256 hex-digest of the list of followers known on the
receiving instance (where “receiving instance” is determined by accounts
sharing the same host name for their ActivityPub actor `id`)
- `url` of a collection that should be fetched by the instance actor
Internally, move away from the webfinger-based `domain` attribute and use
account `uri` prefix to group accounts.
* Add environment variable to disable followers synchronization
Since the whole mechanism relies on some new preconditions that, in some
extremely rare cases, might not be met, add an environment variable
(DISABLE_FOLLOWERS_SYNCHRONIZATION) to disable the mechanism altogether and
avoid followers being incorrectly removed.
The current conditions are:
1. all managed accounts' actor `id` and inbox URL have the same URI scheme and
netloc.
2. all accounts whose actor `id` or inbox URL share the same URI scheme and
netloc as a managed account must be managed by the same Mastodon instance
as well.
As far as Mastodon is concerned, breaking those preconditions require extensive
configuration changes in the reverse proxy and might also cause other issues.
Therefore, this environment variable provides a way out for people with highly
unusual configurations, and can be safely ignored for the overwhelming majority
of Mastodon administrators.
* Only set follower synchronization header on non-public statuses
This is to avoid unnecessary computations and allow Follow-related
activities to be handled by the usual codepath instead of going through
the synchronization mechanism (otherwise, any Follow/Undo/Accept activity
would trigger the synchronization mechanism even if processing the activity
itself would be enough to re-introduce synchronization)
* Change how ActivityPub::SynchronizeFollowersService handles follow requests
If the remote lists a local follower which we only know has sent a follow
request, consider the follow request as accepted instead of sending an Undo.
* Integrate review feeback
- rename X-AS-Collection-Synchronization to Collection-Synchronization
- various minor refactoring and code style changes
* Only select required fields when computing followers_hash
* Use actor URI rather than webfinger domain in synchronization endpoint
* Change hash computation to be a XOR of individual hashes
Makes it much easier to be memory-efficient, and avoid sorting discrepancy issues.
* Marginally improve followers_hash computation speed
* Further improve hash computation performances by using pluck_each
There are edge cases where requests to certain hosts timeout when
using the vanilla HTTP.rb gem, which the goldfinger gem uses. Now
that we no longer need to support OStatus servers, webfinger logic
is so simple that there is no point encapsulating it in a gem, so
we can just use our own Request class. With that, we benefit from
more robust timeout code and IPv4/IPv6 resolution.
Fix#14091
* Add bell button
Fix#4890
* Remove duplicate type from post-deployment migration
* Fix legacy class type mappings
* Improve query performance with better index
* Fix validation
* Remove redundant index from notifications
* Add database support for list show-reply preferences
* Add backend support to read and update list-specific show_replies settings
* Add basic UI to set list replies setting
* Add specs for list replies policy
* Switch "cycling" reply policy link to a set of radio inputs
* Capitalize replies_policy strings
* Change radio button design to be consistent with that of the directory explorer
* Make Array-creation behavior of Paginable more predictable
Paginable.paginate_by_id usually returns ActiveRecord::Relation, but it
returns an Array if min_id option is present. The behavior caused problems
fixed with the following commits:
- 552e886b64
- b63ede5005
- 64ef37b89d
To prevent from recurring similar problems, this commit introduces two
changes:
- The scope now always returns an Array whether min_id option is present
or not.
- The scope is renamed to to_a_paginated_by_id to clarify it returns an
Array.
* Transform Paginable.to_a_paginated_by_id from a scope to a class method
https://api.rubyonrails.org/classes/ActiveRecord/Scoping/Named/ClassMethods.html#method-i-scope
> The method is intended to return an ActiveRecord::Relation object, which
> is composable with other scopes.
Paginable.to_a_paginated_by_id returns an Array and is not appropriate
as a scope.
* feat: add possibility of adding WebAuthn security keys to use as 2FA
This adds a basic UI for enabling WebAuthn 2FA. We did a little refactor
to the Settings page for editing the 2FA methods – now it will list the
methods that are available to the user (TOTP and WebAuthn) and from
there they'll be able to add or remove any of them.
Also, it's worth mentioning that for enabling WebAuthn it's required to
have TOTP enabled, so the first time that you go to the 2FA Settings
page, you'll be asked to set it up.
This work was inspired by the one donde by Github in their platform, and
despite it could be approached in different ways, we decided to go with
this one given that we feel that this gives a great UX.
Co-authored-by: Facundo Padula <facundo.padula@cedarcode.com>
* feat: add request for WebAuthn as second factor at login if enabled
This commits adds the feature for using WebAuthn as a second factor for
login when enabled.
If users have WebAuthn enabled, now a page requesting for the use of a
WebAuthn credential for log in will appear, although a link redirecting
to the old page for logging in using a two-factor code will also be
present.
Co-authored-by: Facundo Padula <facundo.padula@cedarcode.com>
* feat: add possibility of deleting WebAuthn Credentials
Co-authored-by: Facundo Padula <facundo.padula@cedarcode.com>
* feat: disable WebAuthn when an Admin disables 2FA for a user
Co-authored-by: Facundo Padula <facundo.padula@cedarcode.com>
* feat: remove ability to disable TOTP leaving only WebAuthn as 2FA
Following examples form other platforms like Github, we decided to make
Webauthn 2FA secondary to 2FA with TOTP, so that we removed the
possibility of removing TOTP authentication only, leaving users with
just WEbAuthn as 2FA. Instead, users will have to click on 'Disable 2FA'
in order to remove second factor auth.
The reason for WebAuthn being secondary to TOPT is that in that way,
users will still be able to log in using their code from their phone's
application if they don't have their security keys with them – or maybe
even lost them.
* We had to change a little the flow for setting up TOTP, given that now
it's possible to setting up again if you already had TOTP, in order to
let users modify their authenticator app – given that now it's not
possible for them to disable TOTP and set it up again with another
authenticator app.
So, basically, now instead of storing the new `otp_secret` in the
user, we store it in the session until the process of set up is
finished.
This was because, as it was before, when users clicked on 'Edit' in
the new two-factor methods lists page, but then went back without
finishing the flow, their `otp_secret` had been changed therefore
invalidating their previous authenticator app, making them unable to
log in again using TOTP.
Co-authored-by: Facundo Padula <facundo.padula@cedarcode.com>
* refactor: fix eslint errors
The PR build was failing given that linting returning some errors.
This commit attempts to fix them.
* refactor: normalize i18n translations
The build was failing given that i18n translations files were not
normalized.
This commits fixes that.
* refactor: avoid having the webauthn gem locked to a specific version
* refactor: use symbols for routes without '/'
* refactor: avoid sending webauthn disabled email when 2FA is disabled
When an admins disable 2FA for users, we were sending two mails
to them, one notifying that 2FA was disabled and the other to notify
that WebAuthn was disabled.
As the second one is redundant since the first email includes it, we can
remove it and send just one email to users.
* refactor: avoid creating new env variable for webauthn_origin config
* refactor: improve flash error messages for webauthn pages
Co-authored-by: Facundo Padula <facundo.padula@cedarcode.com>
Since #14145, the `set_type_and_extension` has been moved from
`before_post_process` to `before_file_post_process`, but while the former
runs before all validations performed by Paperclip, the latter is dependent
on the order validations and hooks are defined.
In our case, this meant video files could be checked against the generic 10MB
limit, causing validation failures, which, internally, make Paperclip skip
post-processing, and thus, transcoding of the video file.
The actual validation would then happen after the type is correctly set, so
the large file would pass validation, but without being transcoded first.
This commit moves the hook definition so that it is run before checking for
the file size.
* Add UserNote model
* Add UI for user notes
* Put comment in relationships entity
* Add API to create user notes
* Copy user notes to new account when receiving a Move activity
* Address some of the review remarks
* Replace modal by inline edition
* Please CodeClimate
* Button design changes
* Change design again
* Cancel note edition when pressing Escape
* Fixes
* Tweak design again
* Move “Add note” item, and allow users to add notes to themselves
* Rename UserNote into AccountNote, rename “comment” Relationship attribute to “note”
- Change audio files to not be stripped of metadata
- Automatically extract cover art from audio if it exists
- Add `thumbnail` parameter to `POST /api/v1/media`, `POST /api/v2/media` and `PUT /api/v1/media/:id`
- Add `icon` to represent it in attachments in ActivityPub
- Fix `preview_url` containing URL of missing missing image when there is no thumbnail instead of null
- Fix duration of audio not being displayed on public pages until the file is loaded
- Fix audio attachments not being represented in OpenGraph tags
- Fix audio being represented as "1 image" in OpenGraph descriptions
- Fix video metadata being overwritten by paperclip-av-transcoder
- Fix embedded player not using Mastodon's UI
- Fix audio/video progress bars not moving smoothly
- Fix audio/video buffered bars not displaying correctly
* Fix admin page crashing when trying to block an invalid domain name
Fixes#13880
* Fix trailing and leading spaces not being properly stripped for domain blocks
* Improve RSS entries for statuses
- Render polls in both accounts and tags serializers
- Refactor RSS serializers
- Change title preview to include ellipsis when truncated
- Change title preview to show CW instead of toot text
- Add tests
* Remove title from OEmbed serialization
Twitter doesn't serialize title either, and tihs allows us to move the
title formatting code to the RSS serializers.
When deduplicating account usernames for OAuthable users, the routine did check if any account was known with that username, including remote accounts. This caused some unnecessary deduplication, and usernames ending with unexpected trailing _1.
This fixes#13580
Follow-up to #13452, fixing broken `uri.nil?` test.
Also remove the separate check for `uri` presence, as that would result
in a “Please review 2 errors below” while only one would be listed.
* Fix “Email changed” notification sometimes having wrong e-mail
Fixes#6778
The root of the issue is that `send_devise_notification` was called before
the changes were properly commited to the database, causing the mailer to
pick previous values if running too early.
Devise's documentation provides guidance on how to handle that[1][2], however,
I have found it to not be working, as the following happens, in that order:
- `send_devise_notification` is called for the `email_changed` notification.
In that case, `changed?` is false and `saved_changes?` is true, so
if we use the former, we have the same issue.
- the `after_commit` hook is called
- `send_devise_notification` is called for the `confirmation_instructions`
notification.
In that case, `changed?` is still false, and `saved_changes?` still true,
so if we use the latter, that second notification email is simply not
going to be sent (as we would be queuing the notification *after*
executing the after_commit hook).
This is because it may be called from either an `after_update` or
`after_commit` hook, the difference not being a call to `save` but the
transaction actually being committed to the database. This may arguably
be a bug in Devise, or Devise's notification.
The proposed workaround is inspired by Devise's documentation but checks
whether a transaction is open to make the call whether to immediately
send the notification or defer it to the `after_commit` hook.
[1]: https://www.rubydoc.info/github/plataformatec/devise/Devise%2FModels%2FAuthenticatable:send_devise_notification
[2]: 406915cb78/lib/devise/models/authenticatable.rb (L133-L194)
* Fix cases when sending notifications without changing the model
* Defer sending if and only if in transaction including current record
* Fix PostgreSQL load when linking in announcements
Fixes#13245 by caching status lookups
Since statuses are supposed to be known already and we only
need their URLs and a few other things, caching them should
be fine.
Since it's only used by announcements so far, there won't
be much statuses to cache.
* Perform status lookup when saving announcements, not when rendering them
* Change EntityCache#status to fetch URLs instead of looking into the database
* Move announcement link lookup to publishing worker
* Address issues pointed out during review
* Change ActivityPub follower/following collections to not link first page
* Add support for hiding followers and following of remote users
* Switch to using a single `hide_collections` column
* Address code style remarks
Change `all_day` to be a visual client-side cue only
Publish immediately if `scheduled_at` is in the past
Add `published_at` and `updated_at` to announcements JSON
While making browser requests in the other sessions after a password
change or reset does not allow you to be logged in and correctly
invalidates the session making the request, sessions have API tokens
associated with them, which can still be used until that session
is invalidated.
This is a security issue for accounts that were already compromised
some other way because it makes it harder to throw out the hijacker.
* Add announcements
Fix#11006
* Add reactions to announcements
* Add admin UI for announcements
* Add unit tests
* Fix issues
- Add `with_dismissed` param to announcements API
- Fix end date not being formatted when time range is given
- Fix announcement delete causing reactions to send streaming updates
- Fix announcements container growing too wide and mascot too small
- Fix `all_day` being settable when no time range is given
- Change text "Update" to "Announcement"
* Fix scheduler unpublishing announcements before they are due
* Fix filter params not being passed to announcements filter
