Compact Language Detector v3 (CLD3) is the successor of CLD2, which was
used in the previous implementation. CLD3 includes improvements since CLD2,
and supports newer compilers. On the other hand, it has additional
requirements and cld3-ruby, the FFI of CLD3 for Ruby, is still new and may
be still inmature.
Though CLD3 is named after CLD2, it is implemented with a neural network
model, different from the old implementation, which is based on a Naïve
Bayesian classifier.
CLD3 supports newer compilers, such as GCC 6. CLD2 is not compatible with
GCC 6 because it assigns negative values to varibales typed unsigned.
(see internal/cld_generated_cjk_uni_prop_80.cc) The support for GCC 6 and
newer compilers are essential today, when some server operating system
such as Ubuntu Server 16.10 has GCC 6 by default.
On the one hand, CLD3 requires C++11 support. Environments with old
compilers such as Ubuntu Server 14.04 needs to update the system or install
a newer compiler.
CLD3 needs protocol buffers as a new dependency. However,it is not
considered problematic because major server operating systems, CentOS and
Ubuntu Server provide them.
The FFI cld3-ruby was written by me (Akihiko Odaki) for use in Mastodon.
It is still new and may be inmature, but confirmed to pass existing tests.
* Dont use raise_error by itself (avoids warning)
* Add coverage for AccountFilter
* Improve coverage and refactor for Subscription#lease_seconds
* Improve coverage and refactor for NotificationMailer
* Simplify assignment of min/max threshold on subscription
* Fix#2473 - Use sidekiq scheduler to refresh PuSH subscriptions instead of cron
Fix an issue where / in domain would raise exception in TagManager#normalize_domain
PuSH subscriptions refresh done in a round-robin way to avoid hammering a single
server's hub in sequence. Correct handling of failures/retries through Sidekiq (see
also #2613). Optimize Account#with_followers scope. Also, since subscriptions
are now delegated to Sidekiq jobs, an uncaught exception will not stop the entire
refreshing operation halfway through
Fix#2702 - Correct user agent header on outgoing http requests
* Add test for SubscribeService
* Extract #expiring_accounts into method
* Make mastodon:push:refresh no-op
* Queues are now defined in sidekiq.yml
* Queues are now in sidekiq.yml
* Likely fix#2458, fix#2031 - handle out-of-order deletes for statuses
If a delete arrives before the original status, cache that information
for 6h, and if the original status arrives in that window, ignore it
* Add test case
Fix#2196 - Respond with 201 when Salmon accepted, 400 when unverified
Fix#2629 - Correctly handle confirm_domain? for local accounts
Unify rules for extracting author acct from XML, prefer <email>, fall back
to <name> + <uri> (see also #2017, #2172)
* Replace browserify with webpack
* Add react-intl-translations-manager
* Do not minify in development, add offline-plugin for ServiceWorker background cache updates
* Adjust tests and dependencies
* Fix production deployments
* Fix tests
* More optimizations
* Improve travis cache for npm stuff
* Re-run travis
* Add back support for custom.scss as before
* Remove offline-plugin and babili
* Fix issue with Immutable.List().unshift(...values) not working as expected
* Make travis load schema instead of running all migrations in sequence
* Fix missing React import in WarningContainer. Optimize rendering performance by using ImmutablePureComponent instead of
React.PureComponent. ImmutablePureComponent uses Immutable.is() to compare props. Replace dynamic callback bindings in
<UI />
* Add react definitions to places that use JSX
* Add Procfile.dev for running rails, webpack and streaming API at the same time
* Add coverage for create with empty acct value
* Add coverage for create with webfinger failure
* Add coverage for create with webfinger providing bad values
* Add coverage for create when webfinger is good
* Add coverage for session[:remote_follow] having data
* Simplify how remote follow pulls acct from session
* Remote follow behaves more like model
* Move the discovery portions of remote follow out of controller
* Check for suspended accounts
* Naive approached to timeline filtering
* Convert allowed_languages into a db column
* Allow users to choose languages to see statuses in
* Style list items as two columns
* Add a hint to explain language filtering preference
* Add coverage for admin/confirmations controller
* Coverage for statuses controller show action
* Add coverage for admin/domain_blocks controller
* Add coverage for settings/profiles#update
This adds a test for the `Feed#get` method. While the data
transformations in `Feed#get` may seem redundant, they are important to
maintain the order from Redis. The tests I wrote will fail if someone
tries to refactor away this "redundancy" (as I tried to do in the first
iteration of this change).
In #2110, a new attachment type "unknown" was introduced for
attachments that were rejected due to a domain being blocked using
reject_media. However, the "type" field was never set to "unknown"
because a default value of "0" (image) is set for that column,
causing the `type.blank?` expression to always equal false.
This version uses type_changed? instead, causing the type to be set
to "unknown" unless a type has been explicitly set. This introduces
a small change in behaviour causing the type to be set to unknown
before paperclip calls `before_post_process`. Presumably this
behaviour is more appropriate than the current one because the
attachment type has not been determined by that point.
Included are new tests for `ProcessFeedService` and
`UpdateRemoteProfileService` which now check that remote media is
downloaded for non-blocked domains and is rejected for others.
* Add render_views in more places
* Delegate methods from account to user with allow nil true, so that admin accounts show view renders when missing a user
* Use actual account instances in authorize follow controller spec
* Add rough outline of coverage needed for public timeline
* Specs for visibility, replies, boosts
* Specs for silenced account
* Specs for local_only option
* Specs for blocks and mutes
* Add tentative spec around including other silenced account statuses
* Add with_public_visibility scope
* Add simple coverage for tag_timeline
* Tag timeline includes replies
* Replace tag.statuses with a tagged_with scope in tag timeline method
* Use with_public_visibility in tag timeline
* Extract common scope between public and tag timelines to method
* Extract local domain check to local_only scope
* Extract local_only check to starting scope method
* Move list of excluded from timeline account ids to account model
* Simplify excluded accounts list on account model
* Only join accounts when needed
* Rename method for account specific filtering
* Extract method for account exclusions
* Fix bug where silenced accounts were not including statuses from other silenced accounts
* DRY up filter application from account or no account
* timeline_scope can be private
* Add spec showing that account can find its excluded accounts ids
* Add spec which fails if local_only does not have a left outer join
* rubocop
* Instead of parsing shared status contents verbatim, make roundtrip
to purported original URL. Confirm that the "original" URL is from the
same domain as the author it claims to be from.
* Fix obvious typo, add comment
* Use URI look-up first
* Add test, update Goldfinger dependency to make less useless HTTP requests per Webfinger lookup
* OEmbed support for PreviewCard
* Improve ProviderDiscovery code failure treatment
* Do not crawl links if there is a content warning, since those
don't display a link card anyway
* Reset db schema
* Fresh migrate
* Fix rubocop style issues
Fix#1681 - return existing access token when applicable instead of creating new
* Fix test
* Extract http client to helper
* Improve oembed controller
* Add coverage for domain block service with silence
* Get rid of warning about find_each and order
* Move domain_block to attr_reader
* Move optional clear_media into silence_accounts method
* Use blocked_domain method to reduce passed vars
* Extract blocked_domain_accounts method to find accounts on the domain
* Extract media_from_blocked_domain method to find relevant attachments
* Separate destruction of account images and account attachments
* Fixes#1985
- add migration AddMediaAttachmentMeta, which add meta field to media_attachments
- before saving attachment, set file meta if needed
- add meta in api
* add spec
* align the “size” format for image and video
* fix code climate
* fixes media_attachment_spec.rb
* Stricter whitelist rules
* Linting
* Added spec for blacklisting
* Test subdomain blacklist on domain whitelist
* No need to split
* Change spec name
This PR fixes I18n.locale for rake middlewares. Mastodon uses Devise that depends on Warden.
Warden::Manager can be found in rake middleware. It is outside of the controller.
In the case of authentication failed, warden calls throw(:warden). At the time Warden::Manager
delegates request to failure_app to generate response and flash[:alert] after catching it.
Unfortunately, I18n.locale is already reset then because I18n.with_locale is enabled only
inside the controller. If we used I18n.locale=, Devise::FailureApp could get the current locale.
* Change ActivityPub paging to match spec. Clean up ActivityPub outbox changes.
* Fix code style and test failures for OutboxController.
* Attempt to fix CI errors.
duplicates. Web UI regenerates UUID for that header every time the compose
form is changed or successfully submitted
Also, fix Farsi i18n overwriting the English one
* Fix#2119 - Whenever about to send a HTTP request, normalize the URI
* Add test for IDN request in FetchLinkCardService
* Perform IDN normalization on domains before they are stored in the DB
* Add failing spec for oauth/authorized_applications controller
* Use explicit reference to flashes partial from admin layout
Because some of the controllers which use the admin layout do not inherit from
application controller, this partial is not in their view path.
* Make private toots get PuSHed to subscription URLs that belong to domains where you have approved followers
* Authorized followers controller, stub for bulk action
* Soft block in the background
* Add simple test for new controller
* Rename Settings::FollowersController to Settings::FollowerDomainsController, paginate results,
rename "private" post setting to "followers-only", fix pagination style, improve post privacy
preferences style, improve warning style
* Extract compose form warnings into own container, show warning when posting to followers-only with unlocked account
* Add failing spec showing that human_locales does not match what i18n knows about
* Add missing `ar` key for arabic to human locales
* Remove duplicate `id` key from available locales
* Sort keys in human locales list
* Add spec for human_locale helper
The two methods `StreamEntriesHelper#stream_link_target` and
`StreamEntriesHelper#acct` are based on checking whether we are running
in an embedded view.
This adds some test helper code to make the testing easier. We extracted
some "magic strings" to constants to lower the coupling in the specs.
The code that generates CSS is based on a lot of boolean conditions.
The possible combinations of these grows exponentially as we add more
conditions.
Since most of the code is conditional on a single boolean, we tested the
following:
1. All `false`
2. All `true`
3. Each individual flag set to `true`
The methods tested are:
* `StreamEntriesHelper#style_classes`
* `StreamEntriesHelper#microformats_classes`
* `StreamEntriesHelper#microformats_h_class`
* Clean up collapsible components
* Expose user Outboxes and AS2 representations of statuses
* Save work thus far.
* Fix bad merge.
* Save my work
* Clean up pagination.
* First test working.
* Add tests.
* Add Forbidden error template.
* Revert yarn.lock changes.
* Fix code style deviations and use localized instead of hardcoded English text.
* Shows confirmed status in list.
* Adds ability to confirm users in admin UI.
* Added new english translations.
* Addresses feedback from #2245.
* More feedback.
* Add spec coverage for settings/two_factor_auth area
* extract setup method for qr code
* Move otp required check to before action
* Merge method only used once
* Remove duplicate view
* Consolidate creation of @codes for backup
* Move settings/2fq#recovery_codes to settings/recovery_codes#create
* Rename settings/two_factor_auth#disable to #destroy
* Add coverage for the otp required path on 2fa#show
* Clean up the recovery codes list styles
* Move settings/two_factor_auth to settings/two_factor_authentication
* Reorganize the settings two factor auth area
Updated to use a flow like:
- settings/two_factor_authentication goes to a #show view which has a button
either enable or disable 2fa on the account
- the disable button turns off the otp requirement for the user
- the enable button cycles the user secret and redirects to a confirmation page
- the confirmation page is a #new view which shows the QR code for user
- that page posts to #create which verifies the code, and creates the recovery
codes
- that create action shares a view with a recovery codes controller which can be
used separately to reset codes if needed
* Add render_views for admin/settings spec
* Add coverage for admin/settings#update
* Add coverage for admin/settings typecasting open_registrations setting
* Simplify how admin/settings finds the value for updating
* Rely on activerecord to not update a value that hasnt changed
* Add coverage for non-existent setting
* Use a constant for boolean settings
* Fix#2108 - Fix gif uploads
Add specs for media attachment gifv conversion
* Add ffmpeg to travis
* Make travis install ffmpeg, not libav
* Switch travis to trusty
* Add routing specs for accounts followers and following actions
* Use more restful route naming for public account follow pages
Moves two actions:
- accounts#followers to accounts/follower_accounts#index
- accounts#following to accounts/following_accounts#index
Adds routing spec to ensure prior URLs are preserved.
* Extract detect_language to separate class
* Use default locale, not just en
* Add spec to confirm that whatlanguage cant identify empty string
* Allow account locale to override default in language detector
* PostStatusService supplies an account to detect language
* Add a ReportFilter class
* Add reports and targeted_reports relationships to Account
* Use ReportFilter from admin/reports controller
* Link to admin/reports filtered views from admin account show view
* Add indexes to reports.account_id and reports.target_account_id
* Remove unused method #set_counters_maps from api controller
* Remove unused method #set_account_counters_maps from api controller
* Remove unused method Account#followers_domains
* Remove unused User.prolific scope
* Add mastodon:users:admins task to list all admin emails
* Use interpolated query style in Account.triadic_closures
* Coverage for Account.triadic_closures
* Move site title helper to instance helper (name change only)
* Remove newline in <title> tag
* Add site_hostname helper method to wrap up local_domain value
* Use site_hostname helper in places that need local_domain value
These were being skipped with `xit`, but we also just added the `unused` task to
the CI server, and it will be caught there. That seems good enough for this,
instead of requiring a full keys check on every local spec suite run.
This change includes:
- Improve the spec coverage for incoming request to the webfinger action
- For requests without an accept header (ie, what a browser might look like),
return a JSON response.
- For requests with an explicit format of xml or json, return that format.
- For requests using an accept header, return that format.
Also adds failing spec showing webfinger does not return xml, which covers the
issue described in: https://github.com/tootsuite/mastodon/issues/1983
Create MediaAttachment but without actual file download when domain is blocked with reject_media set to true
Clean up old media files when creating a new domain block with reject_media set to true
Return remote_url in media attachments API if local file is not present
Undo domain block action in admin UI
Ability to enable reject_media from admin UI
