Commit graph

1293 commits

Author SHA1 Message Date
ilja 7695010268 Prune Objects --keep-threads option (#350)
This adds an option to the prune_objects mix task.
The original way deleted all non-local public posts older than a certain time frame.
Here we add a different query which you can call using the option --keep-threads.

We query from the activities table all context id's where
    1. the newest activity with this context is still old
    2. none of the activities with this context is is local
    3. none of the activities with this context is bookmarked
and delete all objects with these contexts.

The idea is that posts with local activities (posts, replies, likes, repeats...) may be interesting to keep.
Besides that, a post lives in a certain context (the thread), so we keep the whole thread as well.

Caveats:
* ~~Quotes have a different context. Therefore, when someone quotes a post, it's possible the quoted post will still be deleted.~~ fixed in https://akkoma.dev/AkkomaGang/akkoma/pulls/379
* Although undocumented (in docs/docs/administration/CLI_tasks/database.md/#prune-old-remote-posts-from-the-database), the 'normal' delete action still kept old remote non-public posts. I added an option to keep this behaviour, but this also means that you now have to explicitly provide that option. **This could be considered a breaking change!**
* ~~Note that this removes from the objects table, but not from the activities.~~ See https://akkoma.dev/AkkomaGang/akkoma/pulls/427 for that.

Some statistics from explain analyse:
(cost=1402845.92..1933782.00 rows=3810907 width=62) (actual time=2562455.486..2562455.495 rows=0 loops=1)
 Planning Time: 505.327 ms
 Trigger for constraint chat_message_references_object_id_fkey: time=651939.797 calls=921740
 Trigger for constraint deliveries_object_id_fkey: time=52036.009 calls=921740
 Trigger for constraint hashtags_objects_object_id_fkey: time=20665.778 calls=921740
 Execution Time: 3287933.902 ms

***
**TODO**
1. [x] **Question:** Is it OK to keep it like this in regard to quote posts? If not (ie post quoted by local users should also be kept), should we give quotes the same context as the post they are quoting? (If we don't want to give them the same context, I'll have to see how/if I can do it without being too costly)
    * See https://akkoma.dev/AkkomaGang/akkoma/pulls/379
2. [x] **Question:** the "original" query only deletes public posts (this is undocumented, but you can check the code). This new one doesn't care for scope. From the docs I get that the idea is that posts can be refetched when needed. But I have from a trusted source that Pleroma can't refetch non-public posts. I assume that's the reason why they are kept here. I see different options to deal with this
    1. ~~We keep it as currently implemented and just don't care about scope with this option~~
    2. ~~We add logic to not delete non-public posts either (I'll have to see how costly that becomes)~~
    3. We add an extra --keep-non-public parameter. This is technically speaking breakage (you didn't have to provide a param before for this, now you do), but I'm inclined to not care much because it wasn't documented nor tested in the first place.
3. [x] See if we can do the query using Elixir
4. [x] Test on a bigger DB to see that we don't run into a timeout
5. [x] Add docs

Co-authored-by: ilja <git@ilja.space>
Reviewed-on: https://akkoma.dev/AkkomaGang/akkoma/pulls/350
Co-authored-by: ilja <akkoma.dev@ilja.space>
Co-committed-by: ilja <akkoma.dev@ilja.space>
2023-01-09 22:15:41 +00:00
FloatingGhost 336d06b2a8 Significantly tighten HTTP CSP 2023-01-02 15:21:19 +00:00
FloatingGhost c4b46ca460 Add /api/v1/followed_tags 2022-12-31 18:09:34 +00:00
ilja 745e15468e Use same context for quote posts as the post that's being quoted (#379)
See https://akkoma.dev/AkkomaGang/akkoma/pulls/350#issuecomment-6109

When making quotes through Mast-API, they will now have the same context as the quoted post. This also results in them being showed when fetching the thread. I checked Misskey to see how it's there, and they show the quotes there as well, see e.g. <https://mk.toast.cafe/notes/98u1g0tulg>.

An example from Akkoma:

Co-authored-by: ilja <git@ilja.space>
Reviewed-on: https://akkoma.dev/AkkomaGang/akkoma/pulls/379
Reviewed-by: floatingghost <hannah@coffee-and-dreams.uk>
Co-authored-by: ilja <akkoma.dev@ilja.space>
Co-committed-by: ilja <akkoma.dev@ilja.space>
2022-12-31 18:09:27 +00:00
FloatingGhost c8f2c4b638 add changelog entry for timeouts 2022-12-31 03:52:52 +00:00
floatingghost 9be6caf125 argon2 password hashing (#406)
Co-authored-by: FloatingGhost <hannah@coffee-and-dreams.uk>
Reviewed-on: https://akkoma.dev/AkkomaGang/akkoma/pulls/406
2022-12-30 02:46:58 +00:00
floatingghost a5e98083f2 Add link verification in profile fields (#405)
Co-authored-by: FloatingGhost <hannah@coffee-and-dreams.uk>
Reviewed-on: https://akkoma.dev/AkkomaGang/akkoma/pulls/405
2022-12-29 20:56:06 +00:00
FloatingGhost af7c3fab98 Do not crash on invalid atom in configDB 2022-12-21 00:16:39 +00:00
Atsuko Karagi 4a78c431cf Simplified HTTP signature processing 2022-12-19 20:41:48 +00:00
Atsuko Karagi e17c71a389 Respect restrict_unauthenticated in /api/v1/accounts/lookup 2022-12-19 20:32:16 +00:00
ilja c092fc9fd6 Add translation module for Argos Translate (#351)
Argos Translate is a Python module for translation and can be used as a command line tool.

This is also the engine for LibreTranslate, for which we already have a module.
Here we can use the engine directly from our server without doing requests to a third party or having to install our own LibreTranslate webservice (obviously you do have to install Argos Translate).

One thing that's currently still missing from Argos Translate is auto-detection of languages (see <https://github.com/argosopentech/argos-translate/issues/9>). For now, when no source language is provided, we just return the text unchanged, supposedly translated from the target language. That way you get a near immediate response in pleroma-fe when clicking Translate, after which you can select the source language from a dropdown.

Argos Translate also doesn't seem to handle html very well. Therefore we give admins the option to strip the html before translating. I made this an option because I'm unsure if/how this will change in the future.

Co-authored-by: ilja <git@ilja.space>
Reviewed-on: https://akkoma.dev/AkkomaGang/akkoma/pulls/351
Co-authored-by: ilja <akkoma.dev@ilja.space>
Co-committed-by: ilja <akkoma.dev@ilja.space>
2022-12-19 13:06:39 +00:00
FloatingGhost 52d8183787 drop admin scopes on create app instead of rejecting 2022-12-17 23:14:49 +00:00
FloatingGhost dcac8adb3d Add option to modify HTTP pool size 2022-12-16 18:33:00 +00:00
FloatingGhost 126f1ca69c increase rich media backoff time 2022-12-16 17:31:04 +00:00
FloatingGhost 372eea4e7c add changelog entry for custom emoji 2022-12-16 13:20:48 +00:00
FloatingGhost 426f4271c2 add changelog entry 2022-12-16 11:57:19 +00:00
floatingghost 07a48b9293 giant massive dep upgrade and dialyxir-found error emporium (#371)
Co-authored-by: FloatingGhost <hannah@coffee-and-dreams.uk>
Reviewed-on: https://akkoma.dev/AkkomaGang/akkoma/pulls/371
2022-12-14 12:38:48 +00:00
Norm 8104f46031
Update 'CHANGELOG.md' 2022-12-12 17:28:56 -05:00
FloatingGhost 9d9c26b833 Ensure Gun is Gone 2022-12-11 19:26:21 +00:00
FloatingGhost 1f0ef94271 Bump versions 2022-12-10 14:50:02 +00:00
FloatingGhost 68894089e8 Do not fetch anything from blocked instances 2022-12-10 00:09:45 +00:00
FloatingGhost 739ed14f54 Revert "mandate published on notes"
This reverts commit e49b583147.
2022-12-09 20:59:26 +00:00
FloatingGhost e49b583147 mandate published on notes
fixes #356
2022-12-09 20:27:54 +00:00
FloatingGhost f5a315f04c Add URL and code to :not_found errors
Ref #355
2022-12-09 20:13:31 +00:00
FloatingGhost dcf58a3c53 Do not pass transient undo-y activities through MRF 2022-12-09 20:01:38 +00:00
FloatingGhost 9db4c2429f Remove FollowBotPolicy 2022-12-09 19:59:27 +00:00
FloatingGhost 6f83ae27aa extend reject MRF to check if originating instance is blocked 2022-12-09 19:57:29 +00:00
sfr 7c4b415929 static-fe overhaul (#236)
makes static-fe look more like pleroma-fe, with the stylesheets matching pleroma-dark and pleroma-light based on `prefers-color-scheme`.

- [x] navbar
- [x] about sidebar
- [x] background image
- [x] statuses
  - [x] "reply to" or "edited" tags
- [x] accounts
  - [x] show more / show less
  - [x] posts / with replies / media / followers / following
    - [x] followers/following would require user card snippets
  - [x] admin/bot indicators
- [x] attachments
  - [x] nsfw attachments
- [x] fontawesome icons
- [x] clean up and sort css
- [x] add pleroma-light
- [x] replace hardcoded strings

also i forgot
- [x] repeated headers

how it looks + sneak peek at statuses:
![](https://akkoma.dev/attachments/c0d3a025-6987-4630-8eb9-5f4db6858359)

Co-authored-by: Sol Fisher Romanoff <sol@solfisher.com>
Reviewed-on: https://akkoma.dev/AkkomaGang/akkoma/pulls/236
Co-authored-by: sfr <sol@solfisher.com>
Co-committed-by: sfr <sol@solfisher.com>
2022-12-07 11:20:53 +00:00
floatingghost 4c3971aebd Add changelog entry for hashtag following 2022-12-04 18:35:04 +00:00
floatingghost db60640c5b Fixing up deletes a bit (#327)
Co-authored-by: FloatingGhost <hannah@coffee-and-dreams.uk>
Reviewed-on: https://akkoma.dev/AkkomaGang/akkoma/pulls/327
2022-12-01 15:00:53 +00:00
floatingghost 0cfd5b4e89 Add ability to set a default post expiry (#321)
Co-authored-by: FloatingGhost <hannah@coffee-and-dreams.uk>
Reviewed-on: https://akkoma.dev/AkkomaGang/akkoma/pulls/321
2022-11-28 13:34:54 +00:00
FloatingGhost ee7059c9cf Spin off imports into n oban jobs 2022-11-27 21:45:41 +00:00
floatingghost 98a21debf9 normalise markup by default (#316)
why was this _not_ default?

honestly i'm surprised pleroma hasn't exploded yet

Co-authored-by: FloatingGhost <hannah@coffee-and-dreams.uk>
Reviewed-on: https://akkoma.dev/AkkomaGang/akkoma/pulls/316
2022-11-26 21:06:20 +00:00
floatingghost e3085c495c fix tests broken by relay defaults changing (#314)
Co-authored-by: FloatingGhost <hannah@coffee-and-dreams.uk>
Reviewed-on: https://akkoma.dev/AkkomaGang/akkoma/pulls/314
2022-11-26 20:45:47 +00:00
floatingghost 2fe1484ed3 http timeout config (#307)
Ref https://meta.akkoma.dev/t/increase-timeout-on-libretranslate-request-how/156/2

Co-authored-by: FloatingGhost <hannah@coffee-and-dreams.uk>
Reviewed-on: https://akkoma.dev/AkkomaGang/akkoma/pulls/307
2022-11-24 12:27:16 +00:00
ave 1c4ca20ff7 Change follow_operation schema to use type BooleanLike (#301)
Changes follow_operation schema to use BooleanLike instead of :boolean so that strings like "0" and "1" (used by mastodon.py) can be accepted. Rest of file uses the same. For more info please see https://git.pleroma.social/pleroma/pleroma/-/issues/2999

(I'm also sending this here as I'm not hopeful about upstream not ignoring  it)

Co-authored-by: ave <ave@ave.zone>
Reviewed-on: https://akkoma.dev/AkkomaGang/akkoma/pulls/301
Co-authored-by: ave <ave@noreply.akkoma>
Co-committed-by: ave <ave@noreply.akkoma>
2022-11-24 11:27:01 +00:00
FloatingGhost 89dbc7177b Chores for 2022.11 2022-11-11 16:12:04 +00:00
FloatingGhost c5b6cb746f add requested_by changelog entry 2022-11-10 03:17:00 +00:00
FloatingGhost 7bbaa8f8e0 automatically trim loading *. prefixes on domain blocks 2022-11-07 22:33:18 +00:00
floatingghost 31ad09010e Fix regex usage in MRF (#254)
fixes #235
fixes #228

Co-authored-by: FloatingGhost <hannah@coffee-and-dreams.uk>
Reviewed-on: https://akkoma.dev/AkkomaGang/akkoma/pulls/254
2022-11-06 23:50:32 +00:00
FloatingGhost bbedbaaf5c Changelog 2022-11-06 22:50:11 +00:00
FloatingGhost 1bb8b76311 Fix tests in ldap registration 2022-11-01 14:21:35 +00:00
floatingghost f36d14818d Unilateral remove from followers (#232)
from https://git.pleroma.social/pleroma/pleroma/-/merge_requests/3647/

Co-authored-by: marcin mikołajczak <git@mkljczk.pl>
Co-authored-by: Tusooa Zhu <tusooa@kazv.moe>
Co-authored-by: FloatingGhost <hannah@coffee-and-dreams.uk>
Reviewed-on: https://akkoma.dev/AkkomaGang/akkoma/pulls/232
2022-10-19 10:01:14 +00:00
floatingghost 66f913355a Docker builds (#231)
Co-authored-by: FloatingGhost <hannah@coffee-and-dreams.uk>
Reviewed-on: https://akkoma.dev/AkkomaGang/akkoma/pulls/231
2022-10-16 19:25:54 +00:00
FloatingGhost 60b3c8d17b bump version 2022-10-14 12:49:35 +01:00
FloatingGhost 856c57208b Ensure deletes are handled after everything else 2022-10-11 14:30:08 +01:00
FloatingGhost cb9b0d3720 optimise notifications query 2022-10-11 11:40:43 +01:00
floatingghost c6e63aaf6b Backend settings sync (#226)
Co-authored-by: FloatingGhost <hannah@coffee-and-dreams.uk>
Reviewed-on: https://akkoma.dev/AkkomaGang/akkoma/pulls/226
2022-10-06 16:22:15 +00:00
floatingghost b2aa82cee5 Fix false error in meilisearch index (#221)
the schema changed

https://docs.meilisearch.com/reference/api/documents.html#add-or-update-documents

this wasn't breaking anything, it would just report errors that were actually successes

Co-authored-by: FloatingGhost <hannah@coffee-and-dreams.uk>
Reviewed-on: https://akkoma.dev/AkkomaGang/akkoma/pulls/221
2022-09-20 10:36:21 +00:00
floatingghost 9b2c169cef Merge pull request 'Move remote user interaction changelog entry to correct version' (#219) from norm/akkoma:changelog-remote-user-interaction into develop
Reviewed-on: https://akkoma.dev/AkkomaGang/akkoma/pulls/219
2022-09-19 17:33:32 +00:00