Commit graph

69 commits

Author SHA1 Message Date
timorl 3f54945033
Fix the one test that wasn't just being flaky 2024-04-21 19:43:26 +02:00
timorl cd7af81896
Rename StripLocation to StripMetadata for temporal-proofing reasons 2024-04-16 20:37:00 +02:00
timorl 59d32c10d9
Formatting 2024-04-16 08:02:13 +02:00
timorl b144218dce
Merge branch 'develop' into elseinspe 2024-04-14 20:31:33 +02:00
Oneric 0ec62acb9d Always insert Dedupe upload filter
This actually was already intended before to eradict all future
path-traversal-style exploits and to fix issues with some
characters like akkoma#610 in 0b2ec0ccee. However, Dedupe and
AnonymizeFilename got mixed up. The latter only anonymises the name
in Content-Disposition headers GET parameters (with link_name),
_not_ the upload path.

Even without Dedupe, the upload path is prefixed by an UUID,
so it _should_ already be hard to guess for attackers. But now
we actually can be sure no path shenanigangs occur, uploads
reliably work and save some disk space.

While this makes the final path predictable, this prediction is
not exploitable. Insertion of a back-reference to the upload
itself requires pulling off a successfull preimage attack against
SHA-256, which is deemed infeasible for the foreseeable futures.

Dedupe was already included in the default list in config.exs
since 28cfb2c37a, but this will get overridde by whatever the
config generated by the "pleroma.instance gen" task chose.

Upload+delete tests running in parallel using Dedupe might be flaky, but
this was already true before and needs its own commit to fix eventually.
2024-03-18 22:33:10 -01:00
Oneric fef773ca35 Drop media base_url default and recommend different domain
Same-domain setups enabled now at least two exploits,
so they ought to be discouraged and definitely not be the default.
2024-03-18 22:33:10 -01:00
Oneric 8cf183cb42 Drop Chat tables
Chats were removed in 0f132b802d
2024-02-11 05:15:08 +01:00
FloatingGhost c7aeeec232 fix yet another keyword equality check 2023-08-07 17:00:16 +01:00
FloatingGhost c193b4d507 Remove frankly awful config file test 2023-08-06 16:20:46 +01:00
FloatingGhost 87cc5a2110 Fix uploads test being reliant on being run later 2023-08-04 12:59:33 +01:00
FloatingGhost 64e233ca20 Tag Mock-tests as "mocked" and run them seperately 2023-08-04 12:50:50 +01:00
FloatingGhost fe8c166b8f Remove IO.inspects 2023-08-04 12:01:52 +01:00
FloatingGhost d5de05bbe4 scream 2023-08-03 18:10:27 +01:00
FloatingGhost 98cb255d12 Support elixir1.15
OTP builds to 1.15

Changelog entry

Ensure policies are fully loaded

Fix :warn

use main branch for linkify

Fix warn in tests

Migrations for phoenix 1.17

Revert "Migrations for phoenix 1.17"

This reverts commit 6a3b2f15b7.

Oban upgrade

Add default empty whitelist

mix format

limit test to amd64

OTP 26 tests for 1.15

use OTP_VERSION tag

baka

just 1.15

Massive deps update

Update locale, deps

Mix format

shell????

multiline???

?

max cases 1

use assert_recieve

don't put_env in async tests

don't async conn/fs tests

mix format

FIx some uploader issues

Fix tests
2023-08-03 17:44:09 +01:00
ilja f49e9e6d4c Clean up bookmarks after prune_objects
When doing prune_objects, it's possible that bookmarked objects are deleted.
This gave problems when fetching the bookmark TL.
Here we clean up the bookmarks during pruning in the case were it's possible that bookmarked objects are deleted.
2023-05-21 13:02:28 +02:00
ilja c7fb78cc32 Move deadline and old_insert_date to setup
Several tests for prune_objetcs need a date older than the deadline for pruning, so I moved that to the setup
2023-05-21 12:01:54 +02:00
ilja 328b4d93b7 Changelog + remove some unneeded comments from the tests 2023-02-26 14:43:19 +01:00
ilja 57eef6d764 prune_objects can prune orphaned activities who reference an array of objects
E.g. Flag activities have an array of objects

We prune the activity when NONE of the objects can be found

Note that the cost of finding and deleting these is ~4x higher than finding and deleting the non-array ones

Only string:
Delete on activities  (cost=506573.48..506580.38 rows=0 width=0)

Only Array:
Delete on activities  (cost=3570359.68..4276365.34 rows=0 width=0)

(They are still executed separately, so the total cost is the sum of the two)
2023-02-26 14:41:50 +01:00
ilja a7ec6e039c prune_objects can prune orphaned activities
We add an option to also prune remote activities who don't have existing objects any more they reference.
Rn, we only check for activities who only reference one object, not an array or embeded object.
2023-02-26 14:41:50 +01:00
ilja 7695010268 Prune Objects --keep-threads option (#350)
This adds an option to the prune_objects mix task.
The original way deleted all non-local public posts older than a certain time frame.
Here we add a different query which you can call using the option --keep-threads.

We query from the activities table all context id's where
    1. the newest activity with this context is still old
    2. none of the activities with this context is is local
    3. none of the activities with this context is bookmarked
and delete all objects with these contexts.

The idea is that posts with local activities (posts, replies, likes, repeats...) may be interesting to keep.
Besides that, a post lives in a certain context (the thread), so we keep the whole thread as well.

Caveats:
* ~~Quotes have a different context. Therefore, when someone quotes a post, it's possible the quoted post will still be deleted.~~ fixed in https://akkoma.dev/AkkomaGang/akkoma/pulls/379
* Although undocumented (in docs/docs/administration/CLI_tasks/database.md/#prune-old-remote-posts-from-the-database), the 'normal' delete action still kept old remote non-public posts. I added an option to keep this behaviour, but this also means that you now have to explicitly provide that option. **This could be considered a breaking change!**
* ~~Note that this removes from the objects table, but not from the activities.~~ See https://akkoma.dev/AkkomaGang/akkoma/pulls/427 for that.

Some statistics from explain analyse:
(cost=1402845.92..1933782.00 rows=3810907 width=62) (actual time=2562455.486..2562455.495 rows=0 loops=1)
 Planning Time: 505.327 ms
 Trigger for constraint chat_message_references_object_id_fkey: time=651939.797 calls=921740
 Trigger for constraint deliveries_object_id_fkey: time=52036.009 calls=921740
 Trigger for constraint hashtags_objects_object_id_fkey: time=20665.778 calls=921740
 Execution Time: 3287933.902 ms

***
**TODO**
1. [x] **Question:** Is it OK to keep it like this in regard to quote posts? If not (ie post quoted by local users should also be kept), should we give quotes the same context as the post they are quoting? (If we don't want to give them the same context, I'll have to see how/if I can do it without being too costly)
    * See https://akkoma.dev/AkkomaGang/akkoma/pulls/379
2. [x] **Question:** the "original" query only deletes public posts (this is undocumented, but you can check the code). This new one doesn't care for scope. From the docs I get that the idea is that posts can be refetched when needed. But I have from a trusted source that Pleroma can't refetch non-public posts. I assume that's the reason why they are kept here. I see different options to deal with this
    1. ~~We keep it as currently implemented and just don't care about scope with this option~~
    2. ~~We add logic to not delete non-public posts either (I'll have to see how costly that becomes)~~
    3. We add an extra --keep-non-public parameter. This is technically speaking breakage (you didn't have to provide a param before for this, now you do), but I'm inclined to not care much because it wasn't documented nor tested in the first place.
3. [x] See if we can do the query using Elixir
4. [x] Test on a bigger DB to see that we don't run into a timeout
5. [x] Add docs

Co-authored-by: ilja <git@ilja.space>
Reviewed-on: https://akkoma.dev/AkkomaGang/akkoma/pulls/350
Co-authored-by: ilja <akkoma.dev@ilja.space>
Co-committed-by: ilja <akkoma.dev@ilja.space>
2023-01-09 22:15:41 +00:00
FloatingGhost f752126427 Remove quack, ensure adapter is finch 2022-12-11 23:22:35 +00:00
Ilja 338612d72b Use EXIF data of image to prefill image description
During attachment upload Pleroma returns a "description" field.

* This MR allows Pleroma to read the EXIF data during upload and return the description to the FE using this field.
    * If a description is already present (e.g. because a previous module added it), it will use that
    * Otherwise it will read from the EXIF data. First it will check -ImageDescription, if that's empty, it will check -iptc:Caption-Abstract
    * If no description is found, it will simply return nil, which is the default value
* When people set up a new instance, they will be asked if they want to read metadata and this module will be activated if so

There was an Exiftool module, which has now been renamed to Exiftool.StripLocation
2022-10-23 14:46:16 +02:00
floatingghost 8e4de118c1 Don't persist local undone follow (#194)
same deal but backwards this time

Co-authored-by: FloatingGhost <hannah@coffee-and-dreams.uk>
Reviewed-on: https://akkoma.dev/AkkomaGang/akkoma/pulls/194
2022-08-31 18:00:36 +00:00
FloatingGhost 0f132b802d purge chat and shout endpoints 2022-07-21 11:29:28 +01:00
Tusooa Zhu 4ed0a780b7 Fix digest test 2022-06-29 20:45:05 +01:00
Alex Gleason 54af527759
Upgrade Ecto to v3.6.2, remove deprecated ecto_explain 2021-06-23 13:22:32 -05:00
Mark Felder fe40f6f291 Switch from the deprecated "use Mix.config" to "import Config" 2021-05-20 14:02:58 -05:00
Alexander Strizhakov 03843a5386
migrating config to tmp folder 2021-03-23 14:29:02 +03:00
feld 2217b35bd4 Merge branch 'ecto-rollback-in-test-env' into 'develop'
Possibility to run rollback in test env

See merge request pleroma/pleroma!3212
2021-02-26 16:47:53 +00:00
Mark Felder e854c35e65 Convert tests to all use clear_config instead of Pleroma.Config.put 2021-01-26 11:58:43 -06:00
Mark Felder f868dcf3ac Ensure consistent behavior between pleroma.user activate/deactivate mix tasks 2021-01-25 10:48:28 -06:00
Mark Felder 5e8da27e14 Provide pleroma.user mix task for both activate and deactivate 2021-01-25 10:36:51 -06:00
Mark Felder 28581e03ad Merge branch 'develop' into refactor/deactivated_user_field 2021-01-18 14:58:21 -06:00
Mark Felder 2c0fe2ea9e Remove toggle_confirmation; require explicit state change
Also cosmetic changes to make the code clearer
2021-01-15 13:11:51 -06:00
Mark Felder d36182c088 Change user.confirmation_pending field to user.is_confirmed 2021-01-15 12:44:41 -06:00
Mark Felder 7516660753 Remove toggle_activation 2021-01-15 11:25:32 -06:00
Mark Felder 860b5c7804 Change user.deactivated field to user.is_active 2021-01-15 11:24:46 -06:00
Haelwenn (lanodan) Monnier c4439c630f
Bump Copyright to 2021
grep -rl '# Copyright © .* Pleroma' * | xargs sed -i 's;Copyright © .* Pleroma .*;Copyright © 2017-2021 Pleroma Authors <https://pleroma.social/>;'
2021-01-13 07:49:50 +01:00
lain e1e7e4d379 Object: Rework how Object.normalize works
Now it defaults to not fetching, and the option is named.
2021-01-04 13:38:31 +01:00
lain e4f1d8f48c Merge branch 'cachex-test' into 'develop'
Test framework overhaul (speed, reliability)

See merge request pleroma/pleroma!3209
2020-12-26 10:26:35 +00:00
Alexander Strizhakov 538af14d52
possibility to run rollback in test env 2020-12-22 17:55:30 +03:00
lain b17c36c45a Tests: More test fixes. 2020-12-21 12:59:11 +01:00
lain 9ba60f70d2 Tests: Make as many tests as possible async.
In general, tests that match these criteria can be made async:

- Doesn't use real Cachex.
- Doesn't write to the Config / Application Environment.
- Uses Mock. Using Mox is fine.
- Uses the streamer.
2020-12-21 12:21:40 +01:00
lain b4b68b71fc Tests: Remove unneeded explicit cachex use.
Only use cachex when we're actually testing it.
2020-12-18 13:18:17 +01:00
Alex Gleason 80891e83d8
Merge remote-tracking branch 'upstream/develop' into registration-workflow 2020-12-17 09:05:36 -06:00
Alexander Strizhakov cebe3c7def Fix for dropping posts/notifs in WS when mix task is executed
- start oban in mix tasks with empty queues, plugins and crontab
- fix for update_users_following_followers_counts
- fix for removed logo.png
- typo in resend confirmation emails mix task docs
- fix for uploads mix task (start Majic.Pool)
- fix for creating user mix task (start :fast_html app)
2020-12-14 11:02:32 -06:00
lain 1436a2fa2d Merge branch 'stream-follow-updates' into 'develop'
Stream follow updates

Closes #2299

See merge request pleroma/pleroma!3183
2020-12-09 15:52:31 +00:00
lain ed76323776 Merge branch 'feature/configdb-mix-tasks' into 'develop'
Add mix tasks to give additional ConfigDB recovery and debugging options

See merge request pleroma/pleroma!3174
2020-12-07 18:06:06 +00:00
Mark Felder e3dd0d45b7 Slip in a test to ensure we can use the atom syntax in mix task arguments 2020-12-07 11:21:06 -06:00
Ivan Tashkinov e00c667145 [#3174] Refactoring: ConfigDB fetching functions, ConfigDB tests.
Minor fixes.
2020-12-06 18:02:30 +03:00