Commit graph

60 commits

Author SHA1 Message Date
0ec62acb9d Always insert Dedupe upload filter
This actually was already intended before to eradict all future
path-traversal-style exploits and to fix issues with some
characters like akkoma#610 in 0b2ec0ccee. However, Dedupe and
AnonymizeFilename got mixed up. The latter only anonymises the name
in Content-Disposition headers GET parameters (with link_name),
_not_ the upload path.

Even without Dedupe, the upload path is prefixed by an UUID,
so it _should_ already be hard to guess for attackers. But now
we actually can be sure no path shenanigangs occur, uploads
reliably work and save some disk space.

While this makes the final path predictable, this prediction is
not exploitable. Insertion of a back-reference to the upload
itself requires pulling off a successfull preimage attack against
SHA-256, which is deemed infeasible for the foreseeable futures.

Dedupe was already included in the default list in config.exs
since 28cfb2c37a, but this will get overridde by whatever the
config generated by the "pleroma.instance gen" task chose.

Upload+delete tests running in parallel using Dedupe might be flaky, but
this was already true before and needs its own commit to fix eventually.
2024-03-18 22:33:10 -01:00
fef773ca35 Drop media base_url default and recommend different domain
Same-domain setups enabled now at least two exploits,
so they ought to be discouraged and definitely not be the default.
2024-03-18 22:33:10 -01:00
8cf183cb42 Drop Chat tables
All checks were successful
ci/woodpecker/pr/lint Pipeline was successful
ci/woodpecker/pr/test Pipeline was successful
ci/woodpecker/pr/build-amd64 Pipeline was successful
ci/woodpecker/pr/build-arm64 Pipeline was successful
ci/woodpecker/pr/docs Pipeline was successful
Chats were removed in 0f132b802d
2024-02-11 05:15:08 +01:00
c7aeeec232 fix yet another keyword equality check
Some checks are pending
ci/woodpecker/push/build-arm64 Pipeline is pending
ci/woodpecker/push/docs Pipeline is pending
ci/woodpecker/push/test Pipeline is pending
ci/woodpecker/pr/test Pipeline is pending
ci/woodpecker/push/build-amd64 Pipeline is pending
ci/woodpecker/pr/build-amd64 Pipeline is pending
ci/woodpecker/pr/build-arm64 Pipeline is pending
ci/woodpecker/pr/docs Pipeline is pending
2023-08-07 17:00:16 +01:00
c193b4d507 Remove frankly awful config file test 2023-08-06 16:20:46 +01:00
87cc5a2110 Fix uploads test being reliant on being run later
Some checks are pending
ci/woodpecker/push/build-amd64 Pipeline is pending
ci/woodpecker/push/build-arm64 Pipeline is pending
ci/woodpecker/push/docs Pipeline is pending
ci/woodpecker/push/test Pipeline is pending
ci/woodpecker/pr/test Pipeline was successful
ci/woodpecker/pr/build-amd64 Pipeline was successful
ci/woodpecker/pr/build-arm64 Pipeline was successful
ci/woodpecker/pr/docs Pipeline was successful
2023-08-04 12:59:33 +01:00
64e233ca20 Tag Mock-tests as "mocked" and run them seperately 2023-08-04 12:50:50 +01:00
fe8c166b8f Remove IO.inspects
Some checks failed
ci/woodpecker/push/test Pipeline was successful
ci/woodpecker/push/build-amd64 Pipeline was successful
ci/woodpecker/push/build-arm64 Pipeline was successful
ci/woodpecker/push/docs Pipeline was successful
ci/woodpecker/pr/test Pipeline failed
ci/woodpecker/pr/build-amd64 unknown status
ci/woodpecker/pr/build-arm64 unknown status
ci/woodpecker/pr/docs unknown status
2023-08-04 12:01:52 +01:00
d5de05bbe4 scream
Some checks failed
ci/woodpecker/push/build-amd64 Pipeline is pending
ci/woodpecker/push/build-arm64 Pipeline is pending
ci/woodpecker/push/docs Pipeline is pending
ci/woodpecker/push/test Pipeline is pending
ci/woodpecker/pr/test Pipeline failed
ci/woodpecker/pr/build-arm64 unknown status
ci/woodpecker/pr/build-amd64 unknown status
ci/woodpecker/pr/docs unknown status
2023-08-03 18:10:27 +01:00
98cb255d12 Support elixir1.15
Some checks failed
ci/woodpecker/push/build-amd64 Pipeline is pending
ci/woodpecker/push/build-arm64 Pipeline is pending
ci/woodpecker/push/docs Pipeline is pending
ci/woodpecker/push/test Pipeline is pending
ci/woodpecker/pr/test Pipeline failed
ci/woodpecker/pr/build-amd64 unknown status
ci/woodpecker/pr/build-arm64 unknown status
ci/woodpecker/pr/docs unknown status
OTP builds to 1.15

Changelog entry

Ensure policies are fully loaded

Fix :warn

use main branch for linkify

Fix warn in tests

Migrations for phoenix 1.17

Revert "Migrations for phoenix 1.17"

This reverts commit 6a3b2f15b7.

Oban upgrade

Add default empty whitelist

mix format

limit test to amd64

OTP 26 tests for 1.15

use OTP_VERSION tag

baka

just 1.15

Massive deps update

Update locale, deps

Mix format

shell????

multiline???

?

max cases 1

use assert_recieve

don't put_env in async tests

don't async conn/fs tests

mix format

FIx some uploader issues

Fix tests
2023-08-03 17:44:09 +01:00
ilja
f49e9e6d4c Clean up bookmarks after prune_objects
Some checks are pending
ci/woodpecker/pr/woodpecker Pipeline is pending
When doing prune_objects, it's possible that bookmarked objects are deleted.
This gave problems when fetching the bookmark TL.
Here we clean up the bookmarks during pruning in the case were it's possible that bookmarked objects are deleted.
2023-05-21 13:02:28 +02:00
ilja
c7fb78cc32 Move deadline and old_insert_date to setup
Some checks are pending
ci/woodpecker/pr/woodpecker Pipeline is pending
Several tests for prune_objetcs need a date older than the deadline for pruning, so I moved that to the setup
2023-05-21 12:01:54 +02:00
ilja
328b4d93b7 Changelog + remove some unneeded comments from the tests 2023-02-26 14:43:19 +01:00
ilja
57eef6d764 prune_objects can prune orphaned activities who reference an array of objects
E.g. Flag activities have an array of objects

We prune the activity when NONE of the objects can be found

Note that the cost of finding and deleting these is ~4x higher than finding and deleting the non-array ones

Only string:
Delete on activities  (cost=506573.48..506580.38 rows=0 width=0)

Only Array:
Delete on activities  (cost=3570359.68..4276365.34 rows=0 width=0)

(They are still executed separately, so the total cost is the sum of the two)
2023-02-26 14:41:50 +01:00
ilja
a7ec6e039c prune_objects can prune orphaned activities
We add an option to also prune remote activities who don't have existing objects any more they reference.
Rn, we only check for activities who only reference one object, not an array or embeded object.
2023-02-26 14:41:50 +01:00
7695010268 Prune Objects --keep-threads option (#350)
Some checks are pending
ci/woodpecker/push/woodpecker Pipeline is pending
This adds an option to the prune_objects mix task.
The original way deleted all non-local public posts older than a certain time frame.
Here we add a different query which you can call using the option --keep-threads.

We query from the activities table all context id's where
    1. the newest activity with this context is still old
    2. none of the activities with this context is is local
    3. none of the activities with this context is bookmarked
and delete all objects with these contexts.

The idea is that posts with local activities (posts, replies, likes, repeats...) may be interesting to keep.
Besides that, a post lives in a certain context (the thread), so we keep the whole thread as well.

Caveats:
* ~~Quotes have a different context. Therefore, when someone quotes a post, it's possible the quoted post will still be deleted.~~ fixed in #379
* Although undocumented (in docs/docs/administration/CLI_tasks/database.md/#prune-old-remote-posts-from-the-database), the 'normal' delete action still kept old remote non-public posts. I added an option to keep this behaviour, but this also means that you now have to explicitly provide that option. **This could be considered a breaking change!**
* ~~Note that this removes from the objects table, but not from the activities.~~ See #427 for that.

Some statistics from explain analyse:
(cost=1402845.92..1933782.00 rows=3810907 width=62) (actual time=2562455.486..2562455.495 rows=0 loops=1)
 Planning Time: 505.327 ms
 Trigger for constraint chat_message_references_object_id_fkey: time=651939.797 calls=921740
 Trigger for constraint deliveries_object_id_fkey: time=52036.009 calls=921740
 Trigger for constraint hashtags_objects_object_id_fkey: time=20665.778 calls=921740
 Execution Time: 3287933.902 ms

***
**TODO**
1. [x] **Question:** Is it OK to keep it like this in regard to quote posts? If not (ie post quoted by local users should also be kept), should we give quotes the same context as the post they are quoting? (If we don't want to give them the same context, I'll have to see how/if I can do it without being too costly)
    * See #379
2. [x] **Question:** the "original" query only deletes public posts (this is undocumented, but you can check the code). This new one doesn't care for scope. From the docs I get that the idea is that posts can be refetched when needed. But I have from a trusted source that Pleroma can't refetch non-public posts. I assume that's the reason why they are kept here. I see different options to deal with this
    1. ~~We keep it as currently implemented and just don't care about scope with this option~~
    2. ~~We add logic to not delete non-public posts either (I'll have to see how costly that becomes)~~
    3. We add an extra --keep-non-public parameter. This is technically speaking breakage (you didn't have to provide a param before for this, now you do), but I'm inclined to not care much because it wasn't documented nor tested in the first place.
3. [x] See if we can do the query using Elixir
4. [x] Test on a bigger DB to see that we don't run into a timeout
5. [x] Add docs

Co-authored-by: ilja <git@ilja.space>
Reviewed-on: #350
Co-authored-by: ilja <akkoma.dev@ilja.space>
Co-committed-by: ilja <akkoma.dev@ilja.space>
2023-01-09 22:15:41 +00:00
f752126427 Remove quack, ensure adapter is finch
Some checks are pending
ci/woodpecker/push/woodpecker Pipeline is pending
2022-12-11 23:22:35 +00:00
8e4de118c1 Don't persist local undone follow (#194)
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
same deal but backwards this time

Co-authored-by: FloatingGhost <hannah@coffee-and-dreams.uk>
Reviewed-on: #194
2022-08-31 18:00:36 +00:00
0f132b802d purge chat and shout endpoints
All checks were successful
ci/woodpecker/push/docs Pipeline was successful
ci/woodpecker/push/release Pipeline was successful
ci/woodpecker/push/test Pipeline was successful
ci/woodpecker/pr/release Pipeline was successful
ci/woodpecker/pr/docs Pipeline was successful
ci/woodpecker/pr/test Pipeline was successful
2022-07-21 11:29:28 +01:00
Tusooa Zhu
4ed0a780b7 Fix digest test 2022-06-29 20:45:05 +01:00
Alex Gleason
54af527759
Upgrade Ecto to v3.6.2, remove deprecated ecto_explain 2021-06-23 13:22:32 -05:00
fe40f6f291 Switch from the deprecated "use Mix.config" to "import Config" 2021-05-20 14:02:58 -05:00
Alexander Strizhakov
03843a5386
migrating config to tmp folder 2021-03-23 14:29:02 +03:00
2217b35bd4 Merge branch 'ecto-rollback-in-test-env' into 'develop'
Possibility to run rollback in test env

See merge request pleroma/pleroma!3212
2021-02-26 16:47:53 +00:00
e854c35e65 Convert tests to all use clear_config instead of Pleroma.Config.put 2021-01-26 11:58:43 -06:00
f868dcf3ac Ensure consistent behavior between pleroma.user activate/deactivate mix tasks 2021-01-25 10:48:28 -06:00
5e8da27e14 Provide pleroma.user mix task for both activate and deactivate 2021-01-25 10:36:51 -06:00
28581e03ad Merge branch 'develop' into refactor/deactivated_user_field 2021-01-18 14:58:21 -06:00
Mark Felder
2c0fe2ea9e Remove toggle_confirmation; require explicit state change
Also cosmetic changes to make the code clearer
2021-01-15 13:11:51 -06:00
Mark Felder
d36182c088 Change user.confirmation_pending field to user.is_confirmed 2021-01-15 12:44:41 -06:00
Mark Felder
7516660753 Remove toggle_activation 2021-01-15 11:25:32 -06:00
Mark Felder
860b5c7804 Change user.deactivated field to user.is_active 2021-01-15 11:24:46 -06:00
Haelwenn (lanodan) Monnier
c4439c630f
Bump Copyright to 2021
grep -rl '# Copyright © .* Pleroma' * | xargs sed -i 's;Copyright © .* Pleroma .*;Copyright © 2017-2021 Pleroma Authors <https://pleroma.social/>;'
2021-01-13 07:49:50 +01:00
lain
e1e7e4d379 Object: Rework how Object.normalize works
Now it defaults to not fetching, and the option is named.
2021-01-04 13:38:31 +01:00
lain
e4f1d8f48c Merge branch 'cachex-test' into 'develop'
Test framework overhaul (speed, reliability)

See merge request pleroma/pleroma!3209
2020-12-26 10:26:35 +00:00
Alexander Strizhakov
538af14d52
possibility to run rollback in test env 2020-12-22 17:55:30 +03:00
lain
b17c36c45a Tests: More test fixes. 2020-12-21 12:59:11 +01:00
lain
9ba60f70d2 Tests: Make as many tests as possible async.
In general, tests that match these criteria can be made async:

- Doesn't use real Cachex.
- Doesn't write to the Config / Application Environment.
- Uses Mock. Using Mox is fine.
- Uses the streamer.
2020-12-21 12:21:40 +01:00
lain
b4b68b71fc Tests: Remove unneeded explicit cachex use.
Only use cachex when we're actually testing it.
2020-12-18 13:18:17 +01:00
Alex Gleason
80891e83d8
Merge remote-tracking branch 'upstream/develop' into registration-workflow 2020-12-17 09:05:36 -06:00
Alexander Strizhakov
cebe3c7def Fix for dropping posts/notifs in WS when mix task is executed
- start oban in mix tasks with empty queues, plugins and crontab
- fix for update_users_following_followers_counts
- fix for removed logo.png
- typo in resend confirmation emails mix task docs
- fix for uploads mix task (start Majic.Pool)
- fix for creating user mix task (start :fast_html app)
2020-12-14 11:02:32 -06:00
lain
1436a2fa2d Merge branch 'stream-follow-updates' into 'develop'
Stream follow updates

Closes #2299

See merge request pleroma/pleroma!3183
2020-12-09 15:52:31 +00:00
lain
ed76323776 Merge branch 'feature/configdb-mix-tasks' into 'develop'
Add mix tasks to give additional ConfigDB recovery and debugging options

See merge request pleroma/pleroma!3174
2020-12-07 18:06:06 +00:00
Mark Felder
e3dd0d45b7 Slip in a test to ensure we can use the atom syntax in mix task arguments 2020-12-07 11:21:06 -06:00
Ivan Tashkinov
e00c667145 [#3174] Refactoring: ConfigDB fetching functions, ConfigDB tests.
Minor fixes.
2020-12-06 18:02:30 +03:00
Mark Felder
657002e738 Answer new prompt interactively 2020-12-04 12:07:23 -06:00
Mark Felder
e379ab8277 Add --force flag for delete and reset commands
Bunch of reorganization and consolidation
2020-12-02 16:24:32 -06:00
Mark Felder
13947999ad Use a callback strategy to short circuit the functions and print a nice error 2020-12-02 12:33:34 -06:00
Mark Felder
3e6d9187a7 Add tests for config dumping 2020-12-02 10:43:22 -06:00
Egor Kislitsyn
35ba48494f
Stream follow updates 2020-12-02 00:18:58 +04:00