Commit graph

700 commits

Author SHA1 Message Date
a95af3ee4c exiftool: strip all non-essential tags
Documentation was already clear on this only stripping GPS tags.
But there are more potentially sensitive metadata tags (e.g. author
and possibly description) and the name alone suggests a broader effect.

Thus change the filter to strip all metadata except for colourspace info
and orientation (technically it strips everything and then readds
selected tags).

Explicitly stripping CommonIFD0 is needed since -all does not modify
IFD0 due to TIFF storing some actual image data there. CommonIFD0 then
strips a bunch of commonly used actual metadata tags from IFD0, to my
understanding leaving TIFF image data and custom metadata tags intact.
2024-04-25 23:00:42 +02:00
timorl
09d3ccf770
Read description before stripping metadata
Some checks are pending
ci/woodpecker/pr/build-amd64 Pipeline is pending
ci/woodpecker/pr/build-arm64 Pipeline is pending
ci/woodpecker/pr/docs Pipeline is pending
ci/woodpecker/pr/lint Pipeline is pending
ci/woodpecker/pr/test Pipeline is pending
2024-04-19 20:51:54 +02:00
timorl
cd7af81896
Rename StripLocation to StripMetadata for temporal-proofing reasons
Some checks are pending
ci/woodpecker/pr/build-amd64 Pipeline is pending
ci/woodpecker/pr/build-arm64 Pipeline is pending
ci/woodpecker/pr/docs Pipeline is pending
ci/woodpecker/pr/lint Pipeline is pending
ci/woodpecker/pr/test Pipeline is pending
2024-04-16 20:37:00 +02:00
timorl
b144218dce
Merge branch 'develop' into elseinspe
Some checks failed
ci/woodpecker/pr/lint Pipeline failed
ci/woodpecker/pr/test unknown status
ci/woodpecker/pr/build-arm64 unknown status
ci/woodpecker/pr/build-amd64 unknown status
ci/woodpecker/pr/docs unknown status
2024-04-14 20:31:33 +02:00
2d439034ca Ensure that spoof-inserted does not time out 2024-03-30 12:55:22 +00:00
0648d9ebaa Add mix tasks to detect spoofed posts and users
At least as far as we can
2024-03-26 16:05:20 -01:00
d441101200 Add mix task to detect uploaded spoof payloads 2024-03-26 16:05:20 -01:00
0ec62acb9d Always insert Dedupe upload filter
This actually was already intended before to eradict all future
path-traversal-style exploits and to fix issues with some
characters like akkoma#610 in 0b2ec0ccee. However, Dedupe and
AnonymizeFilename got mixed up. The latter only anonymises the name
in Content-Disposition headers GET parameters (with link_name),
_not_ the upload path.

Even without Dedupe, the upload path is prefixed by an UUID,
so it _should_ already be hard to guess for attackers. But now
we actually can be sure no path shenanigangs occur, uploads
reliably work and save some disk space.

While this makes the final path predictable, this prediction is
not exploitable. Insertion of a back-reference to the upload
itself requires pulling off a successfull preimage attack against
SHA-256, which is deemed infeasible for the foreseeable futures.

Dedupe was already included in the default list in config.exs
since 28cfb2c37a, but this will get overridde by whatever the
config generated by the "pleroma.instance gen" task chose.

Upload+delete tests running in parallel using Dedupe might be flaky, but
this was already true before and needs its own commit to fix eventually.
2024-03-18 22:33:10 -01:00
fef773ca35 Drop media base_url default and recommend different domain
Same-domain setups enabled now at least two exploits,
so they ought to be discouraged and definitely not be the default.
2024-03-18 22:33:10 -01:00
6cb40bee26 Migrate to phoenix 1.7 (#626)
All checks were successful
ci/woodpecker/push/lint Pipeline was successful
ci/woodpecker/push/test Pipeline was successful
ci/woodpecker/push/build-arm64 Pipeline was successful
ci/woodpecker/push/build-amd64 Pipeline was successful
ci/woodpecker/push/docs Pipeline was successful
Closes #612

Co-authored-by: tusooa <tusooa@kazv.moe>
Reviewed-on: #626
Co-authored-by: FloatingGhost <hannah@coffee-and-dreams.uk>
Co-committed-by: FloatingGhost <hannah@coffee-and-dreams.uk>
2023-08-15 10:22:18 +00:00
0b32beb051 Merge pull request 'meilisearch: Move published date to lower priority' (#623) from norm/akkoma:meilisearch-order into develop
Some checks are pending
ci/woodpecker/push/build-amd64 Pipeline is pending
ci/woodpecker/push/build-arm64 Pipeline is pending
ci/woodpecker/push/docs Pipeline is pending
ci/woodpecker/push/test Pipeline is pending
Reviewed-on: #623
2023-08-12 14:36:53 +00:00
7bb41bffb3 Merge pull request 'Reload emoji when using mix pleroma.emoji gen-pack and get-packs' (#563) from norm/akkoma:emoji-reload into develop
Some checks are pending
ci/woodpecker/push/build-amd64 Pipeline is pending
ci/woodpecker/push/build-arm64 Pipeline is pending
ci/woodpecker/push/docs Pipeline is pending
ci/woodpecker/push/test Pipeline is pending
Reviewed-on: #563
2023-08-12 14:07:23 +00:00
d79c92f9c6
meilisearch: Move published date to lower priority
Some checks are pending
ci/woodpecker/pr/build-amd64 Pipeline is pending
ci/woodpecker/pr/build-arm64 Pipeline is pending
ci/woodpecker/pr/docs Pipeline is pending
ci/woodpecker/pr/test Pipeline is pending
Currently, Akkoma sorts by published date first before everything else.
This however makes search results pretty bad since Meilisearch uses a
bucket sort algorithm in order of the ranking rules specified:
https://www.meilisearch.com/docs/learn/core_concepts/relevancy#behavior

Since the `published` attribute is a unix timestamp, the resulting
buckets are pretty small so the other rules essentially have little to
no effect on the rankings of search results.

This fixes that issue by moving the `published:desc` rule further down
so it still sorts by date, but only after considering everything else.

AFAIK attribute and sort doesn't really affect results for Akkoma since
the only attribute considered is the `content` attribute and the `sort`
parameter isn't used in Akkoma searches. Everything else is made to
match more closely to Meilisearch's defaults.
2023-08-11 11:07:14 -04:00
Haelwenn (lanodan) Monnier
4f57c87be4
instance gen: Reduce permissions of pleroma directories and config files
Original: 69caedc591
2023-08-04 14:13:50 -04:00
98cb255d12 Support elixir1.15
Some checks failed
ci/woodpecker/push/build-amd64 Pipeline is pending
ci/woodpecker/push/build-arm64 Pipeline is pending
ci/woodpecker/push/docs Pipeline is pending
ci/woodpecker/push/test Pipeline is pending
ci/woodpecker/pr/test Pipeline failed
ci/woodpecker/pr/build-amd64 unknown status
ci/woodpecker/pr/build-arm64 unknown status
ci/woodpecker/pr/docs unknown status
OTP builds to 1.15

Changelog entry

Ensure policies are fully loaded

Fix :warn

use main branch for linkify

Fix warn in tests

Migrations for phoenix 1.17

Revert "Migrations for phoenix 1.17"

This reverts commit 6a3b2f15b7.

Oban upgrade

Add default empty whitelist

mix format

limit test to amd64

OTP 26 tests for 1.15

use OTP_VERSION tag

baka

just 1.15

Massive deps update

Update locale, deps

Mix format

shell????

multiline???

?

max cases 1

use assert_recieve

don't put_env in async tests

don't async conn/fs tests

mix format

FIx some uploader issues

Fix tests
2023-08-03 17:44:09 +01:00
b99053d2c2 Reload emoji when using mix pleroma.emoji gen-pack and get-packs
Some checks are pending
ci/woodpecker/pr/woodpecker Pipeline is pending
I think it makes more sense that the emoji cache gets reloaded in Akkoma if you add or create emoji packs.
2023-06-04 02:43:18 +00:00
6225f24f5f Merge pull request 'Clean up bookmarks after prune_objects' (#544) from ilja/akkoma:clean_up_bookmarks_after_prune_objects into develop
Some checks are pending
ci/woodpecker/push/woodpecker Pipeline is pending
Reviewed-on: #544
2023-05-22 21:28:48 +00:00
ilja
f49e9e6d4c Clean up bookmarks after prune_objects
Some checks are pending
ci/woodpecker/pr/woodpecker Pipeline is pending
When doing prune_objects, it's possible that bookmarked objects are deleted.
This gave problems when fetching the bookmark TL.
Here we clean up the bookmarks during pruning in the case were it's possible that bookmarked objects are deleted.
2023-05-21 13:02:28 +02:00
522221f7fb Mix format 2023-04-14 17:56:34 +01:00
2a8c1f4192 Add extra diagnostic tasks in
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
2023-03-29 14:11:00 +01:00
ilja
57eef6d764 prune_objects can prune orphaned activities who reference an array of objects
E.g. Flag activities have an array of objects

We prune the activity when NONE of the objects can be found

Note that the cost of finding and deleting these is ~4x higher than finding and deleting the non-array ones

Only string:
Delete on activities  (cost=506573.48..506580.38 rows=0 width=0)

Only Array:
Delete on activities  (cost=3570359.68..4276365.34 rows=0 width=0)

(They are still executed separately, so the total cost is the sum of the two)
2023-02-26 14:41:50 +01:00
ilja
a7ec6e039c prune_objects can prune orphaned activities
We add an option to also prune remote activities who don't have existing objects any more they reference.
Rn, we only check for activities who only reference one object, not an array or embeded object.
2023-02-26 14:41:50 +01:00
7695010268 Prune Objects --keep-threads option (#350)
Some checks are pending
ci/woodpecker/push/woodpecker Pipeline is pending
This adds an option to the prune_objects mix task.
The original way deleted all non-local public posts older than a certain time frame.
Here we add a different query which you can call using the option --keep-threads.

We query from the activities table all context id's where
    1. the newest activity with this context is still old
    2. none of the activities with this context is is local
    3. none of the activities with this context is bookmarked
and delete all objects with these contexts.

The idea is that posts with local activities (posts, replies, likes, repeats...) may be interesting to keep.
Besides that, a post lives in a certain context (the thread), so we keep the whole thread as well.

Caveats:
* ~~Quotes have a different context. Therefore, when someone quotes a post, it's possible the quoted post will still be deleted.~~ fixed in #379
* Although undocumented (in docs/docs/administration/CLI_tasks/database.md/#prune-old-remote-posts-from-the-database), the 'normal' delete action still kept old remote non-public posts. I added an option to keep this behaviour, but this also means that you now have to explicitly provide that option. **This could be considered a breaking change!**
* ~~Note that this removes from the objects table, but not from the activities.~~ See #427 for that.

Some statistics from explain analyse:
(cost=1402845.92..1933782.00 rows=3810907 width=62) (actual time=2562455.486..2562455.495 rows=0 loops=1)
 Planning Time: 505.327 ms
 Trigger for constraint chat_message_references_object_id_fkey: time=651939.797 calls=921740
 Trigger for constraint deliveries_object_id_fkey: time=52036.009 calls=921740
 Trigger for constraint hashtags_objects_object_id_fkey: time=20665.778 calls=921740
 Execution Time: 3287933.902 ms

***
**TODO**
1. [x] **Question:** Is it OK to keep it like this in regard to quote posts? If not (ie post quoted by local users should also be kept), should we give quotes the same context as the post they are quoting? (If we don't want to give them the same context, I'll have to see how/if I can do it without being too costly)
    * See #379
2. [x] **Question:** the "original" query only deletes public posts (this is undocumented, but you can check the code). This new one doesn't care for scope. From the docs I get that the idea is that posts can be refetched when needed. But I have from a trusted source that Pleroma can't refetch non-public posts. I assume that's the reason why they are kept here. I see different options to deal with this
    1. ~~We keep it as currently implemented and just don't care about scope with this option~~
    2. ~~We add logic to not delete non-public posts either (I'll have to see how costly that becomes)~~
    3. We add an extra --keep-non-public parameter. This is technically speaking breakage (you didn't have to provide a param before for this, now you do), but I'm inclined to not care much because it wasn't documented nor tested in the first place.
3. [x] See if we can do the query using Elixir
4. [x] Test on a bigger DB to see that we don't run into a timeout
5. [x] Add docs

Co-authored-by: ilja <git@ilja.space>
Reviewed-on: #350
Co-authored-by: ilja <akkoma.dev@ilja.space>
Co-committed-by: ilja <akkoma.dev@ilja.space>
2023-01-09 22:15:41 +00:00
9be6caf125 argon2 password hashing (#406)
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
Co-authored-by: FloatingGhost <hannah@coffee-and-dreams.uk>
Reviewed-on: #406
2022-12-30 02:46:58 +00:00
5a405bdadf document dump_to_file and load_from_file
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
2022-12-29 20:00:04 +00:00
d1bf8aa9ed Add dump_to_file and load_from_file tasks 2022-12-29 19:56:35 +00:00
07a48b9293 giant massive dep upgrade and dialyxir-found error emporium (#371)
Some checks are pending
ci/woodpecker/push/woodpecker Pipeline is pending
Co-authored-by: FloatingGhost <hannah@coffee-and-dreams.uk>
Reviewed-on: #371
2022-12-14 12:38:48 +00:00
e6da301296 Add diagnostics http
Some checks are pending
ci/woodpecker/push/woodpecker Pipeline is pending
2022-12-11 22:57:18 +00:00
09326ffa56 Diagnostics tasks (#348)
Some checks are pending
ci/woodpecker/push/woodpecker Pipeline is pending
a bunch of ways to get query plans to help with debugging

Co-authored-by: FloatingGhost <hannah@coffee-and-dreams.uk>
Reviewed-on: #348
2022-12-07 11:12:34 +00:00
d55de5debf Remerge of hashtag following (#341)
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
this time with less idiot

Co-authored-by: FloatingGhost <hannah@coffee-and-dreams.uk>
Reviewed-on: #341
2022-12-05 12:58:48 +00:00
ec6bf8c3f7 revert 4a94c9a31e
Some checks are pending
ci/woodpecker/push/woodpecker Pipeline is pending
revert Add ability to follow hashtags (#336)

Co-authored-by: FloatingGhost <hannah@coffee-and-dreams.uk>
Reviewed-on: #336
2022-12-04 20:04:09 +00:00
4a94c9a31e Add ability to follow hashtags (#336)
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
Co-authored-by: FloatingGhost <hannah@coffee-and-dreams.uk>
Reviewed-on: #336
2022-12-04 17:36:59 +00:00
6b882a2c0b Purge Rejected Follow requests in daily task (#334)
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
Co-authored-by: FloatingGhost <hannah@coffee-and-dreams.uk>
Reviewed-on: #334
2022-12-03 23:17:43 +00:00
db60640c5b Fixing up deletes a bit (#327)
Some checks failed
ci/woodpecker/push/woodpecker Pipeline failed
Co-authored-by: FloatingGhost <hannah@coffee-and-dreams.uk>
Reviewed-on: #327
2022-12-01 15:00:53 +00:00
e3085c495c fix tests broken by relay defaults changing (#314)
Some checks are pending
ci/woodpecker/push/woodpecker Pipeline is pending
Co-authored-by: FloatingGhost <hannah@coffee-and-dreams.uk>
Reviewed-on: #314
2022-11-26 20:45:47 +00:00
Ilja
338612d72b Use EXIF data of image to prefill image description
During attachment upload Pleroma returns a "description" field.

* This MR allows Pleroma to read the EXIF data during upload and return the description to the FE using this field.
    * If a description is already present (e.g. because a previous module added it), it will use that
    * Otherwise it will read from the EXIF data. First it will check -ImageDescription, if that's empty, it will check -iptc:Caption-Abstract
    * If no description is found, it will simply return nil, which is the default value
* When people set up a new instance, they will be asked if they want to read metadata and this module will be activated if so

There was an Exiftool module, which has now been renamed to Exiftool.StripLocation
2022-10-23 14:46:16 +02:00
856c57208b Ensure deletes are handled after everything else
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
ci/woodpecker/pr/woodpecker Pipeline was successful
2022-10-11 14:30:08 +01:00
92ba2802fb generate-keys-at-registration-time (#181)
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
Reviewed-on: #181
2022-08-24 14:36:33 +00:00
61641957cb fix compatibility with meilisearch (#164)
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
Reviewed-on: #164
2022-08-16 22:56:49 +00:00
2033d7d4fc ensure extra info in fix_follow_state prints
Some checks are pending
ci/woodpecker/push/woodpecker Pipeline is pending
2022-07-29 19:50:26 +01:00
4c47992686 bugfix/follow-state (#104)
Some checks are pending
ci/woodpecker/push/docs Pipeline is pending
ci/woodpecker/push/release Pipeline is pending
ci/woodpecker/push/test Pipeline is pending
Reviewed-on: #104
2022-07-23 18:58:45 +00:00
d2a185c013 Documentation updates for stable release (#73)
Some checks are pending
ci/woodpecker/push/docs Pipeline is pending
ci/woodpecker/push/release Pipeline is pending
ci/woodpecker/push/test Pipeline is pending
Reviewed-on: #73
2022-07-15 12:27:16 +00:00
7dfc3f3d0e Change default Postgres user/DB to akkoma
All checks were successful
ci/woodpecker/pr/release Pipeline was successful
ci/woodpecker/pr/lint Pipeline was successful
ci/woodpecker/pr/test Pipeline was successful
2022-07-12 12:41:30 -04:00
a9c82b62f2 Fixes for elasticsearch 8 (#52)
Some checks failed
ci/woodpecker/push/lint Pipeline was successful
ci/woodpecker/push/release Pipeline failed
ci/woodpecker/push/test Pipeline was successful
Reviewed-on: #52
2022-07-06 18:57:00 +00:00
a036a01a1e mix format
Some checks failed
ci/woodpecker/push/lint Pipeline was successful
ci/woodpecker/push/release Pipeline was successful
ci/woodpecker/push/test Pipeline failed
ci/woodpecker/pr/lint Pipeline was successful
ci/woodpecker/pr/release Pipeline was successful
ci/woodpecker/pr/test Pipeline was successful
2022-07-04 17:38:16 +01:00
364b6969eb Use finch everywhere (#33)
Some checks failed
ci/woodpecker/push/lint Pipeline failed
ci/woodpecker/push/test unknown status
ci/woodpecker/push/release Pipeline was successful
Reviewed-on: #33
2022-07-04 16:30:38 +00:00
2937495712 fix ES import from live (#29)
Some checks failed
ci/woodpecker/push/lint Pipeline was successful
ci/woodpecker/push/test Pipeline failed
ci/woodpecker/push/release Pipeline was successful
Reviewed-on: #29
2022-06-30 18:44:31 +00:00
bc9e76cce7 Add documentation for ES search
Some checks failed
ci/woodpecker/push/release Pipeline was successful
ci/woodpecker/push/lint Pipeline was successful
ci/woodpecker/pr/release Pipeline was successful
ci/woodpecker/pr/lint Pipeline was successful
ci/woodpecker/pr/test Pipeline failed
ci/woodpecker/push/test Pipeline failed
2022-06-30 17:36:57 +01:00
1ecdb19de5 Refactor ES on top of search behaviour
Some checks failed
ci/woodpecker/push/release Pipeline was successful
ci/woodpecker/push/lint Pipeline was successful
ci/woodpecker/push/test Pipeline failed
2022-06-30 16:28:31 +01:00
Ekaterina Vaartis
563b964690 Change updateId to uid because apparently that's the new name 2022-06-29 20:49:45 +01:00