AkkomaGang/akkoma

Fork 92

Merge search behaviour change, and refactor elasticsearch #28

Merged

floatingghost merged 91 commits from elasticsearch-refactor into develop

2022-06-30 16:55:31 +00:00

floatingghost commented

2022-06-30 16:54:05 +00:00

Owner

No description provided.

floatingghost added 91 commits 2022-06-30 16:54:06 +00:00

Add unicode 14 support 8de65de3b9

and add a test with a unicode 14 emoji

Fix test get_user_apps/1 8b843be03e

For some reason I had a test who suddenly failed, mix test test/pleroma/web/o_auth/app_test.exs:54. A user has a list of applications and this test adds them and then sees if the list it gets back is the same as the apps it added.

When I ran mix test a day before I didn't have this problem and when I pushed code today in a different MR, the pipeline succeeded (see https://git.pleroma.social/ilja/pleroma/-/jobs/205827), yet locally it failed. So it seems the test can sometimes succeed and sometimes fail, which makes it untrustworthy.

The failure I see is because the returned list is in reverse order. I assume that's not per sé wrong. You just want to know if the apps you added are actually there. I fixed the test by first ordering the lists before comparing.

AFAICT (and as far as that's relevant) the test got introduced in commit cb2a072e62

mix: Check .git presence 91d7150031

Delete report notifs when demoting from superuser 1048bc1bb9

When someone isn't a superuser any more, they shouldn't see the reporsts any more either.
Here we delete the report notifications from a user when that user gets updated from being a superuser to a non-superuser.

After code review a20d2847e2

Use patern matching to see if someone was superuser before

Prefer userLanguage cookie over Accept-Language header in detecting locale 775f997c40

https://git.pleroma.social/pleroma/pleroma-meta/-/issues/60

Make remote follow pages translatable ed1d9e91bc

Extract translatable text 7107fdc1b9

Make lint happy 8f08c902a5

Make password reset pages translatable 50913c4dc5

Make tag feed translatable ec57e3480c

Use proper lang attributes in htmls 793922f1d8

Make oauth pages translatable 8f8d2abb12

Make mfa pages translatable c6652fccae

Fix tests 03aac29531

Make static fe translatable 39cfecb188

Make mail and mailer translatable 63a0536d55

Fix digest test 4ed0a780b7

Make all emails translatable c301a95276

Send emails i18n'd using backend-stored user language 7726148472

Allow update_credentials to update User.language 1b77308644

Allow user to register with custom language 72bdb0640f

Document API addition dadc4eabf7

Add changelog 6e8123a330

Fallback to a variant if the language in general is not supported ef73f61b07

For an example, here, zh is not supported, but zh_Hans and zh_Hant
are. If the user asks for zh, we should choose a variant for them
instead of fallbacking to default.

Some browsers (e.g. Firefox) does not allow users to customize
their language codes. For example, there is no zh-Hans, but only
zh, zh-CN, zh-TW, zh-HK, etc. This provides a workaround for
those users suffering from bad design decisions.

Add test for fallbacking to a general language ff0bb3a3ac

Make lint happy 845e5769ce

Support multiple locales formally fa95bc8725

elixir gettext current does not fully support fallback to another language [0].
But it might in the future. We adapt it so that all languages in Accept-Language
headers are received by Pleroma.Web.Gettext. User.languages is now a comma-separated
list.

[0]: https://github.com/elixir-gettext/gettext/issues/303

Support multiple locales from userLanguage cookie 07bd35227a

Lint fcfb5a4967

Support fallbacking to other languages 2df25e6666

Fix eratic test for POST /api/pleroma/admin/reports/:id/notes e3107fee98

It retrieved two ReportNotes and then checked one of them. But the order isn't guaranteed, while the test tested on the content of the first ReportNote.

I made the test on the content more generic

Fix incorrect fallback when English is set to first language 33d821256e

Allow to skip cache in Cache plug 932e5df19e

Ref: fix-local-public

Skip cache when /objects or /activities is authenticated 3fd87b6a75

Ref: fix-local-public

Also use actor_type to determine if an account is a bot in antiFollowbotPolicy 661d0ba481

StealEmojiPolicy: fix String rejected_shortcodes 8f140deb8f

* rejected_shortcodes is defined as a list of strings in the
  configuration description. As such, database-based configuration was
  led to handle those settings as strings, and not as the actually
  expected type, Regex.
* This caused each message passing through this MRF, if a rejected
  shortcode was set and the emoji did not exist already on the instance,
  to fail federating, as an exception was raised, swiftly caught and
  mostly silenced.
* This commit fixes the issue by introducing new behavior: strings are
  now handled as perfect matches for an emoji shortcode (meaning that if
  the emoji-to-be-pulled's shortcode is in the blacklist, it will be
  rejected), while still supporting Regex types as before.

Instruct users to run 'git pull' as the pleroma user 60eec9d0c5

Initial meilisearch implementation, doesn't delete posts yet e961cf2689

Add logging to milisiearch index and make it use desc(id) 41db5c8653

Ensure only indexing public posts and implement clearing and delete e5ac2ffa07

Make the indexing batch differently and more, show number indexed abf82a63ec

Make meilisearch sort on publish date converted to unix time 7b3701e6b9

Tweak search ordering to hopefully return newer results c3a04166a0

Use content instead of source and scrub it a586ce0ddd

Make the chunk size smaller 52a872432d

Also index incoming federated posts b3401ba7bd

Mark only content as searchable for meilisearch 14ef6ce80f

Adjust content indexing to skip more unneeded stuff 117f525fd6

Rework task indexing to share code with the main module 0cf3654907

The code in the main module now scrubs new posts too

Make indexing logs rewrite themselves 5360cc1097

Implement meilisearch auth dbf556cdcf

Add a message with a count of posts to index d5cc272a91

Add the meilisearch.stats command d1079f1aa3

Add search/meilisearch documentation 375154e5de

Fix activity being passed to objec_to_search_data 41f3572152

Add tests for local post indexing for meilisearch 88891e9d76

Add private_key: nil to default meilisearch options b519726571

Reorder ranking rules for (maybe) better results 5ed1759091

Move add_to_index / remove_from_index to Pleroma.Actitivy.Search 0802c0666b

Add more documentation about rum to meilisearch docs 9ab0f130cc

Don't try removing from index again in common_api a937a98df5

It's already removed in the side effects of the pipeline

Use proper deleted object for removing from index 37913c8ddf

Modify some meilisearch variables afcdd2f437

Set content-type to application/json 51faa28568

Only add local posts to index in activity_pub d99a2be351

Remote ones are already added in another place

Add a reindex option 9e7d7ebd48

Signed-off-by: Ekaterina Vaartis <vaartis@kotobank.ch>

Support reindexing meilisearch >=0.24.0 86971fceaa

It has has a different error code key

Fix a typo in search docs 7a9d9cf457

Move the search.ex file so credo doesn't complain 5dd908552c

Style fixes 0769f06bd1

Rename search.ex to database_search.ex and add search/2 ea582fbf93

Rename Activity.Search to Search.DatabaseSearch c128798418

Implement suggestions from the Meilisearch MR bac70a2bc1

- Index unlisted posts
- Move version check outside of the streaming and only do it once
- Use a PUT request instead of checking manually if there is need to insert
- Add error handling, sort of

Add config description for meilisearch 58cc5d13a2

Update search.md documentation with meilisearch indexing steps 426cff3372

Make chunk size configurable cc3319ac1d

Don't support meilisearch < 0.24.0, since it breaks things a4914add8c

Add description for initial_indexing_chunk_size 80e52f4d86

Add a search backend behaviour 54b2a86f47

Use oban for search indexing 5def4a7d49

Fix meilisearch tests and jobs for oban 7aebff799b

Don't try removing deleted users and such from index as posts 7f53aa400b

Change search_indexing = 10 and retries for indexing = 2 774b4e165a

Actually, unlisted posts are indexed 09ea3bb694

Change the meilisearch key auth to conform to 0.25.0 b7462040cc