Merge search behaviour change, and refactor elasticsearch #28

Merged

floatingghost merged 91 commits from elasticsearch-refactor into develop

2022-06-30 16:55:31 +00:00

floatingghost commented

2022-06-30 16:54:05 +00:00

Owner

No description provided.

floatingghost added 91 commits 2022-06-30 16:54:06 +00:00

8de65de3b9 Add unicode 14 support

and add a test with a unicode 14 emoji

8b843be03e Fix test get_user_apps/1

For some reason I had a test who suddenly failed, mix test test/pleroma/web/o_auth/app_test.exs:54. A user has a list of applications and this test adds them and then sees if the list it gets back is the same as the apps it added.

When I ran mix test a day before I didn't have this problem and when I pushed code today in a different MR, the pipeline succeeded (see https://git.pleroma.social/ilja/pleroma/-/jobs/205827), yet locally it failed. So it seems the test can sometimes succeed and sometimes fail, which makes it untrustworthy.

The failure I see is because the returned list is in reverse order. I assume that's not per sé wrong. You just want to know if the apps you added are actually there. I fixed the test by first ordering the lists before comparing.

AFAICT (and as far as that's relevant) the test got introduced in commit cb2a072e62

91d7150031 mix: Check .git presence

1048bc1bb9 Delete report notifs when demoting from superuser

When someone isn't a superuser any more, they shouldn't see the reporsts any more either.
Here we delete the report notifications from a user when that user gets updated from being a superuser to a non-superuser.

a20d2847e2 After code review

Use patern matching to see if someone was superuser before

775f997c40 Prefer userLanguage cookie over Accept-Language header in detecting locale

https://git.pleroma.social/pleroma/pleroma-meta/-/issues/60

ed1d9e91bc Make remote follow pages translatable

7107fdc1b9 Extract translatable text

8f08c902a5 Make lint happy

50913c4dc5 Make password reset pages translatable

ec57e3480c Make tag feed translatable

793922f1d8 Use proper lang attributes in htmls

8f8d2abb12 Make oauth pages translatable

c6652fccae Make mfa pages translatable

03aac29531 Fix tests

39cfecb188 Make static fe translatable

63a0536d55 Make mail and mailer translatable

4ed0a780b7 Fix digest test

c301a95276 Make all emails translatable

7726148472 Send emails i18n'd using backend-stored user language

1b77308644 Allow update_credentials to update User.language

72bdb0640f Allow user to register with custom language

dadc4eabf7 Document API addition

6e8123a330 Add changelog

ef73f61b07 Fallback to a variant if the language in general is not supported

For an example, here, zh is not supported, but zh_Hans and zh_Hant
are. If the user asks for zh, we should choose a variant for them
instead of fallbacking to default.

Some browsers (e.g. Firefox) does not allow users to customize
their language codes. For example, there is no zh-Hans, but only
zh, zh-CN, zh-TW, zh-HK, etc. This provides a workaround for
those users suffering from bad design decisions.

ff0bb3a3ac Add test for fallbacking to a general language

845e5769ce Make lint happy

fa95bc8725 Support multiple locales formally

elixir gettext current does not fully support fallback to another language [0].
But it might in the future. We adapt it so that all languages in Accept-Language
headers are received by Pleroma.Web.Gettext. User.languages is now a comma-separated
list.

[0]: https://github.com/elixir-gettext/gettext/issues/303

07bd35227a Support multiple locales from userLanguage cookie

fcfb5a4967 Lint

2df25e6666 Support fallbacking to other languages

e3107fee98 Fix eratic test for POST /api/pleroma/admin/reports/:id/notes

It retrieved two ReportNotes and then checked one of them. But the order isn't guaranteed, while the test tested on the content of the first ReportNote.

I made the test on the content more generic

33d821256e Fix incorrect fallback when English is set to first language

932e5df19e Allow to skip cache in Cache plug

Ref: fix-local-public

3fd87b6a75 Skip cache when /objects or /activities is authenticated

Ref: fix-local-public

661d0ba481 Also use actor_type to determine if an account is a bot in antiFollowbotPolicy

8f140deb8f StealEmojiPolicy: fix String rejected_shortcodes

* rejected_shortcodes is defined as a list of strings in the
  configuration description. As such, database-based configuration was
  led to handle those settings as strings, and not as the actually
  expected type, Regex.
* This caused each message passing through this MRF, if a rejected
  shortcode was set and the emoji did not exist already on the instance,
  to fail federating, as an exception was raised, swiftly caught and
  mostly silenced.
* This commit fixes the issue by introducing new behavior: strings are
  now handled as perfect matches for an emoji shortcode (meaning that if
  the emoji-to-be-pulled's shortcode is in the blacklist, it will be
  rejected), while still supporting Regex types as before.

60eec9d0c5 Instruct users to run 'git pull' as the pleroma user

e961cf2689 Initial meilisearch implementation, doesn't delete posts yet

41db5c8653 Add logging to milisiearch index and make it use desc(id)

e5ac2ffa07 Ensure only indexing public posts and implement clearing and delete

abf82a63ec Make the indexing batch differently and more, show number indexed

7b3701e6b9 Make meilisearch sort on publish date converted to unix time

c3a04166a0 Tweak search ordering to hopefully return newer results

a586ce0ddd Use content instead of source and scrub it

52a872432d Make the chunk size smaller

b3401ba7bd Also index incoming federated posts

14ef6ce80f Mark only content as searchable for meilisearch

117f525fd6 Adjust content indexing to skip more unneeded stuff

0cf3654907 Rework task indexing to share code with the main module

The code in the main module now scrubs new posts too

5360cc1097 Make indexing logs rewrite themselves

dbf556cdcf Implement meilisearch auth

d5cc272a91 Add a message with a count of posts to index

d1079f1aa3 Add the meilisearch.stats command

375154e5de Add search/meilisearch documentation

41f3572152 Fix activity being passed to objec_to_search_data

88891e9d76 Add tests for local post indexing for meilisearch

b519726571 Add private_key: nil to default meilisearch options

5ed1759091 Reorder ranking rules for (maybe) better results

0802c0666b Move add_to_index / remove_from_index to Pleroma.Actitivy.Search

9ab0f130cc Add more documentation about rum to meilisearch docs

a937a98df5 Don't try removing from index again in common_api

It's already removed in the side effects of the pipeline

37913c8ddf Use proper deleted object for removing from index

afcdd2f437 Modify some meilisearch variables

51faa28568 Set content-type to application/json

d99a2be351 Only add local posts to index in activity_pub

Remote ones are already added in another place

9e7d7ebd48 Add a reindex option

Signed-off-by: Ekaterina Vaartis <vaartis@kotobank.ch>

86971fceaa Support reindexing meilisearch >=0.24.0

It has has a different error code key

7a9d9cf457 Fix a typo in search docs

5dd908552c Move the search.ex file so credo doesn't complain

0769f06bd1 Style fixes

ea582fbf93 Rename search.ex to database_search.ex and add search/2

c128798418 Rename Activity.Search to Search.DatabaseSearch

bac70a2bc1 Implement suggestions from the Meilisearch MR

- Index unlisted posts
- Move version check outside of the streaming and only do it once
- Use a PUT request instead of checking manually if there is need to insert
- Add error handling, sort of

58cc5d13a2 Add config description for meilisearch

426cff3372 Update search.md documentation with meilisearch indexing steps

cc3319ac1d Make chunk size configurable

a4914add8c Don't support meilisearch < 0.24.0, since it breaks things

80e52f4d86 Add description for initial_indexing_chunk_size

54b2a86f47 Add a search backend behaviour

5def4a7d49 Use oban for search indexing

7aebff799b Fix meilisearch tests and jobs for oban

7f53aa400b Don't try removing deleted users and such from index as posts

774b4e165a Change search_indexing = 10 and retries for indexing = 2

09ea3bb694 Actually, unlisted posts are indexed

b7462040cc Change the meilisearch key auth to conform to 0.25.0