2937495712
fix ES import from live ( #29 )
...
Reviewed-on: AkkomaGang/akkoma#29
2022-06-30 18:44:31 +00:00
40bec73db6
Merge pull request 'Merge search behaviour change, and refactor elasticsearch' ( #28 ) from elasticsearch-refactor into develop
...
Reviewed-on: AkkomaGang/akkoma#28
2022-06-30 16:55:30 +00:00
bc9e76cce7
Add documentation for ES search
2022-06-30 17:36:57 +01:00
635a3c223a
Add elasticsearch tests
2022-06-30 16:53:21 +01:00
1ecdb19de5
Refactor ES on top of search behaviour
2022-06-30 16:28:31 +01:00
Ekaterina Vaartis
69d5d1a01b
Update meilisearch docs
2022-06-29 20:49:45 +01:00
Ekaterina Vaartis
563b964690
Change updateId to uid because apparently that's the new name
2022-06-29 20:49:45 +01:00
Ekaterina Vaartis
b7462040cc
Change the meilisearch key auth to conform to 0.25.0
2022-06-29 20:49:45 +01:00
Ekaterina Vaartis
09ea3bb694
Actually, unlisted posts are indexed
2022-06-29 20:49:45 +01:00
Ekaterina Vaartis
774b4e165a
Change search_indexing = 10 and retries for indexing = 2
2022-06-29 20:49:45 +01:00
Ekaterina Vaartis
7f53aa400b
Don't try removing deleted users and such from index as posts
2022-06-29 20:49:45 +01:00
Ekaterina Vaartis
7aebff799b
Fix meilisearch tests and jobs for oban
2022-06-29 20:49:45 +01:00
Ekaterina Vaartis
5def4a7d49
Use oban for search indexing
2022-06-29 20:49:45 +01:00
Ekaterina Vaartis
54b2a86f47
Add a search backend behaviour
2022-06-29 20:49:45 +01:00
Ekaterina Vaartis
80e52f4d86
Add description for initial_indexing_chunk_size
2022-06-29 20:49:45 +01:00
Ekaterina Vaartis
a4914add8c
Don't support meilisearch < 0.24.0, since it breaks things
2022-06-29 20:49:45 +01:00
Ekaterina Vaartis
cc3319ac1d
Make chunk size configurable
2022-06-29 20:49:45 +01:00
Ekaterina Vaartis
426cff3372
Update search.md documentation with meilisearch indexing steps
2022-06-29 20:49:45 +01:00
Ekaterina Vaartis
58cc5d13a2
Add config description for meilisearch
2022-06-29 20:49:45 +01:00
Ekaterina Vaartis
bac70a2bc1
Implement suggestions from the Meilisearch MR
...
- Index unlisted posts
- Move version check outside of the streaming and only do it once
- Use a PUT request instead of checking manually if there is need to insert
- Add error handling, sort of
2022-06-29 20:49:45 +01:00
Ekaterina Vaartis
c128798418
Rename Activity.Search to Search.DatabaseSearch
2022-06-29 20:49:44 +01:00
Ekaterina Vaartis
ea582fbf93
Rename search.ex to database_search.ex and add search/2
2022-06-29 20:49:00 +01:00
Ekaterina Vaartis
0769f06bd1
Style fixes
2022-06-29 20:49:00 +01:00
Ekaterina Vaartis
5dd908552c
Move the search.ex file so credo doesn't complain
2022-06-29 20:48:59 +01:00
Ekaterina Vaartis
7a9d9cf457
Fix a typo in search docs
2022-06-29 20:48:44 +01:00
Ekaterina Vaartis
86971fceaa
Support reindexing meilisearch >=0.24.0
...
It has has a different error code key
2022-06-29 20:48:44 +01:00
Ekaterina Vaartis
9e7d7ebd48
Add a reindex option
...
Signed-off-by: Ekaterina Vaartis <vaartis@kotobank.ch>
2022-06-29 20:48:44 +01:00
Ekaterina Vaartis
d99a2be351
Only add local posts to index in activity_pub
...
Remote ones are already added in another place
2022-06-29 20:48:44 +01:00
Ekaterina Vaartis
51faa28568
Set content-type to application/json
2022-06-29 20:48:44 +01:00
Ekaterina Vaartis
afcdd2f437
Modify some meilisearch variables
2022-06-29 20:48:44 +01:00
Ekaterina Vaartis
37913c8ddf
Use proper deleted object for removing from index
2022-06-29 20:48:44 +01:00
Ekaterina Vaartis
a937a98df5
Don't try removing from index again in common_api
...
It's already removed in the side effects of the pipeline
2022-06-29 20:48:44 +01:00
Ekaterina Vaartis
9ab0f130cc
Add more documentation about rum to meilisearch docs
2022-06-29 20:48:44 +01:00
Ekaterina Vaartis
0802c0666b
Move add_to_index / remove_from_index to Pleroma.Actitivy.Search
2022-06-29 20:48:44 +01:00
Ekaterina Vaartis
5ed1759091
Reorder ranking rules for (maybe) better results
2022-06-29 20:48:44 +01:00
Ekaterina Vaartis
b519726571
Add private_key: nil to default meilisearch options
2022-06-29 20:48:43 +01:00
Ekaterina Vaartis
88891e9d76
Add tests for local post indexing for meilisearch
2022-06-29 20:48:29 +01:00
Ekaterina Vaartis
41f3572152
Fix activity being passed to objec_to_search_data
2022-06-29 20:48:29 +01:00
Ekaterina Vaartis
375154e5de
Add search/meilisearch documentation
2022-06-29 20:48:29 +01:00
Ekaterina Vaartis
d1079f1aa3
Add the meilisearch.stats command
2022-06-29 20:48:29 +01:00
Ekaterina Vaartis
d5cc272a91
Add a message with a count of posts to index
2022-06-29 20:48:29 +01:00
Ekaterina Vaartis
dbf556cdcf
Implement meilisearch auth
2022-06-29 20:48:29 +01:00
Ekaterina Vaartis
5360cc1097
Make indexing logs rewrite themselves
2022-06-29 20:48:29 +01:00
Ekaterina Vaartis
0cf3654907
Rework task indexing to share code with the main module
...
The code in the main module now scrubs new posts too
2022-06-29 20:48:29 +01:00
Ekaterina Vaartis
117f525fd6
Adjust content indexing to skip more unneeded stuff
2022-06-29 20:48:29 +01:00
Ekaterina Vaartis
14ef6ce80f
Mark only content as searchable for meilisearch
2022-06-29 20:48:29 +01:00
Ekaterina Vaartis
b3401ba7bd
Also index incoming federated posts
2022-06-29 20:48:29 +01:00
Ekaterina Vaartis
52a872432d
Make the chunk size smaller
2022-06-29 20:48:29 +01:00
Ekaterina Vaartis
a586ce0ddd
Use content instead of source and scrub it
2022-06-29 20:48:29 +01:00
Ekaterina Vaartis
c3a04166a0
Tweak search ordering to hopefully return newer results
2022-06-29 20:48:29 +01:00
