From 87c9ffa303304b1ddabe70ea5eeb3dd7ef6528d0 Mon Sep 17 00:00:00 2001 From: Daniel Berkompas Date: Fri, 3 Nov 2017 15:50:38 -0700 Subject: [PATCH] Improve README --- README.md | 196 +++++++++++++++++++++++++++------ lib/mix/elasticsearch.build.ex | 18 +++ 2 files changed, 180 insertions(+), 34 deletions(-) diff --git a/README.md b/README.md index 7695c29..0ecb7f4 100644 --- a/README.md +++ b/README.md @@ -4,6 +4,8 @@ A simple, no-nonsense Elasticsearch library for Elixir. Highlights include: - **No DSLs.** Interact directly with the `Elasticsearch` JSON API. - **Zero-downtime index (re)building.** Via `Mix.Tasks.Elasticsearch.Build` task. +- **Dev Tools**. Helpers for runnig Elasticsearch as part of your supervision + tree during development. ## Installation @@ -41,30 +43,28 @@ config :elasticsearch, # Elasticsearch time to catch up. bulk_wait_interval: 15_000, # 15 seconds - # This loader module must implement the Elasticsearch.DataLoader - # behaviour. It will be used to fetch data for each source in each - # indexes' `sources` list, below: - loader: MyApp.ElasticsearchLoader, - # If you want to mock the responses of the Elasticsearch JSON API # for testing or other purposes, you can inject a different module # here. It must implement the Elasticsearch.API behaviour. api_module: Elasticsearch.API.HTTP, # You should configure each index which you maintain in Elasticsearch here. + # This configuration will be read by the `mix elasticsearch.build` task, + # described below. indexes: %{ - # `:cities` becomes the Elixir name for this index, which you'll use in - # queries, etc. - cities: %{ - # This is the base name of the Elasticsearch index. Each index will be - # built with a timestamp included in the name, like "cities-5902341238". - # It will then be aliased to "cities" for easy querying. - alias: "cities", - + # This is the base name of the Elasticsearch index. Each index will be + # built with a timestamp included in the name, like "posts-5902341238". + # It will then be aliased to "posts" for easy querying. + posts: %{ # This file describes the mappings and settings for your index. It will # be posted as-is to Elasticsearch when you create your index, and # therefore allows all the settings you could post directly. - settings: "priv/elasticsearch/cities.json", + settings: "priv/elasticsearch/posts.json", + + # This loader module must implement the Elasticsearch.DataLoader + # behaviour. It will be used to fetch data for each source in each + # indexes' `sources` list, below: + loader: MyApp.ElasticsearchLoader, # This is the list of data sources that should be used to populate this # index. The `:loader` module above will be passed each one of these @@ -72,40 +72,168 @@ config :elasticsearch, # # Each piece of data that is returned by the loader must implement the # Elasticsearch.Document protocol. - sources: [Type1] + sources: [Post] } } ``` +## Protocols & Behaviours + +#### Elasticsearch.DataLoader + +Your app must provide a `Loader` module, which will fetch data to upload to +Elasticsearch. This module must implement the `Elasticsearch.DataLoader` +behaviour. + +```elixir +defmodule MyApp.ElasticsearchLoader do + @behaviour Elasticsearch.DataLoader + + @impl Elasticsearch.DataLoader + def load(MyApp.Post, offset, limit) do + # Return MyApp.Posts, restricted by offset and limit + end +end +``` + +#### Elasticsearch.Document + +Each result returned by your loader must implement the `Elasticsearch.Document` +protocol. + +```elixir +defimpl Elasticsearch.Document, for: MyApp.Post do + def id(post), do: post.id + def type(_post), do: "post" + def parent(_post), do: false + def encode(post) do + %{ + title: post.title, + author: post.author + } + end +end +``` + +#### Elasticsearch.API + +You can plug in a different module to make API requests, as long as it +implements the `Elasticsearch.API` behaviour. + +This can be used in test mode, for example: + +```elixir +# config/test.exs +config :elasticsearch, + api_module: MyApp.ElasticsearchMock +``` + +Your mock can then stub requests and responses from Elasticsearch. + +```elixir +defmodule MyApp.ElasticsearchMock do + @behaviour Elasticsearch.API + + def get("/posts/1", _headers, _opts) do + {:ok, %HTTPoison.Response{ + status_code: 404, + body: %{ + "status" => "not_found" + } + }} + end +end +``` + +## Indexing + +#### Bulk + +Use the `mix elasticsearch.build` task to build indexes using a zero-downtime, +hot-swap technique with Elasticsearch aliases. + +```bash +# This will read the `indexes[posts]` configuration seen above, to build +# an index, `posts-123123123`, which will then be aliased to `posts`. +$ mix elasticsearch.build posts +``` + +See the docs on `Mix.Tasks.Elasticsearch.Build` and `Elasticsearch.Builder` +for more details. + +#### Individual Documents + +Use `Elasticsearch.put_document/2` to upload a document to a particular index. + +```elixir +# MyApp.Post must implement Elasticsearch.Document +Elasticsearch.put_document(%MyApp.Post{}, "index-name") +``` + +To remove documents, use `Elasticsearch.delete_document/2`: + +```elixir +Elasticsearch.delete_document(%MyApp.Post{}, "index-name") +``` + ## Querying -You can query Elasticsearch using raw requests, or with the help of -the `Elasticsearch.Query` struct. +You can query Elasticsearch the `post/2` function: ```elixir # Raw query -Elasticsearch.post("/cities/city/_search", '{"query": {"match_all": {}}}') +Elasticsearch.post("/posts/post/_search", '{"query": {"match_all": {}}}') # Using a map -Elasticsearch.post("/cities/city/_search", %{"query" => %{"match_all" => %{}}}) - -# Using a query -query = %Elasticsearch.Query{ - indexes: [:cities], - types: [:city], - query: %{ - "query" => %{ - "match_all" => %{} - } - } -} - -Elasticsearch.execute(query) +Elasticsearch.post("/posts/post/_search", %{"query" => %{"match_all" => %{}}}) ``` -TODOS: +See the official Elasticsearch [documentation](https://www.elastic.co/guide/en/elasticsearch/reference/6.x/index.html) +for how to write queries. -- [ ] Write tests +## Dev Tools + +This package provides two utilities for developing with Elasticsearch: + +- `mix elasticsearch.install`: A mix task to install Elasticsearch and Kibana + to a folder of your choosing. + +- `Elasticsearch.Executable`. Use this to start and stop Elasticsearch as part + of your supervision tree. + + ```elixir + children = [ + worker(Elasticsearch.Executable, [ + "Elasticsearch", + "./vendor/elasticsearch/bin/elasticsearch", # assuming elasticsearch is in your vendor/ dir + 9200 + ]), + worker(Elasticsearch.Executable, [ + "Kibana", + "./vendor/kibana/bin/kibana", # assuming kibana is in your vendor/ dir + 5601 + ]) + ] + ``` + +## Documentation + +Run `mix docs` to generate local documentation. + +## Contributing + +To contribute code to this project, you'll need to: + +1. Fork the repo +2. Clone your fork +3. Run `bin/setup` +4. Create a branch +5. Commit your changes +6. Open a PR + +## Todos + +- [x] Write tests - [ ] Update documentation in `Elasticsearch` module - [ ] Update documentation in `mix elasticsearch.build` task - [ ] Document how to mock Elasticsearch for testing diff --git a/lib/mix/elasticsearch.build.ex b/lib/mix/elasticsearch.build.ex index 5f9c875..724de25 100644 --- a/lib/mix/elasticsearch.build.ex +++ b/lib/mix/elasticsearch.build.ex @@ -1,6 +1,24 @@ defmodule Mix.Tasks.Elasticsearch.Build do @moduledoc """ Builds Elasticsearch indexes using a zero-downtime, hot-swap technique. + + 1. Build an index for the given `alias`, with a timestamp: `alias-12323123` + 2. Bulk upload data to that index using `loader` and `sources`. + 3. Alias the `alias` to `alias-12323123`. + 4. Remove old indexes beginning with `alias`. + 5. Refresh `alias-12323123`. + + For a functional version of this approach, see + `Elasticsearch.Builder.hot_swap_index/4`. + + ## Example + + $ mix elasticsearch.build posts {index2} {index3} + + To build an index only if it does not exist, use the `--existing` option: + + $ mix elasticsearch.build posts --existing + Index posts already exists. """ require Logger