Skip to content

Commit

Permalink
Update record indexer README
Browse files Browse the repository at this point in the history
  • Loading branch information
barbarahui committed Jan 23, 2024
1 parent b93bc87 commit 406308e
Showing 1 changed file with 21 additions and 2 deletions.
23 changes: 21 additions & 2 deletions record_indexer/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ python index_templates/rikolti_template.py

This creates a template that will be used whenever an index with name matching `rikolti*` is added to the cluster.

## Run indexer
## Run indexer from command line

Create a new index for a collection and add it to the `rikolti-stg` alias:

Expand All @@ -22,10 +22,29 @@ Add the current stage index for a collection to the `rikolti-prd` alias:
python -m record_indexer.move_index_to_prod <collection_id>
```

Note that the index creation code enforces the existence of one stage index at a time. This means we can simply supply the collection ID as input and the process will move the current stage index to production.
## Indexer development using aws-mwaa-local-runner

See the Rikolti README page section on [Airflow Development](https://github.com/ucldc/rikolti/#airflow-development). In particular, make sure that indexer-related env vars are set as described there.

## Index lifecycle

The lifecycle of an index is as follows:

#### Create new index
1. Create a new index named `rikolti-{collection_id}-{version}`, where `version` is the current datetime).
2. Remove any existing indices for the collection from the `rikolti-stg` alias.
3. Add the new index to the `rikolti-stg` alias.
4. Delete any older unaliased indices, retaining the number of unaliased indices specified by `settings.INDEX_RETENTION`.

Note that the index creation code enforces the existence of one stage index at a time.

#### Move staged index to production
1. Identify the current stage index for the collection.
2. Remove any existing indices for the collection from the `rikolti-prd` alias.
3. Add the current stage index to the `rikolti-prd` alias. (This means that at this stage in the lifecycle, the index will be aliased to `rikolti-stg` and `rikolti-prd` at the same time.)

#### Delete old index
This happens during index creation (see step 4. above).



Expand Down

0 comments on commit 406308e

Please sign in to comment.