ElasticSearch loader: date index

The ES loader accepts a static string as an index name in the .conf file. This works, of course, but makes it more challenging to rotate data than if the date could be appended to the index name. Is there a way to get the current system time from the environment and append it in real time? I’m already using env vars for the environment and type (‘good’ or ‘bad’): {ENVIRONMENT}"-enriched-"{RECORD_STATUS} and this works great. It’s pulling values from ECS via Terraform tfvars and/or docker-compose. Now I’m trying to encapsulate a simple date function in an additional env var interpolation, something like ${LocalDate.now()} (to try to eval a Java func) or ${(date)} (Linux date) but neither works. This may be a long shot to begin with because I’m guessing the loader app config is only evaluated once at startup, but maybe that’s enough to sneak an expression in for recurring evaluation?

Just curious if anyone else has tried this and succeeded. It means the difference between dropping an index wholesale (a simple DELETE call or AWS ES ISM policy) and deleting records by querying and comparing derived_tstamp, which is somewhat significant.

Hi @seanhall,

It’s not possible to do such thing.

To handle the rotation of indices, you might want to take a look at index lifecycle management and/or Curator.

Thanks for confirming. I’ve looked at both of those, and they both work best with a date appended to the index; otherwise, as is, the loader will put all records to ‘enriched-good’ and Curator will delete my entire ‘enriched-good’ index or move it to colder storage.

Hi @seanhall the way we handle this internally is by using Elasticsearch Index aliasing. To leverage this you would setup:

  1. Daily indexes for good & bad data (with the data appended to the end)
  2. Alias which is updated after each rotation to point to the latest days index
  • In the loader you use the alias as your index to load into

This means that the ES Loader will point to whatever index you want it too and you can rotate indexes at whatever interval you would like as the loader is just pointing to the alias. One important note is that your alias should only ever point to one index to avoid issues loading data.

Hope this helps!