Hi!
I am having trouble loading my enrich events to my aws es cluster.
configuration error:
Cannot convert 'ENRICHED_EVENTS' to com.snowplowanalytics.stream.loader.Config$StreamType: . at 6
configuration error:
i login to docker to try to refactor my in_stream_type and match what’s decalared from tf.vars but I can’t find the stream type parameter from es-loader config.
Hi Josh, it is 1.0.0
Hi @alvin the latest version of the loader is 2.0.9 which is likely what you should be using instead! The config option for “ENRICHED_EVENTS” is only the latest versions of the loader so it is likely you have a config compatible with v2.x.x but are trying to use it with v1.x.x.
Could you try with a later version and see if that works? If need examples of configs they can be found here: snowplow-elasticsearch-loader/config at master · snowplow/snowplow-elasticsearch-loader · GitHub
Hi Josh,
Yes, I did try to upgrade my Elasticsearch loader version to 0.2.0 and 0.3.0 from the module available at Terraform Registry. However, for some reason, these two latest versions are throwing version constraints issues when I run terraform init -upgrade. Even if I update versions.tf to the latest hashicorp/aws version, the issue persists. Therefore, I reverted back to the older version of 0.1.0.
Please see the attached screenshot for more information.
For reference, I used the exact module on my infrastructure from quickstart-examples/terraform/aws/pipeline/secure at main · snowplow/quickstart-examples · GitHub.
Hi @alvin in the quickstart modules you need to change the version constraint from ~> 3.45.0 to ~> 3.75.0 which should resolve the issue. None of the modules should have a hard fixed version.
Hi Josh,
I was able to change the version constraint from ~> 3.45.0 to ~> 3.75.0 in the QuickStart module’s lock file, and it worked well for the es loader versions 0.1.0 and 0.1.1. terraform init was successful.
However, when I modified the es loader version to 0.2.0 and 0.3.0, I still received an error for the constraints.
Hi @alvin what is the latest commit hash of the quickstart module you have cloned? You still appear to have an AWS constraint of 3.45.0 in there which is not coming from the modules themselves so figuring out where that is coming from is the only real thing to debug here!
@alvin I think I know the module to blame here - it looks like the Snowflake Loader module has had its constraint setup incorrectly. If you are not using that module you should be able to just delete it which will let the ES Loader setup progress correctly. Ill work on getting that fixed as soon as possible!
Hi @josh !
Thank you for the support. I appreciate it!
I tried to remove all snowflake loaders from my current module. Unfortunately, it is still throwing a version constraint error.
Could not retrieve the list of available versions for provider hashicorp/aws: no available releases match the given constraints >= 3.25.0, >= 3.41.0, >= 3.45.0, ~> 3.45.0, >= 3.75.0
I removed all snowflake components under \pipeline\.terraform\modules, main.tf, and variables.tf. I tried both constraints from .terraform.lock.hcl: constraints = ">= 3.25.0, ~> 3.45.0" and constraints = ">= 3.45.0, ~> 3.75.0". I used 0.3.0 (latest) elasticsearch-loader-kinesis-ec2 module for this.
Hi @alvin can you please provide all the Terraform you are trying to execute so this can be debugged? I cannot recreate your issue so its likely I am missing some crucial context here.
It might be easiest at this point to setup a gist / simple github repo with your terraform so I can take a look and get this unblocked for you.