Snowplow Mini Setup

Hi, I am trying to setup Snowplow Mini on GCP and I am stuck at First time Usage > Generating a pair of read/write API keys for local Iglu schema registry

1.I have created an uuid v4 and added it in the Control pannel

I then clicked on my VM instance in GCP to open the command line and downloaded igluctl 0.70 from bintray. The following commands worked fine, until this one where I am stuck:
$ ./igluctl server keygen --vendor-prefix com.acme mini-address/iglu-server 53b4c441-84f7-467e-af4c-074ced53eb20

The api key is the same as the one added in 1.
What I am wondering mini-address is it supposed to be the project-id from google cloud or is that a domain name that I have to setup ?

1 Like

Hi @Elena_Paraschiv,

mini-address refers to public address of your VM instance, it isn’t GCP project id. It doesn’t have to be a domain name, it can be an IP address as well. If you followed our guide, your VM should have an external IP that is accessible publicly. You don’t have to setup a domain name to use your Mini.

Please let us know if we can clarify anything further.

Thank you @oguzhanunlu. That solved it ! :grinning:

1 Like

A few steps further down the line at Accessing the Elasticsearch API , I am stuck at the following commands

username:password are the ones I created in Control Pannel at First time usage > Connecting to the instance for the first time

I think the issue is because I did the optional step before Copying Iglu repo to SnowPlow mini

The first 2 commands worked fine ,
but I got stuck at this one:
/path/to/igluctl static push iglu-central/schemas http://[public_dns]/iglu-server 980ae3ab-3aba-4ffe-a3c2-3b2e24e2ffce --public

Hi @Elena_Paraschiv,

I’m glad to hear that first issue is resolved. Which version of Mini did you deploy? That’ll be useful for us when answering questions.

On the first screenshot, it is likely due to ES not being ready. Have you executed those cURL commands shortly after deploying? If not, I’d recommend restarting all services through control plane.

I also noticed that you used igluctl to push from a non-existing path, iglu-central/schemas, this is a path that doesn’t exist, as shown by ls command output. Note that the actual issue isn’t this since error message tells us that java isn’t found. Igluctl requires JRE to be installed. You can use JRE 8 or 9.

On non-existing path issue, if you clone iglu-central repository to the same path as igluctl, you’ll be able to execute that command, assuming you installed JRE.

Please let us know if we can clarify anything further!

Hey @oguzhanunlu. Thanks for that. I managed to get over those steps with your help.
I am stuck at Accessing Elasticsearch API

I don’t get the 200 OK response

The version of mini deployed is snowplow-mini-0-6-1-large . same as in docs

Hey @Elena_Paraschiv,

It might be due to ES not being ready right after a reboot/restart. Could you restart all services from control plane and hit the same endpoint after a 5-6 seconds?

If that doesn’t work, you can either re-deploy not to investigate it further or ssh into the box and investigate elasticsearch more closely. I’d recommend a re-deploy to save time.

Please let us know how it goes!

Hi @oguzhanunlu , sorry for the late reply. I have been trying multiple times to start afresh, but facing new issues while setting it up from scratch. :cry:

I am just following the steps in the [configuration] , might be the commands there need to be updated (, now I get this

At usage, step 2: Generate pair of read/write API keys
I added the UUID in Iglu ApiKey in control pannel .
The version I am using is snowplow-mini-0-6-1

Hi @Elena_Paraschiv,

I noticed that you are following Mini 0.10.0 documentation but you are using Mini 0.6.1. Please follow 0.6.1 documentation on a fresh deployment and let us know how it goes!

Hi @Elena_Paraschiv,

I just noticed that we have a typo at 0.6.1 documentation at generating api key section.

First step is written as

Navigate to http:///iglu-server

whereas it should be

Navigate to http://mini-address/iglu-server

We’ll fix this typo shortly but I wanted to share with you in case you read it before we fix it.

Thanks for help @oghuzhanunlu. I will try the docs and let you know.

Btw on 0.6.1 the [images are not working] tried on different browser also ( ->

Also at documentation for 0-10-1 0-6-1 version is referenced

Thanks for the feedback @Elena_Paraschiv!

You can see hyperlinks of Mini 0.10.1 images right below the example command.

You’re right that example should also use 0.10.1. For your easy reference, 0.10.1-large image is here.

The fix for 0.6.1 images will be live shortly. Thanks for sharing Elena!

Please keep us posted.

Regarding 0.6.1 generating pair of read/write API keys I don’t understand steps 2 to Try it out!

I added the API key at Authorize button . Do I need to open the POST/api/schemas afterwards? What should I write on those fields ?

Hi @Elena_Paraschiv,

After adding your api key, you should unfold POST /api/auth/keygen section and enter a vendor_prefix, e.g. com.acme. Iglu Server will generate a pair of read-write keys that can be used for schemas whose vendor starts with vendor_prefix.

What you unfolded is another Iglu Server endpoint and that endpoint is used to upload individual schemas to Iglu Server. In short, you provide information about your schema such as vendor, name etc. and then upload it.

Please let us know how it goes!

I added the api key in the Control Pannel, then at /iglu-server top right box and then at api/auth/keygen and clicked ‘Execute’ . It doesn’t seem to return something I get 'Type Error: Could not connect to server ’

Hey @Elena_Paraschiv,

Did you notice the ip address in request URL? It is, not your Mini’s ip address. Please follow before using Swagger UI.

Here are the steps

I explicitly shared the link to Iglu Server config file for Mini 0.6.1 since I noticed that documentation points to config file on master branch, which points to configuration of Iglu Server used by later Mini versions, not 0.6.1. We’ll address this shortly as well.

Thanks for sharing your progress Elena, I appreciate that! Please let us know how it goes!

Thank you so much for your patience @oguzhanunlu. I managed to get through this step :partying_face:

But when I try to run the elastic search steps, I get 502 bad gateway

When I write the curl command my connection is timed out , and in kibana I get this error. Is that another cofiguration I need to do ? I see http:elasticsearch:9200 instead of my mini ip.

Glad to hear that @Elena_Paraschiv!

No, you don’t need another configuration. The reason you see that screen is you connected to Kibana right after you rebooted services or restarted the machine. It takes some time for Elasticsearch & Kibana to connect to each other. If you wait ~5 sec or so and refresh the page, you should see everything well.

Thanks @oguzhanunlu . I tried to wait multiple times, but it didn’t work. Took a video recording of it I might be missing something else . When I run the curl it returns <a href="/elasticsearch/">Moved Permanently</a>.

I found this case which seems similar to mine where setting an elasticsearch.yml is mentioned, but didn’t find that in 0.6.1 docs.

  • also found this tutorial>step 1:3 mentions some configuration in kibana, but port : 5601 doesn’t work for me

It seems Elasticsearch isn’t healthy. One common scenario is launching Mini in a machine with low amount of memory so that Elasticsearch can’t launch fine. Which machine type are you using?

If it isn’t about memory size, then I see 2 options:

  • Re-deploy Mini not to investigate Elasticsearch further
  • SSH into Mini instance and check Elasticsearch container logs

I think 1st option would be a lot faster.

1 Like