Hi, I am trying to do manual setup on GCP using the steps provided on the documentation. My enrich process is struck in initialization process. Request help in identifying what’s wrong? All the manual setups are old(2 years+) and do not confirm to the documentation on snowplow anymore where dataflow processes are deprecated. Any newer step by step implementation would be useful.
admin_@cloudshell:~/snowplow (maurya-agrolis)$ cat resolver.json
{
“schema”: “iglu:com.snowplowanalytics.iglu/resolver-config/jsonschema/1-0-1”,
“data”: {
“cacheSize”: 500,
“repositories”: [
{
“name”: “Iglu Central”,
“priority”: 0,
“vendorPrefixes”: [ “com.snowplowanalytics” ],
“connection”: {
“http”: {
“uri”: “http://iglucentral.com”
}
}
},
{
“name”: “Iglu Central - GCP Mirror”,
“priority”: 1,
“vendorPrefixes”: [ “com.snowplowanalytics” ],
“connection”: {
“http”: {
“uri”: “http://mirror01.iglucentral.com”
}
}
}
]
}
admin_@cloudshell:~/snowplow (maurya-agrolis)$ cat config*
{
“input”: {
“subscription”: “projects/maurya-agrolis/subscriptions/snowplow-good-sub”
}
“output”: {
“good”: {
“topic”: “projects/maurya-agrolis/topics/snowplow-enriched”
}
"bad": {
"topic": "projects/maurya-agrolis/topics/snowplow-bad"
}
}
}cat: configs: Is a directory
admin_@cloudshell:~/snowplow (maurya-agrolis)$ docker run
-it --rm
-v $PWD:/snowplow
-e GOOGLE_APPLICATION_CREDENTIALS=/snowplow/maurya-agrolis-78651008d53d.json
snowplow/snowplow-enrich-pubsub:3.7.0
–enrichments /snowplow/enrichments
–iglu-config /snowplow/resolver.json
–config /snowplow/config.hocon
Unable to find image ‘snowplow/snowplow-enrich-pubsub:3.7.0’ locally
3.7.0: Pulling from snowplow/snowplow-enrich-pubsub
7608715873ec: Pull complete
0587d987c598: Pull complete
13b082a24c8f: Pull complete
813a8123a4ba: Pull complete
3ffd57121cd5: Pull complete
4f4fb700ef54: Pull complete
409e21c2b908: Pull complete
1c5d8aa32dd1: Pull complete
04206ee21b1a: Pull complete
Digest: sha256:88299918e6e70a37d437f5d6d8288556e633ec0b6cb215b9c8d776075db884a1
Status: Downloaded newer image for snowplow/snowplow-enrich-pubsub:3.7.0
[pool-1-thread-1] INFO com.snowplowanalytics.snowplow.enrich.common.fs2.io.FileSystem - Files found in /snowplow/enrichments:
[pool-1-thread-1] INFO com.snowplowanalytics.snowplow.enrich.common.fs2.config.ParsedConfigs - Parsed config file: ConfigFile(PubSub(projects/maurya-agrolis/subscriptions/snowplow-good-sub,1,3000,50000000,1 hour),Outputs(PubSub(projects/maurya-agrolis/topics/snowplow-enriched,None,200 milliseconds,1000,8000000),None,PubSub(projects/maurya-agrolis/topics/snowplow-bad,None,200 milliseconds,1000,8000000)),Concurrency(256,3),None,RemoteAdapterConfigs(10 seconds,45 seconds,10,List()),Monitoring(None,MetricsReporters(None,None,false)),Telemetry(false,15 minutes,POST,collector-g.snowplowanalytics.com,443,true,None,None,None,None,None),FeatureFlags(false,false),None)
[pool-1-thread-1] INFO com.snowplowanalytics.snowplow.enrich.common.fs2.config.ParsedConfigs - Parsed Iglu Client with following registries: Iglu Central, Iglu Central - GCP Mirror
[pool-1-thread-1] INFO com.snowplowanalytics.snowplow.enrich.common.fs2.config.ParsedConfigs - Parsed following enrichments:
[pool-1-thread-1] INFO com.snowplowanalytics.snowplow.enrich.common.fs2.Run - Initialising resources for snowplow-enrich-pubsub 3.7.0
[pool-1-thread-1] INFO com.snowplowanalytics.snowplow.enrich.common.fs2.Environment - Running Enrich
[pool-1-thread-1] INFO com.snowplowanalytics.snowplow.enrich.common.fs2.Assets - Initializing (downloading) enrichments assets
I also notice the messages are getting to snowplow-bad topic. I think the resolver.json may need some fixing here. I do not intend to do any enrichments for now so have left the enrichments directory empty.
This was the test message I sent from my local desktop ==> curl -d “&e=pv&page=curl-test&url=http%3A%2F%2Fjust-testing.com&aid=snowplow-good” -X POST http://34.123.26.180:8080/com.snowplowanalytics.iglu/v1