Running into some issues trying to test out a custom unstruct event that we are trying to build into our pipeline. I am pretty sure I have the schema built ok, and I pushed it to my iglu server with the igluctl tool, but when I try to run the mutator it always errors out with the same unclear message.
Any thoughts from someone more succesful in adding custom events?
Here is my workflow:
Check iglu for my schema
$ curl <GCP.iglu.server.URL>:8080/api/schemas/com.acme_company/viewed_product/jsonschema/2-0-0 -X GET -H "apikey: ########-####-####-####-############" | json_pp
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 729 100 729 0 0 4418 0 --:--:-- --:--:-- --:--:-- 4418
{
"description" : "Schema for Custom Dimension Test",
"type" : "object",
"$schema" : "http://iglucentral.com/schemas/com.snowplowanalytics.self-desc/schema/jsonschema/1-0-0#",
"properties" : {
"contentParagraphCount" : {
"type" : "string"
},
"premiumStatus" : {
"type" : "boolean"
},
"videoTitle" : {
"type" : "string"
},
"contentType" : {
"type" : "string"
},
"domain" : {
"type" : "string"
},
"contentWordCount" : {
"type" : "integer"
},
"contentAuthor" : {
"type" : "string"
},
"premiumEndDate" : {
"type" : "string"
},
"contentPubHour" : {
"type" : "string"
},
"contentId" : {
"type" : "string"
},
"contentPubDate" : {
"type" : "string"
},
"contentLastModifiedDate" : {
"type" : "string"
}
},
"additionalProperties" : false,
"self" : {
"version" : "2-0-0",
"format" : "jsonschema",
"name" : "viewed_product",
"vendor" : "com.acme_company"
}
}
Looks good (AFAIK) but could it have a problem with the version being 2-0-0? I have an identical schema with v.1-0-0 but I am using the 2-0-0 to match the tracker the web dev built on the test website. I wouldn’t expect the version to cause it to crap out.
Lets try out adding these fields to BQ from my local machine (mac) w/ docker desktop running:
$ docker run \
-v /snowplow/config:/snowplow/config \
-e GOOGLE_APPLICATION_CREDENTIALS=/snowplow/config/key.json \
snowplow/snowplow-bigquery-mutator:1.3.0-distroless \
add-column \
--config $(cat /snowplow/config/base64/sp-bigquery-streamloader-config_b64) \
--resolver $(cat /snowplow/config/base64/iglu_resolver_b64) \
--shred-property=CONTEXTS \
--schema="iglu:com.acme_company/viewed_product/jsonschema/2-0-0"
[ioapp-compute-0] INFO com.snowplowanalytics.snowplow.storage.bigquery.mutator.Main - load_tstamp already exists
com.snowplowanalytics.snowplow.storage.bigquery.mutator.Mutator$MutatorError
at com.snowplowanalytics.snowplow.storage.bigquery.mutator.Mutator$.com$snowplowanalytics$snowplow$storage$bigquery$mutator$Mutator$$fetchError(Mutator.scala:167)
at com.snowplowanalytics.snowplow.storage.bigquery.mutator.Mutator.$anonfun$getSchema$1(Mutator.scala:105)
at cats.data.EitherT.$anonfun$bimap$1(EitherT.scala:383)
at timeout @ com.snowplowanalytics.snowplow.storage.bigquery.mutator.Mutator.getSchema(Mutator.scala:105)
at map @ com.snowplowanalytics.iglu.client.Client$.parseDefault(Client.scala:58)
at flatMap @ com.snowplowanalytics.snowplow.storage.bigquery.mutator.Mutator$.initialize(Mutator.scala:157)
at flatMap @ com.snowplowanalytics.snowplow.storage.bigquery.mutator.Mutator.getSchema(Mutator.scala:110)
at flatMap @ com.snowplowanalytics.snowplow.storage.bigquery.mutator.Mutator.addField(Mutator.scala:73)
at map @ com.snowplowanalytics.snowplow.storage.bigquery.mutator.Main$.$anonfun$run$24(Main.scala:66)
at delay @ org.typelevel.log4cats.slf4j.internal.Slf4jLoggerInternal$Slf4jLogger.$anonfun$info$4(Slf4jLoggerInternal.scala:91)
at delay @ org.typelevel.log4cats.slf4j.internal.Slf4jLoggerInternal$Slf4jLogger.isInfoEnabled(Slf4jLoggerInternal.scala:66)
at ifM$extension @ org.typelevel.log4cats.slf4j.internal.Slf4jLoggerInternal$Slf4jLogger.info(Slf4jLoggerInternal.scala:91)
at apply @ com.snowplowanalytics.snowplow.storage.bigquery.mutator.TableReference$BigQueryTable.getTable(TableReference.scala:39)
at flatMap @ com.snowplowanalytics.snowplow.storage.bigquery.mutator.TableReference$BigQueryTable.getTable(TableReference.scala:39)
at map @ com.snowplowanalytics.snowplow.storage.bigquery.mutator.TableReference$BigQueryTable.getFields(TableReference.scala:45)
at flatMap @ com.snowplowanalytics.snowplow.storage.bigquery.mutator.Mutator.addField(Mutator.scala:84)
at liftF @ com.snowplowanalytics.iglu.client.resolver.Resolver$.$anonfun$parse$3(Resolver.scala:269)
at map @ com.snowplowanalytics.iglu.client.Client$.parseDefault(Client.scala:58)
at map @ com.snowplowanalytics.iglu.client.Client$.parseDefault(Client.scala:58)
Hmm, no good.
Similar error when I typed the schema name wrong for a snowplow provided schema I added for testing’s sake (ad_click if you are wondering)
Lets take a look at my iglu_resolver just in case (not sure its relevant but…)
{
"schema": "iglu:com.snowplowanalytics.iglu/resolver-config/jsonschema/1-0-1",
"data": {
"cacheSize": 500,
"repositories": [
{
"name": "Iglu Central",
"priority": 0,
"vendorPrefixes": [ "com.snowplowanalytics" ],
"connection": {
"http": {
"uri": "http://iglucentral.com"
}
}
},
{
"name": "Custom Iglu Server",
"priority": 1,
"vendorPrefixes": [ "com.snowplowanalytics" ],
"connection": {
"http": {
"uri": "GCP.load.balancer.IP",
"apikey": "APIKeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeey"
}
}
}
]
}
}
I have tried the uri with /api appended and it always gives the same error.
I have not been able to try :8080/api because of the illegal char error with decoding a colon
