Mutator Exception Adding Custom Schema

For engineers
1

Hi Guys, I’ve been working with the implementation in GCP for snowplow, and so far so good except adding new Schema.

I created a schema and submitted it. Then after I run:

sudo docker run \
    -v $PWD:/snowplow \
    snowplow/snowplow-bigquery-mutator:1.2.0 \
    add-column \
    --config=$(cat config.hocon | base64 -w 0) \
    --resolver=$(cat resolver.json | base64 -w 0) \
	--shred-property=CONTEXTS \
	--schema="iglu:com.acme/ad_click/jsonschema/1-0-0"

Exception:

[ioapp-compute-0] INFO com.snowplowanalytics.snowplow.storage.bigquery.mutator.Main - load_tstamp already exists
com.snowplowanalytics.snowplow.storage.bigquery.mutator.Mutator$MutatorError
        at com.snowplowanalytics.snowplow.storage.bigquery.mutator.Mutator$.com$snowplowanalytics$snowplow$storage$bigquery$mutator$Mutator$$fetchError(Mutator.scala:167)
        at com.snowplowanalytics.snowplow.storage.bigquery.mutator.Mutator.$anonfun$getSchema$1(Mutator.scala:105)
        at cats.data.EitherT.$anonfun$bimap$1(EitherT.scala:383)
        at timeout @ com.snowplowanalytics.snowplow.storage.bigquery.mutator.Mutator.getSchema(Mutator.scala:105)
        at map @ com.snowplowanalytics.iglu.client.Client$.parseDefault(Client.scala:58)
        at flatMap @ com.snowplowanalytics.snowplow.storage.bigquery.mutator.Mutator$.initialize(Mutator.scala:157)
        at flatMap @ com.snowplowanalytics.snowplow.storage.bigquery.mutator.Mutator.getSchema(Mutator.scala:110)
        at flatMap @ com.snowplowanalytics.snowplow.storage.bigquery.mutator.Mutator.addField(Mutator.scala:73)
        at map @ com.snowplowanalytics.snowplow.storage.bigquery.mutator.Main$.$anonfun$run$24(Main.scala:66)
        at delay @ org.typelevel.log4cats.slf4j.internal.Slf4jLoggerInternal$Slf4jLogger.$anonfun$info$4(Slf4jLoggerInternal.scala:91)
        at delay @ org.typelevel.log4cats.slf4j.internal.Slf4jLoggerInternal$Slf4jLogger.isInfoEnabled(Slf4jLoggerInternal.scala:66)
        at ifM$extension @ org.typelevel.log4cats.slf4j.internal.Slf4jLoggerInternal$Slf4jLogger.info(Slf4jLoggerInternal.scala:91)
        at apply @ com.snowplowanalytics.snowplow.storage.bigquery.mutator.TableReference$BigQueryTable.getTable(TableReference.scala:39)
        at flatMap @ com.snowplowanalytics.snowplow.storage.bigquery.mutator.TableReference$BigQueryTable.getTable(TableReference.scala:39)
        at map @ com.snowplowanalytics.snowplow.storage.bigquery.mutator.TableReference$BigQueryTable.getFields(TableReference.scala:45)
        at flatMap @ com.snowplowanalytics.snowplow.storage.bigquery.mutator.Mutator.addField(Mutator.scala:84)
        at liftF @ com.snowplowanalytics.iglu.client.resolver.Resolver$.$anonfun$parse$3(Resolver.scala:269)
        at map @ com.snowplowanalytics.iglu.client.Client$.parseDefault(Client.scala:58)
        at map @ com.snowplowanalytics.iglu.client.Client$.parseDefault(Client.scala:58)

The resolver is configured like this:

{
  "schema": "iglu:com.snowplowanalytics.iglu/resolver-config/jsonschema/1-0-1",
  "data": {
    "cacheSize": 500,
    "repositories": [
      {
        "name": "Iglu Central",
        "priority": 0,
        "vendorPrefixes": [ "com.snowplowanalytics" ],
        "connection": {
          "http": {
            "uri": "http://iglucentral.com"
          }
        }
      },
      {
        "name": "Custom Iglu Server",
        "priority": 1,
        "vendorPrefixes": [ "com.acme" ],
        "connection": {
          "http": {
            "uri": "http://10.156.0.48:8080",
            "apikey": "XXXXXXXX-b4b9-11ec-b909-XXXXXXXX"
          }
        }
      }
    ]
  }
}

Just in case I executed:

curl -i -H "Accept: application/json" -H "apikey: XXXXXXXX-b4b9-11ec-b909-XXXXXXXX" -X GET http://10.156.0.48:8080/api/schemas

The connection was successful so is not an issue of conectivity

Do you know where this could come from, and is there a way to see the logs?

Thank you!

2

Hi @FabianEpic in your Iglu Resolver I believe the URI for the Iglu Server you have deployed should be:

http://10.156.0.48:8080/api

instead of:

http://10.156.0.48:8080

Could you give that a try?

1 Like
3

It worked!! Thank you so much Josh! Been struggling for hours! Viva la snowplow!

1 Like