Bigquery mutator create command error on GCP VM

For engineers GCP pipeline
1

Hi there,

I’m following Simo’s Ahava guide to setup Snowplow in GCP. I’m almost there but I have some issues when trying to setup a vm instance for the ETL process.

From what I understand I will need to add the following command in my startup script:

java -jar snowplow-bigquery-mutator-1.0.1.jar create --config $(cat bigquery_config.json | base64 -w 0) --resolver $(cat iglu_resolver.json | base64 -w 0)

I’ve tried to run myself for testing purpose and it returns me this error:

Attempt to decode value on failed cursor: DownField(projectId)
DecodingFailure at .http.uri: Iglu Client Failure. Provided URI string violates RFC 2396: [Illegal character in pat
h at index 1: ${iglu_server_url}]

The bigquery_config.json file seems to be well copied on my vm as well as the snowplow-bigquery-mutator-1.0.1.jar file.

Here is my bigquery_config.json file:

{
    "schema": "iglu:com.snowplowanalytics.snowplow.storage/bigquery_config/jsonschema/1-0-0",
    "data": {
        "name": "Snowplow Page View Data",
        "id": "b19fea0e-380c-450c-ba33-d3b507173e25",

        "projectId": "myprojectid,
        "datasetId": "mydataset",
        "tableId": "pageviews",

        "input": "enriched-good-sub",
        "typesTopic": "bq-types",
        "typesSubscription": "bq-types-sub",
        "badRows": "bq-bad-rows",
        "failedInserts": "bq-failed-inserts",

        "load": {
            "mode": "STREAMING_INSERTS",
            "retry": false
        },

        "purpose": "ENRICHED_EVENTS"
    }
}

Thanks.

2

Hi @simonbreton, it looks like you’re using the old config style that was only valid for BigQuery loader versions earlier than 1.0.0. I suggest you continue to use version 1.0.1 of the loader, but update your config to the new format.

To help you out, you can take a look at the example configuration files in the github repo and also the configuration reference on our docs site. We also have an upgrade guide to help you adapt your configuration to the new style.

The new config style uses HOCON format instead of self-describing JSON, which we find to be more user-friendly for configuring applications, and also is more consistent with other snowplow applications.

1 Like