Run bigquery-loader-cli

For engineers Storage targets
1

I came across to this article .
I am running this command: ./bigquery-loader-cli-0.1.0 --create-table “moonshot-1349” “dataset_test_1” “table_test_1” part-00000.gzip

And it throws following error:
FileNotFoundException: java.io.FileNotFoundException: snowplow_bigquery_refresh_token.properties (No such file or directory)
Exception in thread “main” java.lang.NullPointerException
at com.snowplowanalytics.snowplow.bigquery.loader.BigqueryInterface.getCredentials(BigqueryInterface.scala:155)
at com.snowplowanalytics.snowplow.bigquery.loader.BigqueryInterface.loadCredentials(BigqueryInterface.scala:135)
at com.snowplowanalytics.snowplow.bigquery.loader.BigqueryInterface.(BigqueryInterface.scala:110)
at com.snowplowanalytics.snowplow.bigquery.loader.BigqueryLoaderCli$.main(BigqueryLoaderCli.scala:71)
at com.snowplowanalytics.snowplow.bigquery.loader.BigqueryLoaderCli.main(BigqueryLoaderCli.scala)

Am I missing something?

2

Did you set up credentials/clientId in the Google console?

The cli uses the credentials to generate a token to store in snowplow_bigquery_refresh_token.properties

3

Could you help me with how to setup credentials/clientId in google console?

4

Sure, read the instructions in the README:

5

Hi Shin,

I have already finished that process and downloaded json file and renamed it to client_secrets_.json.

6

The instructions actually say to rename it to client_secrets_<projectId>.json

7

I created another client id with application type others. So it;s working now.
It created table in my dataset …But throws following error:
WARNING: Application name is not set. Call Builder#setApplicationName.
IOException while creating dataset_test_1: com.google.api.client.googleapis.json.GoogleJsonResponseException: 409 Conflict
{
“code” : 409,

“errors” : [ {
“domain” : “global”,
“message” : “Already Exists: Dataset moonshot-1349:dataset_test_1”,
“reason” : “duplicate”
} ],
“message” : “Already Exists: Dataset moonshot-1349:dataset_test_1”
}
Table created
Exception in thread “main” java.nio.charset.MalformedInputException: Input length = 1
at java.nio.charset.CoderResult.throwException(CoderResult.java:281)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:339)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
at java.io.InputStreamReader.read(InputStreamReader.java:184)
at java.io.BufferedReader.fill(BufferedReader.java:161)
at java.io.BufferedReader.readLine(BufferedReader.java:324)
at java.io.BufferedReader.readLine(BufferedReader.java:389)
at scala.io.BufferedSource$BufferedLineIterator.hasNext(BufferedSource.scala:67)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
at scala.collection.mutable.ListBuffer.$plus$plus$eq(ListBuffer.scala:176)
at scala.collection.mutable.ListBuffer.$plus$plus$eq(ListBuffer.scala:45)
at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:273)
at scala.collection.AbstractIterator.to(Iterator.scala:1157)
at scala.collection.TraversableOnce$class.toList(TraversableOnce.scala:257)
at scala.collection.AbstractIterator.toList(Iterator.scala:1157)
at com.snowplowanalytics.snowplow.bigquery.loader.TsvParser$.addFieldsToData(TsvParser.scala:48)
at com.snowplowanalytics.snowplow.bigquery.loader.BigqueryLoaderCli$.com$snowplowanalytics$snowplow$bigquery$loader$BigqueryLoaderCli$$sendBatch$1(BigqueryLoaderCli.scala:135)
at com.snowplowanalytics.snowplow.bigquery.loader.BigqueryLoaderCli$$anonfun$uploadData$1.apply(BigqueryLoaderCli.scala:152)
at com.snowplowanalytics.snowplow.bigquery.loader.BigqueryLoaderCli$$anonfun$uploadData$1.apply(BigqueryLoaderCli.scala:152)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:105)
at com.snowplowanalytics.snowplow.bigquery.loader.BigqueryLoaderCli$.uploadData(BigqueryLoaderCli.scala:152)
at com.snowplowanalytics.snowplow.bigquery.loader.BigqueryLoaderCli$.main(BigqueryLoaderCli.scala:75)
at com.snowplowanalytics.snowplow.bigquery.loader.BigqueryLoaderCli.main(BigqueryLoaderCli.scala)