Bigquery-loader-cli --Exception in thread "main" java.util.NoSuchElementException

mongodb · August 16, 2016, 11:59am

I am following https://github.com/snowplow/bigquery-loader-cli documentation and got success to create a table, but getting error while trying to upload data in same table.

Below error I am getting

WARNING: Application name is not set. Call Builder#setApplicationName.
Exception in thread "main" java.util.NoSuchElementException: None.get
	at scala.None$.get(Option.scala:313)
	at scala.None$.get(Option.scala:311)
	at com.snowplowanalytics.snowplow.bigquery.loader.BigqueryLoaderCli$.main(BigqueryLoaderCli.scala:73)
	at com.snowplowanalytics.snowplow.bigquery.loader.BigqueryLoaderCli.main(BigqueryLoaderCli.scala)

And I have tried multiple command to make it run and all command give same above error

java -jar bigquery-loader-cli-0.1.0 snowplow-project-id atomic events ./run=2016-08-16-09-22-28-./part-00000

java -jar bigquery-loader-cli-0.1.0 snowplow-project-id atomic events ./run=2016-08-16-09-22-28-./

Is anything I missed here? Please suggest.

mongodb · August 16, 2016, 3:51pm

can anyone help me to solve this error?

alex · August 16, 2016, 6:21pm

That BigQuery loader is not actively supported by Snowplow, but based on that error, you’re missing the project ID:

github.com

snowplow/bigquery-loader-cli/blob/master/src/main/scala/com.snowplowanalytics.snowplow.storage/BigqueryLoaderCli.scala#L71






def main (args: Array[String]) {
  
  try{ 


    parser.parse(args)


    //initializes, checks for authorization and if not authorized 
    //starts authorization process
    val bigqueryInterface = new BigqueryInterface(projectId.value.get)


    if (createTable.value.get) createDatasetAndTable(projectId.value.get, datasetId.value.get, tableId.value.get, bigqueryInterface)


    uploadData(projectId.value.get, datasetId.value.get, tableId.value.get, tsvFileLocation.value.get, bigqueryInterface)


  }
  catch{
    case e: ArgotUsageException => println(e.message) 
  }

mongodb · August 17, 2016, 7:49am

I have followed this format

java -jar bigquery-loader-cli-0.1.0 < projectId > < datasetId > < tableId > < dataLocation >

Here what I have passed

java -jar bigquery-loader-cli-0.1.0 snowplow-project-id atomic events ./run=2016-08-16-09-22-28-./part-00000

So, project id is already there. Any other thing I missed?

mongodb · August 17, 2016, 10:24am

Well, I removed old dataset and created another dataset. This time I got different error.

Dataset created
Table created
Exception in thread “main” java.lang.Error: There seems to have been a parsing error
at com.snowplowanalytics.snowplow.bigquery.loader.TsvParser$.getValues(TsvParser.scala:63)
at com.snowplowanalytics.snowplow.bigquery.loader.TsvParser$$anonfun$addFieldsToData$1.apply(TsvParser.scala:51)
at com.snowplowanalytics.snowplow.bigquery.loader.TsvParser$$anonfun$addFieldsToData$1.apply(TsvParser.scala:50)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.immutable.List.foreach(List.scala:318)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
at scala.collection.AbstractTraversable.map(Traversable.scala:105)
at com.snowplowanalytics.snowplow.bigquery.loader.TsvParser$.addFieldsToData(TsvParser.scala:50)
at com.snowplowanalytics.snowplow.bigquery.loader.BigqueryLoaderCli$.com$snowplowanalytics$snowplow$bigquery$loader$BigqueryLoaderCli$$sendBatch$1(BigqueryLoaderCli.scala:135)
at com.snowplowanalytics.snowplow.bigquery.loader.BigqueryLoaderCli$$anonfun$uploadData$1.apply(BigqueryLoaderCli.scala:152)
at com.snowplowanalytics.snowplow.bigquery.loader.BigqueryLoaderCli$$anonfun$uploadData$1.apply(BigqueryLoaderCli.scala:152)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:105)
at com.snowplowanalytics.snowplow.bigquery.loader.BigqueryLoaderCli$.uploadData(BigqueryLoaderCli.scala:152)
at com.snowplowanalytics.snowplow.bigquery.loader.BigqueryLoaderCli$.main(BigqueryLoaderCli.scala:75)
at com.snowplowanalytics.snowplow.bigquery.loader.BigqueryLoaderCli.main(BigqueryLoaderCli.scala)

When I googled it, it seems error is coming from exception.

// TODO: switch from throwing error to using scalaz Validation, maybe.
def getValues(line: String): List[String] = {
val values = line.split("\t", -1).toList
if (values.length != 108){
throw new Error(“There seems to have been a parsing error”)
}
values
}

github.com

snowplow/bigquery-loader-cli/blob/master/src/main/scala/com.snowplowanalytics.snowplow.storage/TsvParser.scala

 /*
 * Copyright (c) 2015 Snowplow Analytics Ltd.
 * All rights reserved.
 *
 * This program is licensed to you under the Apache License Version 2.0,
 * and you may not use this file except in compliance with the Apache
 * License Version 2.0.
 * You may obtain a copy of the Apache License Version 2.0 at
 * http://www.apache.org/licenses/LICENSE-2.0.
 *
 * Unless required by applicable law or agreed to in writing,
 * software distributed under the Apache License Version 2.0 is distributed
 * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,
 * either express or implied.
 *
 * See the Apache License Version 2.0 for the specific language
 * governing permissions and limitations there under.
 */
package com.snowplowanalytics.snowplow.bigquery.loader

This file has been truncated. show original

@alex can you help me to understand what are the values that have parsing error?

alex · August 17, 2016, 11:13am

Hi @mongodb - my suspicion is that BigQuery Loader CLI works against an older version of the Snowplow enriched event format (one with a different number of tabs).

Unfortunately this project was an R&D spike which we are not actively supporting (though we learnt a lot for future BigQuery support). I’ll update the GitHub tagline for the repo to make this clear.

Shin · August 17, 2016, 11:38am

For what it’s worth, here are the changes I had to do to support the latest enriched event format (as alex mentions):

gist.github.com

https://gist.github.com/shin-nien/ea5c8d2c48179178c7c5b0ea5741f8f6

bigquery-loader-cli.diff

diff --git a/src/main/scala/com.snowplowanalytics.snowplow.storage/BasicSchema.scala b/src/main/scala/com.snowplowanalytics.snowplow.storage/BasicSchema.scala
index ed77668..1ed5b20 100644
--- a/src/main/scala/com.snowplowanalytics.snowplow.storage/BasicSchema.scala
+++ b/src/main/scala/com.snowplowanalytics.snowplow.storage/BasicSchema.scala
@@ -130,7 +130,32 @@ object SnowplowEnrichedEventSchema {
     ("dvce_screenheight", "INTEGER"),
     ("doc_charset", "STRING"),
     ("doc_width", "INTEGER"),
-    ("doc_height", "INTEGER")
+    ("doc_height", "INTEGER"),

This file has been truncated. show original

But I quickly gave up on it because of other issues e.g. unstructured events, enrichment contexts etc that would mean continuously updating the format.