Error while running postgresql storageloader

For engineers Storage targets
1

Below is the data i am getting from scala stream collector to the stream enrich.

CFe23a	web	2017-07-26 15:22:19.437	2017-07-26 15:22:11.807	2017-07-26 15:22:08.350	page_view	93fbadbd-7554-42ed-bd5a-3ab35232eb24		cf	js-2.8.0	ssc-0.9.0-stdout	kinesis-0.10.0-common-0.24.0		127.0.0.x	3280538411	01d51db4-9dc9-40e9-9323-34dc0c12cedd	1	9708667e-0c7d-47cc-baae-630cf7cd1018												http://localhost/snowplow/SampleExampleTracker.html	Fixed Width 2 Blue	http://localhost/snowplow/	http	localhost	80	/snowplow/SampleExampleTracker.html			http	localhost	80	/snowplow/			internal																																	Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36	Chrome	Chrome	59.0.3071.115	Browser	WEBKIT	en-GB	1	0	0	0	0	0	0	0	0	1	24	1366	623	Windows 10	Windows	Microsoft Corporation	Asia/Kolkata	Computer	0	1366	768	UTF-8	1349	877												2017-07-26 15:22:08.390			{"schema":"iglu:com.snowplowanalytics.snowplow/contexts/jsonschema/1-0-1","data":[{"schema":"iglu:com.snowplowanalytics.snowplow/ua_parser_context/jsonschema/1-0-0","data":{"useragentFamily":"Chrome","useragentMajor":"59","useragentMinor":"0","useragentPatch":"3071","useragentVersion":"Chrome 59.0.3071","osFamily":"Windows","osMajor":null,"osMinor":null,"osPatch":null,"osPatchMinor":null,"osVersion":"Windows","deviceFamily":"Other"}}]}	3bd1506d-4604-4d93-938a-eb155de67df1	2017-07-26 15:22:11.767	com.snowplowanalytics.snowplow	page_view	jsonschema	1-0-0	905a11a06007e238743688f27dd65028	
CFe23a	web	2017-07-26 15:32:09.205	2017-07-26 15:32:09.199	2017-07-26 15:32:09.125	page_ping	8d4080ee-9e51-40b3-b641-d6ffd2c594cc		cf	js-2.8.0	ssc-0.9.0-stdout	kinesis-0.10.0-common-0.24.0		127.0.0.x	3280538411	01d51db4-9dc9-40e9-9323-34dc0c12cedd	1	e958982c-b9e5-4869-8324-b6f79b1258a8												http://localhost/snowplow/SampleExampleTracker.html	Fixed Width 2 Blue	http://localhost/snowplow/	http	localhost	80	/snowplow/SampleExampleTracker.html			http	localhost	80	/snowplow/			internal																													0	0	0	0	Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36	Chrome	Chrome	59.0.3071.115	Browser	WEBKIT	en-GB	1	0	0	0	0	0	0	0	0	1	24	1366	623	Windows 10	Windows	Microsoft Corporation	Asia/Kolkata	Computer	0	1366	768	UTF-8	1349	877												2017-07-26 15:32:09.128			{"schema":"iglu:com.snowplowanalytics.snowplow/contexts/jsonschema/1-0-1","data":[{"schema":"iglu:com.snowplowanalytics.snowplow/ua_parser_context/jsonschema/1-0-0","data":{"useragentFamily":"Chrome","useragentMajor":"59","useragentMinor":"0","useragentPatch":"3071","useragentVersion":"Chrome 59.0.3071","osFamily":"Windows","osMajor":null,"osMinor":null,"osPatch":null,"osPatchMinor":null,"osVersion":"Windows","deviceFamily":"Other"}}]}	3bd1506d-4604-4d93-938a-eb155de67df1	2017-07-26 15:32:09.196	com.snowplowanalytics.snowplow	page_ping	jsonschema	1-0-0	9987b7e5f3116c29dab463b02ebf088a

I am using postgreSQL to store data from collector and bellow is config.yml file

  aws:
access_key_id: ADD HERE
secret_access_key: ADD HERE
s3:
region: eu-west-1 # S3 bucket region
buckets:
in: ADD HERE
archive: ADD HERE
download:
folder: /home/hadoop/snowplow/4-storage/ # Postgres-only config option. Where to store the downloaded files
targets:
   - :name: "PostgreSQL enriched events storage"
	 :type: postgres
	 :host: localhost # Hostname of database server
	 :database: snowplow # Name of database
	 :port: 5432 # Default Postgres port
	 :table: atomic.events
	 :username: power_user
	 :password: hadoop
	 :maxerror: # Not required for Postgres”

when i run the below command
./snowplow-storage-loader --config config/config.yml --resolver config/resolver.json --targets config/targets/ --skip analyze

i am getting below error

Unexpected error: (<unknown>): expected <block end>, but found BlockMappingStart while parsing a block mapping at line 3 column 4
org/jruby/ext/psych/PsychParser.java:219:in `parse'
uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/psych.rb:376:in `parse_stream'
uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/psych.rb:324:in `parse'
uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/psych.rb:251:in `load'
uri:classloader:/storage-loader/lib/snowplow-storage-loader/config.rb:52:in `get_config'
uri:classloader:/storage-loader/bin/snowplow-storage-loader:31:in `<main>'
org/jruby/RubyKernel.java:977:in `load'
uri:classloader:/META-INF/main.rb:1:in `<main>'
org/jruby/RubyKernel.java:959:in `require'
uri:classloader:/META-INF/main.rb:1:in `(root)'
uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:1:in `<main>'

please help me to resolve the error…

2

@shashi,

That error is an indication or a problem parsing your config.yml. I suspect you have a mismatch between the version of the StorageLoader and its configuration file.

In your command, you use --targets config/targets/ which implies you use R88 version (or above). However, the content of your config.yml has “targets” section which was removed in that version. Instead, you would have to use the relevant JSON storage configuration file.

Could you, go over the instructions for the relevant release and correct the mismatch? That means

  • Check the StorageLoader version you use
  • Submit the corresponding version of the configuration file (ensure the correct format)
  • Provide the relevant JSON storage if appropriate
3

hi @ihor thanks for the replay
yes i am using r88 version and please help me which command should i have to run for this version.

4

Hi shashi,

use this configuration sample as your template if you are using snowplow r88. As mentioned above, your storage targets should be configured as separate json files and you should run your storage loader like this:

./snowplow-storage-loader --config /path/to/config.yml --resolver /path/to/resolver.json --targets /path/to/targets/

And your etl-runner like this:

./snowplow-emr-etl-runner --config /path/to/config.yml --resolver /path/to/resolver.json --enrichments /path/to/enrichments/ --targets /path/to/targets/
5

hi kazgurs
thanks for the reply…i will try this command