Storage-loader to Postgres: Unexpected error: no implicit conversion of Symbol into Integer during 'download_events'

For engineers Storage targets
1

storage-loader version: r83_bald_eagle
storage-loader config:

aws:
  access_key_id: <redacted>
  secret_access_key: <redacted>
  s3:
    region: us-east-1
    buckets:
      assets: s3://snowplow-hosted-assets
      jsonpath_assets:
      log: s3://cartful-snowplow-other/logs
      raw:
        in:
          - s3://cartful-snowplow-other/raw/in
        processing: s3://cartful-snowplow-other/raw/processing
        archive: s3://cartful-snowplow-other/raw/archive
      enriched:
        good: s3://cartful-snowplow-other/enriched/good
        bad: s3://cartful-snowplow-other/enriched/bad
        errors: s3://cartful-snowplow-other/enriched/errors
        archive: s3://cartful-snowplow-other/enriched/archive
      shredded:
        good: s3://cartful-snowplow-other/shredded/good
        bad: s3://cartful-snowplow-other/shredded/bad
        errors: s3://cartful-snowplow-other/shredded/errors
        archive: s3://cartful-snowplow-other/shredded/archive
  emr:
    ami_version: 4.5.0
    region: us-east-1
    jobflow_role: EMR_EC2_DefaultRole
    service_role: EMR_DefaultRole
    placement:
    ec2_subnet_id: <redacted>
    ec2_key_name: <redacted>
    bootstrap:
    software:
      hbase:
      lingual:
    jobflow:
      master_instance_type: m4.large
      core_instance_count: 2
      core_instance_type: m4.large
      task_instance_count: 0
      task_instance_type: m4.large
      task_instance_bid: 0.015
    bootstrap_failure_tries: 3
    additional_info:
collectors:
  format: cloudfront
enrich:
  job_name: Snowplow ETL - Thu Nov 17 18:27:21 UTC 2016
  versions:
    hadoop_enrich: 1.8.0
    hadoop_shred: 0.9.0
    hadoop_elasticsearch: 0.1.0
  continue_on_unexpected_error: false
  output_compression: NONE
storage:
  download:
    folder:   /tmp/tmp.vRqYJOAJZ2
  targets:
    - name: "snowplowstorage@snowplowstorage"
      type: postgres
      host: <redacted>
      database: snowplowstorage
      port: 5432
      ssl_mode: disable
      table: atomic.events
      username: snowplowstorage
      password: <redacted>
monitoring:
  tags: {}
  logging:
    level: DEBUG

Which produces this error:

Downloading Snowplow events...
Unexpected error: no implicit conversion of Symbol into Integer
org/jruby/RubyArray.java:1457:in `[]'
uri:classloader:/storage-loader/lib/snowplow-storage-loader/s3_tasks.rb:41:in `download_events'
uri:classloader:/storage-loader/bin/snowplow-storage-loader:42:in `<main>'
org/jruby/RubyKernel.java:973:in `load'
uri:classloader:/META-INF/main.rb:1:in `<main>'
org/jruby/RubyKernel.java:955:in `require'
uri:classloader:/META-INF/main.rb:1:in `(root)'
uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:1:in `<main>'

This is the exact line (link):
s3.host = region_to_safe_host([:aws][:s3][:region])

My YAML document definitely has region set there so I’m not sure what’s going on. Any ideas?

3

I tried compiling from source instead of relying on the binaries provided.

I get a similar issue when I test against the latest release r85-metamorphosis:

Downloading Snowplow events...
Unexpected error: no implicit conversion of Symbol into Integer
org/jruby/RubyArray.java:1458:in `[]'
/snowplow/4-storage/storage-loader/lib/snowplow-storage-loader/s3_tasks.rb:41:in `download_events'
/snowplow/4-storage/storage-loader/bin/snowplow-storage-loader:42:in `<main>'

(same input config)

4

Hey @darren,

It seems you stumbled upon this unfortunate bug: https://github.com/snowplow/snowplow/pull/2888

1 Like
5

@anton - Saving the day once again!! I guess I should have realized this fix. I will patch and continue!! Thanks so much for pointing this out. You’re a life saver.