COMMIT;: ERROR: relation "atomic.manifest" does not exist

For data modelers & consumers Redshift
1

Hello,

While runnning ./snowplow-storage-loader --config config.yml I got the following error:

Loading Snowplow events and shredded types into XXXXXX-snowplow (Redshift cluster)...
Unexpected error: Java::OrgPostgresqlUtil::PSQLException error executing COPY statements: BEGIN;
COPY atomic.events FROM 's3://XXXXXX-snowplow-analytics-data/shredded/good/run=2017-01-19-15-47-11/atomic-events' CREDENTIALS 'aws_access_key_id=xxxxxxxxxxxxxxxxxxxx;aws_secret_access_key=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx' REGION AS 'eu-central-1' DELIMITER '\t' MAXERROR 1 EMPTYASNULL FILLRECORD TRUNCATECOLUMNS  TIMEFORMAT 'auto' ACCEPTINVCHARS GZIP;
INSERT INTO atomic.manifest
          SELECT etl_tstamp, getdate() AS commit_tstamp, count(*) AS event_count, 0 AS shredded_cardinality
          FROM atomic.events
          WHERE etl_tstamp IS NOT null
          GROUP BY 1
          ORDER BY etl_tstamp DESC
          LIMIT 1;

COMMIT;: ERROR: relation "atomic.manifest" does not exist
uri:classloader:/storage-loader/lib/snowplow-storage-loader/redshift_loader.rb:92:in `load_events_and_shredded_types'
uri:classloader:/gems/contracts-0.11.0/lib/contracts/method_reference.rb:43:in `send_to'
uri:classloader:/gems/contracts-0.11.0/lib/contracts/call_with.rb:76:in `call_with'
uri:classloader:/gems/contracts-0.11.0/lib/contracts/method_handler.rb:138:in `block in redefine_method'
uri:classloader:/storage-loader/bin/snowplow-storage-loader:54:in `block in (root)'
uri:classloader:/storage-loader/bin/snowplow-storage-loader:51:in `<main>'
org/jruby/RubyKernel.java:973:in `load'
uri:classloader:/META-INF/main.rb:1:in `<main>'
org/jruby/RubyKernel.java:955:in `require'
uri:classloader:/META-INF/main.rb:1:in `(root)'
uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:1:in `<main>'

I cannot find anything about this so called atomic.mainfest relation anywhere in neither documentation nor code. What is it, and how do I solve the error?

BTW here is my Config file (the storage part):

storage:
  download:
    folder: # not needed. We need redshift
  targets:
    - name: "XXXXXX-snowplow"
      type: redshift
      host: XXXXXX-snowplow.XXXXXX.eu-central-1.redshift.amazonaws.com
      database: XXXXXX
      port: 5439
      ssl_mode: disable
      table: atomic.events
      username: <%= ENV['AWS_REDSHIFT_USERNAME'] %>
      password: <%= ENV['AWS_REDSHIFT_PASSWORD'] %>
      maxerror: 1
      comprows: 200000
monitoring:
  tags: {}
  logging:
    level: DEBUG

The bintray I used: snowplow_emr_r87_chichen_itza_rc3.zip

2

Hi @radubogdan,

The R87 runners are still in development. Please, use http://dl.bintray.com/snowplow/snowplow-generic/:snowplow_emr_r83_bald_eagle.zip instead.

3

Thank you @ihor.

I should’ve figure it out that RC’s can have differences like this :slight_smile: however the R87 emretlrunner didn’t throw any issues and it’s compatible with all the documentation on GitHub.

4

Hi Ihor?

Just confirming that r87 is still not in release as yet? I see that there are 4 RCs but also one thats not an RC. Would you still recommend us using r83 at this time?

Thanks!

5

Hi @kjain, R87 has been released. See the thread here.

6

Thanks Mike!
I can see that the manifest table sql is included in the upgrade instructions as well…