RDB Loader Fails --> Amazon](500310) Invalid operation: ALTER TABLE ADD COLUMN defined as NOT NULL must have a non-null default expression;

For engineers AWS real-time pipeline
1

Hi All,
We are using Snowplow with EMR based Shredder and SQS with RDB Loader running in Amazon EKS.
One of our team member wanted to add not null constraints on an existing column so he added it to the required block in json schema (iglu server)

"required": [ col1,col2 ]

but then we started getting the following error in RDB Loader.

Amazon](500310) Invalid operation: ALTER TABLE ADD COLUMN defined as NOT NULL must have a non-null default expression;

to fix this i tried adding default expression in json schema as follows

"exampleColumn": {
      "description": "boolean flag",
      "type": "boolean",
      "default": false # could be true or false based on the requirement
    }

but the problem did not resolve. I have been searching a lot on different topics but found nothing related to this.

could you kindly help.

2

Hi @ahid_002,

Which version of RDB Loader are you using? We very recently addressed an issue which sounds like exactly what you are describing here. Hopefully if you upgrade to the latest version of RDB Loader (4.0.4) then it will be able to safely handle the schema changes you are describing, without causing errors.

1 Like
3

Hi we are using 2.2 version and I was thinking to upgrade but as you suggested this seems like a valid solution. i will update and see if it works. thank you so much for the help @istreeter .

6

This issue was resolved by upgrading the version to latest stable release and using RDB-Loader-Redshift instead of old RDB-Loader

1 Like
7

Hi @ahid_002
Thanks for updating on how you resolved your issue.
Cheers,
Eddie

1 Like