I updated my aws redshift com_myapp_click table schema with a new column that can be null, now RS loader server is failing to load enriched data into redshift from s3, upon checking err_reason in stl_load_errors table for that particular event it says
Delimiter not found
then i head -n 10 .txt and all the data is tab separated. i am confused if i am missing something, it would be great if Someone from the community can direct me to the solution.
This can occur in Redshift if there is a mismatch between the data in the file (and the number of delimiters) relative to the number of columns in the table.
Did you do this manually? If so you may need to double check the table as RDB loader will do this automatically for you so you may have ended up with a duplicate column.
How many delimiters / columns are in your file relative to the number of columns in your table?
Hi @mike , yess i added the column manually. i guess i can update the version of schema from 1-0-1 to 1-0-2 and add the column in schema. the loader will update the redshift table automatically right??
Yes - you shouldn’t need to change any Redshift tables manually (if you do so chances are the loader will get confused and you might end up with tables in a state where it can no longer manage them).
Hi @mike Thank you for the help, do i have to restart the loader server or will it add the column when it loads the data enriched against new schema ?
also my current schema is at 1-0-9, should i update it to 1-0-10 or 1-1-0?
because 1-0-10 is placed after 1-0-1 in my iglu registry, (don’t know if it should be the last schema in the list to work)