We are currently using the RDB-Databricks Loader in combination with the Stream Transformer for Kinesis. For our enriched stream these two work seamlessly and load the data without issue. For our bad stream, however, we’re seeing the following error in the loader:
INFO DataDiscovery: Empty discovery at s3://databricks-prod/transformed-invalid/run=2022-12-07-21-00-00-e352ae60c1ed-e8c5-464c-af8a/. Acknowledging the message without loading attempt
The Stream Transformer is working as expected and producing folders for each run as expected:
I’m not sure if there is a bad table that needs to be created first? I know for the enriched stream there’s an events table to be made before loading. I can’t seem to find any documentation for loading invalid messages using the RDB Loader.