I’m encountering an issue with our Databricks cluster setup that has affected our RDB Loader 5.6.0 deployment. The cluster, which usually spins up automatically, has stopped doing so over the past week. Notably, I can confirm that SQS messages are still available, and the associated EC2 instance for RDB Loader is running without any issues.
After reviewing the setup, I found that no logs are appearing in CloudWatch, which suggests that the cluster may not be initializing as expected. Here are some relevant details:
Databricks Table Properties:
delta.columnMapping.mode = ‘name’
delta.minReaderVersion = ‘2’
delta.minWriterVersion = ‘5’
Databricks Runtime Version: Recently upgraded from 14.x to 15.4 LTS
Given this information, I’m wondering if the recent upgrade to Databricks runtime 15.4 LTS may be affecting the RDB Loader’s automated cluster spin-up functionality. Could there be any compatibility issues between RDB Loader 5.6.0 and Databricks 15.4 LTS, or specific configuration requirements to address this change?