I am getting the following error while running my streaming loader snowflake in aws. THe servers are continously restarting and error is coming, however there is no issue in loading the data in Snowflake.
net.snowflake.ingest.utils.SFException: One or more channels [SNOWPLOW.ATOMIC_COMMUNITY.EVENTS.SNOWPLOW] might contain uncommitted rows due to server side errors, please consider reopening the channels to replay the data loading by using the latest persistent offset token.
NOTE: Picked up JDK_JAVA_OPTIONS: -XX:InitialRAMPercentage=75 -XX:MaxRAMPercentage=75
[io-compute-0] INFO com.snowplowanalytics.snowplow.runtime.HealthProbe - Health service listening on port 8000
[io-compute-0] INFO com.snowplowanalytics.snowplow.snowflake.processing.TableManager - Opening JDBC connection to jdbc:snowflake:/abc.snowflakecomputing.com:443
[io-compute-blocker-0] INFO com.snowplowanalytics.snowplow.snowflake.processing.TableManager - Creating table SNOWPLOW.ATOMIC_COMMUNITY.events if it does not already exist...
[io-compute-0] INFO com.snowplowanalytics.snowplow.snowflake.processing.Channel - Opening channel snowplow
[io-compute-1] INFO com.snowplowanalytics.snowplow.runtime.Metrics - events_good = 0
[io-compute-1] INFO com.snowplowanalytics.snowplow.runtime.Metrics - events_bad = 0
[io-compute-1] INFO com.snowplowanalytics.snowplow.runtime.Metrics - latency_millis = 0
[ingest-build-upload-thread-0] WARN net.snowflake.ingest.internal.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[ingest-register-thread] WARN net.snowflake.ingest.streaming.internal.SnowflakeStreamingIngestClientInternal - [SF_INGEST] Channel has been invalidated because of failure response, name=SNOWPLOW, channel_sequencer=34965, status_code=20, message=The channel is owned by another writer at this time. Either re-open the channel to gain exclusive write ownership or close the channel and let the other writer continue, executionCount=0
[ingest-register-thread] WARN net.snowflake.ingest.streaming.internal.SnowflakeStreamingIngestChannelInternal - Channel is invalidated, name=SNOWPLOW.ATOMIC_COMMUNITY.EVENTS.SNOWPLOW, channel sequencer=34965, row sequencer=2, message=invalidate with matched sequencer
[ingest-register-thread] WARN net.snowflake.ingest.streaming.internal.SnowflakeStreamingIngestClientInternal - [SF_INGEST] Channel has been invalidated because of failure response, name=SNOWPLOW, channel_sequencer=34965, status_code=20, message=The channel is owned by another writer at this time. Either re-open the channel to gain exclusive write ownership or close the channel and let the other writer continue, executionCount=0
[ingest-register-thread] WARN net.snowflake.ingest.streaming.internal.SnowflakeStreamingIngestChannelInternal - Channel is invalidated, name=SNOWPLOW.ATOMIC_COMMUNITY.EVENTS.SNOWPLOW, channel sequencer=34965, row sequencer=2, message=invalidate with matched sequencer
[io-compute-0] INFO com.snowplowanalytics.snowplow.snowflake.processing.Channel - Closing channel SNOWPLOW.ATOMIC_COMMUNITY.EVENTS.SNOWPLOW
[io-compute-1] INFO com.snowplowanalytics.snowplow.snowflake.processing.Channel - Opening channel snowplow
[io-compute-0] INFO com.snowplowanalytics.snowplow.runtime.Metrics - events_good = 16038
[io-compute-0] INFO com.snowplowanalytics.snowplow.runtime.Metrics - events_bad = 0
Hi @Shobhit_Tiwari I cannot tell what the root cause just from the section of logs you shared. But I might be able to tell the root cause if I see some context around those logs.
Please can you share the logs starting from the line where it says NOTE: Picked up JDK_JAVA_OPTIONS: running until where the first error happens after that?
Which version of the loader are you using?
I am using snowflake-streaming-loader-ec2 v0.1.2
@istreeter ,
I am using snowflake-streaming-loader-ec2 v0.1.2
NOTE: Picked up JDK_JAVA_OPTIONS: -XX:InitialRAMPercentage=75 -XX:MaxRAMPercentage=75
[io-compute-0] INFO com.snowplowanalytics.snowplow.runtime.HealthProbe - Health service listening on port 8000
[io-compute-0] INFO com.snowplowanalytics.snowplow.snowflake.processing.TableManager - Opening JDBC connection to jdbc:snowflake://*****.snowflakecomputing.com:443
[io-compute-blocker-0] INFO com.snowplowanalytics.snowplow.snowflake.processing.TableManager - Creating table SNOWPLOW.ATOMIC_COMMUNITY.events if it does not already exist...
[io-compute-0] INFO com.snowplowanalytics.snowplow.snowflake.processing.Channel - Opening channel snowplow
[io-compute-1] INFO com.snowplowanalytics.snowplow.runtime.Metrics - events_good = 0
[io-compute-1] INFO com.snowplowanalytics.snowplow.runtime.Metrics - events_bad = 0
[io-compute-1] INFO com.snowplowanalytics.snowplow.runtime.Metrics - latency_millis = 0
[io-compute-0] INFO com.snowplowanalytics.snowplow.runtime.Metrics - events_good = 0
[io-compute-0] INFO com.snowplowanalytics.snowplow.runtime.Metrics - events_bad = 0
[io-compute-0] INFO com.snowplowanalytics.snowplow.runtime.Metrics - latency_millis = 0
[io-compute-1] INFO com.snowplowanalytics.snowplow.runtime.Metrics - events_good = 0
[io-compute-1] INFO com.snowplowanalytics.snowplow.runtime.Metrics - events_bad = 0
[io-compute-1] INFO com.snowplowanalytics.snowplow.runtime.Metrics - latency_millis = 0
[io-compute-1] INFO com.snowplowanalytics.snowplow.runtime.Metrics - events_good = 0
[io-compute-1] INFO com.snowplowanalytics.snowplow.runtime.Metrics - events_bad = 0
[io-compute-1] INFO com.snowplowanalytics.snowplow.runtime.Metrics - latency_millis = 0
[io-compute-0] INFO com.snowplowanalytics.snowplow.runtime.Metrics - events_good = 0
[io-compute-0] INFO com.snowplowanalytics.snowplow.runtime.Metrics - events_bad = 0
[io-compute-0] INFO com.snowplowanalytics.snowplow.runtime.Metrics - latency_millis = 0
[io-compute-0] INFO com.snowplowanalytics.snowplow.runtime.Metrics - events_good = 0
[io-compute-0] INFO com.snowplowanalytics.snowplow.runtime.Metrics - events_bad = 0
[io-compute-0] INFO com.snowplowanalytics.snowplow.runtime.Metrics - latency_millis = 0
[io-compute-0] INFO com.snowplowanalytics.snowplow.runtime.Metrics - events_good = 0
[io-compute-0] INFO com.snowplowanalytics.snowplow.runtime.Metrics - events_bad = 0
[io-compute-0] INFO com.snowplowanalytics.snowplow.runtime.Metrics - latency_millis = 0
[ingest-build-upload-thread-0] WARN net.snowflake.ingest.internal.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[ingest-register-thread] WARN net.snowflake.ingest.streaming.internal.SnowflakeStreamingIngestClientInternal - [SF_INGEST] Channel has been invalidated because of failure response, name=SNOWPLOW, channel_sequencer=34965, status_code=20, message=The channel is owned by another writer at this time. Either re-open the channel to gain exclusive write ownership or close the channel and let the other writer continue, executionCount=0
[ingest-register-thread] WARN net.snowflake.ingest.streaming.internal.SnowflakeStreamingIngestChannelInternal - Channel is invalidated, name=SNOWPLOW.ATOMIC_COMMUNITY.EVENTS.SNOWPLOW, channel sequencer=34965, row sequencer=2, message=invalidate with matched sequencer
[ingest-register-thread] WARN net.snowflake.ingest.streaming.internal.SnowflakeStreamingIngestClientInternal - [SF_INGEST] Channel has been invalidated because of failure response, name=SNOWPLOW, channel_sequencer=34965, status_code=20, message=The channel is owned by another writer at this time. Either re-open the channel to gain exclusive write ownership or close the channel and let the other writer continue, executionCount=0
[ingest-register-thread] WARN net.snowflake.ingest.streaming.internal.SnowflakeStreamingIngestChannelInternal - Channel is invalidated, name=SNOWPLOW.ATOMIC_COMMUNITY.EVENTS.SNOWPLOW, channel sequencer=34965, row sequencer=2, message=invalidate with matched sequencer
[io-compute-0] INFO com.snowplowanalytics.snowplow.snowflake.processing.Channel - Closing channel SNOWPLOW.ATOMIC_COMMUNITY.EVENTS.SNOWPLOW
[io-compute-1] INFO com.snowplowanalytics.snowplow.snowflake.processing.Channel - Opening channel snowplow
[io-compute-0] INFO com.snowplowanalytics.snowplow.runtime.Metrics - events_good = 16038
[io-compute-0] INFO com.snowplowanalytics.snowplow.runtime.Metrics - events_bad = 0
@istreeter I am using snowflake-streaming-loader-ec2 v0.1.2
NOTE: Picked up JDK_JAVA_OPTIONS: -XX:InitialRAMPercentage=75 -XX:MaxRAMPercentage=75
[io-compute-0] INFO com.snowplowanalytics.snowplow.runtime.HealthProbe - Health service listening on port 8000
[io-compute-0] INFO com.snowplowanalytics.snowplow.snowflake.processing.TableManager - Opening JDBC connection to jdbc:snowflake://abcd.snowflakecomputing.com:443
[io-compute-blocker-0] INFO com.snowplowanalytics.snowplow.snowflake.processing.TableManager - Creating table SNOWPLOW.ATOMIC_COMMUNITY.events if it does not already exist...
[io-compute-0] INFO com.snowplowanalytics.snowplow.snowflake.processing.Channel - Opening channel snowplow
[io-compute-1] INFO com.snowplowanalytics.snowplow.runtime.Metrics - events_good = 0
[io-compute-1] INFO com.snowplowanalytics.snowplow.runtime.Metrics - events_bad = 0
[io-compute-1] INFO com.snowplowanalytics.snowplow.runtime.Metrics - latency_millis = 0
[io-compute-0] INFO com.snowplowanalytics.snowplow.runtime.Metrics - events_good = 0
[io-compute-0] INFO com.snowplowanalytics.snowplow.runtime.Metrics - events_bad = 0
[io-compute-0] INFO com.snowplowanalytics.snowplow.runtime.Metrics - latency_millis = 0
[io-compute-1] INFO com.snowplowanalytics.snowplow.runtime.Metrics - events_good = 0
[io-compute-1] INFO com.snowplowanalytics.snowplow.runtime.Metrics - events_bad = 0
[io-compute-1] INFO com.snowplowanalytics.snowplow.runtime.Metrics - latency_millis = 0
[io-compute-1] INFO com.snowplowanalytics.snowplow.runtime.Metrics - events_good = 0
[io-compute-1] INFO com.snowplowanalytics.snowplow.runtime.Metrics - events_bad = 0
[io-compute-1] INFO com.snowplowanalytics.snowplow.runtime.Metrics - latency_millis = 0
[io-compute-0] INFO com.snowplowanalytics.snowplow.runtime.Metrics - events_good = 0
[io-compute-0] INFO com.snowplowanalytics.snowplow.runtime.Metrics - events_bad = 0
[io-compute-0] INFO com.snowplowanalytics.snowplow.runtime.Metrics - latency_millis = 0
[io-compute-0] INFO com.snowplowanalytics.snowplow.runtime.Metrics - events_good = 0
[io-compute-0] INFO com.snowplowanalytics.snowplow.runtime.Metrics - events_bad = 0
[io-compute-0] INFO com.snowplowanalytics.snowplow.runtime.Metrics - latency_millis = 0
[io-compute-0] INFO com.snowplowanalytics.snowplow.runtime.Metrics - events_good = 0
[io-compute-0] INFO com.snowplowanalytics.snowplow.runtime.Metrics - events_bad = 0
[io-compute-0] INFO com.snowplowanalytics.snowplow.runtime.Metrics - latency_millis = 0
[ingest-build-upload-thread-0] WARN net.snowflake.ingest.internal.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[ingest-register-thread] WARN net.snowflake.ingest.streaming.internal.SnowflakeStreamingIngestClientInternal - [SF_INGEST] Channel has been invalidated because of failure response, name=SNOWPLOW, channel_sequencer=34965, status_code=20, message=The channel is owned by another writer at this time. Either re-open the channel to gain exclusive write ownership or close the channel and let the other writer continue, executionCount=0
[ingest-register-thread] WARN net.snowflake.ingest.streaming.internal.SnowflakeStreamingIngestChannelInternal - Channel is invalidated, name=SNOWPLOW.ATOMIC_COMMUNITY.EVENTS.SNOWPLOW, channel sequencer=34965, row sequencer=2, message=invalidate with matched sequencer
[ingest-register-thread] WARN net.snowflake.ingest.streaming.internal.SnowflakeStreamingIngestClientInternal - [SF_INGEST] Channel has been invalidated because of failure response, name=SNOWPLOW, channel_sequencer=34965, status_code=20, message=The channel is owned by another writer at this time. Either re-open the channel to gain exclusive write ownership or close the channel and let the other writer continue, executionCount=0
[ingest-register-thread] WARN net.snowflake.ingest.streaming.internal.SnowflakeStreamingIngestChannelInternal - Channel is invalidated, name=SNOWPLOW.ATOMIC_COMMUNITY.EVENTS.SNOWPLOW, channel sequencer=34965, row sequencer=2, message=invalidate with matched sequencer
[io-compute-0] INFO com.snowplowanalytics.snowplow.snowflake.processing.Channel - Closing channel SNOWPLOW.ATOMIC_COMMUNITY.EVENTS.SNOWPLOW
[io-compute-1] INFO com.snowplowanalytics.snowplow.snowflake.processing.Channel - Opening channel snowplow
[io-compute-0] INFO com.snowplowanalytics.snowplow.runtime.Metrics - events_good = 16038
[io-compute-0] INFO com.snowplowanalytics.snowplow.runtime.Metrics - events_bad = 0
@istreeter I am trying to post the logs but the bot is automatcally removing it : (
NOTE: Picked up JDK_JAVA_OPTIONS: -XX:InitialRAMPercentage=75 -XX:MaxRAMPercentage=75
[io-compute-0] INFO com.snowplowanalytics.snowplow.runtime.HealthProbe - Health service listening on port 8000
[io-compute-0] INFO com.snowplowanalytics.snowplow.snowflake.processing.TableManager - Opening JDBC connection to jdbc:snowflake://abc.snowflakecomputing.com:443
[io-compute-blocker-0] INFO com.snowplowanalytics.snowplow.snowflake.processing.TableManager - Creating table SNOWPLOW.ATOMIC_COMMUNITY.events if it does not already exist...
[io-compute-0] INFO com.snowplowanalytics.snowplow.snowflake.processing.Channel - Opening channel snowplow
[io-compute-1] INFO com.snowplowanalytics.snowplow.runtime.Metrics - events_good = 0
[io-compute-1] INFO com.snowplowanalytics.snowplow.runtime.Metrics - events_bad = 0
[io-compute-1] INFO com.snowplowanalytics.snowplow.runtime.Metrics - latency_millis = 0
[io-compute-0] INFO com.snowplowanalytics.snowplow.runtime.Metrics - events_good = 0
[io-compute-0] INFO com.snowplowanalytics.snowplow.runtime.Metrics - events_bad = 0
[io-compute-0] INFO com.snowplowanalytics.snowplow.runtime.Metrics - latency_millis = 0
[io-compute-1] INFO com.snowplowanalytics.snowplow.runtime.Metrics - events_good = 0
[io-compute-1] INFO com.snowplowanalytics.snowplow.runtime.Metrics - events_bad = 0
[io-compute-1] INFO com.snowplowanalytics.snowplow.runtime.Metrics - latency_millis = 0
[io-compute-1] INFO com.snowplowanalytics.snowplow.runtime.Metrics - events_good = 0
[io-compute-1] INFO com.snowplowanalytics.snowplow.runtime.Metrics - events_bad = 0
[io-compute-1] INFO com.snowplowanalytics.snowplow.runtime.Metrics - latency_millis = 0
[io-compute-0] INFO com.snowplowanalytics.snowplow.runtime.Metrics - events_good = 0
[io-compute-0] INFO com.snowplowanalytics.snowplow.runtime.Metrics - events_bad = 0
[io-compute-0] INFO com.snowplowanalytics.snowplow.runtime.Metrics - latency_millis = 0
[io-compute-0] INFO com.snowplowanalytics.snowplow.runtime.Metrics - events_good = 0
[io-compute-0] INFO com.snowplowanalytics.snowplow.runtime.Metrics - events_bad = 0
[io-compute-0] INFO com.snowplowanalytics.snowplow.runtime.Metrics - latency_millis = 0
[io-compute-0] INFO com.snowplowanalytics.snowplow.runtime.Metrics - events_good = 0
[io-compute-0] INFO com.snowplowanalytics.snowplow.runtime.Metrics - events_bad = 0
[io-compute-0] INFO com.snowplowanalytics.snowplow.runtime.Metrics - latency_millis = 0
[ingest-build-upload-thread-0] WARN net.snowflake.ingest.internal.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[ingest-register-thread] WARN net.snowflake.ingest.streaming.internal.SnowflakeStreamingIngestClientInternal - [SF_INGEST] Channel has been invalidated because of failure response, name=SNOWPLOW, channel_sequencer=34965, status_code=20, message=The channel is owned by another writer at this time. Either re-open the channel to gain exclusive write ownership or close the channel and let the other writer continue, executionCount=0
[ingest-register-thread] WARN net.snowflake.ingest.streaming.internal.SnowflakeStreamingIngestChannelInternal - Channel is invalidated, name=SNOWPLOW.ATOMIC_COMMUNITY.EVENTS.SNOWPLOW, channel sequencer=34965, row sequencer=2, message=invalidate with matched sequencer
[ingest-register-thread] WARN net.snowflake.ingest.streaming.internal.SnowflakeStreamingIngestClientInternal - [SF_INGEST] Channel has been invalidated because of failure response, name=SNOWPLOW, channel_sequencer=34965, status_code=20, message=The channel is owned by another writer at this time. Either re-open the channel to gain exclusive write ownership or close the channel and let the other writer continue, executionCount=0
[ingest-register-thread] WARN net.snowflake.ingest.streaming.internal.SnowflakeStreamingIngestChannelInternal - Channel is invalidated, name=SNOWPLOW.ATOMIC_COMMUNITY.EVENTS.SNOWPLOW, channel sequencer=34965, row sequencer=2, message=invalidate with matched sequencer
[io-compute-0] INFO com.snowplowanalytics.snowplow.snowflake.processing.Channel - Closing channel SNOWPLOW.ATOMIC_COMMUNITY.EVENTS.SNOWPLOW
[io-compute-1] INFO com.snowplowanalytics.snowplow.snowflake.processing.Channel - Opening channel snowplow
[io-compute-0] INFO com.snowplowanalytics.snowplow.runtime.Metrics - events_good = 16038
[io-compute-0] INFO com.snowplowanalytics.snowplow.runtime.Metrics - events_bad = 0
NOTE: Picked up JDK_JAVA_OPTIONS: -XX:InitialRAMPercentage=75 -XX:MaxRAMPercentage=75
[io-compute-0] INFO com.snowplowanalytics.snowplow.runtime.HealthProbe - Health service listening on port 8000
[io-compute-0] INFO com.snowplowanalytics.snowplow.snowflake.processing.TableManager - Opening JDBC connection to jdbc:snowflake:/abc.snowflakecomputing.com:443
[io-compute-blocker-0] INFO com.snowplowanalytics.snowplow.snowflake.processing.TableManager - Creating table SNOWPLOW.ATOMIC_COMMUNITY.events if it does not already exist...
[io-compute-0] INFO com.snowplowanalytics.snowplow.snowflake.processing.Channel - Opening channel snowplow
[io-compute-1] INFO com.snowplowanalytics.snowplow.runtime.Metrics - events_good = 0
[io-compute-1] INFO com.snowplowanalytics.snowplow.runtime.Metrics - events_bad = 0
[io-compute-1] INFO com.snowplowanalytics.snowplow.runtime.Metrics - latency_millis = 0
[ingest-build-upload-thread-0] WARN net.snowflake.ingest.internal.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[ingest-register-thread] WARN net.snowflake.ingest.streaming.internal.SnowflakeStreamingIngestClientInternal - [SF_INGEST] Channel has been invalidated because of failure response, name=SNOWPLOW, channel_sequencer=34965, status_code=20, message=The channel is owned by another writer at this time. Either re-open the channel to gain exclusive write ownership or close the channel and let the other writer continue, executionCount=0
[ingest-register-thread] WARN net.snowflake.ingest.streaming.internal.SnowflakeStreamingIngestChannelInternal - Channel is invalidated, name=SNOWPLOW.ATOMIC_COMMUNITY.EVENTS.SNOWPLOW, channel sequencer=34965, row sequencer=2, message=invalidate with matched sequencer
[ingest-register-thread] WARN net.snowflake.ingest.streaming.internal.SnowflakeStreamingIngestClientInternal - [SF_INGEST] Channel has been invalidated because of failure response, name=SNOWPLOW, channel_sequencer=34965, status_code=20, message=The channel is owned by another writer at this time. Either re-open the channel to gain exclusive write ownership or close the channel and let the other writer continue, executionCount=0
[ingest-register-thread] WARN net.snowflake.ingest.streaming.internal.SnowflakeStreamingIngestChannelInternal - Channel is invalidated, name=SNOWPLOW.ATOMIC_COMMUNITY.EVENTS.SNOWPLOW, channel sequencer=34965, row sequencer=2, message=invalidate with matched sequencer
[io-compute-0] INFO com.snowplowanalytics.snowplow.snowflake.processing.Channel - Closing channel SNOWPLOW.ATOMIC_COMMUNITY.EVENTS.SNOWPLOW
[io-compute-1] INFO com.snowplowanalytics.snowplow.snowflake.processing.Channel - Opening channel snowplow
[io-compute-0] INFO com.snowplowanalytics.snowplow.runtime.Metrics - events_good = 16038
[io-compute-0] INFO com.snowplowanalytics.snowplow.runtime.Metrics - events_bad = 0
Hi @Shobhit_Tiwari thanks those logs were helpful. I think you are hitting a problem where you have more than one loader trying to write into Snowflake at the same time, and each loader is trying to use the same channel name.
Does that sound plausible? Are you running >1 loader?
I opened a PR in the terraform module which hopefully fixes this problem. It works by setting the snowflake channel name equal to the EC2 host name, so each loader uses a unique name.
Please can you try deploying your terraform pointing to the unique-channel-names branch of the terraform-aws-snowflake-streaming-loader-ec2 terraform module?
Please let us know if that fixes the problem!
1 Like
Hi @istreeter ,
Yes we are running two loaders over here.