Kinesis LZO S3 Sink 0.5.0 Enable logging

For engineers Storage targets
1

Hi All,

How to enable logging for Kinesis LZO S3 Sink 0.5.0 ?

I have set logging level as DEBUG in the config file however only INFO logs are generated. I get following warning as I start application.

log4j:WARN No appenders could be found for logger (com.amazonaws.AmazonWebServiceClient).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

I was stuck with a problem for 2 days, I could see data being written to Kinesis by stream collector, but it was not written S3 and without logs I was clueless.
Today I deployed earlier version i.e. Kinesis LZO S3 Sink 0.4.1 and I got error almost immediately that user did not have access to dynamo DB. I was using same config file. Is something changed since 0.4.1 related to logging ?

Thanks in advance.

1 Like
3

Hello Rajan,

Kinesis S3 uses a log4j properties file to do its logging.

Prior to version 0.5.0, kinesis-s3 used to incorporate an external log4j.properties file on which we had no control (an artifact of some dependency). This problem has been fixed with 0.5.0.

However, since kinesis-s3 is distributed as an executable jar file, this means we cannot pass JVM arguments (like -Dlog4j.properties) when launching it.

As a result, the only solution for now is to add your own log4j.properties file to this folder https://github.com/snowplow/kinesis-s3/tree/master/src/main/resources and rebuild the fat jar from source. The instructions for building kinesis-s3 can be found here.

The longer term solution is to distribute kinesis-s3 as a normal jar so that people will be able to provide their log4j configuration file when launching:

java -Dlog4j.properties=log4j.properties -jar snowplow-kinesis-s3-0.5.0.jar --config config.hocon

1 Like
4

Hello Ben.

Oh I see. Thanks for your help.

5

You can still execute the binary with java -jar, just examine the first few lines of the executable for embedded shell script arguments it passes to java on launch. Adjust accordingly.

2 Likes
6

Hello @dashirov-ga,
It works like charm. Thanks. It saved me from rebuilding the application.

Rajan

7

Happy I could help!

8

Thanks for this thread.

I was able to get debug logging enabled by doing:

cat <<EOT >> log4j.properties
log4j.rootLogger=DEBUG, A1
log4j.appender.A1=org.apache.log4j.ConsoleAppender
log4j.appender.A1.layout=org.apache.log4j.PatternLayout
log4j.appender.A1.layout.ConversionPattern=%d [%t] %-5p %c -  %m%n
EOT

./bin/snowplow-s3-loader -debug -Dlog4j.debug -Dlog4j.configuration=file:/opt/docker/log4j.properties  --config /snowplow/config/s3_loader.hocon

when running s3-loader 0.21.0