Error while uploading snowplow logs to s3

I am running snowplow-kinesis-s3-0.4.0 to upload logs to s3 but i am getting an error,

“com.amazonaws.AmazonClientException: Unable to execute HTTP request: s3-http”

not sure, what is the reason of the error because l am following your setup guide step by step
Config

sink {
aws {
access-key: “key”
secret-key: “key”
}

kinesis {
in {
stream-name: “good”
initial-position: “TRIM_HORIZON”
max-records: “1000”
}

out {
  stream-name: "bad"
}
region: "us-west-2"
app-name: "snownrich"

}

s3 {
region: “http://s3-us-west-2.amazonaws.com/
bucket: “bucketname/folder”
max-timeout: “300”
format: “lzo”
}

buffer {
byte-limit: 4500000
record-limit: 500 # Not supported by Kafka; will be ignored
time-limit: 60000
}
logging {
level: “error”
}

monitoring {
snowplow {
collector-uri: “xx.xx.xx.xx”
collector-port: 80
app-id: “collector-monitor”
method: “GET”
}
}
}

Error

com.snowplowanalytics.snowplow.storage.kinesis.s3.S3Emitter com$snowplowanalytics$snowplow$storage$kinesis$s3$S3Emitter$$attemptEmit$1
SEVERE: S3Emitter threw an unexpected exception
com.amazonaws.AmazonClientException: Unable to execute HTTP request: s3-http
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:478)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:302)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3672)
at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1436)
at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1302)
at com.snowplowanalytics.snowplow.storage.kinesis.s3.S3Emitter.com$snowplowanalytics$snowplow$storage$kinesis$s3$S3Emitter$$attemptEmit$1(S3Emitter.scala:157)
at com.snowplowanalytics.snowplow.storage.kinesis.s3.S3Emitter$$anonfun$emit$1.apply(S3Emitter.scala:189)
at com.snowplowanalytics.snowplow.storage.kinesis.s3.S3Emitter$$anonfun$emit$1.apply(S3Emitter.scala:189)
at scala.collection.immutable.List.foreach(List.scala:318)
at com.snowplowanalytics.snowplow.storage.kinesis.s3.S3Emitter.emit(S3Emitter.scala:189)
at com.amazonaws.services.kinesis.connectors.KinesisConnectorRecordProcessor.emit(KinesisConnectorRecordProcessor.java:159)
at com.amazonaws.services.kinesis.connectors.KinesisConnectorRecordProcessor.processRecords(KinesisConnectorRecordProcessor.java:132)
at com.amazonaws.services.kinesis.clientlibrary.lib.worker.V1ToV2RecordProcessorAdapter.processRecords(V1ToV2RecordProcessorAdapter.java:42)
at com.amazonaws.services.kinesis.clientlibrary.lib.worker.ProcessTask.call(ProcessTask.java:172)
at com.amazonaws.services.kinesis.clientlibrary.lib.worker.MetricsCollectingTaskDecorator.call(MetricsCollectingTaskDecorator.java:48)
at com.amazonaws.services.kinesis.clientlibrary.lib.worker.MetricsCollectingTaskDecorator.call(MetricsCollectingTaskDecorator.java:23)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.UnknownHostException: s3-http
at java.net.InetAddress.getAllByName0(InetAddress.java:1273)
at java.net.InetAddress.getAllByName(InetAddress.java:1185)
at java.net.InetAddress.getAllByName(InetAddress.java:1119)
at org.apache.http.impl.conn.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:44)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.resolveHostname(DefaultClientConnectionOperator.java:259)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:159)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:304)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:611)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:446)
at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:706)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:467)
… 19 more

Feb 22, 2017 7:29:24 AM com.amazonaws.services.kinesis.metrics.impl.CWPublisherRunnable publishMetrics
INFO: Successfully published 13 datums.
Feb 22, 2017 7:29:28 AM com.snowplowanalytics.snowplow.storage.kinesis.s3.S3Emitter forceShutdown
SEVERE: Shutting down application as unable to connect to S3 for over 300 ms

Does s3 bucket exist in the specified region? Does your user have privilages to write to the bucket? This looks like privilages issue…

no its not privileges issue because i just have uploaded files to my s3 bucket using same IAM keys using s3cmd.
region is also correct: us-west-2

Can you verify form the instance running kinesis-s3 storage to verify if s3 endpoint is not firewalled?

hey, i just have tested the s3cmd on the same server where i am running

command:
./s3cmd ls s3://snowplow-xxx-xxx/

output:

DIR s3://snowplow-xxx-xxx/logs/
2017-02-22 07:26 16606 s3://snowplow-xxx-xxx/logo.png

s3cmd config:
region: us-west-2
using same IAM id and key

if issue was because of firewall, i wont be able to ls or upload file’s to s3 from same server using s3cmd

i am not sure about this but following error is

Feb 22, 2017 6:17:02 PM com.amazonaws.http.AmazonHttpClient executeHelper
INFO: Unable to execute HTTP request: snowplowlogbucket.s3-http

java.net.UnknownHostException: snowplowlogbucket.s3-http

but it should be some complete aws s3 URL instead of just snowplowlogbucket.s3-http.

maybe that’s why snowplow-kinesis-s3-0.4.0 si not able to execute http request.

Hi @geetanshjindal - from your config you have put region: "http://s3-us-west-2.amazonaws.com/" - this should simply be “us-west-2”.

Could you please update that and try again?

yes its working now, Thanks.