Hi,
We’re running Clojure Collector 1.1.0 on a Amazon Elastic Beanstalk, ELB with m4.large instances.
Never got any significant issue for the past 18 months. Until yesterday.
Yesterday we got a CPU spike and latency increase but the ELB load another instance and manage to deal with the situation by itself.
Today, same behaviour, CPU spike, latency increase and 2 of 3 instances went to Severe state and also ELB got some problems dealing with it.
Restarted app server seemed to fix it. Looking the log file found some odd stuff.
Multiple entries indicating error parsing invalid character.
-------------------------------------
/var/log/tomcat8/catalina.2018-02-21.log
-------------------------------------
21-Feb-2018 11:52:19.931 INFO [http-nio-8080-exec-129] org.apache.coyote.http11.AbstractHttp11Processor.process Error parsing HTTP request header
Note: further occurrences of HTTP header parsing errors will be logged at DEBUG level.
java.lang.IllegalArgumentException: Invalid character found in the request target. The valid characters are defined in RFC 7230 and RFC 3986
at org.apache.coyote.http11.AbstractNioInputBuffer.parseRequestLine(AbstractNioInputBuffer.java:283)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1045)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:684)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1533)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1489)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
-------------------------------------
/var/log/httpd/error_log
-------------------------------------
[Wed Feb 21 11:53:33 2018] [error] server reached MaxClients setting, consider raising the MaxClients setting
-------------------------------------
/var/log/httpd/elasticbeanstalk-error_log
-------------------------------------
[Wed Feb 21 11:54:06 2018] [error] [client 172.XXX.XXX.246] (70007)The timeout specified has expired: proxy: error reading status line from remote server localhost:8080
[Wed Feb 21 11:54:06 2018] [error] [client 172.XXX.XXX.246] proxy: Error reading from remote server returned by /r/tp2
[Wed Feb 21 11:54:06 2018] [error] [client 172.XXX.XXX.246] (70007)The timeout specified has expired: proxy: error reading status line from remote server localhost:8080
[Wed Feb 21 11:54:06 2018] [error] [client 172.XXX.XXX.246] proxy: Error reading from remote server returned by /r/tp2
[Wed Feb 21 11:54:06 2018] [error] [client 172.XXX.XXX.246] (70007)The timeout specified has expired: proxy: error reading status line from remote server localhost:8080
[Wed Feb 21 11:54:06 2018] [error] [client 172.XXX.XXX.246] proxy: Error reading from remote server returned by /i
[Wed Feb 21 11:54:06 2018] [error] [client 172.XXX.XXX.246] (70007)The timeout specified has expired: proxy: error reading status line from remote server localhost:8080
[Wed Feb 21 11:54:06 2018] [error] [client 172.XXX.XXX.246] proxy: Error reading from remote server returned by /i
[Wed Feb 21 11:54:06 2018] [error] [client 172.XXX.XXX.246] (70007)The timeout specified has expired: proxy: error reading status line from remote server localhost:8080
[Wed Feb 21 11:54:06 2018] [error] [client 172.XXX.XXX.246] proxy: Error reading from remote server returned by /r/tp2
[Wed Feb 21 11:54:06 2018] [error] [client 172.XXX.XXX.246] (70007)The timeout specified has expired: proxy: error reading status line from remote server localhost:8080
[Wed Feb 21 11:54:06 2018] [error] [client 172.XXX.XXX.246] proxy: Error reading from remote server returned by /i
[Wed Feb 21 11:54:06 2018] [error] [client 172.XXX.XXX.246] (70007)The timeout specified has expired: proxy: error reading status line from remote server localhost:8080
[Wed Feb 21 11:54:06 2018] [error] [client 172.XXX.XXX.246] proxy: Error reading from remote server returned by /r/tp2
Anyone got any clue what is going on?
We hope that this unpleasant situation will not be repeated again tomorrow.
TIA