memory usage again

Hello, I have read several posts about memory usage, there seem to be a variety of issues. I have been using traccar server on centos for a while now, 2 years on this server, and for some time there has been a gradually increasing memory usage. Up until this week, a server reboot every couple months would solve the problem. Now the java process goes sky high within a few days. The machine has 6 processors (cores) and 64 GB RAM. It is setup to log tracking to mysql, and there are currently 4 devices reporting every 5 minutes. Previously there were up to 9 devices.

I am the only one on this physical server and to my knowledge there are no other java processes besides traccar.

from top

4173 root      20   0   29.4g 854856   7464 S 501.3  1.3   2163:23 java

Looking through the wrapper.log files I see some different errors.

Here is a sample from last night

[root@srv2 logs]# cat wrapper.log.20200824
FINEST|4173/0|Service traccar|20-08-24 20:00:49|[qtp1590129041-11656] WARN org.eclipse.jetty.http.HttpParser - Illegal character 0x16 in state=START for buffer HeapByteBuffer@3f801468[p=1,l=122,c=16384,r=121]={\x16<<<\x03\x01\x00u\x01\x00\x00q\x03\x03\xBb\xAdq\xEf\xFcu\x1f...\x00\x08\x04\x01\x04\x03\x02\x01\x02\x03\xFf\x01\x00\x01\x00>>>er-Agent: Mozilla...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}
FINEST|4173/0|Service traccar|20-08-24 20:00:49|[qtp1590129041-11656] WARN org.eclipse.jetty.http.HttpParser - badMessage: 400 Illegal character 0x16 for HttpChannelOverHttp@68eacc9d{r=0,c=false,a=IDLE,uri=}
FINEST|4173/0|Service traccar|20-08-24 20:18:41|[qtp1590129041-11698] WARN org.eclipse.jetty.http.HttpParser - Illegal character 0x16 in state=START for buffer HeapByteBuffer@426b52f1[p=1,l=122,c=16384,r=121]={\x16<<<\x03\x01\x00u\x01\x00\x00q\x03\x03Io\xCb\xC8W\x1eT...\x00\x08\x04\x01\x04\x03\x02\x01\x02\x03\xFf\x01\x00\x01\x00>>>25253Ds%2525253Ah...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}
FINEST|4173/0|Service traccar|20-08-24 20:18:41|[qtp1590129041-11698] WARN org.eclipse.jetty.http.HttpParser - badMessage: 400 Illegal character 0x16 for HttpChannelOverHttp@93ac6d2{r=0,c=false,a=IDLE,uri=}
FINEST|4173/0|Service traccar|20-08-24 22:23:45|[qtp1590129041-12007] WARN org.eclipse.jetty.http.HttpParser - Illegal character 0x3 in state=START for buffer HeapByteBuffer@9b71251[p=1,l=47,c=16384,r=46]={\x03<<<\x00\x00/*\xE0\x00\x00\x00\x00\x00Cookie:...nistr\r\n\x01\x00\x08\x00\x03\x00\x00\x00>>>Ds%2525253Ah8%255...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}
FINEST|4173/0|Service traccar|20-08-24 22:23:45|[qtp1590129041-12007] WARN org.eclipse.jetty.http.HttpParser - badMessage: 400 Illegal character 0x3 for HttpChannelOverHttp@60cf8acb{r=0,c=false,a=IDLE,uri=}
FINEST|4173/0|Service traccar|20-08-24 22:23:45|[qtp1590129041-12006] WARN org.eclipse.jetty.http.HttpParser - Illegal character 0x3 in state=START for buffer HeapByteBuffer@47fb388d[p=1,l=47,c=16384,r=46]={\x03<<<\x00\x00/*\xE0\x00\x00\x00\x00\x00Cookie:...nistr\r\n\x01\x00\x08\x00\x03\x00\x00\x00>>>25253Ah8%255Eo%25...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}
FINEST|4173/0|Service traccar|20-08-24 22:23:45|[qtp1590129041-12006] WARN org.eclipse.jetty.http.HttpParser - badMessage: 400 Illegal character 0x3 for HttpChannelOverHttp@4c3863b8{r=0,c=false,a=IDLE,uri=}
FINEST|4173/0|Service traccar|20-08-24 22:23:45|[qtp1590129041-12006] WARN org.eclipse.jetty.http.HttpParser - Illegal character 0x3 in state=START for buffer HeapByteBuffer@426b52f1[p=1,l=47,c=16384,r=46]={\x03<<<\x00\x00/*\xE0\x00\x00\x00\x00\x00Cookie:...nistr\r\n\x01\x00\x08\x00\x03\x00\x00\x00>>>/\xC0+\xC0\x11\xC0\x07\xC0\x13\xC0\t\xC0\x14\xC0\n\x00\x05...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}
FINEST|4173/0|Service traccar|20-08-24 22:23:45|[qtp1590129041-12006] WARN org.eclipse.jetty.http.HttpParser - badMessage: 400 Illegal character 0x3 for HttpChannelOverHttp@4807fee7{r=0,c=false,a=IDLE,uri=}
FINEST|4173/0|Service traccar|20-08-24 22:23:45|[Fatal Error] :1:1: Premature end of file.
FINEST|4173/0|Service traccar|20-08-24 22:23:45|[Fatal Error] :1:1: Premature end of file.
FINEST|4173/0|Service traccar|20-08-24 22:23:45|[Fatal Error] :1:1: Premature end of file.
FINEST|4173/0|Service traccar|20-08-24 22:23:45|[Fatal Error] :1:1: Premature end of file.
FINEST|4173/0|Service traccar|20-08-24 22:23:45|[qtp1590129041-12006] WARN org.eclipse.jetty.http.HttpParser - Illegal character 0x3 in state=START for buffer HeapByteBuffer@33bac9f4[p=1,l=47,c=16384,r=46]={\x03<<<\x00\x00/*\xE0\x00\x00\x00\x00\x00Cookie:...nistr\r\n\x01\x00\x08\x00\x03\x00\x00

Any ideas?

Thanks
Mike

mwpclark4 years ago

Here is the workaround that worked for me this morning.

/opt/traccar/bin/stopDaemon.sh

First I stopped the daemon. But the java usage did not stop.

I killed the java process with kill -9 PID

cpu usage dropped like a stone, went below 1 within a couple minutes.

I restarted the daemon with

/opt/traccar/bin/startDaemon.sh

A new java process was started but is not consuming excessive cpu.

The device tracking system is working.

Not ideal but better than server reboot.

Anton Tananaev4 years ago

There's definitely something wrong. With those specs you should be able to handle thousands of devices.

Two recommendations to start with:

Upgrade to the latest version
Follow optimization guide, if you haven't done it already