memory usage again

mwpclark 5 years ago

Hello, I have read several posts about memory usage, there seem to be a variety of issues. I have been using traccar server on centos for a while now, 2 years on this server, and for some time there has been a gradually increasing memory usage. Up until this week, a server reboot every couple months would solve the problem. Now the java process goes sky high within a few days. The machine has 6 processors (cores) and 64 GB RAM. It is setup to log tracking to mysql, and there are currently 4 devices reporting every 5 minutes. Previously there were up to 9 devices.

I am the only one on this physical server and to my knowledge there are no other java processes besides traccar.

from top

4173 root      20   0   29.4g 854856   7464 S 501.3  1.3   2163:23 java

Looking through the wrapper.log files I see some different errors.

Here is a sample from last night

[root@srv2 logs]# cat wrapper.log.20200824
FINEST|4173/0|Service traccar|20-08-24 20:00:49|[qtp1590129041-11656] WARN org.eclipse.jetty.http.HttpParser - Illegal character 0x16 in state=START for buffer HeapByteBuffer@3f801468[p=1,l=122,c=16384,r=121]={\x16<<<\x03\x01\x00u\x01\x00\x00q\x03\x03\xBb\xAdq\xEf\xFcu\x1f...\x00\x08\x04\x01\x04\x03\x02\x01\x02\x03\xFf\x01\x00\x01\x00>>>er-Agent: Mozilla...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}
FINEST|4173/0|Service traccar|20-08-24 20:00:49|[qtp1590129041-11656] WARN org.eclipse.jetty.http.HttpParser - badMessage: 400 Illegal character 0x16 for HttpChannelOverHttp@68eacc9d{r=0,c=false,a=IDLE,uri=}
FINEST|4173/0|Service traccar|20-08-24 20:18:41|[qtp1590129041-11698] WARN org.eclipse.jetty.http.HttpParser - Illegal character 0x16 in state=START for buffer HeapByteBuffer@426b52f1[p=1,l=122,c=16384,r=121]={\x16<<<\x03\x01\x00u\x01\x00\x00q\x03\x03Io\xCb\xC8W\x1eT...\x00\x08\x04\x01\x04\x03\x02\x01\x02\x03\xFf\x01\x00\x01\x00>>>25253Ds%2525253Ah...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}
FINEST|4173/0|Service traccar|20-08-24 20:18:41|[qtp1590129041-11698] WARN org.eclipse.jetty.http.HttpParser - badMessage: 400 Illegal character 0x16 for HttpChannelOverHttp@93ac6d2{r=0,c=false,a=IDLE,uri=}
FINEST|4173/0|Service traccar|20-08-24 22:23:45|[qtp1590129041-12007] WARN org.eclipse.jetty.http.HttpParser - Illegal character 0x3 in state=START for buffer HeapByteBuffer@9b71251[p=1,l=47,c=16384,r=46]={\x03<<<\x00\x00/*\xE0\x00\x00\x00\x00\x00Cookie:...nistr\r\n\x01\x00\x08\x00\x03\x00\x00\x00>>>Ds%2525253Ah8%255...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}
FINEST|4173/0|Service traccar|20-08-24 22:23:45|[qtp1590129041-12007] WARN org.eclipse.jetty.http.HttpParser - badMessage: 400 Illegal character 0x3 for HttpChannelOverHttp@60cf8acb{r=0,c=false,a=IDLE,uri=}
FINEST|4173/0|Service traccar|20-08-24 22:23:45|[qtp1590129041-12006] WARN org.eclipse.jetty.http.HttpParser - Illegal character 0x3 in state=START for buffer HeapByteBuffer@47fb388d[p=1,l=47,c=16384,r=46]={\x03<<<\x00\x00/*\xE0\x00\x00\x00\x00\x00Cookie:...nistr\r\n\x01\x00\x08\x00\x03\x00\x00\x00>>>25253Ah8%255Eo%25...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}
FINEST|4173/0|Service traccar|20-08-24 22:23:45|[qtp1590129041-12006] WARN org.eclipse.jetty.http.HttpParser - badMessage: 400 Illegal character 0x3 for HttpChannelOverHttp@4c3863b8{r=0,c=false,a=IDLE,uri=}
FINEST|4173/0|Service traccar|20-08-24 22:23:45|[qtp1590129041-12006] WARN org.eclipse.jetty.http.HttpParser - Illegal character 0x3 in state=START for buffer HeapByteBuffer@426b52f1[p=1,l=47,c=16384,r=46]={\x03<<<\x00\x00/*\xE0\x00\x00\x00\x00\x00Cookie:...nistr\r\n\x01\x00\x08\x00\x03\x00\x00\x00>>>/\xC0+\xC0\x11\xC0\x07\xC0\x13\xC0\t\xC0\x14\xC0\n\x00\x05...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}
FINEST|4173/0|Service traccar|20-08-24 22:23:45|[qtp1590129041-12006] WARN org.eclipse.jetty.http.HttpParser - badMessage: 400 Illegal character 0x3 for HttpChannelOverHttp@4807fee7{r=0,c=false,a=IDLE,uri=}
FINEST|4173/0|Service traccar|20-08-24 22:23:45|[Fatal Error] :1:1: Premature end of file.
FINEST|4173/0|Service traccar|20-08-24 22:23:45|[Fatal Error] :1:1: Premature end of file.
FINEST|4173/0|Service traccar|20-08-24 22:23:45|[Fatal Error] :1:1: Premature end of file.
FINEST|4173/0|Service traccar|20-08-24 22:23:45|[Fatal Error] :1:1: Premature end of file.
FINEST|4173/0|Service traccar|20-08-24 22:23:45|[qtp1590129041-12006] WARN org.eclipse.jetty.http.HttpParser - Illegal character 0x3 in state=START for buffer HeapByteBuffer@33bac9f4[p=1,l=47,c=16384,r=46]={\x03<<<\x00\x00/*\xE0\x00\x00\x00\x00\x00Cookie:...nistr\r\n\x01\x00\x08\x00\x03\x00\x00

Any ideas?

Thanks
Mike

mwpclark 5 years ago

Here is the workaround that worked for me this morning.

/opt/traccar/bin/stopDaemon.sh

First I stopped the daemon. But the java usage did not stop.

I killed the java process with kill -9 PID

cpu usage dropped like a stone, went below 1 within a couple minutes.

I restarted the daemon with

/opt/traccar/bin/startDaemon.sh

A new java process was started but is not consuming excessive cpu.

The device tracking system is working.

Not ideal but better than server reboot.

Anton Tananaev 5 years ago

There's definitely something wrong. With those specs you should be able to handle thousands of devices.

Two recommendations to start with:

Upgrade to the latest version
Follow optimization guide, if you haven't done it already