Server stopped working

5of96 years ago

Hi, yesterday my server 3.17 stopped working with this in the tracker-server.log:

2018-06-03 17:55:44  WARN: Geocoding failed - java.util.concurrent.TimeoutExcept
ion: Request timed out to maps.googleapis.com/172.217.21.234:443 of 60000 ms - E
xecutionException (...)

It worked again when I restarted it today but until this moment it just accepted the messages from the trackers like this:

2018-06-03 23:58:03 DEBUG: [59C2D1D6: 5032 < 80.187.100.111] HEX: 3130.....

But didn't wrote them into the database, so all requests during this error are lost?!
What could be the problem?
Thank you

Anton Tananaev6 years ago

I hear two issues. If Traccar stop working you have to check wrapper log file. As for the second issue, I need to see log fragment with full HEX message and a few lines around it.

5of96 years ago

Hi,
it wasn't a full stop of the server, the website still worked, but the positions get no longer updated.
Here the only entry of the wrapper:
/opt/traccar/logs/wrapper.log.20180603

INFO|1019/0|Service traccar|18-06-03 22:39:00|[HikariPool-1 housekeeper] WARN co
m.zaxxer.hikari.pool.HikariPool - HikariPool-1 - Thread starvation or clock leap
 detected (housekeeper delta=1m1s859ms388µs93ns).

And here the full entrys of the tracker.log:

2018-06-03 17:54:37 DEBUG: [A6CC4548: 5030 < 2.247.254.234] HEX: 233836313130373
03330373834393035234d543630302330303030234155544f23310d0a23643632393230366261623
0244750524d432c3135353433342e30302c412c343832382e323832372c4e2c30313231342e37323
3352c452c342e36372c3133352e31362c3033303631382c2c2c412a35390d0a23230d0a
2018-06-03 17:54:37  INFO: [A6CC4548] id: 861107030784905, time: 2018-06-03 17:5
4:34, lat: xx.47140, lon: xx.24537, speed: 4.7, course: 135.2
2018-06-03 17:54:44 DEBUG: [A6CC4548: 5030 < 2.247.254.234] HEX: 233836313130373
03330373834393035234d543630302330303030234155544f53544f5023310d0a236436323932303
662616230244750524d432c3135353434332e30302c412c343832382e323831392c4e2c303132313
42e373237332c452c302e30322c2c3033303631382c2c2c412a34460d0a23230d0a
2018-06-03 17:54:44  WARN: Geocoding failed - Empty address - GeocoderException 
(JsonGeocoder:69 < *:34 < *:96 < ...)
2018-06-03 17:54:56 DEBUG: [59C2D1D6: 5032 < 80.187.100.111] HEX: 31303030303030
3031302c32303138303630333135353435342c31322e3334303637352c34382e3532353938362c30
2c3335382c3534302c31342c320d0a
2018-06-03 17:55:44  WARN: Geocoding failed - java.util.concurrent.TimeoutExcept
ion: Request timed out to maps.googleapis.com/172.217.21.234:443 of 60000 ms - E
xecutionException (...)
2018-06-03 17:55:55 DEBUG: [A6CC4548: 5030 < 2.247.254.234] HEX: 233836313130373
03330373834393035234d543630302330303030234155544f4c4f5723310d0a23643632393230366
5316330244750524d432c3135353535342e30302c412c343832382e323638372c4e2c30313231342
e373237342c452c302e31362c2c3033303631382c2c2c412a34330d0a23230d0a
2018-06-03 18:00:00 DEBUG: [59C2D1D6: 5032 < 80.187.100.111] HEX: 31303030303030
3031302c32303138303630333135353935372c31322e3334303639312c34382e3532353938332c30
2c3335382c3534302c31342c320d0a
2018-06-03 18:05:04 DEBUG: [59C2D1D6: 5032 < 80.187.100.111] HEX: 31303030303030
3031302c32303138303630333136303530322c31322e3334303730302c34382e3532353938312c30
2c3335382c3533392c31342c320d0a
2018-06-03 18:07:14 DEBUG: [A6CC4548: 5030 < 2.247.254.234] HEX: 233836313130373
03330373834393035234d543630302330303030234155544f535441525423310d0a2364363239323
03665316330244750524d432c3136303731332e30302c412c343832382e323833322c4e2c3031323
1342e373235322c452c302e33302c2c3033303631382c2c2c412a34340d0a23230d0a
2018-06-03 18:07:17 DEBUG: [A6CC4548: 5030 < 2.247.254.234] HEX: 233836313130373
03330373834393035234d543630302330303030234155544f23310d0a23643632393230366531633
0244750524d432c3136303731342e30302c412c343832382e323832372c4e2c30313231342e37323
7322c452c312e35352c33302e34352c3033303631382c2c2c412a36420d0a23230d0a
2018-06-03 18:07:21 DEBUG: [A6CC4548: 5030 < 2.247.254.234] HEX: 233836313130373
03330373834393035234d543630302330303030234155544f23310d0a23643632393230366261623
0244750524d432c3136303732312e30302c412c343832382e323834342c4e2c30313231342e37333
9392c452c372e32302c38312e33302c3033303631382c2c2c412a36300d0a23230d0a
2018-06-03 18:07:25 DEBUG: [A6CC4548: 5030 < 2.247.254.234] HEX: 233836313130373
Anton Tananaev6 years ago

Issue in wrapper log means that Traccar doesn't have enough resources. As for the decoding problem, it seems like you have problem with connecting to Google API, so I would recommend disabling it.

5of96 years ago

I changed the resources from ~250mb to ~650mb ram for wrapper but till now it stopped working two more times. Is it possible to get the accepted but not imported positions into the database or are they lost? Why doesn't it stop fully to prevent accepting messages and doing nothing with them? Do you know a way to restart the service automaticaly in such a case?

5of96 years ago

Updated to openjdk 11 and wrapper has now access to 850mb ram, but doesn't ever uses it, not during working time and not when it hangs. The case that it accepts messages but doesn't use them is pretty bad, because they're all lost, if it wouldn't accept them, the tracker would save them itself... Is there a way to get those not processed messages processed later? Would be a good thing for a new version!?

Jun 14 10:38:47 gps.voit.li traccar[508]: YAJSW: yajsw-stable-12.12
Jun 14 10:38:47 gps.voit.li traccar[508]: OS : Linux/4.15.0-22-generic/amd64
Jun 14 10:38:47 gps.voit.li traccar[508]: JVM : Oracle Corporation/10.0.1//usr/lib/jvm/java-11-openjdk-amd64/64
Anton Tananaev6 years ago

Java 11 is no even released yet. I would recommend using Java 8 or 9.

How many devices do you have?

5of96 years ago

But it seems that it doesn't hang that often with the latest java version of Ubuntu 18.04...
I have 8 devices, and 3 of them working continously, between 1000 and 5000 messages a day

5of96 years ago

It stopped again yesterday with no comprehensibly reason. Do you need any more logfiles or something like that to comprehense that problem? Or do you want me to test anything else?
Thank you

Anton Tananaev6 years ago

Not sure what you can try, but from your logs I can see that there are some network issues:

Request timed out to maps.googleapis.com/172.217.21.234:443 of 60000 ms

Maybe you should investigate those.

5of96 years ago

But these had been with 3.16, too...

5of96 years ago

Hi,
the error is gone... I changed back to Java 8 and gave the wrapper now over 1GB RAM, but despite that I made a script started by a cronjob which restart the server in case there is no position update for more than a hour, maybe others find this useful:

#!/bin/bash
LastPositionTime=`echo "SELECT servertime FROM positions ORDER BY id DESC LIMIT 1;" | mysql gps -u gps | tail -n 1`

JETZT=`date --date='now' +%s`

ANFANG=`date --date="$LastPositionTime" +%s`

echo Letzte Positionsmeldung vor $(( $JETZT - $ANFANG)) sec.
echo Letzte Position von $LastPositionTime

if [ $(( $JETZT - $ANFANG)) -gt 4200 ]; then

echo "Traccar Neustart wegen: Last Position at $LastPositionTime" | /usr/bin/mail -s "Traccar Neustart" florian@mailserver.????

systemctl restart traccar.service

fi