Strange Traccar issue with incoming data packets

Victor Butlera year ago

Few weeks back a quite strange issue occurred and I am still unable to find the root cause:

  • At 23:15 the traccar log file stops, no warning messages, no errors
  • At 00:12 on the next day (about 1 hour later), log file resumes. No warnings, no errors, no server restart indications.
  • However, no incoming data is recorded anymore, neither in the log file, nor in the database.
  • The only recorded information in the log file is for user login, reports and actions, i.e. web interface runs fine.
  • Running the top command shows the traccar java process has been running for weeks.
  • The lack of recorded data packets continue until Traccar server restart. Everything goes back to normal after that.
  • UDP protocol is being used.

Example of the log files at the time the issue occurred and before the server restart (notice how the log stops at 23:15):

End of log file for 2023-10-27:

2023-10-27 23:15:03  INFO: [U618096c8] id: XXXXXXXXXX, time: 2023-10-27 23:14:58, lat: XXX, lon: XXX, speed: 63.2, course: 300.0
2023-10-27 23:15:03  INFO: [U618096c8: teltonika < XX.XX.XX.XX] HEX data
2023-10-27 23:15:03  INFO: [U618096c8: teltonika > XX.XX.XX.XX] Response

Start of log file for 2023-10-28:

2023-10-28 00:12:01  INFO: user: 4, action: login, from: XX.XX.XX.XX
2023-10-28 00:12:02  INFO: user: 4, action: edit, object: user, id: 4
2023-10-28 00:12:05  INFO: user: 4, report: events, from: 2023-10-27 00:12, to: 2023-10-28 00:12, devices: [12, 13], groups: []
2023-10-28 00:12:16  INFO: user: 4, report: stops, from: 2023-10-27 05:00, to: 2023-10-28 00:12, devices: [12], groups: []
2023-10-28 00:12:26  INFO: user: 4, report: stops, from: 2023-10-27 05:00, to: 2023-10-27 05:00, devices: [12], groups: []
2023-10-28 00:12:39  INFO: user: 8, action: login, from: XX.XX.XX.XX
2023-10-28 00:12:40  INFO: user: 8, action: edit, object: user, id: 8
2023-10-28 02:16:42  INFO: user: 3, action: unlink, owner: device, id: 12, property: geofence, id: 35

It definitely wasn't a device issue, the traccar server also appeared to be running, just no data packets were being recorded, as if the port was closed/blocked. The issue being fixed by the traccar server restart indicates some issue related to the java process but I cannot find any trace of such.

Any idea what the cause might have been or where to look at?

Anton Tananaeva year ago

You probably have to check jstack result. I suspect all device handling threads must be blocked on something.

Victor Butlera year ago

Good advice, it's too bad I didn't run it back when the issue occurred.

How can I recognize the device handling threads (in case the issue repeats again)?