Devices Intermittently Stop Sending Data to Traccar

Description:

I'm encountering a persistent issue where multiple Teltonika devices intermittently stop sending position data to my Traccar server (version: 6.2). This happens even though the devices are actively transmitting data to another platform (gps-server.com) simultaneously, device stop sending data randomly.

Troubleshooting Performed:

Device Issues Ruled Out:

The devices are functioning properly, proven by their continuous data transmission to gps-server.com.

Traccar Configuration Verified:

I've thoroughly reviewed my traccar.xml configuration, adjusted timeouts (server.timeout, status.timeout) to accommodate potential delays.

System Resources Confirmed Sufficient:

My VPS has ample resources (8 cores, 24GB RAM) and the sysctl.conf settings are optimized for network performance.

Firewall Not Blocking Traffic:

There are no active firewalls on the VPS.

Additional Information:

Device Models:

FMC920, FMB920, FMB130

Hosting:

Vps Contabo

Operating System:

Ubuntu 22.02

Relevant Configuration Files:

sysctl.conf

net.ipv6.conf.all.disable_ipv6 = 1 
kernel.panic = 10 
vm.max_map_count = 250000 
fs.file-max = 1000000 
net.ipv4.ip_local_port_range = 1024 65535 
net.ipv4.tcp_keepalive_time = 60 
net.ipv4.tcp_keepalive_intvl = 10 
net.ipv4.tcp_keepalive_probes = 5 
net.core.somaxconn = 8192 
net.core.netdev_max_backlog = 4096 
net.ipv4.tcp_fin_timeout = 30 
net.ipv4.tcp_tw_reuse = 1 
net.ipv4.tcp_rmem = 8192 131072 33554432 
net.ipv4.tcp_wmem = 8192 131072 33554432 
net.ipv4.tcp_sack = 1 
net.ipv4.tcp_fack = 1

traccar.xml (relevant entries)

...
    key='server.timeout'>30
    key='status.timeout'>300

    key='coordinates.filter'>true
    key='coordinates.minError'>20

    key='event.overspeed.notRepeat'>true
    key='event.motion.speedThreshold'>1.11

    key='filter.enable'>true
    key='filter.invalid'>true
    key='filter.zero'>true
    key='filter.duplicate'>true
    key='filter.distance'>20
    key='filter.maxSpeed'>25000

    key='distance.enable'>true
    key='geocoder.enable'>false
    
    key='report.trip.minimalNoDataDuration'>300
    key='report.periodLimit'>7776000
    key='report.fastThreshold'>90000
...

ulimit -a output

real-time non-blocking time  (microseconds, -R) 
unlimited core file size              (blocks, -c) 0 
data seg size               (kbytes, -d) unlimited 
scheduling priority                 (-e) 0 
file size                   (blocks, -f) unlimited 
pending signals                     (-i) 95884 
max locked memory           (kbytes, -l) 3077408 
max memory size             (kbytes, -m) unlimited 
open files                          (-n) 400000 
pipe size                (512 bytes, -p) 8 
POSIX message queues         (bytes, -q) 819200 
real-time priority                  (-r) 0 
stack size                  (kbytes, -s) 8192 
cpu time                   (seconds, -t) unlimited 
max user processes                  (-u) 95884 
virtual memory              (kbytes, -v) unlimited 
file locks                          (-x) unlimited 

Device configuration snippet (specific to Teltonika models)

Open Link Timeout : 300s 
Response timeout: 30s 
Network Ping Timeout:0s 
When Moving device send position every 10 s

JVM conf:  -java       --Xms20g       --Xmx20g       --Djava.net.preferIPv4Stack= true

Logs: I can provide relevant Traccar logs if needed for further analysis.

Here is logs on a period when a specific device didn't send position to traccar but send it to gps-server live:

2024-08-30 12:30:05  INFO: Position filtered by Distance filters from device: 350612074333343 
2024-08-30 12:30:05  INFO: [T9478a17b] id: 350612074333343, time: 2024-08-30 12:28:27, lat: 34.96829, lon: -3.37329, speed: 18.4, course: 228.0 
2024-08-30 12:30:05  INFO: [T9478a17b] id: 350612074333343, time: 2024-08-30 12:28:29, lat: 34.96822, lon: -3.37351, speed: 20.0, course: 245.0 
2024-08-30 12:30:35  INFO: Event id: 350612074333343, time: 2024-08-30 12:30:35, type: deviceOffline, notifications: 0 
2024-08-30 13:20:02  INFO: Event id: 350612074333343, time: 2024-08-30 13:20:02, type: deviceOnline, notifications: 0 
2024-08-30 13:20:32  INFO: Event id: 350612074333343, time: 2024-08-30 13:20:32, type: deviceOffline, notifications: 0 
2024-08-30 14:20:02  INFO: Event id: 350612074333343, time: 2024-08-30 14:20:02, type: deviceOnline, notifications: 0 
2024-08-30 14:20:32  INFO: Event id: 350612074333343, time: 2024-08-30 14:20:32, type: deviceOffline, notifications: 0 
2024-08-30 16:00:51  INFO: Event id: 350612074333343, time: 2024-08-30 16:00:51, type: deviceOnline, notifications: 0 
2024-08-30 16:00:52  INFO: [Tba4fe1ce] id: 350612074333343, time: 2024-08-30 15:42:05, lat: 35.04020, lon: -2.96931, course: 315.0

Between 14:00 and 16:30, the device was successfully sending position data to gps-server. I tested by configuring the device to connect exclusively to Traccar and then exclusively to gps-server. The device only transmitted data when connected to gps-server.

Here is some random line of logs:

...
2024-08-30 12:32:11  INFO: [T0c6c9b19] error - Adjusted frame length exceeds 1099: 50165 - discarded - TooLongFrameException (... < WrapperInboundHandler:57 < ... < StandardLoggingHandler:62 < ... < NetworkMessageHandler:37 < ...)
2024-08-30 12:32:26  INFO: [T50e0078b] error - Adjusted frame length exceeds 1099: 8473 - discarded - TooLongFrameException (... < WrapperInboundHandler:57 < ... < StandardLoggingHandler:62 < ... < NetworkMessageHandler:37 < ...)
2024-08-30 12:32:46  INFO: [T4b8f454b] error - Connection reset by peer - IOException (...)
2024-08-30 12:32:51  INFO: [T6842f44a] error - Connection reset by peer - IOException (...)
2024-08-30 12:32:51  INFO: [T70823f58] error - Connection reset by peer - IOException (...)
...
2024-08-30 14:21:23  INFO: [T0ad70957] error - begin 21, end 24, length 5 - StringIndexOutOfBoundsException (... < Gps103ProtocolDecoder:413 < ExtendedObjectDecoder:75 < ... < WrapperContext:102 < ...)
...
2024-08-30 14:23:54  INFO: [Tcc8c957e] error - readerIndex: 86, writerIndex: 82 (expected: 0 <= readerIndex <= writerIndex <= capacity(82)) - IndexOutOfBoundsException (... < TeltonikaProtocolDecoder:359 < *:548 < *:692 < *:737 < *:725 < ExtendedObjectDecoder:75 < ... < WrapperContext:102 < ...)

Questions:

Have you encountered similar issues with Teltonika devices and Traccar?
Based on the provided information, are there any potential causes for the intermittent disconnect?
What additional troubleshooting steps would you recommend to diagnose the root cause of this problem?

Thank you for your time and expertise.

I recommend upgrading to the latest version first.

Hi Anton,
Thanks for your quick response. I apologize for the delay, I wanted to monitor the situation for a longer period before getting back to you.
I upgraded to Traccar version 6.4 yesterday and monitored the system throughout the day. Unfortunately, the issue persists. It seems similar to the one reported here: https://github.com/traccar/traccar/issues/5393, but it's happening with Teltonika devices. I've also conducted thorough tests to rule out any device-related issues.
If you have any further suggestions or troubleshooting steps please don't hesitate, I'd appreciate it.

What do you see in the logs?

Here is one of the devices I was monitoring and facing the issue: Imei : 866069065516856
The device went offline between 13:42:29 and 15:14:08, but it was still moving and sending data to another server.

2024-08-31 12:41:59  INFO: [Tc19652ce] id: 866069065516856, time: 2024-08-31 12:41:54, lat: 35.23868, lon: -2.89693, course: 58.0
2024-08-31 12:42:29  INFO: Event id: 866069065516856, time: 2024-08-31 12:42:29, type: deviceOffline, notifications: 0
2024-08-31 13:41:57  INFO: Event id: 866069065516856, time: 2024-08-31 13:41:57, type: deviceOnline, notifications: 0
2024-08-31 13:41:58  INFO: Position filtered by Distance filters from device: 866069065516856
2024-08-31 13:42:28  INFO: Event id: 866069065516856, time: 2024-08-31 13:42:28, type: deviceOffline, notifications: 0
2024-08-31 15:14:08  INFO: Event id: 866069065516856, time: 2024-08-31 15:14:08, type: deviceOnline, notifications: 0
2024-08-31 15:14:17  INFO: Position filtered by Distance filters from device: 866069065516856
2024-08-31 15:14:17  INFO: [Tf815fef5] id: 866069065516856, time: 2024-08-31 14:37:03, lat: 35.23868, lon: -2.89693, course: 125.0
2024-08-31 15:14:17  INFO: [Tf815fef5] id: 866069065516856, time: 2024-08-31 14:38:07, lat: 35.23868, lon: -2.89693, course: 125.0
2024-08-31 15:14:17  INFO: [Tf815fef5] id: 866069065516856, time: 2024-08-31 14:38:11, lat: 35.23868, lon: -2.89693, course: 125.0

Filtered with the connection: Tc19652ce

2024-08-31 12:41:58  INFO: [Tc19652ce] connected
2024-08-31 12:41:58  INFO: [Tc19652ce: teltonika < 105.74.10.146] 000f383636303639303635353136383536
2024-08-31 12:41:59  INFO: [Tc19652ce: teltonika > 105.74.10.146] 01
2024-08-31 12:41:59  INFO: [Tc19652ce: teltonika < 105.74.10.146] 00000000000000588e0100000191a874d25000fe45f84d1500ff540014003a10000000f0000c000500ef0000f00000150400c800004501000500b5000900b60006004231c10043100d00440000000200f10000ebf20010005375e600000000010000dcdf
2024-08-31 12:41:59  INFO: [Tc19652ce] id: 866069065516856, time: 2024-08-31 12:41:54, lat: 35.23868, lon: -2.89693, course: 58.0
2024-08-31 12:41:59  INFO: [Tc19652ce: teltonika > 105.74.10.146] 00000001
2024-08-31 12:42:29  INFO: [Tc19652ce] timed out
2024-08-31 12:42:29  INFO: [Tc19652ce] disconnected

Filtered with 105.74.10.146

2024-08-31 12:41:59  INFO: [Tc19652ce: teltonika > 105.74.10.146] 00000001
2024-08-31 13:41:57  INFO: [Tcc112aad: teltonika < 105.74.10.146] 000f383636303639303635353136383536
2024-08-31 13:41:57  INFO: [Tcc112aad: teltonika > 105.74.10.146] 01
2024-08-31 13:41:58  INFO: [Tcc112aad: teltonika < 105.74.10.146] 00000000000000588e0100000191a8abc0d000fe45f84d1500ff540014003a0f00000000000c000500ef0000f00000150400c800004501000500b5000c00b600060042326a0043101200440000000200f10000ebf20010005375e600000000010000d995
2024-08-31 13:41:58  INFO: [Tcc112aad: teltonika > 105.74.10.146] 00000001
2024-08-31 15:14:08  INFO: [Tf815fef5: teltonika < 105.74.10.146] 000f383636303639303635353136383536
2024-08-31 15:14:08  INFO: [Tf815fef5: teltonika > 105.74.10.146] 01
2024-08-31 15:14:17  INFO: [Tf815fef5: teltonika < 105.74.10.146] 00000000000004a98e0e00000191a8de3e9800fe45f84d1500ff54001a003a1000000000000c000500ef0000f00000150300c800004501000500b5000a00b60006004231ae0043101200440000000200f10000ebf20010005375e60000000000000191a8de3ea200fe45f4331500fe27001a007d10000000f0000c000500ef0000f00100150300c800004501000500b5000a00b60006004231af0043101200440000000200f10000ebf20010005375e60000000000000191a8df389800fe45f4331500fe27001a007d10000000f0000c000500ef0000f00000150300c800004501000500b5000a00b60006004231e80043101200440000000200f10000ebf20010005375e60000000000000191a8df483800fe45f4331500fe27001a007d10000000f0000c000500ef0000f00100150300c800004501000500b5000a00b60006004231a40043101100440000000200f10000ebf20010005375e60000000000000191a8df7b0000fe45f4331500fe27001a007d10000000ef000c000500ef0100f00100150400c800004501000500b5000a00b60006004235320043101200440000000200f10000ebf20010005375e60000000000000191a8e0617800fe45f4331500fe27001a007d10000000f0000c000500ef0100f00000150300c800004501000500b5000a00b600060042370d0043101200440000000200f10000ebf20010005375e60000000000000191a8e078e800fe45f4331500fe27001a00f410000700f0000c000500ef0100f00100150400c800004501000500b5000a00b60006004232ac0043101200440000000200f10000ebf20010005375ec0000000000000191a8e07cd000fe45f07d1500fe8c001a00e210000700ef000c000500ef0000f00100150400c800004501000500b5000a00b60006004231940043101100440000000200f10000ebf20010005375f50000000000000191a8e0982800fe45ed8f1500fe5a001a00fb10000a0000000c000500ef0000f00100150400c800004501000500b5000a00b60006004233390043101200440000000200f10000ebf20010005375fa0000000000000191a8e09c1000fe45ec201500fe17001a00ef10000c0000000c000500ef0000f00100150400c800004501000500b5000d00b600060042322d0043101200440000000200f10000ebf20010005375ff0000000000000191a8e0abb000fe45e4611500f9ec001900e21000120000000c000500ef0000f00100150500c800004501000500b5000d00b60006004230ae0043101200440000000200f10000ebf20010005376150000000000000191a8e0bf3800fe45dd271500f359001900ed10000c0000000c000500ef0000f00100150500c800004501000500b5000d00b60006004233da0043101200440000000200f10000ebf200100053762e0000000000000191a8e0bf4200fe45dd271500f359001900ed10000c00ef000c000500ef0100f00100150500c800004501000500b5000d00b60006004232460043101200440000000200f10000ebf200100053762e0000000000000191a8e0c32000fe45dbea1500f2f5001900fc1000090000000c000500ef0000f00100150500c800004501000500b5000d00b60006004231310043101200440000000200f10000ebf2001000537631000000000e0000cee3
2024-08-31 15:14:18  INFO: [Tf815fef5: teltonika > 105.74.10.146] 0000000e
2024-08-31 15:16:42  INFO: [T6b613cc5: teltonika < 105.74.10.146] 000f383636303639303635353136383536
2024-08-31 15:16:43  INFO: [T6b613cc5: teltonika > 105.74.10.146] 01

Filtred with: Tcc112aad to see what happened during 13:42:29

2024-08-31 13:41:57  INFO: [Tcc112aad] connected
2024-08-31 13:41:57  INFO: [Tcc112aad: teltonika < 105.74.10.146] 000f383636303639303635353136383536
2024-08-31 13:41:57  INFO: [Tcc112aad: teltonika > 105.74.10.146] 01
2024-08-31 13:41:58  INFO: [Tcc112aad: teltonika < 105.74.10.146] 00000000000000588e0100000191a8abc0d000fe45f84d1500ff540014003a0f00000000000c000500ef0000f00000150400c800004501000500b5000c00b600060042326a0043101200440000000200f10000ebf20010005375e600000000010000d995
2024-08-31 13:41:58  INFO: [Tcc112aad: teltonika > 105.74.10.146] 00000001
2024-08-31 13:42:28  INFO: [Tcc112aad] timed out
2024-08-31 13:42:28  INFO: [Tcc112aad] disconnected

I apologize for any data loss that may have occurred during my analysis. I couldn't find any errors in the logs at that time. Please let me know if there's any additional information I can provide.

I don't see any indication that it's a Traccar issue. If you still think that device is sending some data and you don't see it in Traccar for whatever reason, I recommend using something like Wireshark to show us the proof.

It more likely that the host might not be accepting all connections for some reason. I'll try to capture TCP dumps every 15 minutes for about 2-3 hours tomorrow and then provide an update.

"...but it was still moving and sending data to another server."
Are you have the traccar sever as secondary server in your device configuration?
Can you share screen shot of GPRS settings of Teltonica configurator?

@Anton Tananaev I'll attach the tcpdump showing a device before the connection with the server was stopped

In the meantime, we let the device ( blocked ) on traccar side, and monitor the other gps server, the device was working fine we let it for about 2 hours, after that we restarted the device and It starts sending again to Traccar ( restarting the device didn't work every time sometimes even after restarting it won't send), That could lead us to mistakenly believe the issue is on the device side, for that we configured some devices to only send to Traccar without implementing any backup server and the problem still occurs, we took that one of the devices and add the backup server to it it starts sending to the other server, then return it back to Traccar and to our surprise it did not send.
For the log side, we couldn't find any errors, we're suspecting the host not accepting the data, more than suspecting a Traccar side issue but we could confirm nothing until now.
Another observation, is that the device probably loose coverage before the problem occurs, we can't confirm that neither for sure, but we prefer to take it in consideration.

@Kaloyan Kanev We're configuring our devices with SMS, and we're using the 2004 and 2007 to set the servers ip and 2005 and 2008 to set ports for the backup type we use the 2010:2, which mean that the device should send to the both server. for a test purpose we cancelled the second server in some devices, or add it to check if the device block on the second server too.

This is not what we wanted to check, right? If you think it's a server issue, you have provide evidence that your device is trying to connect or send some data, but Traccar doesn't accept it and doesn't show anything in the logs.

"you have provide evidence that your device is trying to connect or send some data"
I don't have the same expertise you have on the field, I couldn't find a filter to know what the exact device did, I could not find the ip he'll use ( if he's trying to connect after ) and it's emblematic cause we have multiple devices (about 1600). When the device make a new connection after he has another ip address, I tried with meta.imei or meta.imsi but seems not to work, I'm searching for a way to filter, I'll be back with news later.

Well, you have to isolate it somehow. Possibly get a fixed IP from your provider or get a SIM from another provider so you can filter based on that.

I have some useful data, we configured the device to send to another instance of Traccar we use for testing in parallel of the primary one, and here it is the log:

As we can see all is working fine, I'll attach the log from the primary traccar:

Here we can notice a gap time in the log, the following position:

2024-09-01 17:11:55  INFO: [Tbb1d122a] id: 350612074319516, time: 2024-09-01 17:11:47, lat: 35.15005, lon: -2.97964, course: 0.0

Which was sent at 17:11:55 to the second server, was sent to the first only at 17:20, when I check on the tcpdump I found that there was no connection with the host at 17:11 but there was multiple at 17:13 without resulting on havinf positions only some event due to the connection made.
( PS: the difference between the log time and the tcpdump time is 1 hour, 17:00 in the log is equivalent to 18:00 in tcpdump)

You have to provide more context. Which IP is your device?

Yes for sure, the device ip is : 105.67.5.41