Data packet coming in chunks

Hi, I recently upgraded to server 4.2. I notice a weird pattern in the data. They are coming in a bunch. As a result, the WebSocket pushes data every 3-4 minutes. That's hampering the real-time tracking. I have around 400 devices sending data at 20s intervals. The server is hosted on a t2large instance. All suggested optimisations are done. I am pasting some data dumps below. What can be the possible causes for this ?

2019-01-11 16:50:14  INFO: [39222ede: 5013 > 42.110.131.161] HEX: 2a48512c3836353230353033343131313430302c56342c56312c323031393031313131363530313423
2019-01-11 16:50:14  INFO: [39222ede: 5013 > 42.110.131.161] HEX: 2a48512c3836353230353033343131313430302c56342c56312c323031393031313131363530313423
2019-01-11 16:50:14  INFO: [8dee3d67] id: 865205034107622, time: 2019-01-11 16:49:54, lat: 22.87748, lon: 87.79084, course: 299.0
2019-01-11 16:50:14  INFO: [627fd342: 5013 < 42.110.131.228] HEX: 2a48512c3836353230353033333031353230362c48544254230d0a2a48512c3836353230353033333031353230362c56312c3136343133372c412c323331392e39383136322c4e2c30383832372e37323633322c452c33302e33312c32382c3131303131392c4646464646464646230d0a2a48512c3836353230353033333031353230362c56312c3136343135372c412c323332302e31323737342c4e2c30383832372e38313133362c452c32382e38392c32382c3131303131392c4646464646464646230d0a2a48512c3836353230353033333031353230362c56312c3136343231372c412c323332302e32323230352c4e2c30383832372e38363631332c452c302e30302c32392c3131303131392c4646464646464646230d0a2a48512c3836353230353033333031353230362c56312c3136343233372c412c323332302e32353236382c4e2c30383832372e38383338322c452c31332e30322c31392c3131303131392c4646464646464646230d0a2a48512c3836353230353033333031353230362c56312c3136343235372c412c323332302e33333833322c4e2c30383832372e39333236372c452c32302e35342c32392c3131303131392c4646464646464646230d0a2a48512c3836353230353033333031353230362c56312c3136343331372c412c323332302e34353432302c4e2c30383832382e30303034302c452c32362e38312c32372c3131303131392c4646464646464646230d0a2a48512c3836353230353033333031353230362c48544254230d0a2a48512c3836353230353033333031353230362c56312c3136343333372c412c323332302e35393034382c4e2c30383832382e30373932382c452c32382e34392c32372c3131303131392c4646464646464646230d0a2a48512c3836353230353033333031353230362c56312c3136343335372c412c323332302e37323933352c4e2c30383832382e31353736342c452c32392e30342c32372c3131303131392c4646464646464646230d0a2a48512c3836353230353033333031353230362c56312c3136343431372c412c323332302e38373232362c4e2c30383832382e32333932372c452c32382e33392c32362c3131303131392c4646464646464646230d0a2a48512c3836353230353033333031353230362c56312c3136343433372c412c323332312e30313239372c4e2c30383832382e33313435362c452c32372e32362c32372c3131303131392c4646464646464646230d0a2a48512c3836353230353033333031353230362c56312c3136343435372c412c323332312e31353136302c4e2c30383832382e33383636362c452c32382e38332c32322c3131303131392c4646464646464646230d0a2a48512c3836353230353033333031353230362c56312c3136
2019-01-11 16:50:14  INFO: [627fd342: 5013 > 42.110.131.228] HEX: 2a48512c3836353230353033333031353230362c56342c56312c323031393031313131363530313423
2019-01-11 16:50:14  INFO: [29f36ac8] id: 865205034110709, time: 2019-01-11 16:45:44, lat: 22.37987, lon: 87.36274, speed: 15.8, course: 60.0
2019-01-11 16:50:14  INFO: [58bc2987] id: 865205034114594, time: 2019-01-11 16:46:57, lat: 23.53575, lon: 84.91956, course: 122.0
2019-01-11 16:50:14  INFO: [58bc2987: 5013 > 42.110.129.62] HEX: 2a48512c3836353230353033343131343539342c56342c56312c323031393031313131363530313423
2019-01-11 16:50:14  INFO: [39222ede: 5013 > 42.110.131.161] HEX: 2a48512c3836353230353033343131313430302c56342c56312c323031393031313131363530313423
2019-01-11 16:50:15  INFO: [39222ede: 5013 > 42.110.131.161] HEX: 2a48512c3836353230353033343131313430302c56342c56312c323031393031313131363530313523
2019-01-11 16:50:15  INFO: [627fd342: 5013 > 42.110.131.228] HEX: 2a48512c3836353230353033333031353230362c56342c56312c323031393031313131363530313523
2019-01-11 16:50:15  INFO: [29f36ac8] id: 865205034110709, time: 2019-01-11 16:46:04, lat: 22.38033, lon: 87.36347, course: 43.0
2019-01-11 16:50:15  INFO: [39222ede: 5013 > 42.110.131.161] HEX: 2a48512c3836353230353033343131313430302c56342c56312c323031393031313131363530313523
2019-01-11 16:50:15  INFO: [627fd342: 5013 > 42.110.131.228] HEX: 2a48512c3836353230353033333031353230362c56342c56312c323031393031313131363530313523
2019-01-11 16:50:15  INFO: [58bc2987] id: 865205034114594, time: 2019-01-11 16:47:17, lat: 23.53575, lon: 84.91956, course: 122.0
2019-01-11 16:50:15  INFO: [58bc2987: 5013 > 42.110.129.62] HEX: 2a48512c3836353230353033343131343539342c56342c56312c323031393031313131363530313523
2019-01-11 16:50:15  INFO: [39222ede: 5013 > 42.110.131.161] HEX: 2a48512c3836353230353033343131313430302c56342c56312c323031393031313131363530313523
2019-01-11 16:50:15  INFO: [627fd342: 5013 > 42.110.131.228] HEX: 2a48512c3836353230353033333031353230362c56342c56312c323031393031313131363530313523
2019-01-11 16:50:15  INFO: [29f36ac8] id: 865205034110709, time: 2019-01-11 16:46:24, lat: 22.38074, lon: 87.36435, speed: 15.2, course: 63.0
2019-01-11 16:50:15  INFO: [39222ede: 5013 > 42.110.131.161] HEX: 2a48512c3836353230353033343131313430302c56342c56312c323031393031313131363530313523
2019-01-11 16:50:16  INFO: [627fd342: 5013 > 42.110.131.228] HEX: 2a48512c3836353230353033333031353230362c56342c56312c323031393031313131363530313623
2019-01-11 16:50:16  INFO: [627fd342: 5013 > 42.110.131.228] HEX: 2a48512c3836353230353033333031353230362c56342c56312c323031393031313131363530313623
2019-01-11 16:50:16  INFO: [29f36ac8] id: 865205034110709, time: 2019-01-11 16:46:44, lat: 22.38144, lon: 87.36591, speed: 15.8, course: 66.0
2019-01-11 16:50:16  INFO: [58bc2987] id: 865205034114594, time: 2019-01-11 16:47:37, lat: 23.53575, lon: 84.91956, course: 122.0
2019-01-11 16:50:16  INFO: [58bc2987: 5013 > 42.110.129.62] HEX: 2a48512c3836353230353033343131343539342c56342c56312c323031393031313131363530313623
2019-01-11 16:50:16  INFO: [39222ede: 5013 > 42.110.131.161] HEX: 2a48512c3836353230353033343131313430302c56342c56312c323031393031313131363530313623
2019-01-11 16:50:16  INFO: [39222ede: 5013 > 42.110.131.161] HEX: 2a48512c3836353230353033343131313430302c56342c56312c323031393031313131363530313623
2019-01-11 16:50:16  INFO: [627fd342: 5013 > 42.110.131.228] HEX: 2a48512c3836353230353033333031353230362c56342c56312c323031393031313131363530313623
2019-01-11 16:50:17  INFO: [29f36ac8] id: 865205034110709, time: 2019-01-11 16:47:04, lat: 22.38161, lon: 87.36734, speed: 13.9, course: 97.0
2019-01-11 16:50:17  INFO: [39222ede: 5013 > 42.110.131.161] HEX: 2a48512c3836353230353033343131313430302c56342c56312c323031393031313131363530313723
2019-01-11 16:50:17  INFO: [627fd342: 5013 > 42.110.131.228] HEX: 2a48512c3836353230353033333031353230362c56342c56312c323031393031313131363530313723
2019-01-11 16:50:17  INFO: [58bc2987] id: 865205034114594, time: 2019-01-11 16:47:57, lat: 23.53575, lon: 84.91956, course: 122.0
2019-01-11 16:50:17  INFO: [58bc2987: 5013 > 42.110.129.62] HEX: 2a48512c3836353230353033343131343539342c56342c56312c323031393031313131363530313723
2019-01-11 16:50:17  INFO: [39222ede: 5013 > 42.110.131.161] HEX: 2a48512c3836353230353033343131313430302c56342c56312c323031393031313131363530313723
2019-01-11 16:50:17  INFO: [627fd342: 5013 > 42.110.131.228] HEX: 2a48512c3836353230353033333031353230362c56342c56312c323031393031313131363530313723
2019-01-11 16:50:17  INFO: [29f36ac8] id: 865205034110709, time: 2019-01-11 16:47:24, lat: 22.38142, lon: 87.36892, speed: 19.9, course: 96.0
2019-01-11 16:50:17  INFO: [d04c1fdc] connected
2019-01-11 16:50:17  INFO: [39222ede: 5013 > 42.110.131.161] HEX: 2a48512c3836353230353033343131313430302c56342c56312c323031393031313131363530313723
2019-01-11 16:50:17  INFO: [627fd342: 5013 > 42.110.131.228] HEX: 2a48512c3836353230353033333031353230362c56342c56312c323031393031313131363530313723
2019-01-11 16:50:17  INFO: [b51888d2: 5013 < 42.110.131.149] HEX: 2a48512c3836353230353033333131323233352c56312c3136343633352c412c323234382e38343139362c4e2c30383831342e36373632322c452c302e30302c302c3131303131392c4646464646424646230d0a2a48512c3836353230353033333131323233352c56312c3136343635352c412c323234382e38343139362c4e2c30383831342e36373632322c452c302e30302c302c3131303131392c4646464646424646230d0a2a48512c3836353230353033333131323233352c56312c3136343731352c412c323234382e38343139362c4e2c30383831342e36373632322c452c302e30302c302c3131303131392c4646464646424646230d0a2a48512c3836353230353033333131323233352c56312c3136343733352c412c323234382e38343139362c4e2c30383831342e36373632322c452c302e30302c302c3131303131392c4646464646424646230d0a2a48512c3836353230353033333131323233352c56312c3136343735352c412c323234382e38343139362c4e2c30383831342e36373632322c452c302e30302c302c3131303131392c4646464646424646230d0a2a48512c3836353230353033333131323233352c56312c3136343831352c412c323234382e38343139362c4e2c30383831342e36373632322c452c302e30302c302c3131303131392c4646464646424646230d0a2a48512c3836353230353033333131323233352c48544254230d0a2a48512c3836353230353033333131323233352c56312c3136343833352c412c323234382e38343139362c4e2c30383831342e36373632322c452c302e30302c302c3131303131392c4646464646424646230d0a2a48512c3836353230353033333131323233352c56312c3136343835352c412c323234382e38343139362c4e2c30383831342e36373632322c452c302e30302c302c3131303131392c4646464646424646230d0a2a48512c3836353230353033333131323233352c56312c3136343931352c412c323234382e38343139362c4e2c30383831342e36373632322c452c302e30302c302c3131303131392c4646464646424646230d0a2a48512c3836353230353033333131323233352c56312c3136343933352c412c323234382e38343139362c4e2c30383831342e36373632322c452c302e30302c302c3131303131392c4646464646424646230d0a2a48512c3836353230353033333131323233352c56312c3136343935362c412c323234382e38343139362c4e2c30383831342e36373632322c452c302e30302c302c3131303131392c4646464646424646230d0a
2019-01-11 16:50:17  INFO: [b51888d2: 5013 > 42.110.131.149] HEX: 2a48512c3836353230353033333131323233352c56342c56312c323031393031313131363530313723
2019-01-11 16:50:17  INFO: [627fd342: 5013 > 42.110.131.228] HEX: 2a48512c3836353230353033333031353230362c56342c56312c323031393031313131363530313723
2019-01-11 16:50:17  INFO: [3e10d038] id: 865205034111590, time: 2019-01-11 16:43:38, lat: 23.38047, lon: 85.30487, speed: 11.1, course: 102.0
2019-01-11 16:50:17  INFO: [58bc2987] id: 865205034114594, time: 2019-01-11 16:48:17, lat: 23.53575, lon: 84.91956, course: 122.0
2019-01-11 16:50:17  INFO: [58bc2987: 5013 > 42.110.129.62] HEX: 2a48512c3836353230353033343131343539342c56342c56312c323031393031313131363530313723
2019-01-11 16:50:17  INFO: [627fd342: 5013 < 42.110.131.228] HEX: 343531372c412c323332312e32393531302c4e2c30383832382e34353834312c452c32382e34302c32352c3131303131392c4646464646464646230d0a2a48512c3836353230353033333031353230362c48544254230d0a2a48512c3836353230353033333031353230362c56312c3136343533372c412c323332312e34343031322c4e2c30383832382e35333332332c452c32392e32362c32352c3131303131392c4646464646464646230d0a2a48512c3836353230353033333031353230362c56312c3136343535372c412c323332312e35383431342c4e2c30383832382e36313132362c452c32372e39312c32352c3131303131392c4646464646464646230d0a2a48512c3836353230353033333031353230362c56312c3136343631372c412c323332312e37323533312c4e2c30383832382e36383439352c452c32372e39392c32352c3131303131392c4646464646464646230d0a2a48512c3836353230353033333031353230362c56312c3136343633372c412c323332312e38363733372c4e2c30383832382e37353931332c452c32382e36312c32362c3131303131392c4646464646464646230d0a2a48512c3836353230353033333031353230362c56312c3136343635372c412c323332322e30303937352c4e2c30383832382e38333331332c452c32372e34382c32352c3131303131392c4646464646464646230d0a2a48512c3836353230353033333031353230362c56312c3136343731382c412c323332322e31333138322c4e2c30383832382e38393634362c452c32312e30302c32342c3131303131392c4646464646464646230d0a2a48512c3836353230353033333031353230362c48544254230d0a2a48512c3836353230353033333031353230362c56312c3136343733382c412c323332322e32343733342c4e2c30383832382e39353636312c452c32342e35362c32342c3131303131392c4646464646464646230d0a2a48512c3836353230353033333031353230362c56312c3136343735382c412c323332322e33373932332c4e2c30383832392e30323537322c452c32382e34342c32372c3131303131392c4646464646464646230d0a2a48512c3836353230353033333031353230362c56312c3136343831382c412c323332322e35323535332c4e2c30383832392e31303038322c452c32382e39382c32352c3131303131392c4646464646464646230d0a2a48512c3836353230353033333031353230362c56312c3136343833382c412c323332322e36363631302c4e2c30383832392e31373337372c452c32352e33342c32352c3131303131392c4646464646464646230d0a2a48512c3836353230353033333031353230362c56312c3136343835382c412c323332322e37363835332c4e2c30383832392e32323430342c452c32312e38372c32332c3131303131392c4646464646464646230d0a2a48512c3836353230353033333031353230362c56312c3136343931382c412c323332322e38383036372c4e2c30383832392e32373334322c452c31372e38382c31382c3131303131392c4646464646464646230d0a2a48512c3836353230353033333031353230362c48544254230d0a2a48512c3836353230353033333031353230362c56312c3136343933382c412c323332322e39353630382c4e2c30383832392e33303435302c452c31312e37302c32342c3131303131392c4646464646464646230d0a2a48512c3836353230353033333031353230362c56312c3136343935382c412c323332332e30333735342c4e2c30383832392e33333736382c452c32302e34312c32302c3131303131392c4646464646464646230d0a
2019-01-11 16:50:17  INFO: [627fd342: 5013 > 42.110.131.228] HEX: 2a48512c3836353230353033333031353230362c56342c56312c323031393031313131363530313723
2019-01-11 16:50:18  INFO: [3e10d038] id: 865205034111590, time: 2019-01-11 16:43:58, lat: 23.38026, lon: 85.30604, speed: 13.4, course: 98.0
2019-01-11 16:50:18  INFO: [b51888d2] id: 865205033112235, time: 2019-01-11 16:46:35, lat: 22.81403, lon: 88.24460, course: 0.0
2019-01-11 16:50:18  INFO: [b51888d2: 5013 > 42.110.131.149] HEX: 2a48512c3836353230353033333131323233352c56342c56312c323031393031313131363530313823
2019-01-11 16:50:18  INFO: [627fd342: 5013 > 42.110.131.228] HEX: 2a48512c3836353230353033333031353230362c56342c56312c323031393031313131363530313823
2019-01-11 16:50:18  INFO: [58bc2987] id: 865205034114594, time: 2019-01-11 16:48:37, lat: 23.53575, lon: 84.91956, course: 122.0
2019-01-11 16:50:18  INFO: [627fd342: 5013 > 42.110.131.228] HEX: 2a48512c3836353230353033333031353230362c56342c56312c323031393031313131363530313823
2019-01-11 16:50:18  INFO: [3e10d038] id: 865205034111590, time: 2019-01-11 16:44:18, lat: 23.38001, lon: 85.30747, speed: 13.9, course: 99.0
2019-01-11 16:50:18  INFO: [58bc2987: 5013 > 42.110.129.62] HEX: 2a48512c3836353230353033343131343539342c56342c56312c323031393031313131363530313823
2019-01-11 16:50:18  INFO: [627fd342: 5013 > 42.110.131.228] HEX: 2a48512c3836353230353033333031353230362c56342c56312c323031393031313131363530313823
2019-01-11 16:50:18  INFO: [b51888d2] id: 865205033112235, time: 2019-01-11 16:46:55, lat: 22.81403, lon: 88.24460, course: 0.0
2019-01-11 16:50:18  INFO: [b51888d2: 5013 > 42.110.131.149] HEX: 2a48512c3836353230353033333131323233352c56342c56312c323031393031313131363530313823
2019-01-11 16:50:18  INFO: [627fd342: 5013 > 42.110.131.228] HEX: 2a48512c3836353230353033333031353230362c56342c56312c323031393031313131363530313823

Sounds like a device issue to me. Why is it under server?

All 400 devices are reporting data this way? Is there any possibility of data accumulating in the buffer give the fact that almost all devices report location packets to only one single port every 20s?

As for the forum category, where do you suggest that I ask this ?

Unless your server hardware can't handle the load, data should not accumulate. For device issues there is a separate forum.

Where can the bottleneck be? I am using a 2 vcpu, 8 GB ram system running on Ubuntu. All suggested optimizations have been implemented.

Usually bottleneck is database or it can be third party service like reverse geocoding.

Thanks, I'll disable reverse geocoding and check if that works out. The DB is an AWS RDS on an m4large instance. Hopefullt, that can handle the load.

Tried disabling the reverse geocoding. The problem persists. I can't understand why the data is coming in bursts. This wasn't the case earlier.

Any ideas anyone? How this issue can be debugged?

Put one of the devices on a separate server and see if it happens.

@Anton I pointed a device to demo.traccar.org to test. It's pretty accurate in terms of receiving data packets. Checked the websocket data. I feel the tcp buffer is choking up due to a large number of requests coming to a specific port in such short durations. But I don't know how to debug that. Any pointers ?

Hi,

I reduced the number of devices to 2 on the same server. Seems to work flawlessly. Is there any way to check why the packets are arriving in chunks?

Attaching logs ( after reducing no of active devices to 2 )