System useg 100% Traccar 6.4

Navneet Choudhary2 months ago

I am using traccar 6.4 (Latest) with 2intel Cpu and 4 GB ram 120GB SSD with 1000 Devices.

I can see a pattern in the digital ocean logs

Can anyone help me how to resolve this issue and what can be the problem?

Previous version 5.12 was working well for 1000 devices.

Screenshot 2024-09-16 at 13.10.41.png

diagnoz2 months ago

Look to log and find events when cpu were 100%
Problem perhabs with users who execute reports.

Anton Tananaev2 months ago

You can also collect jstack to see what's happening when CPU is at 100%.

Navneet Choudhary2 months ago

Here is the system log using top command

Screenshot 2024-09-17 at 17.26.00.png

Navneet Choudhary2 months ago

here is snap with htop command.

Screenshot 2024-09-17 at 17.28.27.pngs

Anton Tananaev2 months ago

What about jstack that I asked?

jinzo2 months ago

htop/top are displaying some "netsys" binary (quick search does not find anything, but I would guess it's not part of Traccar) as using the most CPU?

Navneet Choudhary2 months ago

upon finding PID for netsys and using jstack this is the reply i got

14739: Unable to open socket file /tmp/.java_pid14739: target process 14739 doesn't respond within 10500ms or HotSpot VM not loaded

ps: i have tried killing all netsys process but it will come again after few mins and start using 100% CPU

My key query is: is netsys anywhere related to traccar or it is completely foreign process

there might be chances that my droplet has been compromised may be.

jinzo2 months ago

Netsys is (probably) not a java/JVM, therefore jstack did not work (you would need to point jstack to your traccar process). Also - curious why you have more traccar processes running?

But yes, I think your system is pwned and netsys binary is some kind of malware.

Navneet Choudhary2 months ago

OK Final solution is : As netsys was not part of traccar and it was spamming the cpu so i think my droplet was compromised by some remote attackers that it why it was behaving like that.

I created a fresh droplet imported the sqldump database and done a fresh traccar installation and it is working fine without any issue.
PS: i used a reversed ip so that it was easy to migrate existing running devices.

and as Anton asked me for jstack for the process as it was compromised so it was not giving any jstack data for that process.

thank you