Server CPU Usage spiked to 100% always

Nizamuddin Mohamed 2 years ago

Hello all,

First of all, thank you, Anton, for the awesome work you've been doing.

I've been using Traccar for almost 3 years now. I have around 200 devices sending their location to the server every 10 seconds. The server has been running smoothly until the last few days. All of a sudden, my server's CPU usage jumped up to 100% from an average of 10% to 30%. I'm using Teltonika and Ruptela devices. Nothing has changed on the server side to trigger this spike in CPU usage. I added a couple more devices, and that's all. I have followed the MySQL optimization methods mentioned in the documentation.

Server Specs: 8 vCPUs, 16GB RAM, CentOS 7
Database: Local MySQL

Thinks I've tried so far with no luck:

  • Restarted Traccar service, MySQL service
  • Restarted all server
  • Upgraded Traccar version from 5.6 to 5.9 (Issue started when it was 5.6)

htop Screenshot: https://photos.app.goo.gl/cVrT33i2aDXtHpmZ7
traccar.conf: https://photos.app.goo.gl/ehxCv7i991fTBPXE9

I would greatly appreciate any assistance in figuring this out.

Anton Tananaev 2 years ago

Why do you have so many instances of Traccar?

Nizamuddin Mohamed 2 years ago

Thank you Anton for the reply,

All those processes spin up as soon as I start the traccar.service, somehow. Isn't it supposed to be like that? I wonder if there are any ways to limit the number of processes. I have around 20 processes running just for Traccar, all of which start automatically with the traccar service.

Below is traccar.service:

[Unit]
Description=traccar
After=network.target

[Service]
Type=simple
WorkingDirectory=/opt/traccar
ExecStart=/opt/traccar/jre/bin/java -Xmx5G -jar tracker-server.jar conf/traccar.xml
SyslogIdentifier=traccar
SuccessExitStatus=143
WatchdogSec=600
Restart=on-failure
RestartSec=10

[Install]
WantedBy=multi-user.target

Added -Xmx5G as suggested on the official documentation.

Anton Tananaev 2 years ago

Are those processes or threads?

Track-trace 2 years ago

Isn't database.positionsHistoryDays hanging your server ?

Anton Tananaev 2 years ago

That parameter doesn't exist on the mentioned versions.

Nizamuddin Mohamed 2 years ago

Thanks for the responses.

@Anton
Probably thread, i mean process as it has pid
@Track-trace
I found database.positionsHistoryDays in this forum, I've put it there since the beginning. Will remove it and restart to see if it makes any difference.

Anton Tananaev 2 years ago

If those are separate processes, something is seriously wrong. There should be only one.

Track-trace 2 years ago

Those are Threads, he was using HTOP when he posted the image of the processes. When @Nizamuddin Mohamed will just use TOP it will only show one Java process. My traccar server shows 46 threads in HTOP for Java traccar.

Nizamuddin Mohamed 2 years ago

Those are threads, In htop, threads would show as processes. I removed database.positionsHistoryDays from traccar.xml and restarted but no luck. The CPU bumps to 100% as soon as the the Traccar service is running.

ps aux | grep traccar : https://photos.app.goo.gl/qf2acJJ896YVguQX8
htop combined threads: https://photos.app.goo.gl/9eu6QpvQPQKk1kBg7

Anton Tananaev 2 years ago

Do you have any computed attributes? Do how have geofences?

Nizamuddin Mohamed 2 years ago

No computed attributes, but i have around 15 geofences.

Anton Tananaev 2 years ago

If your geofences are complicated (many points), that might be the cause.

Nizamuddin Mohamed 2 years ago

Thanks again for the reply. Removed all the Geofences, restarted the MySQL service, and restarted the Traccar service. However, the issue still remains. I also tried removing the position history older than 45 days, but it didn't make any difference.

Anton Tananaev 2 years ago

As I mentioned previously, database.positionsHistoryDays doesn't exist, so it doesn't affect anything.

OK, more questions:

  1. Do you use any external services, like geocoding and geolocation? If yes, try disabling them.
  2. Is your database local or remote?