Server CPU Usage spiked to 100% always

Nizamuddin Mohameda year ago
  1. I had enabled Nominatim earlier, but removed it first since the issue started.
  2. Database is Local.

Here is an error found in Traccar logs: https://photos.app.goo.gl/3pEajYvnyGhQcmCM9
I wonder if it's related.

Anton Tananaeva year ago

It could be related.

Nizamuddin Mohameda year ago

Thanks for the reply @Anton.
Are there any ways we can debug this issue ?
Tried to decode the hex with Traccar hex-decoder, but it shows Binary Hex can't be converted to Text.
I'll update here if any related progress.

Anton Tananaeva year ago

One option is to use jstack to see what those threads are doing.

Nizamuddin Mohameda year ago

I'm trying a way to use jstack in CentOS, meanwhile I've noticed the service is restarting on every 15 minutes or so. 

Oct 25 07:03:17 ********* systemd[1]: Started traccar.
Oct 25 07:03:20 ********* traccar[3223]: Database is up to date, no changesets to execute
Oct 25 07:03:20 ********* traccar[3223]: UPDATE SUMMARY
Oct 25 07:03:20 ********* traccar[3223]: Run:                          0
Oct 25 07:03:20 ********* traccar[3223]: Previously run:              44
Oct 25 07:03:20 ********* traccar[3223]: Filtered out:                 0
Oct 25 07:03:20 ********* traccar[3223]: -------------------------------
Oct 25 07:03:20 ********* traccar[3223]: Total change sets:           44
Oct 25 07:21:23 ********* systemd[1]: traccar.service watchdog timeout (limit 10min)!
Oct 25 07:21:23 ********* systemd[1]: traccar.service: main process exited, code=killed, status=6/ABRT
Oct 25 07:21:23 ********* systemd[1]: Unit traccar.service entered failed state.
Oct 25 07:21:23 ********* systemd[1]: traccar.service failed.
Oct 25 07:21:33 ********* systemd[1]: traccar.service holdoff time over, scheduling restart.
Oct 25 07:21:33 ********* systemd[1]: Stopped traccar.
Oct 25 07:21:33 ********* systemd[1]: Started traccar.
Oct 25 07:21:36 ********* traccar[5094]: Database is up to date, no changesets to execute
Oct 25 07:21:36 ********* traccar[5094]: UPDATE SUMMARY
Oct 25 07:21:36 ********* traccar[5094]: Run:                          0
Oct 25 07:21:36 ********* traccar[5094]: Previously run:              44
Oct 25 07:21:36 ********* traccar[5094]: Filtered out:                 0
Oct 25 07:21:36 ********* traccar[5094]: -------------------------------
Oct 25 07:21:36 ********* traccar[5094]: Total change sets:           44
Oct 25 07:39:39 ********* systemd[1]: traccar.service watchdog timeout (limit 10min)!
Oct 25 07:39:39 ********* systemd[1]: traccar.service: main process exited, code=killed, status=6/ABRT
Oct 25 07:39:39 ********* systemd[1]: Unit traccar.service entered failed state.
Oct 25 07:39:39 ********* systemd[1]: traccar.service failed.
Oct 25 07:39:49 ********* systemd[1]: traccar.service holdoff time over, scheduling restart.
Track-tracea year ago

To root out any external cause, i would just copy traccar server to another VPS. Backup the database and restore the database on the other vps. Then just start traccar server to see if the same happens..

Nizamuddin Mohameda year ago

Another Issue that've found in traccar logs are below, filtered with keyword error

2023-10-25 17:35:36  INFO: [Tee203dc4] error - Cannot invoke "org.traccar.model.Device.getCalendarId()" because "device" is null - NullPointerException (FilterHandler:264 < *:283 < ... < RemoteAddressHandler:53 < ... < BaseDataHandler:29 < ...)
2023-10-25 17:35:36  INFO: [T7647f8e1] error - Cannot invoke "org.traccar.model.Device.getCalendarId()" because "device" is null - NullPointerException (FilterHandler:264 < *:283 < ... < RemoteAddressHandler:53 < ... < BaseDataHandler:29 < ...)
2023-10-25 17:35:36  INFO: [Tee203dc4] error - Cannot invoke "org.traccar.model.Device.getCalendarId()" because "device" is null - NullPointerException (FilterHandler:264 < *:283 < ... < RemoteAddressHandler:53 < ... < BaseDataHandler:29 < ...)
2023-10-25 17:35:36  INFO: [T4f4cde78] error - Cannot invoke "org.traccar.model.Device.getCalendarId()" because "device" is null - NullPointerException (FilterHandler:264 < *:283 < ... < RemoteAddressHandler:53 < ... < BaseDataHandler:29 < ...)
2023-10-25 17:35:36  INFO: [T4f4cde78] error - Cannot invoke "org.traccar.model.Device.getCalendarId()" because "device" is null - NullPointerException (FilterHandler:264 < *:283 < ... < RemoteAddressHandler:53 < ... < BaseDataHandler:29 < ...)
2023-10-25 17:35:36  INFO: [T4f4cde78] error - Cannot invoke "org.traccar.model.Device.getCalendarId()" because "device" is null - NullPointerException (FilterHandler:264 < *:283 < ... < RemoteAddressHandler:53 < ... < BaseDataHandler:29 < ...)
2023-10-25 17:35:36  INFO: [T4f4cde78] error - Cannot invoke "org.traccar.model.Device.getCalendarId()" because "device" is null - NullPointerException (FilterHandler:264 < *:283 < ... < RemoteAddressHandler:53 < ... < BaseDataHandler:29 < ...)
2023-10-25 17:35:36  INFO: [T4f4cde78] error - Cannot invoke "org.traccar.model.Device.getCalendarId()" because "device" is null - NullPointerException (FilterHandler:264 < *:283 < ... < RemoteAddressHandler:53 < ... < BaseDataHandler:29 < ...)
2023-10-25 17:35:36  INFO: [Tee203dc4] error - Cannot invoke "org.traccar.model.Device.getCalendarId()" because "device" is null - NullPointerException (FilterHandler:264 < *:283 < ... < RemoteAddressHandler:53 < ... < BaseDataHandler:29 < ...)
2023-10-25 17:35:36  INFO: [T4f4cde78] error - Cannot invoke "org.traccar.model.Device.getCalendarId()" because "device" is null - NullPointerException (FilterHandler:264 < *:283 < ... < RemoteAddressHandler:53 < ... < BaseDataHandler:29 < ...)
2023-10-25 17:35:36  INFO: [T503a4dc0] error - Index: -1, Size: 0 - IndexOutOfBoundsException (... < RuptelaProtocolDecoder:245 < ExtendedObjectDecoder:75 < ... < WrapperContext:102 < ...)
2023-10-25 17:35:37  INFO: [T4f4cde78] error - Cannot invoke "org.traccar.model.Device.getCalendarId()" because "device" is null - NullPointerException (FilterHandler:264 < *:283 < ... < RemoteAddressHandler:53 < ... < BaseDataHandler:29 < ...)
2023-10-25 17:35:37  INFO: [T63516dbc] error - Index: -1, Size: 0 - IndexOutOfBoundsException (... < RuptelaProtocolDecoder:245 < ExtendedObjectDecoder:75 < ... < WrapperContext:102 < ...)
2023-10-25 17:35:37  INFO: [T8d371725] error - Index: -1, Size: 0 - IndexOutOfBoundsException (... < RuptelaProtocolDecoder:245 < ExtendedObjectDecoder:75 < ... < WrapperContext:102 < ...)
2023-10-25 17:35:37  INFO: [Tb7ed485b] error - Index: -1, Size: 0 - IndexOutOfBoundsException (... < RuptelaProtocolDecoder:245 < ExtendedObjectDecoder:75 < ... < WrapperContext:102 < ...)
2023-10-25 17:35:37  INFO: [Ta11f7b7b] error - Index: -1, Size: 0 - IndexOutOfBoundsException (... < RuptelaProtocolDecoder:245 < ExtendedObjectDecoder:75 < ... < WrapperContext:102 < ...)
2023-10-25 17:35:37  INFO: [T41f627b5] error - Index: -1, Size: 0 - IndexOutOfBoundsException (... < RuptelaProtocolDecoder:245 < ExtendedObjectDecoder:75 < ... < WrapperContext:102 < ...)
2023-10-25 17:35:37  INFO: [Ta2781167] error - Index: -1, Size: 0 - IndexOutOfBoundsException (... < RuptelaProtocolDecoder:245 < ExtendedObjectDecoder:75 < ... < WrapperContext:102 < ...)
2023-10-25 17:35:37  INFO: [Td004b089] error - Index: -1, Size: 0 - IndexOutOfBoundsException (... < RuptelaProtocolDecoder:245 < ExtendedObjectDecoder:75 < ... < WrapperContext:102 < ...)
2023-10-25 17:35:37  INFO: [Te1a0e1b1] error - Index: -1, Size: 0 - IndexOutOfBoundsException (... < RuptelaProtocolDecoder:245 < ExtendedObjectDecoder:75 < ... < WrapperContext:102 < ...)
2023-10-25 17:35:37  INFO: [T0d7624ef] error - Index: -1, Size: 0 - IndexOutOfBoundsException (... < RuptelaProtocolDecoder:245 < ExtendedObjectDecoder:75 < ... < WrapperContext:102 < ...)
2023-10-25 17:35:37  INFO: [Ta8c6e033] error - Index: -1, Size: 0 - IndexOutOfBoundsException (... < RuptelaProtocolDecoder:245 < ExtendedObjectDecoder:75 < ... < WrapperContext:102 < ...)
2023-10-25 17:35:37  INFO: [T4c56a046] error - Index: -1, Size: 0 - IndexOutOfBoundsException (... < RuptelaProtocolDecoder:245 < ExtendedObjectDecoder:75 < ... < WrapperContext:102 < ...)
2023-10-25 17:35:37  INFO: [Td74603ee] error - Index: -1, Size: 0 - IndexOutOfBoundsException (... < RuptelaProtocolDecoder:245 < ExtendedObjectDecoder:75 < ... < WrapperContext:102 < ...)
2023-10-25 17:35:37  INFO: [Tddc52142] error - Index: -1, Size: 0 - IndexOutOfBoundsException (... < RuptelaProtocolDecoder:245 < ExtendedObjectDecoder:75 < ... < WrapperContext:102 < ...)
2023-10-25 17:35:37  INFO: [T41a0ee6c] error - Index: -1, Size: 0 - IndexOutOfBoundsException (... < RuptelaProtocolDecoder:245 < ExtendedObjectDecoder:75 < ... < WrapperContext:102 < ...)
2023-10-25 17:35:37  INFO: [T92cc1525] error - Index: -1, Size: 0 - IndexOutOfBoundsException (... < RuptelaProtocolDecoder:245 < ExtendedObjectDecoder:75 < ... < WrapperContext:102 < ...)
2023-10-25 17:35:37  INFO: [Tf214b1ff] error - Index: -1, Size: 0 - IndexOutOfBoundsException (... < RuptelaProtocolDecoder:245 < ExtendedObjectDecoder:75 < ... < WrapperContext:102 < ...)
2023-10-25 17:35:37  INFO: [T15dd5b5b] error - Index: -1, Size: 0 - IndexOutOfBoundsException (... < RuptelaProtocolDecoder:245 < ExtendedObjectDecoder:75 < ... < WrapperContext:102 < ...)
2023-10-25 17:35:37  INFO: [T654095df] error - Index: -1, Size: 0 - IndexOutOfBoundsException (... < RuptelaProtocolDecoder:245 < ExtendedObjectDecoder:75 < ... < WrapperContext:102 < ...)
2023-10-25 17:35:37  INFO: [T1bb99cc0] error - Index: -1, Size: 0 - IndexOutOfBoundsException (... < RuptelaProtocolDecoder:245 < ExtendedObjectDecoder:75 < ... < WrapperContext:102 < ...)
2023-10-25 17:35:37  INFO: [T53af1099] error - Index: -1, Size: 0 - IndexOutOfBoundsException (... < RuptelaProtocolDecoder:245 < ExtendedObjectDecoder:75 < ... < WrapperContext:102 < ...)
2023-10-25 17:35:37  INFO: [Tbd72f237] error - Index: -1, Size: 0 - IndexOutOfBoundsException (... < RuptelaProtocolDecoder:245 < ExtendedObjectDecoder:75 < ... < WrapperContext:102 < ...)
2023-10-25 17:35:37  INFO: [T81832e7b] error - Index: -1, Size: 0 - IndexOutOfBoundsException (... < RuptelaProtocolDecoder:245 < ExtendedObjectDecoder:75 < ... < WrapperContext:102 < ...)
2023-10-25 17:35:37  INFO: [Td68b0326] error - Index: -1, Size: 0 - IndexOutOfBoundsException (... < RuptelaProtocolDecoder:245 < ExtendedObjectDecoder:75 < ... < WrapperContext:102 < ...)
2023-10-25 17:35:37  INFO: [Tbbdc4827] error - Index: -1, Size: 0 - IndexOutOfBoundsException (... < RuptelaProtocolDecoder:245 < ExtendedObjectDecoder:75 < ... < WrapperContext:102 < ...)
2023-10-25 17:35:37  INFO: [Tb22beede] error - Index: -1, Size: 0 - IndexOutOfBoundsException (... < RuptelaProtocolDecoder:245 < ExtendedObjectDecoder:75 < ... < WrapperContext:102 < ...)
2023-10-25 17:35:37  INFO: [Tf7b8b1e9] error - Index: -1, Size: 0 - IndexOutOfBoundsException (... < RuptelaProtocolDecoder:245 < ExtendedObjectDecoder:75 < ... < WrapperContext:102 < ...)
Nizamuddin Mohameda year ago

@Tracker-trace
Thanks for the advise. I guess that would be my next step.
I've blocked all the ports from external access to check if it's related to any DDoS attack, but there were no change in CPU usage.

Track-tracea year ago

With external cause i did mean not traccar itself but for instances related to java. So just running the same traccar server on another clean vpn server you could quickly rule that out.

jinzoa year ago

Your Ruptela error (https://photos.app.goo.gl/3pEajYvnyGhQcmCM9) is probably related to this: https://github.com/traccar/traccar/pull/5190
This does cause the Ruptela devices to send a lot of data (and also stop updating) but it was introduced only in 5.9 - so it cannot be your root cause for issues if the CPU usage started with 5.6. One way or another I would either upgrade to the latest "master" version or downgrade to 5.8