Traccar 5.9 restarts several times during the day.
What did you check sofar ?
Not reaching CPU or memory limits ? (you can check that with Top or Htop)
Meaby a Cron job that closed the process ?
@Track-trace The CPU and memory limits are soft as shown in the top. Thanks for the help, but any suggestions on where I can check?
top - 11:14:58 up 2:21, 1 user, load average: 0.12, 0.11, 0.10
Tasks: 40 total, 1 running, 39 sleeping, 0 stopped, 0 zombie
%Cpu(s): 1.4 us, 0.4 sy, 0.0 ni, 98.2 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
MiB Mem : 6144.0 total, 4740.4 free, 916.6 used, 487.1 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 5224.2 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
236 root 20 0 4732308 429408 20000 S 3.3 6.8 5:25.97 java
251 mysql 20 0 2876408 474252 36200 S 3.3 7.5 4:19.15 mysqld
1860 www-data 20 0 223820 13692 5800 S 0.3 0.2 0:00.02 apache2
1 root 20 0 99940 11104 8432 S 0.0 0.2 0:00.64 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd/3ee21d
3 root 20 0 0 0 0 S 0.0 0.0 0:00.00 khelper
41 root 20 0 31328 13580 12612 S 0.0 0.2 0:00.12 systemd-journal
91 root 20 0 21420 4892 3916 S 0.0 0.1 0:00.05 systemd-udevd
139 systemd+ 20 0 16104 6608 5712 S 0.0 0.1 0:00.06 systemd-network
168 root 20 0 4288 2720 2464 S 0.0 0.0 0:00.00 cron
169 message+ 20 0 8132 3860 3504 S 0.0 0.1 0:00.03 dbus-daemon
173 syslog 20 0 152400 4116 3624 S 0.0 0.1 0:00.01 rsyslogd
179 root 20 0 15040 7248 6344 S 0.0 0.1 0:00.08 systemd-logind
183 systemd+ 20 0 25544 12752 8560 S 0.0 0.2 0:00.27 systemd-resolve
235 root 20 0 223476 31004 23264 S 0.0 0.5 0:00.89 apache2
238 root 20 0 3220 1072 980 S 0.0 0.0 0:00.00 agetty
239 root 20 0 2800 1100 1012 S 0.0 0.0 0:00.00 agetty
240 root 20 0 15408 9244 7616 S 0.0 0.1 0:00.00 sshd
402 root 20 0 16748 9472 8128 S 0.0 0.2 0:00.02 systemd
403 root 20 0 102524 2924 24 S 0.0 0.0 0:00.00 (sd-pam)
422 root 20 0 18368 12296 8484 S 0.0 0.2 0:00.23 sshd
433 root 20 0 7888 5404 4324 S 0.0 0.1 0:00.05 sftp-server
490 root 20 0 9668 2528 2188 S 0.0 0.0 0:00.00 xinetd
866 www-data 20 0 223848 14696 6736 S 0.0 0.2 0:03.98 apache2
1309 root 20 0 16656 10584 8476 S 0.0 0.2 0:00.13 sshd
1319 root 20 0 5028 4108 3424 S 0.0 0.1 0:00.01 bash
1461 www-data 20 0 223828 14696 6740 S 0.0 0.2 0:00.93 apache2
1647 www-data 20 0 223872 15056 7052 S 0.0 0.2 0:00.42 apache2
1689 www-data 20 0 223828 14772 6812 S 0.0 0.2 0:00.30 apache2
1741 www-data 20 0 223872 14692 6736 S 0.0 0.2 0:00.08 apache2
1818 www-data 20 0 223820 14772 6808 S 0.0 0.2 0:00.09 apache2
1819 www-data 20 0 223820 14696 6744 S 0.0 0.2 0:00.09 apache2
1827 www-data 20 0 223820 14716 6756 S 0.0 0.2 0:00.10 apache2
1830 www-data 20 0 223936 14692 6732 S 0.0 0.2 0:00.08 apache2
1831 www-data 20 0 223820 14772 6800 S 0.0 0.2 0:00.10 apache2
1842 www-data 20 0 223820 14772 6812 S 0.0 0.2 0:00.01 apache2
1847 www-data 20 0 223820 14696 6736 S 0.0 0.2 0:00.05 apache2 root@GDRASTREAMENTO:/etc/systemd/system# systemctl --failed
UNIT LOAD ACTIVE SUB DESCRIPTION
● systemd-networkd-wait-online.service loaded failed failed Wait for Network to be Configured
LOAD = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB = The low-level unit activation state, values depend on unit type.
1 loaded units listed.
Have you checked logs?
Excuse my ignorance Anton, are you referring to Traccar's own logs? Could you please suggest a specific search? Thanks!
2023-12-16 11:36:01 INFO: [T3b07b466] error - Adjusted frame length exceeds 1024: 18245 - discarded - TooLongFrameException (... < WrapperInboundHandler:57 < ... < StandardLoggingHandler:44 < ... < NetworkMessageHandler:37 < ...)2023-12-16 11:36:01 INFO: [T3b07b466: enfora < 35.203.211.41] 474554202f20485454502f312e310d0a486f73743a203130342e3235312e3232322e3138373a353030380d0a557365722d4167656e743a20457870616e73652c20612050616c6f20416c746f204e6574776f726b7320636f6d70616e792c207365617263686573206163726f73732074686520676c6f62616c2049507634207370616365206d756c7469706c652074696d6573207065722064617920746f206964656e7469667920637573746f6d657273262333393b2070726573656e636573206f6e2074686520496e7465726e65742e20496620796f7520776f756c64206c696b6520746f206265206578636c756465642066726f6d206f7572207363616e732c20706c656173652073656e64204950206164647265737365732f646f6d61696e7320746f3a207363616e696e666f4070616c6f616c746f6e6574776f726b732e636f6d0d0a0d0a
2023-12-16 11:36:01 INFO: [T3b07b466] error - Adjusted frame length exceeds 1024: 18245 - discarded - TooLongFrameException (... < WrapperInboundHandler:57 < ... < StandardLoggingHandler:44 < ... < NetworkMessageHandler:37 < ...)You have to check what happened around the time server restarted.
Anton, the time it restarted was 12/16/2023 at around 8:12 am, the closest thing I found in the logs would be this error. I even re-validated the configuration XML files on this site as seen in another topic you suggest. Validation OK.
Dec 16 08:45:49 GDRASTREAMENTO traccar[663]: 2023-12-16 08:45:49 ERROR: Main method error - - SAXParseException (... < Config:46 < <gener:-1 < *:-1 < ... < Main:120 < ...)
Dec 16 08:46:00 GDRASTREAMENTO traccar[686]: 2023-12-16 08:46:00 ERROR: Main method error - - SAXParseException (... < Config:46 < <gener:-1 < *:-1 < ... < Main:120 < ...)
...
Dec 16 08:51:14 GDRASTREAMENTO traccar[1033]: 2023-12-16 08:51:14 ERROR: Main method error - - SAXParseException (... < Config:46 < <gener:-1 < *:-1 < ... < Main:120 < ...)
Dec 16 08:51:25 GDRASTREAMENTO traccar[1056]: 2023-12-16 08:51:25 ERROR: Main method error - - SAXParseException (... < Config:46 < <gener:-1 < *:-1 < ... < Main:120 < ...)Hi Anton, could you help me please? Would commenting whatchdog in systemd help?
You can try.
watchdog commented did not resolve.
only Traccar fails? what about the other processes? mysql apache ssh? did you check it?
These processes failed but not necessarily at the time the traccar stopped working.
● systemd-networkd-wait-online.service loaded failed failed Wait for Network to be Configured
LOAD = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB = The low-level unit activation state, values depend on unit type.
1 loaded units listed.Dec 16 08:47:00 GDRASTREAMENTO systemd[1]: apache2.service: Killing process 654 (apache2) with signal SIGKILL.
Dec 16 08:47:00 GDRASTREAMENTO systemd[1]: apache2.service: Killing process 661 (apache2) with signal SIGKILL.
Dec 16 08:47:00 GDRASTREAMENTO systemd[1]: apache2.service: Killing process 712 (apache2) with signal SIGKILL.
Dec 16 08:47:00 GDRASTREAMENTO systemd[1]: apache2.service: Killing process 714 (apache2) with signal SIGKILL.
Dec 16 08:47:00 GDRASTREAMENTO systemd[1]: apache2.service: Killing process 716 (apache2) with signal SIGKILL.[Sat Dec 16 20:31:26.320622 2023] [proxy:error] [pid 3588] (70014)End of file found: [client 191.247.0.187:53485] AH03307: ap_proxy_transfer_between_connections: error on origin - ap_pass_brigade
[Sat Dec 16 20:31:26.320666 2023] [proxy:error] [pid 3588] [client 191.247.0.187:53485] AH10221: proxy: websocket: client flushing failed (-102)
[Sat Dec 16 20:31:26.860838 2023] [proxy:error] [pid 3573] (70014)End of file found: [client 191.247.0.187:53488] AH03307: ap_proxy_transfer_between_connections: error on origin - ap_pass_brigade
[Sat Dec 16 20:31:26.860907 2023] [proxy:error] [pid 3573] [client 191.247.0.187:53488] AH10221: proxy: websocket: client flushing failed (-102)
[Sat Dec 16 20:39:25.558070 2023] [proxy:error] [pid 3685] (70014)End of file found: [client 177.124.159.214:27770] AH03307: ap_proxy_transfer_between_connections: error on origin - ap_pass_brigade
[Sat Dec 16 20:39:25.558139 2023] [proxy:error] [pid 3685] [client 177.124.159.214:27770] AH10221: proxy: websocket: client flushing failed (-102)
[Sat Dec 16 21:32:33.530674 2023] [proxy:error] [pid 3961] (70014)End of file found: [client 177.37.151.174:6672] AH03307: ap_proxy_transfer_between_connections: error on origin - ap_pass_brigade
[Sat Dec 16 21:32:33.530727 2023] [proxy:error] [pid 3961] [client 177.37.151.174:6672] AH10221: proxy: websocket: client flushing failed (-102)
[Sat Dec 16 22:56:04.938481 2023] [proxy:error] [pid 4543] (70014)End of file found: [client 177.26.247.13:45662] AH03307: ap_proxy_transfer_between_connections: error on origin - ap_pass_brigade
[Sat Dec 16 22:56:04.938540 2023] [proxy:error] [pid 4543] [client 177.26.247.13:45662] AH10221: proxy: websocket: client flushing failed (-102)2023-12-17T02:07:41.974948Z 141 [Warning] [MY-013360] [Server] Plugin mysql_native_password reported: ''mysql_native_password' is deprecated and will be removed in a future release. Please use caching_sha2_password instead'
2023-12-17T02:07:46.429863Z 142 [Warning] [MY-013360] [Server] Plugin mysql_native_password reported: ''mysql_native_password' is deprecated and will be removed in a future release. Please use caching_sha2_password instead'
2023-12-17T02:07:55.460730Z 143 [Warning] [MY-013360] [Server] Plugin mysql_native_password reported: ''mysql_native_password' is deprecated and will be removed in a future release. Please use caching_sha2_password instead'Dec 16 16:12:24 GDRASTREAMENTO systemd[1]: Started MySQL Community Server.
Dec 15 20:26:23 GDRASTREAMENTO systemd[1]: mysql.service: Found left-over process 316 (mysqld) in control group while starting unit. Ignoring.
Dec 15 20:26:23 GDRASTREAMENTO systemd[1]: mysql.service: Found left-over process 316 (mysqld) in control group while starting unit. Ignoring.
Dec 15 20:56:48 GDRASTREAMENTO systemd[1]: mysql.service: Failed with result 'exit-code'.
Dec 15 20:56:48 GDRASTREAMENTO systemd[1]: Failed to start MySQL Community Server.The apache is OK
root@GDRASTREAMENTO:~# apachectl configtest
Syntax OK
root@GDRASTREAMENTO:~#Maybe you have pending O.S. updates?
If so, the recommendation is to proceed with care since an O.S. update can leave obsolete processes, it is necessary to look for a lot of information before proceeding.
The OS is updated!
Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease
Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB]
Get:3 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB]
Hit:4 http://archive.canonical.com/ubuntu jammy InRelease
Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1263 kB]
Fetched 1492 kB in 1s (1820 kB/s)
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
All packages are up to date
Dear sirs, could anyone help me with this situation that is occurring with traccar? The same restarts several times and looking through the log I see this problem below. I have a VPS with the following configurations: 6GB of RAM, 4VCPU and 170 devices. Thank you in advance for any information that could reverse this situation.