Watchdog systemd

5of94 years ago

The new watchdog for systemd makes some problems so I had to disable it. It restarts always:
Logs:

Jul 06 15:26:34 gps. systemd[1]: traccar.service: Watchdog timeout (limit 1min 40s)!
Jul 06 15:26:34 gps.i systemd[1]: traccar.service: Killing process 21763 (java) with signal SIGABRT.
Jul 06 15:26:34 gps. systemd[1]: traccar.service: Main process exited, code=killed, status=6/ABRT
Jul 06 15:26:34 gps.i systemd[1]: traccar.service: Failed with result 'watchdog'.
Jul 06 15:26:45 gps.i systemd[1]: traccar.service: Scheduled restart job, restart counter is at 13.

Anton Tananaev4 years ago

Do you mind sharing details about your system?

5of94 years ago

It's Ubuntu 20.04, and here start Infos from Traccar:

2020-07-06 15:27:46  INFO: Operating system name: Linux version: 5.4.0-40-generic architecture: amd64
2020-07-06 15:27:46  INFO: Java runtime name: OpenJDK 64-Bit Server VM vendor: Oracle Corporation version: 11.0.5+10-LTS
2020-07-06 15:27:46  INFO: Memory limit heap: 359mb non-heap: 0mb
2020-07-06 15:27:46  INFO: Character encoding: UTF-8 charset: UTF-8
2020-07-06 15:27:46  INFO: Version: 4.9
2020-07-06 15:27:46  INFO: Starting server...
2020-07-06 15:27:47  INFO: jetty-9.4.30.v20200611; built: 2020-06-11T12:34:51.929Z; git: 271836e4c1f4612f12b7bb13ef5a92a927634b0d; jvm 11.0.5+10-LTS
2020-07-06 15:27:47  INFO: Started o.t.w.@1179fc8c{/,null,AVAILABLE}
2020-07-06 15:27:47  INFO: DefaultSessionIdManager workerName=node0
2020-07-06 15:27:47  INFO: No SessionScavenger set, using defaults
2020-07-06 15:27:47  INFO: node0 Scavenging every 600000ms
2020-07-06 15:27:51  INFO: Started o.e.j.s.ServletContextHandler@65a2e14e{/,null

Anton Tananaev4 years ago

Can we see a full log fragment from one restart to another?

5of94 years ago

2020-07-06 15:26:47 INFO: HikariPool-1 - Starting...
2020-07-06 15:26:48 INFO: HikariPool-1 - Start completed.
2020-07-06 15:26:50 INFO: Clearing database change log checksums
2020-07-06 15:26:50 INFO: SELECT COUNT() FROM gps.DATABASECHANGELOGLOCK
2020-07-06 15:26:50 INFO: SELECT COUNT() FROM gps.DATABASECHANGELOGLOCK
2020-07-06 15:26:50 INFO: SELECT LOCKED FROM gps.DATABASECHANGELOGLOCK WHERE ID=1
2020-07-06 15:26:50 INFO: Successfully acquired change log lock
2020-07-06 15:26:51 INFO: SELECT MD5SUM FROM gps.DATABASECHANGELOG WHERE MD5SUM IS NOT NULL LIMIT 1
2020-07-06 15:26:51 INFO: UPDATE gps.DATABASECHANGELOG SET MD5SUM = NULL
2020-07-06 15:26:51 INFO: Successfully released change log lock
2020-07-06 15:26:51 INFO: SELECT COUNT() FROM gps.DATABASECHANGELOGLOCK
2020-07-06 15:26:51 INFO: SELECT COUNT() FROM gps.DATABASECHANGELOGLOCK
2020-07-06 15:26:51 INFO: SELECT LOCKED FROM gps.DATABASECHANGELOGLOCK WHERE ID=1
2020-07-06 15:26:51 INFO: Successfully acquired change log lock
2020-07-06 15:26:51 INFO: Can not use class liquibase.parser.core.yaml.YamlChangeLogParser as a Liquibase service because org.yaml.snakeyaml.constructor.BaseConstructor is not in the classpath
2020-07-06 15:26:51 INFO: Can not use class liquibase.parser.core.json.JsonChangeLogParser as a Liquibase service because org.yaml.snakeyaml.constructor.BaseConstructor is not in the classpath
2020-07-06 15:26:55 INFO: SELECT MD5SUM FROM gps.DATABASECHANGELOG WHERE MD5SUM IS NOT NULL LIMIT 1
2020-07-06 15:26:55 INFO: SELECT COUNT() FROM gps.DATABASECHANGELOG
2020-07-06 15:26:55 INFO: Reading from gps.DATABASECHANGELOG
2020-07-06 15:26:55 INFO: SELECT * FROM gps.DATABASECHANGELOG ORDER BY DATEEXECUTED ASC, ORDEREXECUTED ASC
2020-07-06 15:26:55 INFO: UPDATE gps.DATABASECHANGELOG SET MD5SUM = '8:ea6f202534dc7845fd1a49e1d8384d9c' WHERE ID = 'changelog-3.3' AND AUTHOR = 'author' AND FILENAME = 'changelog-3.3'
2020-07-06 15:26:55 INFO: UPDATE gps.DATABASECHANGELOG SET MD5SUM = '8:194e3ae7ec6d201917136e04a26e237b' WHERE ID = 'changelog-3.5' AND AUTHOR = 'author' AND FILENAME = 'changelog-3.5'
2020-07-06 15:26:55 INFO: UPDATE gps.DATABASECHANGELOG SET MD5SUM = '8:1ec7941b03749f24e365bc6126b07e1a' WHERE ID = 'changelog-3.6' AND AUTHOR = 'author' AND FILENAME = 'changelog-3.6'
2020-07-06 15:26:55 INFO: UPDATE gps.DATABASECHANGELOG SET MD5SUM = '8:2e444e379f31d0b8e09273ec7ffe87a0' WHERE ID = 'changelog-3.7' AND AUTHOR = 'author' AND FILENAME = 'changelog-3.7'
2020-07-06 15:26:55 INFO: UPDATE gps.DATABASECHANGELOG SET MD5SUM = '8:974c33d2fb399ef6477c3897450fb078' WHERE ID = 'changelog-3.7-notmssql' AND AUTHOR = 'author' AND FILENAME = 'changelog-3.7'
2020-07-06 15:26:56 INFO: UPDATE gps.DATABASECHANGELOG SET MD5SUM = '8:74d68027951f8d2ae6bb2e24df8365bc' WHERE ID = 'changelog-3.8' AND AUTHOR = 'author' AND FILENAME = 'changelog-3.8'
2020-07-06 15:26:56 INFO: UPDATE gps.DATABASECHANGELOG SET MD5SUM = '8:34822842d65deb843a7d16f857d52ecc' WHERE ID = 'changelog-3.9' AND AUTHOR = 'author' AND FILENAME = 'changelog-3.9'
2020-07-06 15:26:56 INFO: UPDATE gps.DATABASECHANGELOG SET MD5SUM = '8:e1ddbe83e1ecf856a912755fc118f82e' WHERE ID = 'changelog-3.10' AND AUTHOR = 'author' AND FILENAME = 'changelog-3.10'
2020-07-06 15:26:56 INFO: UPDATE gps.DATABASECHANGELOG SET MD5SUM = '8:191c21d8f0f921845cf93bbc9d0639b9' WHERE ID = 'changelog-3.10-notmssql' AND AUTHOR = 'author' AND FILENAME = 'changelog-3.10'
2020-07-06 15:26:56 INFO: UPDATE gps.DATABASECHANGELOG SET MD5SUM = '8:ad1f63566e8d08812fbf0b93a118ef6e' WHERE ID = 'changelog-3.10-mssql' AND AUTHOR = 'author' AND FILENAME = 'changelog-3.10'
2020-07-06 15:26:56 INFO: UPDATE gps.DATABASECHANGELOG SET MD5SUM = '8:127b36b9d32a9d236df51d19b18c3766' WHERE ID = 'changelog-3.7-mssql' AND AUTHOR = 'author' AND FILENAME = 'changelog-3.10'
2020-07-06 15:26:56 INFO: UPDATE gps.DATABASECHANGELOG SET MD5SUM = '8:7800f890b9706a480bd5a79b591b6ca7' WHERE ID = 'changelog-3.11' AND AUTHOR = 'author' AND FILENAME = 'changelog-3.11'
2020-07-06 15:26:57 INFO: UPDATE gps.DATABASECHANGELOG SET MD5SUM = '8:5ce520811d626ad325a014b9fcbb1a13' WHERE ID = 'changelog-3.12' AND AUTHOR = 'author' AND FILENAME = 'changelog-3.12'
2020-07-06 15:26:57 INFO: UPDATE gps.DATABASECHANGELOG SET MD5SUM = '8:a3bf7fabcde29e106fe2f89829a76a84' WHERE ID = 'changelog-3.12-notmssql' AND AUTHOR = 'author' AND FILENAME = 'changelog-3.12'
2020-07-06 15:26:57 INFO: UPDATE gps.DATABASECHANGELOG SET MD5SUM = '8:cfc881bd2dadb561aa9c1a467bc8cc1c' WHERE ID = 'changelog-3.12-pgsql' AND AUTHOR = 'author' AND FILENAME = 'changelog-3.12'
2020-07-06 15:26:57 INFO: UPDATE gps.DATABASECHANGELOG SET MD5SUM = '8:1be7e6c0520f8be53ef1b099d96afba5' WHERE ID = 'changelog-3.14' AND AUTHOR = 'author' AND FILENAME = 'changelog-3.14'
2020-07-06 15:26:57 INFO: UPDATE gps.DATABASECHANGELOG SET MD5SUM = '8:ac8ccec176027e98bbf45dc6c15d9b05' WHERE ID = 'changelog-3.15' AND AUTHOR = 'author' AND FILENAME = 'changelog-3.15'
2020-07-06 15:26:57 INFO: UPDATE gps.DATABASECHANGELOG SET MD5SUM = '8:b59407d70bfe9bf57bc9968adfbbdf1c' WHERE ID = 'changelog-3.16' AND AUTHOR = 'author' AND FILENAME = 'changelog-3.16'
2020-07-06 15:26:57 INFO: UPDATE gps.DATABASECHANGELOG SET MD5SUM = '8:3f14c3b08068eb7628d0d3e2941eb2d3' WHERE ID = 'changelog-3.3-admin' AND AUTHOR = 'author' AND FILENAME = 'changelog-3.17'
2020-07-06 15:26:57 INFO: UPDATE gps.DATABASECHANGELOG SET MD5SUM = '8:8aa3e56afe6ba86cd004fa801507bc28' WHERE ID = 'changelog-3.17' AND AUTHOR = 'author' AND FILENAME = 'changelog-3.17'
2020-07-06 15:26:57 INFO: UPDATE gps.DATABASECHANGELOG SET MD5SUM = '8:90aedfa378aa717f8d8ae541f97b87b2' WHERE ID = 'changelog-4.0-renaming' AND AUTHOR = 'author' AND FILENAME = 'changelog-4.0'
2020-07-06 15:26:58 INFO: UPDATE gps.DATABASECHANGELOG SET MD5SUM = '8:ac63c4153f5b2ee5c7a07056da269571' WHERE ID = 'changelog-4.0' AND AUTHOR = 'author' AND FILENAME = 'changelog-4.0'
2020-07-06 15:26:58 INFO: UPDATE gps.DATABASECHANGELOG SET MD5SUM = '8:d59fa93049c031a8bcfbc02958c1942c' WHERE ID = 'changelog-4.0-clean' AND AUTHOR = 'author' AND FILENAME = 'changelog-4.0-clean'
2020-07-06 15:26:58 INFO: UPDATE gps.DATABASECHANGELOG SET MD5SUM = '8:1adabae3238e02ccc97cc422359a58a0' WHERE ID = 'changelog-4.0-clean-common' AND AUTHOR = 'author' AND FILENAME = 'changelog-4.0-clean'
2020-07-06 15:26:58 INFO: UPDATE gps.DATABASECHANGELOG SET MD5SUM = '8:3974bfe5a2e962c0cd663433c832c16a' WHERE ID = 'changelog-4.0-pre' AND AUTHOR = 'author' AND FILENAME = 'changelog-4.0'
2020-07-06 15:26:58 INFO: UPDATE gps.DATABASECHANGELOG SET MD5SUM = '8:65fb49c5be37693183708351c507dd50' WHERE ID = 'changelog-4.0-common' AND AUTHOR = 'author' AND FILENAME = 'changelog-4.0'
2020-07-06 15:26:58 INFO: UPDATE gps.DATABASECHANGELOG SET MD5SUM = '8:9831511507d8ae1d6759c8ccf506a27a' WHERE ID = 'changelog-4.0-pg' AND AUTHOR = 'author' AND FILENAME = 'changelog-4.0'
2020-07-06 15:26:58 INFO: UPDATE gps.DATABASECHANGELOG SET MD5SUM = '8:b148f52efe9c6a3e74a56e33e257a3e2' WHERE ID = 'changelog-4.1-mssql' AND AUTHOR = 'author' AND FILENAME = 'changelog-4.1'
2020-07-06 15:26:58 INFO: UPDATE gps.DATABASECHANGELOG SET MD5SUM = '8:a23040cfc84d4b8021f40ee214a9d78b' WHERE ID = 'changelog-4.7' AND AUTHOR = 'author' AND FILENAME = 'changelog-4.7'
2020-07-06 15:26:58 INFO: UPDATE gps.DATABASECHANGELOG SET MD5SUM = '8:805ef202791dcc246373ca16f7c19c4f' WHERE ID = 'changelog-4.9' AND AUTHOR = 'author' AND FILENAME = 'changelog-4.9'
2020-07-06 15:26:58 INFO: SELECT COUNT() FROM gps.DATABASECHANGELOG
2020-07-06 15:26:58 INFO: Reading from gps.DATABASECHANGELOG
2020-07-06 15:26:58 INFO: SELECT * FROM gps.DATABASECHANGELOG ORDER BY DATEEXECUTED ASC, ORDEREXECUTED ASC
2020-07-06 15:26:59 INFO: Successfully released change log lock
2020-07-06 15:26:59 INFO: Logging initialized @14570ms to org.eclipse.jetty.util.log.Slf4jLog
2020-07-06 15:27:00 INFO: Your platform does not provide complete low-level API for accessing direct buffers reliably. Unless explicitly requested, heap buffer will always be preferred to avoid potential system instability.
2020-07-06 15:27:02 INFO: Operating system name: Linux version: 5.4.0-40-generic architecture: amd64
2020-07-06 15:27:02 INFO: Java runtime name: OpenJDK 64-Bit Server VM vendor: Oracle Corporation version: 11.0.5+10-LTS
2020-07-06 15:27:02 INFO: Memory limit heap: 359mb non-heap: 0mb
2020-07-06 15:27:02 INFO: Character encoding: UTF-8 charset: UTF-8
2020-07-06 15:27:02 INFO: Version: 4.9
2020-07-06 15:27:02 INFO: Starting server...
2020-07-06 15:27:03 INFO: jetty-9.4.30.v20200611; built: 2020-06-11T12:34:51.929Z; git: 271836e4c1f4612f12b7bb13ef5a92a927634b0d; jvm 11.0.5+10-LTS
2020-07-06 15:27:03 INFO: Started o.t.w.@1179fc8c{/,null,AVAILABLE}
2020-07-06 15:27:03 INFO: DefaultSessionIdManager workerName=node0
2020-07-06 15:27:03 INFO: No SessionScavenger set, using defaults
2020-07-06 15:27:03 INFO: node0 Scavenging every 600000ms
2020-07-06 15:27:06 INFO: Started o.e.j.s.ServletContextHandler@65a2e14e{/,null,AVAILABLE}
2020-07-06 15:27:06 INFO: Started ServerConnector@2abbd0d9{HTTP/1.1, (http/1.1)}{0.0.0.0:8081}
2020-07-06 15:27:06 INFO: Started @21649ms
2020-07-06 15:27:07 INFO: Health check enabled with period 80000
2020-07-06 15:27:29 INFO: Shutting down server...

Anton Tananaev4 years ago

Maybe you need to increase the timeout? Your server seems to be pretty slow.

thomas4 years ago

Same problem here on a cheap VM running debian, host operated by netcup.de (the smallest VM for 2,36€/month), if I directly run traccar it keeps runnning. I am just checking out longer timeouts :-)

Anton Tananaev4 years ago

You can also remove health checks in the systemd config.

5of94 years ago

Thank you, maybe this could be a chance. Put the watchdog in systemd config file to 600 at slow machines, worked for the first short test!

Juan Sánchez4 years ago

Hello,

How can i remove healthcheck on systemd?

Regards,

Mindreader4 years ago

Where can we remove it ?

thomas4 years ago

On my Debian it's quite simple, just edit the service-file:

nano /etc/systemd/system/traccar.service

My file looks like that now:

[Unit]
Description=traccar
After=network.target

[Service]
Type=simple
WorkingDirectory=/opt/traccar
ExecStart=/opt/traccar/jre/bin/java -jar tracker-server.jar conf/traccar.xml
SyslogIdentifier=traccar
SuccessExitStatus=143
WatchdogSec=300
Restart=on-failure
RestartSec=10

[Install]
WantedBy=multi-user.target

after editing I once needed to execute "systemctl daemon-reload"

Now with the 300sec no problems with the services for about 3 hours now :-)

5of94 years ago

Other possibility is to disable systemd watchdog as you say and build an easy diy watchdog as I did before: Cronjob which checks the last database entry date and if its longer as it should be (you know how often positions should come minimum) than restart the service...

Juan Sánchez4 years ago

On raspberry pi 3b, 1000000 is not enough

5of94 years ago

Other possibility than edit systemd file should be traccar config:

<key='web.healthCheck'>false</entry>

Am I right!? Maybe this should be in the config manual file with text like this:
To disable the systemd watchdog on linux machines (for example on slow machines) this can be set to false