Server periodically drops

Tomáš Holan6 years ago

Hi,
I have problem with traccar server running on Debian. I set-up server month ago and it periodically shutdown after few days without error message.
32GB RAM
8 Cores
SSD

See log, there is only message with "Shutting down server..." and with INFO flag! I think it should be with WARN flag if there's a problem.

2018-09-11 23:14:39 DEBUG: [E1C199F1: 5013 < 46.135.22.27] HEX: 2a48512c363032383035393734352c56312c3231313433382c562c343931362e353032322c4e2c30313731392e393436382c452c3030312e36362c3030302c3131303931382c464646$
2018-09-11 23:14:39  INFO: [E1C199F1] id: 6028059745, time: 2018-09-11 23:14:38, lat: 49.27504, lon: 17.33245, speed: 1.7, course: 0.0
2018-09-11 23:19:37 DEBUG: [E1C199F1: 5013 < 46.135.22.27] HEX: 2a48512c363032383035393734352c56312c3231313933362c412c343931362e353230382c4e2c30313731392e393332332c452c3030302e32392c3030302c3131303931382c464646$
2018-09-11 23:19:37  INFO: [E1C199F1] id: 6028059745, time: 2018-09-11 23:19:36, lat: 49.27535, lon: 17.33221, speed: 0.3, course: 0.0
2018-09-11 23:20:29 DEBUG: [E1C199F1: 5013 < 46.135.22.27] HEX: 2a48512c363032383035393734352c56312c3231323032382c412c343931362e353130382c4e2c30313731392e393330382c452c3030312e31362c3030302c3131303931382c464646$
2018-09-11 23:20:29  INFO: [E1C199F1] id: 6028059745, time: 2018-09-11 23:20:28, lat: 49.27518, lon: 17.33218, speed: 1.2, course: 0.0
2018-09-11 23:24:27  INFO: [E1C199F1] disconnected
2018-09-11 23:24:28  INFO: [8DA878F9] connected
2018-09-11 23:24:35 DEBUG: [8DA878F9: 5013 < 46.135.22.27] HEX: 2a48512c363032383035393734352c56312c3231323433342c412c343931362e353132332c4e2c30313731392e393331352c452c3030302e39372c3030302c3131303931382c464646$
2018-09-11 23:24:35  INFO: [8DA878F9] id: 6028059745, time: 2018-09-11 23:24:34, lat: 49.27521, lon: 17.33219, speed: 1.0, course: 0.0
2018-09-11 23:27:44  INFO: Shutting down server...
2018-09-11 23:27:45  INFO: [F9057BB4] disconnected
2018-09-11 23:27:45  INFO: [F289C47B] disconnected
2018-09-11 23:27:45  INFO: [41E6B787] disconnected
2018-09-11 23:27:45  INFO: [6FBB51D2] disconnected
2018-09-11 23:27:45  INFO: [8DA878F9] disconnected
2018-09-11 23:27:46  INFO: [6E957FDC] disconnected
2018-09-11 23:27:48  INFO: [534F5E70] disconnected

I think about some watchdog which will autostart server if it drops but first I want to know there's no problem like memory leaking etc.
Any suggestion?

Anton Tananaev6 years ago

Have you actually checked wrapper logs to see why it stops?

Tomáš Holan6 years ago

Grrrr didn't saw that.

Here is it but, it didn't give me more info.

INFO|48674/0|Service traccar|18-09-11 23:27:41|Missing wrapper ping within timeout of 30000
WARNING|48674/0|Service traccar|18-09-11 23:27:42|ping between java application and wrapper timed out. if this this is due to server overload consider increasing wrapper.ping.timeout
INFO|48674/0|Service traccar|18-09-11 23:27:44|wrapper manager received stop command
INFO|48674/0|Service traccar|18-09-11 23:27:45|[Thread-16] INFO org.eclipse.jetty.server.ServerConnector - Stopped ServerConnector@245253d8{HTTP/1.1}{0.0.0.0:8084}
INFO|48674/0|Service traccar|18-09-11 23:27:45|[Thread-16] INFO org.eclipse.jetty.server.handler.ContextHandler - Stopped o.t.w.@2519026b{/,null,UNAVAILABLE}
INFO|48674/0|Service traccar|18-09-11 23:27:45|[Thread-16] INFO org.eclipse.jetty.server.handler.ContextHandler - Stopped o.e.j.s.ServletContextHandler@35bfa1bb{/api,null,UNAVAILABLE}
Anton Tananaev6 years ago

Timeout usually means that you don't have enough resources.

Tomáš Holan6 years ago

Trust me I have enough.
As I wrote. 32GB RAM ave. usage 8GB (never go to more then 15GB)
8 Cores ave. usage 20%.

I read here topic where guy had same problem and got same answer, but i had 1 GB RAM(on RPi I think) so 8 Xeon Core with 32GB RAM and 300GB SSD + 4 TB HDD is enough.

Theoretically there can be some temp traffic(too much incoming mails -> antivir scanning or some like that) but this can be only for few seconds.

Anton Tananaev6 years ago

Have you configured JVM to have enough memory?

Tomáš Holan6 years ago

No, I thought that startDaemon.sh alocate enough if there is plenty of free RAM.

Anton Tananaev6 years ago

You clearly haven't a "must read" guide for hosting a production grade Traccar server:

https://www.traccar.org/optimization/

Tomáš Holan6 years ago

OK, I setup 3GB for maxmemory, Thanks. Will see after few days.

Tomáš Holan6 years ago

So, today it drops again

INFO|118660/0|Service traccar|18-09-16 12:54:22|[HikariPool-1 housekeeper] WARN com.zaxxer.hikari.pool.HikariPool - HikariPool-1 - Thread starvation or clock leap detected (housekeeper delta=15m45s515ms354µs260$
INFO|118660/0|Service traccar|18-09-16 12:54:22|Missing wrapper ping within timeout of 30000
INFO|118660/0|Service traccar|18-09-16 12:54:25|[HikariPool-1 housekeeper] WARN com.zaxxer.hikari.pool.HikariPool - HikariPool-1 - Thread starvation or clock leap detected (housekeeper delta=3m16s352ms465µs13ns$
WARNING|118660/0|Service traccar|18-09-16 12:54:25|ping between java application and wrapper timed out. if this this is due to server overload consider increasing wrapper.ping.timeout
INFO|118660/0|Service traccar|18-09-16 12:54:27|wrapper manager received stop command
INFO|118660/0|Service traccar|18-09-16 12:54:28|[Thread-16] INFO org.eclipse.jetty.server.ServerConnector - Stopped ServerConnector@245253d8{HTTP/1.1}{0.0.0.0:8084}
INFO|118660/0|Service traccar|18-09-16 12:54:28|[Thread-16] INFO org.eclipse.jetty.server.handler.ContextHandler - Stopped o.t.w.@2519026b{/,null,UNAVAILABLE}
INFO|118660/0|Service traccar|18-09-16 12:54:29|[Thread-16] INFO org.eclipse.jetty.server.handler.ContextHandler - Stopped o.e.j.s.ServletContextHandler@35bfa1bb{/api,null,UNAVAILABLE}
INFO|wrapper|Service traccar|18-09-16 12:54:30|waitpid 118660 0
INFO|wrapper|Service traccar|18-09-16 12:54:30|exit code posix process: 0 application(status/signal): 0/0

File wrapper.conf:

wrapper.java.command=java
wrapper.java.app.jar=tracker-server.jar
wrapper.java.maxmemory=3072
wrapper.app.parameter.1=./conf/traccar.xml
wrapper.java.additional.1=-Dfile.encoding=UTF-8
wrapper.logfile=logs/wrapper.log.YYYYMMDD
wrapper.logfile.rollmode=DATE
wrapper.ntservice.name=traccar
wrapper.ntservice.displayname=Traccar
wrapper.ntservice.description=Traccar
wrapper.daemon.run_level_dir=${if (new File('/etc/rc0.d').exists()) return '/etc/rcX.d' else return '/etc/init.d/rcX.d'}
Anton Tananaev6 years ago

The issue is still the same. Your service times out.

Tomáš Holan6 years ago

Yeah, I can read that too. I'm asking if is that normal? Memory is set to 3GB and I have connected only one device. We plan to connect around 20 devices and it have to be stable. So I can increase memory limit, but question is, will that help? Is there chance of some memory leak or some more options to get what is wrong? I can't look at server 24h/7d for catch random services shutdown, it's impossible to catch it.

Anton Tananaev6 years ago

If you have only one device, I doubt that it's a memory issue.

Tomáš Holan6 years ago

OK, this messing my time. For save people's time tell them on start you don't want help them. It save time for you and for them.

And for clear. I searched stackoverflow and this forum for couple of days before create this topic. Tried a lot of answer and nothing help. What I want to say is that I tried to fix that for my self cause I know what it is to help people on some SW forum(mainly OSS SW). So, sorry that I missed memory setup (Again didn't think it is in RAM, cause had only one device).

Anton Tananaev6 years ago

You never mentioned that you have only one device. I gave you suggestion about RAM because it's the most common cause of this kind of problem. I tried to help and there is no need to be rude if you don't appreciate my help. If what I suggested didn't help you, it doesn't mean that I didn't want to help. I'm certainly not interested in wasting anyone's time.