In my Traccar deployment, I have noticed that thread memory is one of the first things to exhaust in large deployments with ~1K device connections, based on the logs:
2024-01-09 21:30:56.405 2024-01-09 13:29:28 WARN: HikariPool-1 - Thread starvation or clock leap detected (housekeeper delta=1m4s441ms686µs53ns).
The default thread memory limit in Java is generally 1MB, which is probably not sufficient, given that Traccar opens a new thread for every connected device. Even if the VM is setup with loads of memory (via -Xms and -Xmx), Traccar will malfunction. This issue has been bugging us for years, forcing us to restart the server every week, and I feel like this might be the solution. That's why I propose adding it to the docs.
The -Xss
option in Java affects the stack memory limit. Increasing it (e.g. to 2MB) can be done like this:
java -Xss2M -jar tracker-server.jar conf/traccar.xml
Errata of the first post:
- There is no thread memory, but rather stack memory.
- Stack memory gets exhausted because every thread uses stack.
- 1MB is the default stack memory limit, not thread memory limit.
I don't understand the reasoning. Thread starvation has nothing to do with memory. It points to CPU.
Thanks for clearing it up, I'll check my CPU, then. But still, I feel like increasing the stack memory limit should at least somewhat improve stability or performance in large scale deployments, shouldn't it? >1K threads open should have an impact in stack memory.
No, it shouldn't. If you hit the stack limit, the service will crash with out of memory exception.
In my Traccar deployment, I have noticed that thread memory is one of the first things to exhaust in large deployments with ~1K device connections, based on the logs:
The default thread memory limit in Java is generally 1MB, which is probably not sufficient, given that Traccar opens a new thread for every connected device. Even if the VM is setup with loads of memory (via -Xms and -Xmx), Traccar will malfunction. This issue has been bugging us for years, forcing us to restart the server every week, and I feel like this might be the solution. That's why I propose adding it to the docs.