-Xss option should be added to Optimization docs

Iván Ávalos9 months ago

In my Traccar deployment, I have noticed that thread memory is one of the first things to exhaust in large deployments with ~1K device connections, based on the logs:

2024-01-09 21:30:56.405	2024-01-09 13:29:28  WARN: HikariPool-1 - Thread starvation or clock leap detected (housekeeper delta=1m4s441ms686µs53ns).

The default thread memory limit in Java is generally 1MB, which is probably not sufficient, given that Traccar opens a new thread for every connected device. Even if the VM is setup with loads of memory (via -Xms and -Xmx), Traccar will malfunction. This issue has been bugging us for years, forcing us to restart the server every week, and I feel like this might be the solution. That's why I propose adding it to the docs.

Iván Ávalos9 months ago

The -Xss option in Java affects the stack memory limit. Increasing it (e.g. to 2MB) can be done like this:

java -Xss2M -jar tracker-server.jar conf/traccar.xml
Iván Ávalos9 months ago

Errata of the first post:

  • There is no thread memory, but rather stack memory.
  • Stack memory gets exhausted because every thread uses stack.
  • 1MB is the default stack memory limit, not thread memory limit.
Anton Tananaev9 months ago

I don't understand the reasoning. Thread starvation has nothing to do with memory. It points to CPU.

Iván Ávalos9 months ago

Thanks for clearing it up, I'll check my CPU, then. But still, I feel like increasing the stack memory limit should at least somewhat improve stability or performance in large scale deployments, shouldn't it? >1K threads open should have an impact in stack memory.

Anton Tananaev9 months ago

No, it shouldn't. If you hit the stack limit, the service will crash with out of memory exception.