WARN: Too many open files - IOException (...)

I have gone through the optimization procedure on the documentation to the later but am still getting this error.

What can be the issue?

WARN: Too many open files - IOException (...) 

cat /proc/$(pidof java)/limits | grep "open files"
Max open files            50000                50000                files 

/etc/systemd/user.conf
added: DefaultLimitNOFILE=50000

/etc/systemd/system.conf
added: DefaultLimitNOFILE=50000

/etc/sysctl.conf
added:

vm.max_map_count = 250000
fs.file-max = 250000
net.ipv4.ip_local_port_range = 1024 65535

What is the limit for your process and how many open connections do you have?

ulimit  -u
39710
ulimit  -l
16384

ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 39710
max locked memory       (kbytes, -l) 16384
max memory size         (kbytes, -m) unlimited
open files                      (-n) 50000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 39710
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

 top

top - 15:30:53 up 17 min,  5 users,  load average: 0.25, 0.95, 0.97
Tasks: 392 total,   1 running, 217 sleeping,   0 stopped,   3 zombie
%Cpu(s):  2.1 us,  0.6 sy,  0.0 ni, 97.3 id,  0.0 wa,  0.0 hi,  0.1 si,  0.0 st
KiB Mem : 10228708 total,  3699568 free,  5747028 used,   782112 buff/cache
KiB Swap:  4194300 total,  4194300 free,        0 used.  4168316 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                                                 
 1512 traccar   20   0 20.753g 4.694g  19872 S  56.0 48.1  20:29.95 java  

sudo cat /proc/1512/limits
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        0                    unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             39710                39710                processes 
Max open files            50000                50000                files     
Max locked memory         16777216             16777216             bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       39710                39710                signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us

netstat -ant | awk '{print $6}' | sort | uniq -c | sort -n
      1 Foreign
      1 established)
      8 FIN_WAIT2
     26 LISTEN
   2697 ESTABLISHED
   2887 CLOSE_WAIT
   5874 TIME_WAIT

That's current number of open connections. How do you know it didn't reach 50k at the time of the error?

will running this

netstat -ant | awk '{print $6}' | sort | uniq -c | sort -n

give me an overview when the error occurs again on number of connections? I also had other services running on the same machine. I have stopped all of them so that I only have the traccar service running, I have to instances, the other instances doesn't have the issue. I will keep you posted on findings. The other services might have been the problem too...

The limit should be per process, so I don't think other services would cause any issues.

Noted. For some reason I thought 50K was the max, didn't know you can put more than that. Is there a formula for getting this value from a hardware perspective?

How do you know it didn't reach 50k at the time of the error?

That is what am trying to look into. I only have about 5k devices, and not all transmit wasn't expecting it to exhaust the limit. Today we haven't faced the issue. Am keeping an eye on it. So far so good.

Maybe same question again but how to calculate optimal hardware and settings for 250 teltonika trackers at 1Hz and 30k viewers trugh webinterface(public)

Hi

Are you using TCP for your GPS devices to send their locations to the server? When I used TCP but the GPS device had connectivity options I noticed that the existing TCP connections were never dropped by the server when the GPS device re-established a new connection. For this reason I reach my connection limit and I received the same "Too many open files" message.

I mentioned the issue here, but I have not heard back from the developer:
https://www.traccar.org/forums/topic/traccar-holding-on-to-failed-tcp-connections-until-it-crashes/

I did find that configuring the devices to use UDP instead of TCP 'solves' the problem. But I would prefer to go back to TCP if we can fix the way the server handles the connections.