error "Too many open files"

Manuel Suárez6 years ago

Hola Anton, buenos dias, he aplicado las soluciones que describes aqui: https://www.traccar.org/optimization/
sin ningun exito ya que tras haber aplicado todas las configuraciones sigo teniendo un error que tira el servico de traccar, aunque debo mencionar que no es el mismo que al principio.

2018-12-03 17:12:21  WARN: An exceptionCaught() event was fired, and it reached                                                                                                                                                              at the tail of the pipeline. It usually means the last handler in the pipeline d                                                                                                                                                             id not handle the exception. - Too many open files - IOException (...)
2018-12-03 17:12:22  WARN: Too many open files - IOException (...)
2018-12-03 17:12:22  WARN: An exceptionCaught() event was fired, and it reached                                                                                                                                                              at the tail of the pipeline. It usually means the last handler in the pipeline d                                                                                                                                                             id not handle the exception. - Too many open files - IOException (...)
2018-12-03 17:12:23  WARN: Too many open files - IOException (...)
2018-12-03 17:12:23  WARN: An exceptionCaught() event was fired, and it reached                                                                                                                                                              at the tail of the pipeline. It usually means the last handler in the pipeline d                                                                                                                                                             id not handle the exception. - Too many open files - IOException (...)
2018-12-03 17:12:24  WARN: Too many open files - IOException (...)
2018-12-03 17:12:24  WARN: An exceptionCaught() event was fired, and it reached                                                                                                                                                              at the tail of the pipeline. It usually means the last handler in the pipeline d                                                                                                                                                             id not handle the exception. - Too many open files - IOException (...)
2018-12-03 17:12:25  WARN: Too many open files - IOException (...)
Anton Tananaev6 years ago

If this is addressed to me, you should write in English.

Manuel Suárez6 years ago

An apology I do not notice in what language I publish it Anton

I have applied the solutions you describe here: https://www.traccar.org/optimization/
without any success since after having applied all the configurations I still have an error that pulls the traccar service, although I must mention that it is not the same as at the beginning.

2018-12-03 17:12:21  WARN: An exceptionCaught() event was fired, and it reached                                                                                                                                                              at the tail of the pipeline. It usually means the last handler in the pipeline d                                                                                                                                                             id not handle the exception. - Too many open files - IOException (...)
2018-12-03 17:12:22  WARN: Too many open files - IOException (...)
2018-12-03 17:12:22  WARN: An exceptionCaught() event was fired, and it reached                                                                                                                                                              at the tail of the pipeline. It usually means the last handler in the pipeline d                                                                                                                                                             id not handle the exception. - Too many open files - IOException (...)
2018-12-03 17:12:23  WARN: Too many open files - IOException (...)
2018-12-03 17:12:23  WARN: An exceptionCaught() event was fired, and it reached                                                                                                                                                              at the tail of the pipeline. It usually means the last handler in the pipeline d                                                                                                                                                             id not handle the exception. - Too many open files - IOException (...)
2018-12-03 17:12:24  WARN: Too many open files - IOException (...)
2018-12-03 17:12:24  WARN: An exceptionCaught() event was fired, and it reached                                                                                                                                                              at the tail of the pipeline. It usually means the last handler in the pipeline d                                                                                                                                                             id not handle the exception. - Too many open files - IOException (...)
2018-12-03 17:12:25  WARN: Too many open files - IOException (...)
Manuel Suárez6 years ago

What is more strange to me is that I happened with version 4.1 and 4.2, from version 4.0 to the past I have not had this problem and I have several seecidores with traccar

Anton Tananaev6 years ago

The solution that you applied clearly didn't work. What is the open file limit for the specific process? What is the number of open connections?

Manuel Suárez6 years ago

Seguí el manual lo más exacto posible

Limite de conexiones:

root soft  nofile 50000
root hard  nofile 50000

Escale mas alla de 65k:

vm.max_map_count = 250000
fs.file-max = 250000
net.ipv4.ip_local_port_range = 1024 65535

Parametros de configuracion de servicio:

ExecStart = / opt / traccar / jre / bin / java -Xmx10G -jar tracker-server.jar conf / traccar.xml
Ernesto Vallejo6 years ago

You might be looking for a fix in the wrong direction.
It might be related to your hardware.
Could you elaborate on OS and hardware?

Manuel Suárez6 years ago

Hello, thanks for the support, the operating system is Ubuntu 16.04 64 bits and the hardware of my server:

  • Xeon 4 Core x 3.5 GHZ (E3-1230V6)
  • 16 GB of RAM
    -1 TB on HDD

and I only have about 30 devices configured in the system, Anton mentioned that the error could be generated even if only one device was configured, what has me very stuck is that, although I have applied the improvements according to the documentation of traccar, I did not has worked In fact, I had to download to version 4.0 and in this version I have not given any problems, but I would like to return to 4.2 to continue using the functions of that version.

Thiago Araújo6 years ago

Hello. After upgrading traccar from 3.17 to 4.2 I also started to have the issue of max open files.

The problem was the new systemctl service configuration. Although the system is configured to permit 50000 open files, the process was limited to just 4096 files.

The solution is to add a directive to file /etc/systemd/system/traccar.service

LimitNOFILE=50000

After that you should run the command:

systemctl --system daemon-reload
Alex5 years ago

Hello Anton, I have next problem with traccar service, /etc/systemd/system/traccar.service

CGroup: /system.slice/traccar.service 
       └─17320 /usr/bin/java -cp /opt/traccar/target/tracker-server.jar org.traccar.Main /opt/traccar/debug.xml

 Nov 29 13:25:54 ip-172-31-40-161 traccar[17320]: 2019-11-29 13:25:54  WARN: Too many open files - IOException (...)
 Nov 29 13:25:54 ip-172-31-40-161 traccar[17320]: 2019-11-29 13:25:54  WARN: An exceptionCaught() event was fired, and it reached at the tail of the pip
 Nov 29 13:25:55 ip-172-31-40-161 traccar[17320]: 2019-11-29 13:25:55  WARN: Too many open files - IOException (...)
 Nov 29 13:25:55 ip-172-31-40-161 traccar[17320]: 2019-11-29 13:25:55  WARN: An exceptionCaught() event was fired, and it reached at the tail of the pip
 Nov 29 13:25:56 ip-172-31-40-161 traccar[17320]: 2019-11-29 13:25:56  WARN: Too many open files - IOException (...)
 Nov 29 13:25:56 ip-172-31-40-161 traccar[17320]: 2019-11-29 13:25:56  WARN: An exceptionCaught() event was fired, and it reached at the tail of the pip
 Nov 29 13:25:57 ip-172-31-40-161 traccar[17320]: 2019-11-29 13:25:57  WARN: Too many open files - IOException (...)
 Nov 29 13:25:57 ip-172-31-40-161 traccar[17320]: 2019-11-29 13:25:57  WARN: An exceptionCaught() event was fired, and it reached at the tail of the pip
 Nov 29 13:25:58 ip-172-31-40-161 traccar[17320]: 2019-11-29 13:25:58  WARN: Too many open files - IOException (...)
 Nov 29 13:25:58 ip-172-31-40-161 traccar[17320]: 2019-11-29 13:25:58  WARN: An exceptionCaught() event was fired, and it reached at the tail of the pip

In traccar.service

SyslogIdentifier=traccar
SuccessExitStatus=143
LimitNOFILE=50000
Restart=always
Arsen A.5 years ago

Hello All,

Also have the same problem with v4.6, 15 devices only, hard and soft limits set to 50000.
I use Sinotrack ST-901 devices, wich constantly get disconnect and connect again

WARN: [0c4427b9] error - Connection reset by peer - IOException (...)
INFO: [0c4427b9] disconnected

Do you think this might be the root cause, as each reconnection does not close some resource?

cristianormoraes5 years ago

Same problem here, any solution?

Matt5 years ago

Hi

Are you using TCP for your GPS devices to send their locations to the server? When I used TCP but the GPS device had connectivity options I noticed that the existing TCP connections were never dropped by the server when the GPS device re-established a new connection. For this reason I reach my connection limit and I received the same "Too many open files" message.

I mentioned the issue here, but I have not heard back from the developer:
https://www.traccar.org/forums/topic/traccar-holding-on-to-failed-tcp-connections-until-it-crashes/

I did find that configuring the devices to use UDP instead of TCP 'solves' the problem. But I would prefer to go back to TCP if we can fix the way the server handles the connections.

burak akarsubasi4 years ago

thiagoaraujo84's step is important for centos 7.x ,otherwise server will not apply your configurations. and you see soft and hard limit as 4096 (or other default) when run cat /proc/$(pidof java)/limits | grep "open files"