Log flooded with ResourceNotFoundException, ClosedChannelException

Hi,

I have checked for connections in use and configurations etc. for the cause of the following errors. But there is no success. It happens suddenly. Even server is not sending commands to the devices. What could be the possible cause? There is so much space and RAM available into the server. Using Traccar 3.16.

Unable to find resource 'sms/ignitionOn.vm' - ResourceNotFoundException (... < NotificationFormatter:79 < *:102 < *:113 < NotificationSms:39

error - Broken pipe - IOException (... < Gt06ProtocolDecoder:174 < *:628 < *:486 < *:770 < ExtendedObjectDecoder:58 < ...)
error - ClosedChannelException (... < Gt06ProtocolDecoder:174 < *:628 < *:486 < *:770 < ExtendedObjectDecoder:58 < ...)

ResourceNotFoundException means that you are missing the file.

ClosedChannelException means that device closed connection (not gracefully).

The file is in its place. It works sometimes, sometimes not. Same with
ClosedChannelException.

And it happens suddenly just 2 hours ago. It was working fine before that

If the file is there but you still get this error then you probably have some serious problem with operating system or hardware. Possibly your hard drive is dead.

Dear Anton,

After monitoring the server and digging around I found that the issue is happening due to the so many open files.

I have checked the open connections via netstat -ant | grep ESTABLISHED | wc -l .
They are ~600.

Then checked the open files via lsof | wc -l.
They are ~150000. They are all due to the Traccar (Approx ~148000).

So, I restarted the server and found that they are increasing exponentially from ~1000 upto ~200000.

My config file have following configuration for server timeout.

<entry key='server.timeout'>300</entry>

So, they are stuck upto ~200000 due to this configuration.

So, the ResourceNotFoundException were happening due to the open file limits.

I have increased the limits via

vm.max_map_count = 600000
fs.file-max = 600000
net.ipv4.ip_local_port_range = 1024 65535

So, now there is only one exception left which is:

error - Broken pipe - IOException (... < Gt06ProtocolDecoder:174 < *:628 < *:486 < *:770 < ExtendedObjectDecoder:58 < ...)
error - ClosedChannelException (... < Gt06ProtocolDecoder:174 < *:628 < *:486 < *:770 < ExtendedObjectDecoder:58 < ...)

And I think it's the main problem.
Please help me to get out from this problem because it's causing too many open files. Can't we close the connection here by using try-catch block?
I don't have knowledge about netty.
Also can we found out which devices are causes problems via logs? So, we can remove the particular devices. May be it can happened due to their ISP settings.

P.S: I'm using Traccar 3.16 with some modifications into the Web API's part. There is nothing to do with the Protocols decoders and encoders and basic configurations etc.

If you get a lot of those errors it probably means that you have some device or network issues.

If it's a device issue then can we find out which devices are causing this issue via log so I can remove them. Or ask the ISP provider if it is happening from device side?

And if it's from server side then might be a tool to check out the state of outward connection?

You should be able to find device by connection id (in square brackets).

Sorry Anton, I don't understand. Will you please give me some code so that I can put it as it is into the Gt06ProtocolDecoder catch block?

If you need some customisation you should go with paid support option.