sessionsByEndpoint conflict with HAProxy

mason chak10 months ago

Hi,
I have an HAProxy setup that accepts connections from GPS devices on port 8080 for the GT06 protocol and on port 8081 for the H02 protocol. Additionally, I have servers setup that utilizes horizontal scaling with Redis, and everything works fine under low traffic conditions. However, when the number of connections increases, sometimes the location of, for example, Device A gets stored as Device B's position, leading to some kind of session conflict in my Traccar system.

For instance, a device using the H02 protocol might randomly record positions from devices using the GT06 protocol.

Could you assist me with this problem? If you need more information, I can provide it. Thank you!

Anton Tananaev10 months ago

Are you using TCP for both protocols?

mason chak10 months ago

yes, both are TCP.

Anton Tananaev10 months ago

Then I'm not sure how it's possible. Sessions are stored by remoteAddress, which includes IP and port, so it will be unique for different connections.

mason chak10 months ago

Backend servers see the HAProxy IP and ports. The IP is HAProxy's and is the same for all connections, only the port is unique.

Anton Tananaev10 months ago

I'm not sure I understand your point. My statement still stands, from what I can tell.

mason chak10 months ago

I mean, the backend server cannot see the devices' IPs because of HAProxy. Then, all connections from devices are in this form (haproxyServerIp:port). For example, my HAProxy server IP is 192.168.1.10. Traccar sees device1's IP like this => 192.168.1.10:51500, and device2's IP like this => 192.168.1.10:58932.

And so on, all devices that connect to the backend server have the same IP.

I think for the session to work properly, the IP must be unique beside the unique port number, right? Am I right that the same IP can cause such a conflict?

Anton Tananaev10 months ago

Make sure you carefully read my earlier comments.

Thiago Araújo9 months ago

As shown below, something is happening with getDeviceSession() that is returning the wrong session to the same device.

ID	Protocol	DeviceId	FixTime
2772414696	suntech	19402	2024-03-07 23:01:20.116
2772420728	gt06	19402	2024-03-07 23:03:05.610
2772425260	suntech	19402	2024-03-07 23:04:20.157
2772430977	gt06	19402	2024-03-07 23:06:06.393
2772435224	suntech	19402	2024-03-07 23:07:20.478
2772440818	gt06	19402	2024-03-07 23:09:07.131
2772444779	suntech	19402	2024-03-07 23:10:20.196

I'm using the latest Traccar version. Previously I was using version 4.13 and this problem didn't happen.

Thiago Araújo9 months ago

My guess is this problem is happening because of the migration of this logic from BaseProtocolDecoder, which had its lifecycle managed by threads of the socket channel pipeline, to a new singleton on ConnectionManager. Does this make sense?

Anton Tananaev9 months ago

That doesn't explain it.

Thiago Araújo9 months ago

So what could be causing this device session confusion?

Anton Tananaev9 months ago

I don't know. It probably requires some debugging, not guessing.

mason chak9 months ago

@thiagoaraujo84 hi, i have some questions.

  1. Do you have any HAProxy or other load balancer set up?
  2. Do you encounter this problem with other protocols? Or is the problem somehow connected to the GT06 protocol in your case?
mason chak9 months ago

@Anton Tananaev
can you explain this line of code for me?

connectionManager

In this line, by putting a new session in endpointSessions, we actually keep the old, wrong session instead of clearing it. This can cause conflicts. This line causes sessionByEndpoint to become something like this:

sessionsByEndpoint= (ConcurrentHashMap@8419) size =1
 -> key = (InetSocketAddress@11052) "/192.168.1.1:55010"
 -> value = (ConcurrentHashMap@10953) size=2
        > "351510091165632" -> (DeviceSession@11046)
        > "900900900900900" -> (DeviceSession@11048)