API Permissions calls getting very slow

The API POST Permissions calls are getting slower and slower for the current Traccar setup I have:

Traccar server 5.6 - standard version
About 700 devices and 500 users
One admin user, all devices are assigned to that user.
Each device and user are further assigned to a unique "USER" group (500 groups in total), which are then assigned to the Admin user.

The issue is: when trying to add Computed Attributes to one of the 500 "USER" groups, the API calls are getting slower with each new device or user.

Currently, the average wait time for the server response is about 7 seconds per POST call, which is way too much:

Request URL: https://traccar.server.com/api/permissions
Request Method: POST
Payload: {groupId: "521", attributeId: 15}
Waiting for server response: 6.76 seconds

Is there a way to fix this?

I found this topic here and it seems it might be cache related, still not sure what can be done to fix it.

One option is avoid assigning all devices to a single user.

The proper fix would require smarter cache invalidation, so it would need to be done in the code. If you're planning to implement it yourself, please send a pull request. If you need help and you're interested in sponsoring the work, send us an email.

Well, if not possible outside the box, I will look into getting this resolved in the code. But I need to understand what the root cause is.

Based on your comment and what I read earlier, am I right to understand that on every API permissions call, the cache is invalidated and then recreated once again? Which would be otherwise fine, but in case where one user has all devices (or all users), the cache is getting too big hence the large amount of time required per call?

I would recommend checking the code.

That's very helpful. Any suggestion about where the relevant cache code can be found?

CacheManager

So...Traccar instance is now upgraded to v. 5.8 - the API permissions call issue still exists.

Further to that, I did some testing with quite interesting results - all tests done both on Traccar 5.6 and then 5.8.:

1. I am running a similar development environment - same server, same Traccar version but slight differences in the server configuration and database. So I copied the latest database 1 to 1 from production and tried the same API call. No issues with slow execution.

2. Then, I created a new server instance identical to the production one. I copied the entire instance, so everything is exactly the same. The only difference is the different IP and the fact that there is no incoming traffic from the GPS devices. API permissions calls are executed within milliseconds. So obviously the Cache Manager is not to blame here (unless I am missing something).

3. I thought it might be the incoming traffic that is causing the slow performance (currently ~25+ seconds to execute a single API call). So I temporarily closed the devices port on the production environment. Issue still exists.

4. The execution of the same database query directly in MySQL, takes a few milliseconds. So it's not the DB.

Curiously, I noticed that once I place an API permissions request, the incoming traffic is halted until the call is executed. I assume this is normal, still it cannot explain why the API call takes so much time to get executed.

Two questions:

1. Based on those tests, I assume we can rule out the CacheManager, correct?

2. Is there anything you can suggest on how to find out the reason for the slow API call?

How did you do point 3? Are you sure you restarted the service?

I did it using the server IPv4 Firewall - I have rules that allow only specific ports. I just removed the port in question. The traffic stopped immediately.

Note: I am monitoring the traccar log live, so I am positive that the incoming traffic stopped right away. Once I allowed the port, the traffic resumed.

Because I monitor the traccar log live, I also noticed the temporary traffic stop once I send the API permissions call.

Well, try with a restart. Just cutting the traffic on the firewall doesn't clear the cache, so you have exactly the same problem as before. That's why you don't see any difference.

I thought the same so I restarted the entire server, issue still exists.

Here is the Production environment API call (via modern interface).

Here is the identical Production copy without the API issue.

So any wild ideas? Anything in the traccar/server/apache log that might help to pinpoint the issue?

No other ideas. I still think it's most likely the caching.

I am doing some more tests today, I will let you know the results.

In the meantime, can you confirm whether the cache is cleared/recreated when the traccar service is stopped/restarted?