Server Stopped Working (Warn: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit)

code_af2 years ago

Hello to all,

Today I wanted to send a positionSingle command to a watch protocol's device, during watching the log file, suddenly saw that all devices are disconnecting from the server while all them were online, see the logs before:

2022-09-19 14:35:51  INFO: user: 306, action: login, from: 91.97.97.44
2022-09-19 14:37:16  INFO: [Ta3f01520] id: 8800000004, command type: positionSingle sent
2022-09-19 14:37:16  INFO: [Ta3f01520: watch > 20.14.13.210] 5b53472a383830303030303030342a303030322a43525d
2022-09-19 14:37:41  INFO: [Te21d2770] timed out
2022-09-19 14:37:41  INFO: [Te21d2770] disconnected
2022-09-19 14:38:17  INFO: [T54fde7f5] timed out
2022-09-19 14:38:17  INFO: [T54fde7f5] disconnected
2022-09-19 14:38:45  INFO: user: 306, report: events, from: 2022-08-20 14:38, to: 2022-09-19 14:38, devices: [-1], groups: []
2022-09-19 14:38:45  INFO: user: 306, report: events, from: 2022-08-20 14:38, to: 2022-09-19 14:38, devices: [952], groups: []
2022-09-19 14:38:48  INFO: [T89a2c85c] timed out
2022-09-19 14:38:48  INFO: [T89a2c85c] disconnected
2022-09-19 14:38:53  INFO: [Tcc0acd73] timed out
2022-09-19 14:38:53  INFO: [Tcc0acd73] disconnected
2022-09-19 14:38:56  INFO: [Td9794c6e] timed out
2022-09-19 14:38:56  INFO: [Td9794c6e] disconnected
2022-09-19 14:39:05  INFO: [T70eb653e] timed out
2022-09-19 14:39:05  INFO: [T70eb653e] disconnected
2022-09-19 14:39:07  INFO: [Tcba87f48] timed out
2022-09-19 14:39:07  INFO: [Tcba87f48] disconnected
2022-09-19 14:39:08  INFO: [T81c5323e] timed out
2022-09-19 14:39:08  INFO: [T81c5323e] disconnected
2022-09-19 14:39:10  INFO: [T3ecd7613] timed out
2022-09-19 14:39:10  INFO: [T3ecd7613] disconnected
2022-09-19 14:39:26  INFO: user: 306, action: login, from: 91.97.97.44
2022-09-19 14:39:27  INFO: [Td0a5d097] timed out
2022-09-19 14:39:27  INFO: [Td0a5d097] disconnected
2022-09-19 14:39:34  INFO: [Ta3f01520] timed out
2022-09-19 14:39:34  INFO: [Ta3f01520] disconnected
2022-09-19 14:39:49  INFO: [Te1d00c35] timed out
2022-09-19 14:39:49  INFO: [Te1d00c35] disconnected
2022-09-19 14:40:08  INFO: [T769f2e2a] timed out
2022-09-19 14:40:08  INFO: [T769f2e2a] disconnected
2022-09-19 14:40:13  INFO: user: 306, report: events, from: 2022-08-20 14:40, to: 2022-09-19 14:40, devices: [-1], groups: []
2022-09-19 14:40:13  INFO: user: 306, report: events, from: 2022-08-20 14:40, to: 2022-09-19 14:40, devices: [952], groups: []
2022-09-19 14:40:39  INFO: [T1678f300] timed out
2022-09-19 14:40:39  INFO: [T1678f300] disconnected
2022-09-19 14:40:51  INFO: [Ta86b79c4] timed out
2022-09-19 14:40:51  INFO: [Ta86b79c4] disconnected
2022-09-19 14:40:57  WARN: Notification failed - Connection timed out (Connection timed out) - ConnectException (... < NotificatorFirebase:86 < NotificationManager:104 < ... < *:101 < ... < *:100 < *:124 < ConnectionManager:315 < ...)
2022-09-19 14:40:57  WARN: Failed to get notificator 
2022-09-19 14:40:57  WARN: You are using null notificatior, please check your configuration, notification not sent
2022-09-19 14:40:57  WARN: Failed to get notificator 
2022-09-19 14:40:57  WARN: You are using null notificatior, please check your configuration, notification not sent
2022-09-19 14:41:07  INFO: [Tc6849b4e] timed out
2022-09-19 14:41:07  INFO: [Tc6849b4e] disconnected
2022-09-19 14:41:24  INFO: [Tae5f08ea] timed out
2022-09-19 14:41:24  INFO: [Tae5f08ea] disconnected
2022-09-19 14:41:43  INFO: user: 306, report: events, from: 2022-08-20 14:41, to: 2022-09-19 14:41, devices: [-1], groups: []
2022-09-19 14:41:44  INFO: user: 306, report: events, from: 2022-08-20 14:41, to: 2022-09-19 14:41, devices: [952], groups: []
2022-09-19 14:43:08  WARN: Notification failed - Connection timed out (Connection timed out) - ConnectException (... < NotificatorFirebase:86 < NotificationManager:104 < ... < *:101 < ... < *:100 < *:124 < ConnectionManager:315 < ...)
2022-09-19 14:43:08  WARN: Failed to get notificator 
2022-09-19 14:43:08  WARN: You are using null notificatior, please check your configuration, notification not sent
2022-09-19 14:43:23  INFO: user: 306, action: login, from: 91.97.97.44

I tried to restart the traccar service, but the service stucked at restarting, as you see bellow...

2022-09-19 14:43:38  INFO: Stopping server...

Then I tried again to restart the traccar service with same linux command sudo systemctl restart traccar.service and then saw bellow logs with warning of DOS ALERT:

2022-09-19 14:45:08  INFO: Operating system name: Linux version: 5.4.0-125-generic architecture: amd64
2022-09-19 14:45:09  INFO: Java runtime name: OpenJDK 64-Bit Server VM vendor: ojdkbuild version: 11.0.15+10-LTS
2022-09-19 14:45:09  INFO: Memory limit heap: 8014mb non-heap: 0mb
2022-09-19 14:45:09  INFO: Character encoding: UTF-8 charset: UTF-8
2022-09-19 14:45:09  INFO: Version: 5.3
2022-09-19 14:45:09  INFO: Starting server...
2022-09-19 14:45:09  INFO: HikariPool-1 - Starting...
2022-09-19 14:45:09  WARN: Registered driver with driverClassName=com.mysql.jdbc.Driver was not found, trying direct instantiation.
2022-09-19 14:45:09  INFO: HikariPool-1 - Added connection com.mysql.cj.jdbc.ConnectionImpl@6ac4944a
2022-09-19 14:45:09  INFO: HikariPool-1 - Start completed.
2022-09-19 14:45:09  INFO: Clearing database change log checksums
2022-09-19 14:45:09  INFO: Successfully acquired change log lock
2022-09-19 14:45:10  INFO: Successfully released change log lock
2022-09-19 14:45:10  INFO: Successfully acquired change log lock
2022-09-19 14:45:10  INFO: Cannot load service: liquibase.parser.ChangeLogParser: liquibase.parser.core.json.JsonChangeLogParser Unable to get public no-arg constructor
2022-09-19 14:45:10  INFO: Cannot load service: liquibase.parser.ChangeLogParser: liquibase.parser.core.yaml.YamlChangeLogParser Unable to get public no-arg constructor
2022-09-19 14:45:10  INFO: Reading from traccar.DATABASECHANGELOG
2022-09-19 14:45:11  INFO: Reading from traccar.DATABASECHANGELOG
2022-09-19 14:45:11  INFO: Cannot load service: liquibase.hub.HubService: Provider liquibase.hub.core.StandardHubService could not be instantiated
2022-09-19 14:45:11  INFO: Successfully released change log lock
2022-09-19 14:45:11  INFO: Your platform does not provide complete low-level API for accessing direct buffers reliably. Unless explicitly requested, heap buffer will always be preferred to avoid potential system instability.
2022-09-19 14:45:12  INFO: jetty-10.0.7; built: 2021-10-06T19:34:02.766Z; git: da8a4553af9dd84080931fa0f8c678cd2d60f3d9; jvm 11.0.15+10-LTS
2022-09-19 14:45:12  INFO: Session workerName=node0
2022-09-19 14:45:12  WARN: JAXBContext implementation could not be found. WADL feature is disabled.
2022-09-19 14:45:13  INFO: Started o.e.j.s.ServletContextHandler@40a7974{/,null,AVAILABLE}
2022-09-19 14:45:13  INFO: Started ServerConnector@150fc7a7{HTTP/1.1, (http/1.1)}{0.0.0.0:8082}
2022-09-19 14:45:13  INFO: Started Server@919086{STARTING}[10.0.7,sto=0] @4747ms
2022-09-19 14:45:13  INFO: Health check enabled with period 480000
2022-09-19 14:45:14  INFO: user: 186, action: login, from: 213.12.3.201
2022-09-19 14:45:15  INFO: Stopping server...
2022-09-19 14:45:16  INFO: Operating system name: Linux version: 5.4.0-125-generic architecture: amd64
2022-09-19 14:45:16  INFO: Java runtime name: OpenJDK 64-Bit Server VM vendor: ojdkbuild version: 11.0.15+10-LTS
2022-09-19 14:45:16  INFO: Memory limit heap: 8014mb non-heap: 0mb
2022-09-19 14:45:16  INFO: Character encoding: UTF-8 charset: UTF-8
2022-09-19 14:45:16  INFO: Version: 5.3
2022-09-19 14:45:16  INFO: Starting server...
2022-09-19 14:45:16  INFO: HikariPool-1 - Starting...
2022-09-19 14:45:16  WARN: Registered driver with driverClassName=com.mysql.jdbc.Driver was not found, trying direct instantiation.
2022-09-19 14:45:16  INFO: HikariPool-1 - Added connection com.mysql.cj.jdbc.ConnectionImpl@6ac4944a
2022-09-19 14:45:16  INFO: HikariPool-1 - Start completed.
2022-09-19 14:45:16  INFO: Clearing database change log checksums
2022-09-19 14:45:17  INFO: Successfully acquired change log lock
2022-09-19 14:45:17  INFO: Successfully released change log lock
2022-09-19 14:45:17  INFO: Successfully acquired change log lock
2022-09-19 14:45:17  INFO: Cannot load service: liquibase.parser.ChangeLogParser: liquibase.parser.core.json.JsonChangeLogParser Unable to get public no-arg constructor
2022-09-19 14:45:17  INFO: Cannot load service: liquibase.parser.ChangeLogParser: liquibase.parser.core.yaml.YamlChangeLogParser Unable to get public no-arg constructor
2022-09-19 14:45:18  INFO: Reading from traccar.DATABASECHANGELOG
2022-09-19 14:45:18  INFO: Reading from traccar.DATABASECHANGELOG
2022-09-19 14:45:18  INFO: Cannot load service: liquibase.hub.HubService: Provider liquibase.hub.core.StandardHubService could not be instantiated
2022-09-19 14:45:18  INFO: Successfully released change log lock
2022-09-19 14:45:19  INFO: Your platform does not provide complete low-level API for accessing direct buffers reliably. Unless explicitly requested, heap buffer will always be preferred to avoid potential system instability.
2022-09-19 14:45:19  INFO: jetty-10.0.7; built: 2021-10-06T19:34:02.766Z; git: da8a4553af9dd84080931fa0f8c678cd2d60f3d9; jvm 11.0.15+10-LTS
2022-09-19 14:45:19  INFO: Session workerName=node0
2022-09-19 14:45:20  WARN: JAXBContext implementation could not be found. WADL feature is disabled.
2022-09-19 14:45:20  INFO: Started o.e.j.s.ServletContextHandler@65a4b9d6{/,null,AVAILABLE}
2022-09-19 14:45:20  INFO: Started ServerConnector@798cf6d2{HTTP/1.1, (http/1.1)}{0.0.0.0:8082}
2022-09-19 14:45:20  INFO: Started Server@7ead1d80{STARTING}[10.0.7,sto=0] @4919ms
2022-09-19 14:45:20  INFO: Health check enabled with period 480000
2022-09-19 14:45:23  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@2bb89898[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.955S, count=25], session=node0ingqgwxwmnuv1m64hrtgke5s62134.node0, user=null
2022-09-19 14:45:23  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@549bd163[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.942S, count=25], session=node0lbdtufsgr90x1nqqvjnyb0t982071.node0, user=null
2022-09-19 14:45:23  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@483d893d[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.967S, count=25], session=node0lbdtufsgr90x1nqqvjnyb0t982071.node0, user=null
2022-09-19 14:45:23  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@3dfc8d53[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.983S, count=25], session=node0btkasuz3415qx2pq8rn0ssq0.node0, user=null
2022-09-19 14:45:25  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@67069727[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.989S, count=25], session=node0lbdtufsgr90x1nqqvjnyb0t982071.node0, user=null
2022-09-19 14:45:25  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@3212b144[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.988S, count=25], session=node0lbdtufsgr90x1nqqvjnyb0t982071.node0, user=null
2022-09-19 14:45:25  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@2f55381b[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.998S, count=25], session=node0ingqgwxwmnuv1m64hrtgke5s62134.node0, user=null
2022-09-19 14:45:25  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@26e0e1e[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.994S, count=25], session=node0lbdtufsgr90x1nqqvjnyb0t982071.node0, user=null
2022-09-19 14:45:26  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@6d55527[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.976S, count=25], session=node0lbdtufsgr90x1nqqvjnyb0t982071.node0, user=null
2022-09-19 14:45:26  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@47c86214[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.994S, count=25], session=node0ingqgwxwmnuv1m64hrtgke5s62134.node0, user=null
2022-09-19 14:45:26  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@527904e9[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.946S, count=25], session=node0btkasuz3415qx2pq8rn0ssq0.node0, user=null
2022-09-19 14:45:26  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@145ba0c8[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.975S, count=25], session=node0lbdtufsgr90x1nqqvjnyb0t982071.node0, user=null
2022-09-19 14:45:26  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@43a17c99[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.976S, count=25], session=node0ingqgwxwmnuv1m64hrtgke5s62134.node0, user=null
2022-09-19 14:45:27  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@1edd83dc[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.989S, count=25], session=node0ingqgwxwmnuv1m64hrtgke5s62134.node0, user=null
2022-09-19 14:45:27  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@52ca1896[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.978S, count=25], session=node0lbdtufsgr90x1nqqvjnyb0t982071.node0, user=null
2022-09-19 14:45:28  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@6c4d178b[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.97S, count=25], session=node0ingqgwxwmnuv1m64hrtgke5s62134.node0, user=null
2022-09-19 14:45:28  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@7836dc22[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.967S, count=25], session=node0lbdtufsgr90x1nqqvjnyb0t982071.node0, user=null
2022-09-19 14:45:29  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@37068951[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.99S, count=25], session=node0btkasuz3415qx2pq8rn0ssq0.node0, user=null
2022-09-19 14:45:29  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@1c871510[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.984S, count=25], session=node0ingqgwxwmnuv1m64hrtgke5s62134.node0, user=null
2022-09-19 14:45:29  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@6c176aaf[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.985S, count=25], session=node0lbdtufsgr90x1nqqvjnyb0t982071.node0, user=null
2022-09-19 14:45:30  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@5fd827f7[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.998S, count=25], session=node0lbdtufsgr90x1nqqvjnyb0t982071.node0, user=null
2022-09-19 14:45:30  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@410c122a[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.969S, count=25], session=node0lbdtufsgr90x1nqqvjnyb0t982071.node0, user=null
2022-09-19 14:45:30  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@17f7f88f[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.972S, count=25], session=node0btkasuz3415qx2pq8rn0ssq0.node0, user=null
2022-09-19 14:45:30  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@31e984fb[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.98S, count=25], session=node0ingqgwxwmnuv1m64hrtgke5s62134.node0, user=null
2022-09-19 14:45:31  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@14422212[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.982S, count=25], session=node0ingqgwxwmnuv1m64hrtgke5s62134.node0, user=null
2022-09-19 14:45:31  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@1181e089[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.96S, count=25], session=node0lbdtufsgr90x1nqqvjnyb0t982071.node0, user=null
2022-09-19 14:45:31  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@1520af20[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.992S, count=25], session=node0btkasuz3415qx2pq8rn0ssq0.node0, user=null
2022-09-19 14:45:31  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@384daa72[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.958S, count=25], session=node0ingqgwxwmnuv1m64hrtgke5s62134.node0, user=null
2022-09-19 14:45:32  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@6c74949d[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.969S, count=25], session=node0ingqgwxwmnuv1m64hrtgke5s62134.node0, user=null
2022-09-19 14:45:32  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@4dc32c26[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.918S, count=25], session=node0lbdtufsgr90x1nqqvjnyb0t982071.node0, user=null
2022-09-19 14:45:32  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@2b94533b[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.957S, count=25], session=node0btkasuz3415qx2pq8rn0ssq0.node0, user=null
2022-09-19 14:45:33  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@2e99a759[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.994S, count=25], session=node0lbdtufsgr90x1nqqvjnyb0t982071.node0, user=null
2022-09-19 14:45:34  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@3456d4ae[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.991S, count=25], session=node0ingqgwxwmnuv1m64hrtgke5s62134.node0, user=null
2022-09-19 14:45:34  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@3a9466d6[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.967S, count=25], session=node0lbdtufsgr90x1nqqvjnyb0t982071.node0, user=null
2022-09-19 14:45:36  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@1a471a93[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.925S, count=25], session=node0btkasuz3415qx2pq8rn0ssq0.node0, user=null
2022-09-19 14:45:36  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@7f55dc8e[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.938S, count=25], session=node0lbdtufsgr90x1nqqvjnyb0t982071.node0, user=null
2022-09-19 14:45:36  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@18a49249[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.894S, count=25], session=node0ingqgwxwmnuv1m64hrtgke5s62134.node0, user=null
2022-09-19 14:45:36  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@73b42df3[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.999S, count=25], session=node0btkasuz3415qx2pq8rn0ssq0.node0, user=null
2022-09-19 14:45:38  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@676c3bdf[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.979S, count=25], session=node0btkasuz3415qx2pq8rn0ssq0.node0, user=null
2022-09-19 14:45:38  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@2131dee5[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.955S, count=25], session=node0lbdtufsgr90x1nqqvjnyb0t982071.node0, user=null
2022-09-19 14:45:39  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@665092ed[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.993S, count=25], session=node0btkasuz3415qx2pq8rn0ssq0.node0, user=null
2022-09-19 14:45:39  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@778e75f9[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.988S, count=25], session=node0ingqgwxwmnuv1m64hrtgke5s62134.node0, user=null
2022-09-19 14:45:39  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@4b6024ec[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.998S, count=25], session=node0lbdtufsgr90x1nqqvjnyb0t982071.node0, user=null
2022-09-19 14:45:40  WARN: DOS ALERT: Request delayed=100ms, ip=[0:0:0:0:0:0:0:1], overlimit=OverLimit@17efba96[type=IP, id=[0:0:0:0:0:0:0:1], duration=PT0.98S, count=25], session=node0lbdtufsgr90x1nqqvjnyb0t982071.node0, user=null
2022-09-19 14:45:46  INFO: user: 186, action: login, from: 213.16.32.32

Then I tried the third time to restart the traccar service, and after restarting it again, it is running normal up to now!

2022-09-19 14:46:07  INFO: Operating system name: Linux version: 5.4.0-125-generic architecture: amd64
2022-09-19 14:46:07  INFO: Java runtime name: OpenJDK 64-Bit Server VM vendor: ojdkbuild version: 11.0.15+10-LTS
2022-09-19 14:46:07  INFO: Memory limit heap: 8014mb non-heap: 0mb
2022-09-19 14:46:07  INFO: Character encoding: UTF-8 charset: UTF-8
2022-09-19 14:46:07  INFO: Version: 5.3
2022-09-19 14:46:07  INFO: Starting server...
2022-09-19 14:46:07  INFO: HikariPool-1 - Starting...
2022-09-19 14:46:08  WARN: Registered driver with driverClassName=com.mysql.jdbc.Driver was not found, trying direct instantiation.
2022-09-19 14:46:08  INFO: HikariPool-1 - Added connection com.mysql.cj.jdbc.ConnectionImpl@6ac4944a
2022-09-19 14:46:08  INFO: HikariPool-1 - Start completed.
2022-09-19 14:46:08  INFO: Clearing database change log checksums
2022-09-19 14:46:08  INFO: Successfully acquired change log lock
2022-09-19 14:46:08  INFO: Successfully released change log lock
2022-09-19 14:46:08  INFO: Successfully acquired change log lock
2022-09-19 14:46:08  INFO: Cannot load service: liquibase.parser.ChangeLogParser: liquibase.parser.core.json.JsonChangeLogParser Unable to get public no-arg constructor
2022-09-19 14:46:08  INFO: Cannot load service: liquibase.parser.ChangeLogParser: liquibase.parser.core.yaml.YamlChangeLogParser Unable to get public no-arg constructor
2022-09-19 14:46:09  INFO: Reading from traccar.DATABASECHANGELOG
2022-09-19 14:46:10  INFO: Reading from traccar.DATABASECHANGELOG
2022-09-19 14:46:10  INFO: Cannot load service: liquibase.hub.HubService: Provider liquibase.hub.core.StandardHubService could not be instantiated
2022-09-19 14:46:10  INFO: Successfully released change log lock
2022-09-19 14:46:10  INFO: Your platform does not provide complete low-level API for accessing direct buffers reliably. Unless explicitly requested, heap buffer will always be preferred to avoid potential system instability.
2022-09-19 14:46:11  INFO: jetty-10.0.7; built: 2021-10-06T19:34:02.766Z; git: da8a4553af9dd84080931fa0f8c678cd2d60f3d9; jvm 11.0.15+10-LTS
2022-09-19 14:46:11  INFO: Session workerName=node0
2022-09-19 14:46:11  WARN: JAXBContext implementation could not be found. WADL feature is disabled.
2022-09-19 14:46:11  INFO: Started o.e.j.s.ServletContextHandler@3dea1ecc{/,null,AVAILABLE}
2022-09-19 14:46:11  INFO: Started ServerConnector@150fc7a7{HTTP/1.1, (http/1.1)}{0.0.0.0:8082}
2022-09-19 14:46:11  INFO: Started Server@919086{STARTING}[10.0.7,sto=0] @4697ms
2022-09-19 14:46:11  INFO: Health check enabled with period 480000

I didn't get where this problem comes from, or how to solve this for the future, does anyone else faced the same problem?

Thanks in advance for your help!

Anton Tananaev2 years ago

It seems like you mixed up several completely unrelated things into one. A single command is probably unrelated to all disconnected devices. DOS alert is probably unrelated to restarting the service. etc

code_af2 years ago

Thank you @Anton for you reply.

Maybe they are unrelated to each other, but don't you think there is a reason or problem exists for them? I am concerned for two things:

  1. DOS Alert, was it really a DDOS attack or something? Is there a way to find more about this on our server side, maybe where do you think will be helpful to look into it?

  2. Disconnecting devices are concerned but I don't think so if it was because of single command, but how it was possible that all devices were timed-out and disconnected from the server in a single minute, I am sure that these all devices didn't turn-off at the same time? Even I can see the firebase connection was also timed-out, therefore I am concerned that how it was possible?

Anton Tananaev2 years ago

My point is that I don't think it's a good idea to discuss all the issues at the same time.

code_af2 years ago

Okay. Then I will open a separate discussion for each issue.

Fabiano Gastaldia year ago

Hi, i had the issue, see:
https://www.traccar.org/forums/topic/real-case-of-dos-alert-high-memory-usage-traccar-server-linux-56/