These are the different error messages that can appear as tickets and what they mean:
Critical Error reported by myremotemon3
- Error connecting to client: Network down to 101.102.103.104. No communication for more than 80 seconds
There was a connection and it has not had a keep alive for at least 80 seconds. This usually indicates that the connection between the sites has gone down.
- Error connecting to client: Monitoring not connecting to 101.102.103.104:27007: Error-code is 110=Connection timed out
This indicates that the connection is not going through to the remote side and is timing out. There is nothing to respond from the remote side. Either the connection between the sites is off or the server is off or the AEM is off.
Critical Error reported by dtxbicom at “CLIENTNAME” for branch “BRANCHNAME”
- Error getting JSON: 500 110 Connection timed out
This is indicating the the API request to the PABX timed out. The PABX didn’t respond with in the timeout period. This may happen if there is an issue with the connection between the sites or if the PABX is off or if the API service is off. Usually this can occur once and then recover and is not a problem.
If this happens with every run of the dtxbicom, it might be that the PABX IP is incorrect in the INI file or the PABX is off. Either may be a major issue and should be investigated.
This is the log entry that you should search for and the line just above it will show the PABX IP that is giving this error.
Here is an example grep to show all the instances of the error and the line just before it.
zegrep "JSON.*No route to host" /opt/datatex/bicom/logs/`logdate`/dtxbicom-calls.log* -B 1
2023-01-09 18:53:06 1.0.13.2 https://101.102.103.105/?apikey=***&action=pbxware.tenant.list
2023-01-09 18:53:22 1.0.13.2 Error getting JSON: 500 113 No route to host
- Error getting JSON: 500 110 Connection timed out
This indicates the connection was working and now timed out. In the example grep here it shows that there were 4 timeouts int he entire day and they all occurred between 05:14 and 05:23 in the morning.
The IP is the same for all the queries so all to the same PABX but the server values were different. They indicates that the iss was not just with one tenant. The next time the service runs, the connections should re-attempt to get all the data again and since the error didn’t repeat for the same servers and pages, it means that the date was retrieved.
zegrep "JSON.*Connection timed out" /opt/datatex/bicom/logs/`logdate`/dtxbicom-calls.log* -B 1
2023-01-13 05:14:08 1.0.13.2 ParseCDR(https://101.102.103.105/?apikey=***&action=pbxware.cdr.download&server=8&start=Jan-12-2023&end=Jan-13-2023&page=108)
2023-01-13 05:14:10 1.0.13.2 Error getting JSON: 500 110 Connection timed out2023-01-13 05:15:41 1.0.13.2 ParseCDR(https://101.102.103.105/?apikey=***&action=pbxware.cdr.download&server=8&start=Jan-12-2023&end=Jan-13-2023&page=124)
2023-01-13 05:15:42 1.0.13.2 Error getting JSON: 500 110 Connection timed out2023-01-13 05:21:30 1.0.13.2 ParseCDR(https://101.102.103.105/?apikey=***&action=pbxware.cdr.download&server=8&start=Jan-12-2023&end=Jan-13-2023&page=185)
2023-01-13 05:21:32 1.0.13.2 Error getting JSON: 500 110 Connection timed out2023-01-13 05:23:00 1.0.13.2 ParseCDR(https://101.102.103.105/?apikey=***&action=pbxware.cdr.download&server=9&start=Jan-12-2023&end=Jan-13-2023&page=10)
2023-01-13 05:23:05 1.0.13.2 Error getting JSON: 500 110 Connection timed out
- Error adding/getting recording status for 101.102.103.105, 1667553217.221039: : DoInternalConnect :
-connection rejected by remote interface
This is an indication that dtxbicom could not write the results it received to the loca database. Usually it is as a result of trying to make to many database connections to quickly in sucession with each other. This usually revoers and the next time that data is downloaded for the CDR, it should write the records away.
If this issue repeats a lot, then this is a major issue.
zegrep "DoInternalConnect" /opt/datatex/bicom/logs/`logdate`/dtxbicom-calls.log.gz -B 1 -A 2
2022-11-04 19:47:54 1.0.13.2 ParseCDR(file:///opt/datatex/bicom/pabx/pabx121/tenants/206/cdr/20221104/cdr-105.json)
2022-11-04 19:47:57 1.0.13.2 Error adding/getting recording status for 101.102.103.105, 1667553217.221039: : DoInternalConnect :
-connection rejected by remote interface
2022-11-04 19:47:59 1.0.13.2 ParseCDR(file:///opt/datatex/bicom/pabx/pabx121/tenants/206/cdr/20221104/cdr-174.json)
- Error parsing “Invalid Server ID is specified.”: Access violation
Not sure what causes this yet, since there is no log for the time that this message took place on the server where I saw this.