Zabbix plugin tcp timeout after Grafana upgrade to latest version

[Matt197922]

Hi,

I was running Grafana v7.01 with the Zabbix plugin 3.12.4. No issues here. i could reach the Zabbix API succesfully.

I then upgraded Grafana to the vert latest version v10.4, and updated the Zabbix plugin to it's latest release v4.4.6.

Somehow when i test the data sources i'm ending with a tcp timeout.

i tried to even lower the Zabbix plugin version, but the error persisted. I even changed the OS from CentOS7 to Ubuntu LTS but i still see the problem.

What could have changed to get this error?

To connect to this Zabbix instance i don't use any authentication, just a user and password for the API.

From the logs (debug mode) i just see this:

ogger=oauth_token.sync t=2024-03-11T15:51:27.328852996Z level=debug msg="Singleflight request for OAuth token sync" key=user:1 logger=oauthtoken t=2024-03-11T15:51:27.328943674Z level=debug msg="Singleflight request for getting a new access token" key=oauth-refresh-token-1 logger=oauthtoken t=2024-03-11T15:51:27.331371762Z level=debug msg="No oauth token found for user" userId=1 username= logger=token t=2024-03-11T15:51:27.333246473Z level=debug msg=FeatureEnabled feature=accesscontrol.enforcement enabled=false licenseStatus=NotFound hasLicense=false hasValidLicense=false products=[] logger=accesscontrol.service t=2024-03-11T15:51:27.333432116Z level=debug msg="Fetch permissions from store" key=rbac-permissions-1-user-1 logger=accesscontrol.service t=2024-03-11T15:51:27.341815935Z level=debug msg="Cache permissions" key=rbac-permissions-1-user-1 logger=accesscontrol.evaluator t=2024-03-11T15:51:27.342476145Z level=debug msg="Matched scope" userscope=datasources:* targetscope=datasources:uid:adfclv4p3fi0wc logger=datasources t=2024-03-11T15:51:27.342837313Z level=debug msg="Received command to update data source" url=http://10.196.0.30/api_jsonrpc.php logger=datasource t=2024-03-11T15:51:27.34289481Z level=debug msg="Applying default URL parsing for this data source type" type=alexanderzobnin-zabbix-datasource url=http://10.196.0.30/api_jsonrpc.php logger=sqlstore.session t=2024-03-11T15:51:27.352313027Z level=debug msg="reusing existing session" transaction=true logger=sqlstore.session t=2024-03-11T15:51:27.352878311Z level=debug msg="reusing existing session" transaction=true logger=sqlstore.session t=2024-03-11T15:51:27.353319822Z level=debug msg="reusing existing session" transaction=true logger=secrets.kvstore t=2024-03-11T15:51:27.366733252Z level=debug msg="got secret value" orgId=1 type=datasource namespace="Zabbix JPU" logger=sqlstore.session t=2024-03-11T15:51:27.37322338Z level=debug msg="reusing existing session" transaction=true logger=sqlstore.session t=2024-03-11T15:51:27.379346466Z level=debug msg="reusing existing session" transaction=true logger=secrets.kvstore t=2024-03-11T15:51:27.37978515Z level=debug msg="secret value updated" orgId=1 type=datasource namespace="Zabbix JPU" logger=sqlstore.transactions t=2024-03-11T15:51:27.379841444Z level=debug msg="skip committing the transaction because it belongs to a session created in the outer scope" logger=sqlstore.transactions t=2024-03-11T15:51:27.379874311Z level=debug msg="skip committing the transaction because it belongs to a session created in the outer scope" logger=secrets.kvstore t=2024-03-11T15:51:27.388723371Z level=debug msg="got secret value from cache" orgId=1 type=datasource namespace="Zabbix JPU" logger=oauth_token.sync t=2024-03-11T15:51:27.464094996Z level=debug msg="Singleflight request for OAuth token sync" key=user:1 logger=oauthtoken t=2024-03-11T15:51:27.464217029Z level=debug msg="Expiration check has been cached, no need to refresh" userID=1 logger=token t=2024-03-11T15:51:27.46429154Z level=debug msg=FeatureEnabled feature=accesscontrol.enforcement enabled=false licenseStatus=NotFound hasLicense=false hasValidLicense=false products=[] logger=accesscontrol.service t=2024-03-11T15:51:27.464373253Z level=debug msg="Using cached permissions" key=rbac-permissions-1-user-1 logger=token t=2024-03-11T15:51:27.471932211Z level=debug msg=FeatureEnabled feature=dspermissions.enforcement enabled=false licenseStatus=NotFound hasLicense=false hasValidLicense=false products=[] logger=token t=2024-03-11T15:51:27.472203531Z level=debug msg=FeatureEnabled feature=accesscontrol enabled=false licenseStatus=NotFound hasLicense=false hasValidLicense=false products=[] logger=token t=2024-03-11T15:51:27.472241326Z level=debug msg=FeatureEnabled feature=accesscontrol.enforcement enabled=false licenseStatus=NotFound hasLicense=false hasValidLicense=false products=[] logger=token t=2024-03-11T15:51:27.472263904Z level=debug msg=FeatureEnabled feature=admin enabled=false licenseStatus=NotFound hasLicense=false hasValidLicense=false products=[] logger=token t=2024-03-11T15:51:27.472291993Z level=debug msg=FeatureEnabled feature=analytics enabled=false licenseStatus=NotFound hasLicense=false hasValidLicense=false products=[] logger=token t=2024-03-11T15:51:27.472311998Z level=debug msg=FeatureEnabled feature=analytics.writers enabled=false licenseStatus=NotFound hasLicense=false hasValidLicense=false products=[] logger=token t=2024-03-11T15:51:27.472332813Z level=debug msg=FeatureEnabled feature=auditing enabled=false licenseStatus=NotFound hasLicense=false hasValidLicense=false products=[] logger=token t=2024-03-11T15:51:27.472353097Z level=debug msg=FeatureEnabled feature=caching enabled=false licenseStatus=NotFound hasLicense=false hasValidLicense=false products=[] logger=token t=2024-03-11T15:51:27.472374038Z level=debug msg=FeatureEnabled feature=caching.api enabled=false licenseStatus=NotFound hasLicense=false hasValidLicense=false products=[] logger=token t=2024-03-11T15:51:27.472394202Z level=debug msg=FeatureEnabled feature=caching.queries enabled=false licenseStatus=NotFound hasLicense=false hasValidLicense=false products=[] logger=token t=2024-03-11T15:51:27.472416424Z level=debug msg=FeatureEnabled feature=caching.resources enabled=false licenseStatus=NotFound hasLicense=false hasValidLicense=false products=[] logger=token t=2024-03-11T15:51:27.472454408Z level=debug msg=FeatureEnabled feature=config.vault enabled=false licenseStatus=NotFound hasLicense=false hasValidLicense=false products=[] logger=token t=2024-03-11T15:51:27.472492932Z level=debug msg=FeatureEnabled feature=dspermissions enabled=false licenseStatus=NotFound hasLicense=false hasValidLicense=false products=[] logger=token t=2024-03-11T15:51:27.472525582Z level=debug msg=FeatureEnabled feature=dspermissions.enforcement enabled=false licenseStatus=NotFound hasLicense=false hasValidLicense=false products=[] logger=token t=2024-03-11T15:51:27.472551985Z level=debug msg=FeatureEnabled feature=encryption.aesgcm enabled=false licenseStatus=NotFound hasLicense=false hasValidLicense=false products=[] logger=token t=2024-03-11T15:51:27.472572083Z level=debug msg=FeatureEnabled feature=enterprise.plugins enabled=false licenseStatus=NotFound hasLicense=false hasValidLicense=false products=[] logger=token t=2024-03-11T15:51:27.472598194Z level=debug msg=FeatureEnabled feature=grafanacloud enabled=false licenseStatus=NotFound hasLicense=false hasValidLicense=false products=[] logger=token t=2024-03-11T15:51:27.472619106Z level=debug msg=FeatureEnabled feature=kms.encryption enabled=false licenseStatus=NotFound hasLicense=false hasValidLicense=false products=[] logger=token t=2024-03-11T15:51:27.472646186Z level=debug msg=FeatureEnabled feature=ldapdebug enabled=false licenseStatus=NotFound hasLicense=false hasValidLicense=false products=[] logger=token t=2024-03-11T15:51:27.472677862Z level=debug msg=FeatureEnabled feature=ldapsync enabled=false licenseStatus=NotFound hasLicense=false hasValidLicense=false products=[] logger=token t=2024-03-11T15:51:27.47270649Z level=debug msg=FeatureEnabled feature=publicDashboardsEmailSharing enabled=false licenseStatus=NotFound hasLicense=false hasValidLicense=false products=[] logger=token t=2024-03-11T15:51:27.472727118Z level=debug msg=FeatureEnabled feature=provisioning enabled=false licenseStatus=NotFound hasLicense=false hasValidLicense=false products=[] logger=token t=2024-03-11T15:51:27.472748303Z level=debug msg=FeatureEnabled feature=recordedqueries enabled=false licenseStatus=NotFound hasLicense=false hasValidLicense=false products=[] logger=token t=2024-03-11T15:51:27.472768476Z level=debug msg=FeatureEnabled feature=reports enabled=false licenseStatus=NotFound hasLicense=false hasValidLicense=false products=[] logger=token t=2024-03-11T15:51:27.472788312Z level=debug msg=FeatureEnabled feature=reports.creation enabled=false licenseStatus=NotFound hasLicense=false hasValidLicense=false products=[] logger=token t=2024-03-11T15:51:27.472816643Z level=debug msg=FeatureEnabled feature=reports.email enabled=false licenseStatus=NotFound hasLicense=false hasValidLicense=false products=[] logger=token t=2024-03-11T15:51:27.472836584Z level=debug msg=FeatureEnabled feature=reports.pdf enabled=false licenseStatus=NotFound hasLicense=false hasValidLicense=false products=[] logger=token t=2024-03-11T15:51:27.47285787Z level=debug msg=FeatureEnabled feature=saml enabled=false licenseStatus=NotFound hasLicense=false hasValidLicense=false products=[] logger=token t=2024-03-11T15:51:27.472878499Z level=debug msg=FeatureEnabled feature=teamsync enabled=false licenseStatus=NotFound hasLicense=false hasValidLicense=false products=[] logger=token t=2024-03-11T15:51:27.472899193Z level=debug msg=FeatureEnabled feature=teamgroupsync enabled=false licenseStatus=NotFound hasLicense=false hasValidLicense=false products=[] logger=token t=2024-03-11T15:51:27.472918951Z level=debug msg=FeatureEnabled feature=userlimits enabled=false licenseStatus=NotFound hasLicense=false hasValidLicense=false products=[] logger=token t=2024-03-11T15:51:27.472938767Z level=debug msg=FeatureEnabled feature=whitelabeling enabled=false licenseStatus=NotFound hasLicense=false hasValidLicense=false products=[] logger=oauth_token.sync t=2024-03-11T15:51:27.712217525Z level=debug msg="Singleflight request for OAuth token sync" key=user:1 logger=oauthtoken t=2024-03-11T15:51:27.712380776Z level=debug msg="Expiration check has been cached, no need to refresh" userID=1 logger=token t=2024-03-11T15:51:27.712468964Z level=debug msg=FeatureEnabled feature=accesscontrol.enforcement enabled=false licenseStatus=NotFound hasLicense=false hasValidLicense=false products=[] logger=accesscontrol.service t=2024-03-11T15:51:27.712548605Z level=debug msg="Using cached permissions" key=rbac-permissions-1-user-1 logger=accesscontrol.evaluator t=2024-03-11T15:51:27.713411715Z level=debug msg="Matched scope" userscope=datasources:* targetscope=datasources:uid:adfclv4p3fi0wc logger=secrets.kvstore t=2024-03-11T15:51:27.714988698Z level=debug msg="got secret value from cache" orgId=1 type=datasource namespace="Zabbix JPU" logger=oauth_token.sync t=2024-03-11T15:51:27.848994885Z level=debug msg="Singleflight request for OAuth token sync" key=user:1 logger=oauthtoken t=2024-03-11T15:51:27.849164267Z level=debug msg="Expiration check has been cached, no need to refresh" userID=1 logger=token t=2024-03-11T15:51:27.849233881Z level=debug msg=FeatureEnabled feature=accesscontrol.enforcement enabled=false licenseStatus=NotFound hasLicense=false hasValidLicense=false products=[] logger=accesscontrol.service t=2024-03-11T15:51:27.849320703Z level=debug msg="Using cached permissions" key=rbac-permissions-1-user-1 logger=datasources t=2024-03-11T15:51:27.849770021Z level=debug msg="Querying for data source via SQL store" id=69 orgId=1 logger=token t=2024-03-11T15:51:27.851211267Z level=debug msg=FeatureEnabled feature=dspermissions.enforcement enabled=false licenseStatus=NotFound hasLicense=false hasValidLicense=false products=[] logger=secrets.kvstore t=2024-03-11T15:51:27.851423637Z level=debug msg="got secret value from cache" orgId=1 type=datasource namespace="Zabbix JPU" logger=plugin.alexanderzobnin-zabbix-datasource t=2024-03-11T15:51:27.85727426Z level=debug msg="Initializing new data source instance" logger=plugin.alexanderzobnin-zabbix-datasource t=2024-03-11T15:51:27.85839512Z level=debug msg="Zabbix request" method=apiinfo.version logger=ngalert.scheduler t=2024-03-11T15:51:30.002444677Z level=debug msg="No changes detected. Skip updating" logger=ngalert.scheduler t=2024-03-11T15:51:30.00384527Z level=debug msg="Rule is ready to run on the current tick" uid=c69ced34-08c0-4da3-a684-e1014d2621dc tick=171017229 frequency=6 offset=3 logger=ngalert.scheduler t=2024-03-11T15:51:30.003946527Z level=debug msg="Rule is ready to run on the current tick" uid=a7ea555d-ce42-4f9b-a78c-97ef0a66c2da tick=171017229 frequency=6 offset=3 logger=ngalert.scheduler t=2024-03-11T15:51:30.004009096Z level=debug msg="Rule is ready to run on the current tick" uid=e5583529-fc13-474b-bc82-95a542ae4bd7 tick=171017229 frequency=6 offset=3 logger=ngalert.scheduler rule_uid=c69ced34-08c0-4da3-a684-e1014d2621dc org_id=1 t=2024-03-11T15:51:30.004233965Z level=debug msg="Skip rule evaluation because it is paused" logger=ngalert.scheduler rule_uid=a7ea555d-ce42-4f9b-a78c-97ef0a66c2da org_id=1 t=2024-03-11T15:51:33.338125058Z level=debug msg="Skip rule evaluation because it is paused" logger=ngalert.scheduler rule_uid=e5583529-fc13-474b-bc82-95a542ae4bd7 org_id=1 t=2024-03-11T15:51:36.67188291Z level=debug msg="Skip rule evaluation because it is paused" logger=plugin.alexanderzobnin-zabbix-datasource t=2024-03-11T15:51:37.860003347Z level=error msg="Error querying Zabbix version" error="Post \"http://10.196.0.30/api_jsonrpc.php\": dial tcp 10.196.0.30:80: i/o timeout" logger=plugin.alexanderzobnin-zabbix-datasource t=2024-03-11T15:51:37.860258708Z level=debug msg="Zabbix request" method=apiinfo.version logger=ngalert.scheduler t=2024-03-11T15:51:40.001695524Z level=debug msg="No changes detected. Skip updating" logger=ngalert.scheduler t=2024-03-11T15:51:40.00193569Z level=debug msg="Rule is ready to run on the current tick" uid=c9f9c701-8d7f-4d3e-bcf2-62dd2180d7c3 tick=171017230 frequency=6 offset=4 logger=ngalert.scheduler t=2024-03-11T15:51:40.002043158Z level=debug msg="Rule is ready to run on the current tick" uid=b2c64a40-6cea-4a31-bdfa-d6e453c7ebc3 tick=171017230 frequency=6 offset=4 logger=ngalert.scheduler t=2024-03-11T15:51:40.002167869Z level=debug msg="Rule is ready to run on the current tick" uid=d8927ddc-0fe9-43ea-a9af-1fe19a995cc4 tick=171017230 frequency=6 offset=4 logger=ngalert.scheduler t=2024-03-11T15:51:40.002390624Z level=debug msg="Rule is ready to run on the current tick" uid=b9235f18-fc1d-4f7b-837e-6475439ce16e tick=171017230 frequency=6 offset=4 logger=ngalert.scheduler t=2024-03-11T15:51:40.002606175Z level=debug msg="Rule is ready to run on the current tick" uid=fd700b07-1571-463e-8095-90091edc19f8 tick=171017230 frequency=6 offset=4 logger=ngalert.scheduler rule_uid=c9f9c701-8d7f-4d3e-bcf2-62dd2180d7c3 org_id=1 t=2024-03-11T15:51:40.002880585Z level=debug msg="Skip rule evaluation because it is paused" logger=ngalert.scheduler rule_uid=b2c64a40-6cea-4a31-bdfa-d6e453c7ebc3 org_id=1 t=2024-03-11T15:51:42.003075292Z level=debug msg="Skip rule evaluation because it is paused" logger=ngalert.scheduler rule_uid=d8927ddc-0fe9-43ea-a9af-1fe19a995cc4 org_id=1 t=2024-03-11T15:51:44.003437397Z level=debug msg="Skip rule evaluation because it is paused" logger=ngalert.scheduler rule_uid=b9235f18-fc1d-4f7b-837e-6475439ce16e org_id=1 t=2024-03-11T15:51:46.003100904Z level=debug msg="Skip rule evaluation because it is paused" logger=plugin.alexanderzobnin-zabbix-datasource t=2024-03-11T15:51:47.870070934Z level=error msg="Zabbix API request error" error="Post \"http://10.196.0.30/api_jsonrpc.php\": dial tcp 10.196.0.30:80: i/o timeout" logger=context userId=1 orgId=1 uname=admin t=2024-03-11T15:51:47.871838355Z level=error msg="Request Completed" method=POST path=/api/datasources/69/resources/zabbix-api status=500 remote_addr=192.168.71.27 time_ms=20024 duration=20.024331278s size=127 referer=http://10.192.254.17:3000/connections/datasources/edit/adfclv4p3fi0wc/ handler=/api/datasources/:id/resources/* status_source=downstream logger=ngalert.scheduler rule_uid=fd700b07-1571-463e-8095-90091edc19f8 org_id=1 t=2024-03-11T15:51:48.003703481Z level=debug msg="Skip rule evaluation because it is paused"

Software versions

Grafana	Zabbix	Grafana-Zabbix Plugin
10.4	5.4.4	4.4.6

Thanks for taking the time to read through this!

Rgds,

Matthew

[zoltanbedi]

Try to set up your datasource again if you know your credentials. A lot has changed since 3.12.4.

[Matt197922]

Hi, thanks for responding. But i found the issue. the data source was working via browser using a different IP address. When Grafana was updated it needed a routing to allow access to this Zabbix.

[zoltanbedi]

Thanks for letting us know @Matt197922