Deploy > Kubernetes > Logging

[sandeep-r1]

Hi Team,
We are getting error while deploying logging service. Please find the error below. Please help us here to resolve this issue.

TASK [logging : helm install] **************************************************
fatal: [localhost]: FAILED! => {
    "changed": true,
    "cmd": "helm upgrade --install --atomic --timeout 10m --force filebeat /var/lib/jenkins/workspace/Deploy/alex/dev/Kubernetes/Logging/kubernetes/helm_charts/logging/filebeat -n logging -f /tmp/filebeat.yaml",
    "delta": "0:10:00.602130",
    "end": "2023-06-01 14:10:12.332110",
    "rc": 1,
    "start": "2023-06-01 14:00:11.729980"
}

STDOUT:

Release "filebeat" does not exist. Installing it now.


STDERR:

Error: release filebeat failed, and has been uninstalled due to atomic being set: timed out waiting for the condition


MSG:

non-zero return code


PLAY RECAP *********************************************************************
localhost                  : ok=1    changed=0    unreachable=0    failed=1    skipped=1    rescued=0    ignored=0  

[santhosh-tg]

@sandeep-r1 Can you please check the pod logs while the deployment is happening and see why its failing ?

[srikanth-6520]

@santhosh-tg ouath2-proxy got success , but filebeat only getting fail , below we have provided pod logs of filebeat service.

2023-06-09T04:39:50.663Z INFO [kubernetes] kubernetes/util.go:99 kubernetes: Using node ip-172-16-1-171.ap-south-1.compute.internal provided in the config {"libbeat.processor": "add_kubernetes_metadata"}
2023-06-09T04:39:50.676Z INFO [publisher_pipeline_output] pipeline/output.go:143 Connecting to backoff(async(tcp://172.16.1.148:5044))
2023-06-09T04:39:50.676Z INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2023-06-09T04:39:50.676Z INFO [publisher] pipeline/retry.go:223 done
2023-06-09T04:39:52.442Z ERROR [publisher_pipeline_output] pipeline/output.go:154 Failed to connect to backoff(async(tcp://172.16.1.148:5044)): dial tcp 172.16.1.148:5044: connect: connection refused
2023-06-09T04:39:52.442Z INFO [publisher_pipeline_output] pipeline/output.go:145 Attempting to reconnect to backoff(async(tcp://172.16.1.148:5044)) with 1 reconnect attempt(s)
2023-06-09T04:39:52.442Z INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2023-06-09T04:39:52.442Z INFO [publisher] pipeline/retry.go:223 done
2023-06-09T04:39:55.769Z ERROR [publisher_pipeline_output] pipeline/output.go:154 Failed to connect to backoff(async(tcp://172.16.1.148:5044)): dial tcp 172.16.1.148:5044: connect: connection refused
2023-06-09T04:39:55.769Z INFO [publisher_pipeline_output] pipeline/output.go:145 Attempting to reconnect to backoff(async(tcp://172.16.1.148:5044)) with 2 reconnect attempt(s)
2023-06-09T04:39:55.769Z INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2023-06-09T04:39:55.769Z INFO [publisher] pipeline/retry.go:223 done
2023-06-09T04:40:03.335Z ERROR [publisher_pipeline_output] pipeline/output.go:154 Failed to connect to backoff(async(tcp://172.16.1.148:5044)): dial tcp 172.16.1.148:5044: connect: connection refused
2023-06-09T04:40:03.335Z INFO [publisher_pipeline_output] pipeline/output.go:145 Attempting to reconnect to backoff(async(tcp://172.16.1.148:5044)) with 3 reconnect attempt(s)
2023-06-09T04:40:03.335Z INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2023-06-09T04:40:03.335Z INFO [publisher] pipeline/retry.go:223 done
2023-06-09T04:40:15.358Z ERROR [publisher_pipeline_output] pipeline/output.go:154 Failed to connect to backoff(async(tcp://172.16.1.148:5044)): dial tcp 172.16.1.148:5044: connect: connection refused
2023-06-09T04:40:15.359Z INFO [publisher_pipeline_output] pipeline/output.go:145 Attempting to reconnect to backoff(async(tcp://172.16.1.148:5044)) with 4 reconnect attempt(s)
2023-06-09T04:40:15.359Z INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2023-06-09T04:40:15.359Z INFO [publisher] pipeline/retry.go:223 done
2023-06-09T04:40:20.642Z INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cgroup":{"cpu":{"cfs":{"period":{"us":100000},"quota":{"us":100000}},"id":"cri-containerd-d6b8a2d4ef30965df327465d9dd73bf5e4b5803540b1e4267fef1b3be597ea3c.scope","stats":{"periods":54,"throttled":{"ns":214114098,"periods":2}}},"cpuacct":{"id":"cri-containerd-d6b8a2d4ef30965df327465d9dd73bf5e4b5803540b1e4267fef1b3be597ea3c.scope","total":{"ns":596153322}},"memory":{"id":"cri-containerd-d6b8a2d4ef30965df327465d9dd73bf5e4b5803540b1e4267fef1b3be597ea3c.scope","mem":{"limit":{"bytes":1048576000},"usage":{"bytes":114929664}}}},"cpu":{"system":{"ticks":90,"time":{"ms":98}},"total":{"ticks":330,"time":{"ms":338},"value":330},"user":{"ticks":240,"time":{"ms":240}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":32},"info":{"ephemeral_id":"cda59350-c3a8-4315-9217-8b5d2d4b969e","uptime":{"ms":30250}},"memstats":{"gc_next":48718432,"memory_alloc":25481664,"memory_sys":75842560,"memory_total":80161880,"rss":91353088},"runtime":{"goroutines":169}},"filebeat":{"events":{"active":4145,"added":4155,"done":10},"harvester":{"open_files":21,"running":21,"started":21}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0},"type":"logstash"},"pipeline":{"clients":2,"events":{"active":4117,"filtered":24,"published":4116,"retry":8192,"total":4141}}},"registrar":{"states":{"current":10,"update":10},"writes":{"success":10,"total":10}},"system":{"cpu":{"cores":4},"load":{"1":0.12,"15":0.01,"5":0.08,"norm":{"1":0.03,"15":0.0025,"5":0.02}}}}}}
2023-06-09T04:40:42.379Z ERROR [publisher_pipeline_output] pipeline/output.go:154 Failed to connect to backoff(async(tcp://172.16.1.148:5044)): dial tcp 172.16.1.148:5044: connect: connection refused
2023-06-09T04:40:42.379Z INFO [publisher_pipeline_output] pipeline/output.go:145 Attempting to reconnect to backoff(async(tcp://172.16.1.148:5044)) with 5 reconnect attempt(s)
2023-06-09T04:40:42.380Z INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2023-06-09T04:40:42.380Z INFO [publisher] pipeline/retry.go:223 done

[santhosh-tg]

@sandeep-r1 did you manage to move past this error or still facing the issue ?

[sandeep-r1]

@santhosh-tg we are still facing this issue. Kindly need your help.