Deploy/dev/Kubernetes/Logging/

[dikshauat]

@keshavprasadms,

While Running the logging job getting the following error.

[dikshauat]

Started by user Diksha POC
Checking out git https://github.com/project-sunbird/sunbird-devops.git into /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Logging@script/fdfb149ced34077fca57da60c0df0dd05ff3d2c9d26fff4d9502ef43db314dd2 to read kubernetes/pipelines/logging/Jenkinsfile
The recommended git tool is: NONE
No credentials specified

git rev-parse --resolve-git-dir /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Logging@script/fdfb149ced34077fca57da60c0df0dd05ff3d2c9d26fff4d9502ef43db314dd2/.git # timeout=10
Fetching changes from the remote Git repository
git config remote.origin.url https://github.com/project-sunbird/sunbird-devops.git # timeout=10
Using shallow fetch with depth 1
Fetching upstream changes from https://github.com/project-sunbird/sunbird-devops.git
git --version # timeout=10
git --version # 'git version 2.17.1'
git fetch --tags --progress --depth=1 -- https://github.com/project-sunbird/sunbird-devops.git +refs/heads/:refs/remotes/origin/ # timeout=10
git rev-parse origin/release-4.6.0_RC4^{commit} # timeout=10
git rev-parse release-4.6.0_RC4^{commit} # timeout=10
Checking out Revision 298c0e0cb57f5d8fa32bbc3676ae57c6c2127a6f (release-4.6.0_RC4)
git config core.sparsecheckout # timeout=10
git checkout -f 298c0e0cb57f5d8fa32bbc3676ae57c6c2127a6f # timeout=10
Commit message: "Issue #SB-28497 feat:updated enrollment list size env (#3180)"
First time build. Skipping changelog.
Loading library deploy-conf@heads/shared-lib
Attempting to resolve heads/shared-lib from remote references...
git --version # timeout=10
git --version # 'git version 2.17.1'
git ls-remote -- https://github.com/project-sunbird/sunbird-devops.git # timeout=10
The recommended git tool is: NONE
No credentials specified
git rev-parse --resolve-git-dir /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Logging@libs/4d80a6c970e5bba7406a7e32771e5156e33649aa2faaf628a0193bf413a1701e/.git # timeout=10
Fetching changes from the remote Git repository
git config remote.origin.url https://github.com/project-sunbird/sunbird-devops.git # timeout=10
Using shallow fetch with depth 1
Fetching upstream changes from https://github.com/project-sunbird/sunbird-devops.git
git --version # timeout=10
git --version # 'git version 2.17.1'
git fetch --no-tags --progress --depth=1 -- https://github.com/project-sunbird/sunbird-devops.git +refs/heads/:refs/remotes/origin/ refs/heads/shared-lib # timeout=10
Checking out Revision 778da7ce7f97efd003eed98071921a837d209343 (778da7ce7f97efd003eed98071921a837d209343)
git config core.sparsecheckout # timeout=10
git checkout -f 778da7ce7f97efd003eed98071921a837d209343 # timeout=10
Commit message: "fix: move variables to be accessible across both try blocks"
git rev-list --no-walk 778da7ce7f97efd003eed98071921a837d209343 # timeout=10
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Logging
[Pipeline] {
[Pipeline] stage
[Pipeline] { (checkout public repo)
[Pipeline] echo
Found .git folder. Clearing it..
[Pipeline] sh

• git clean -fxd
  Removing ansible/inventory/env/hosts
  Removing ansible/inventory/env/secrets.yml
  Skipping repository private/
  Removing private@tmp/
  [Pipeline] checkout
  The recommended git tool is: NONE
  No credentials specified

git rev-parse --resolve-git-dir /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Logging/.git # timeout=10
Fetching changes from the remote Git repository
git config remote.origin.url https://github.com/project-sunbird/sunbird-devops.git # timeout=10
Using shallow fetch with depth 1
Fetching upstream changes from https://github.com/project-sunbird/sunbird-devops.git
git --version # timeout=10
git --version # 'git version 2.17.1'
git fetch --tags --progress --depth=1 -- https://github.com/project-sunbird/sunbird-devops.git +refs/heads/:refs/remotes/origin/ # timeout=10
git rev-parse origin/release-4.6.0_RC4^{commit} # timeout=10
git rev-parse release-4.6.0_RC4^{commit} # timeout=10
Checking out Revision 298c0e0cb57f5d8fa32bbc3676ae57c6c2127a6f (release-4.6.0_RC4)
git config core.sparsecheckout # timeout=10
git checkout -f 298c0e0cb57f5d8fa32bbc3676ae57c6c2127a6f # timeout=10
Commit message: "Issue #SB-28497 feat:updated enrollment list size env (#3180)"
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (deploy)
[Pipeline] sh

• pwd
  [Pipeline] sh
• echo Deploy/dev/Kubernetes/Logging
  [Pipeline] sh
• echo Deploy/dev/Kubernetes/Logging
  [Pipeline] sh
• echo Deploy/dev/Kubernetes/Logging
  [Pipeline] ansiColor
  [Pipeline] {

[Pipeline] stage
[Pipeline] { (ansible-run)
[Pipeline] echo
{currentWs=/var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Logging, env=dev, module=Kubernetes, jobName=Logging, ansiblePlaybook=/var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Logging/kubernetes/ansible/deploy_core_service.yml, ansibleExtraArgs=--extra-vars "chart_path=/var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Logging/kubernetes/helm_charts/logging/fluent-bit release_name=fluent-bit role_name=logging" --vault-password-file /var/lib/jenkins/secrets/vault-pass -v}
[Pipeline] dir
Running in /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Logging/private
[Pipeline] {
[Pipeline] echo
Info: Branch override is disabled
[Pipeline] checkout
The recommended git tool is: NONE
using credential github-cred

git rev-parse --resolve-git-dir /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Logging/private/.git # timeout=10
Fetching changes from the remote Git repository
git config remote.origin.url https://github.com/dikshauat/Diksh-POC.git # timeout=10
Using shallow fetch with depth 1
Fetching upstream changes from https://github.com/dikshauat/Diksh-POC.git
git --version # timeout=10
git --version # 'git version 2.17.1'
using GIT_ASKPASS to set credentials
git fetch --no-tags --progress --depth=1 -- https://github.com/dikshauat/Diksh-POC.git +refs/heads/:refs/remotes/origin/ # timeout=10
git rev-parse origin/release-4.3.0^{commit} # timeout=10
Checking out Revision c9d6aed9966398f4b3105b42d8b5733969f54243 (origin/release-4.3.0)
git config core.sparsecheckout # timeout=10
git checkout -f c9d6aed9966398f4b3105b42d8b5733969f54243 # timeout=10
Commit message: "Update secrets.yml"
git rev-list --no-walk c9d6aed9966398f4b3105b42d8b5733969f54243 # timeout=10
[Pipeline] }
[Pipeline] // dir
[Pipeline] sh

• rsync -Lkr /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Logging/private/ansible/inventory/dev/Kubernetes/common.yml /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Logging/private/ansible/inventory/dev/Kubernetes/hosts /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Logging/private/ansible/inventory/dev/Kubernetes/secrets.yml /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Logging/ansible/inventory/env/
• [ -f /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Logging/ansible/inventory/env/kubernetes.yaml ]
• ansible-playbook -i /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Logging/ansible/inventory/env/hosts /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Logging/kubernetes/ansible/deploy_core_service.yml --extra-vars chart_path=/var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Logging/kubernetes/helm_charts/logging/fluent-bit release_name=fluent-bit role_name=logging --vault-password-file /var/lib/jenkins/secrets/vault-pass -v
  No config file found; using defaults
  [WARNING]: Invalid characters were found in group names but not replaced, use
  -vvvv to see details

[WARNING]: * Failed to parse /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/
Logging/ansible/inventory/env/hosts with yaml plugin: Syntax Error while
loading YAML. found unexpected ':' The error appears to be in '/var/lib/jenk
ins/workspace/Deploy/dev/Kubernetes/Logging/ansible/inventory/env/hosts': line
1, column 5, but may be elsewhere in the file depending on the exact syntax
problem. The offending line appears to be: [all:vars] ^ here

[WARNING]: * Failed to parse /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/
Logging/ansible/inventory/env/hosts with ini plugin: /var/lib/jenkins/workspace
/Deploy/dev/Kubernetes/Logging/ansible/inventory/env/hosts:145: Section
[core:children] includes undefined group: postgresql-slave

[WARNING]: Unable to parse /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Log
ging/ansible/inventory/env/hosts as an inventory source

[WARNING]: No inventory was parsed, only implicit localhost is available

[WARNING]: provided hosts list is empty, only localhost is available. Note that
the implicit localhost does not match 'all'

PLAY [local] *******************************************************************
[WARNING]: While constructing a mapping from /var/lib/jenkins/workspace/Deploy/
dev/Kubernetes/Logging/ansible/inventory/env/group_vars/all.yml, line 4, column
1, found a duplicate dict key (app_es_snapshot_host). Using last defined value
only.

[WARNING]: While constructing a mapping from /var/lib/jenkins/workspace/Deploy/
dev/Kubernetes/Logging/ansible/inventory/env/group_vars/all.yml, line 4, column
1, found a duplicate dict key (sunbird_es_host). Using last defined value only.

[WARNING]: While constructing a mapping from /var/lib/jenkins/workspace/Deploy/
dev/Kubernetes/Logging/ansible/inventory/env/group_vars/all.yml, line 4, column
1, found a duplicate dict key (log_es_snapshot_host). Using last defined value
only.

[WARNING]: While constructing a mapping from /var/lib/jenkins/workspace/Deploy/
dev/Kubernetes/Logging/ansible/inventory/env/secrets.yml, line 3, column 1,
found a duplicate dict key (learning_content_drive_apiKey). Using last defined
value only.

TASK [check for kong db migration] *********************************************
skipping: [localhost] => {
"changed": false,
"skip_reason": "Conditional result was False"
}

TASK [logging : tempating variables] *******************************************
[DEPRECATION WARNING]: Distribution Ubuntu 18.04 on host localhost should use
/usr/bin/python3, but is using /usr/bin/python for backward compatibility with
prior Ansible releases. A future Ansible release will default to using the
discovered platform python for this host. See https://docs.ansible.com/ansible/
2.8/reference_appendices/interpreter_discovery.html for more information. This
feature will be removed in version 2.12. Deprecation warnings can be disabled
by setting deprecation_warnings=False in ansible.cfg.
changed: [localhost] => {
"ansible_facts": {
"discovered_interpreter_python": "/usr/bin/python"
},
"changed": true,
"checksum": "f27a55fa9693fd5d83de3eb528990e20cbb04fac",
"dest": "/tmp/fluent-bit.yaml",
"gid": 116,
"group": "jenkins",
"md5sum": "a9147a43d90faa4e41a20894b1e39983",
"mode": "0644",
"owner": "jenkins",
"size": 44,
"src": "/var/lib/jenkins/.ansible/tmp/ansible-tmp-1664264032.237682-1000-189207170132509/source",
"state": "file",
"uid": 112
}

TASK [logging : helm install] **************************************************
fatal: [localhost]: FAILED! => {
"changed": true,
"cmd": "helm upgrade --install --atomic --timeout 10m --force fluent-bit /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/Logging/kubernetes/helm_charts/logging/fluent-bit -n logging -f /tmp/fluent-bit.yaml",
"delta": "0:10:02.037563",
"end": "2022-09-27 13:13:54.938944",
"rc": 1,
"start": "2022-09-27 13:03:52.901381"
}

STDOUT:

Release "fluent-bit" does not exist. Installing it now.

STDERR:

Error: release fluent-bit failed, and has been uninstalled due to atomic being set: timed out waiting for the condition

MSG:

non-zero return code

PLAY RECAP *********************************************************************
localhost : ok=1 changed=1 unreachable=0 failed=1 skipped=1 rescued=0 ignored=0

[Pipeline] }
[Pipeline] // stage
[Pipeline] }

[Pipeline] // ansiColor
[Pipeline] }
[Pipeline] // stage
[Pipeline] ansiColor
[Pipeline] {

[Pipeline] stage
[Pipeline] { (slack_notify)
[Pipeline] sh

• git rev-parse --short HEAD
  [Pipeline] sh
• rev
• cut -d / -f1
• rev
• git name-rev --name-only HEAD
  [Pipeline] sh
• echo Deploy/dev/Kubernetes/Logging
  [Pipeline] echo
  Could not find env specific Slack channel. Check for global slack channel..
  [Pipeline] echo
  Could not find slack environment variable. Skipping slack notification..
  [Pipeline] }
  [Pipeline] // stage
  [Pipeline] }

[Pipeline] // ansiColor
[Pipeline] ansiColor
[Pipeline] {

[Pipeline] stage
[Pipeline] { (email_notify)
[Pipeline] sh

• echo Deploy/dev/Kubernetes/Logging
  [Pipeline] echo
  Could not find env specific email group. Check for global email group..
  [Pipeline] echo
  Could not find global email group variable. Skipping email notification..
  [Pipeline] }
  [Pipeline] // stage
  [Pipeline] }

[Pipeline] // ansiColor
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: script returned exit code 2
Finished: FAILURE

[beepdot]

When the deployment is ongoing, please check the fluent-bit pod logs in logging namespace and see what is causing the pods to crash. You will be able easily figure out the error.