apimanager crashing

[dikshauat]

Hi ,

For resolving a RESTAPI related issue , I deleted the apimanager pod and re-run the apimanager deployment job . But now the pod is crashing . Following is the error I am getting on kubectl.

Kindly help me fixing it

[beepdot]

Please share apimanager pod logs.

[dikshauat]

Here is the required information

Started by user Diksha POC
Checking out git https://github.com/project-sunbird/sunbird-devops.git into /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/APIManager@script/fdfb149ced34077fca57da60c0df0dd05ff3d2c9d26fff4d9502ef43db314dd2 to read kubernetes/pipelines/deploy_core/Jenkinsfile
The recommended git tool is: NONE
No credentials specified

git rev-parse --resolve-git-dir /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/APIManager@script/fdfb149ced34077fca57da60c0df0dd05ff3d2c9d26fff4d9502ef43db314dd2/.git # timeout=10
Fetching changes from the remote Git repository
git config remote.origin.url https://github.com/project-sunbird/sunbird-devops.git # timeout=10
Using shallow fetch with depth 1
Fetching upstream changes from https://github.com/project-sunbird/sunbird-devops.git
git --version # timeout=10
git --version # 'git version 2.17.1'
git fetch --tags --progress --depth=1 -- https://github.com/project-sunbird/sunbird-devops.git +refs/heads/:refs/remotes/origin/ # timeout=10
git rev-parse origin/release-4.6.0_RC4^{commit} # timeout=10
git rev-parse release-4.6.0_RC4^{commit} # timeout=10
Checking out Revision 298c0e0cb57f5d8fa32bbc3676ae57c6c2127a6f (release-4.6.0_RC4)
git config core.sparsecheckout # timeout=10
git checkout -f 298c0e0cb57f5d8fa32bbc3676ae57c6c2127a6f # timeout=10
Commit message: "Issue #SB-28497 feat:updated enrollment list size env (#3180)"
git rev-list --no-walk 298c0e0cb57f5d8fa32bbc3676ae57c6c2127a6f # timeout=10
Loading library deploy-conf@heads/shared-lib
Attempting to resolve heads/shared-lib from remote references...
git --version # timeout=10
git --version # 'git version 2.17.1'
git ls-remote -- https://github.com/project-sunbird/sunbird-devops.git # timeout=10
The recommended git tool is: NONE
No credentials specified
git rev-parse --resolve-git-dir /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/APIManager@libs/4d80a6c970e5bba7406a7e32771e5156e33649aa2faaf628a0193bf413a1701e/.git # timeout=10
Fetching changes from the remote Git repository
git config remote.origin.url https://github.com/project-sunbird/sunbird-devops.git # timeout=10
Using shallow fetch with depth 1
Fetching upstream changes from https://github.com/project-sunbird/sunbird-devops.git
git --version # timeout=10
git --version # 'git version 2.17.1'
git fetch --no-tags --progress --depth=1 -- https://github.com/project-sunbird/sunbird-devops.git +refs/heads/:refs/remotes/origin/ refs/heads/shared-lib # timeout=10
Checking out Revision 778da7ce7f97efd003eed98071921a837d209343 (778da7ce7f97efd003eed98071921a837d209343)
git config core.sparsecheckout # timeout=10
git checkout -f 778da7ce7f97efd003eed98071921a837d209343 # timeout=10
Commit message: "fix: move variables to be accessible across both try blocks"
git rev-list --no-walk 778da7ce7f97efd003eed98071921a837d209343 # timeout=10
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/APIManager
[Pipeline] {
[Pipeline] stage
[Pipeline] { (checkout public repo)
[Pipeline] echo
Found .git folder. Clearing it..
[Pipeline] sh

• git clean -fxd
  Removing metadata.json
  Skipping repository private/
  Removing private@tmp/
  [Pipeline] checkout
  The recommended git tool is: NONE
  No credentials specified

git rev-parse --resolve-git-dir /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/APIManager/.git # timeout=10
Fetching changes from the remote Git repository
git config remote.origin.url https://github.com/project-sunbird/sunbird-devops.git # timeout=10
Using shallow fetch with depth 1
Fetching upstream changes from https://github.com/project-sunbird/sunbird-devops.git
git --version # timeout=10
git --version # 'git version 2.17.1'
git fetch --tags --progress --depth=1 -- https://github.com/project-sunbird/sunbird-devops.git +refs/heads/:refs/remotes/origin/ # timeout=10
git rev-parse origin/release-4.6.0_RC4^{commit} # timeout=10
git rev-parse release-4.6.0_RC4^{commit} # timeout=10
Checking out Revision 298c0e0cb57f5d8fa32bbc3676ae57c6c2127a6f (release-4.6.0_RC4)
git config core.sparsecheckout # timeout=10
git checkout -f 298c0e0cb57f5d8fa32bbc3676ae57c6c2127a6f # timeout=10
Commit message: "Issue #SB-28497 feat:updated enrollment list size env (#3180)"
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (deploy)
[Pipeline] ansiColor
[Pipeline] {

[Pipeline] sh

• echo Deploy/dev/Kubernetes/APIManager
  [Pipeline] sh
• echo Deploy/dev/Kubernetes/APIManager
  [Pipeline] sh
• echo Deploy/dev/Kubernetes/APIManager
  [Pipeline] stage
  [Pipeline] { (parameter checks)
  [Pipeline] echo
  Found environment variable named hub_org with value as: dikshauat
  [Pipeline] copyArtifacts
  Copied 1 artifact from "ArtifactUpload » dev » Core » APIManager" build number 6
  [Pipeline] sh
• jq -r .image_name metadata.json
  [Pipeline] sh
• jq -r .node_name metadata.json
  [Pipeline] echo
  image_tag not specified, using the image_tag specified in metadata.json.
  [Pipeline] sh
• jq -r .image_tag metadata.json
  [Pipeline] }
  [Pipeline] // stage
  [Pipeline] }

[Pipeline] // ansiColor
[Pipeline] sh

• echo Deploy/dev/Kubernetes/APIManager
  [Pipeline] sh
• pwd
  [Pipeline] ansiColor
  [Pipeline] {

[Pipeline] stage
[Pipeline] { (ansible-run)
[Pipeline] echo
{env=dev, module=Kubernetes, jobName=APIManager, absolute_job_path=ArtifactUpload/dev/Core/APIManager, agent=built-in, image_name=kong, image_tag=0.14.1, currentWs=/var/lib/jenkins/workspace/Deploy/dev/Kubernetes/APIManager, ansiblePlaybook=/var/lib/jenkins/workspace/Deploy/dev/Kubernetes/APIManager/kubernetes/ansible/deploy_core_service.yml, ansibleExtraArgs=--extra-vars "chart_path=/var/lib/jenkins/workspace/Deploy/dev/Kubernetes/APIManager/kubernetes/helm_charts/core/apimanager release_name=apimanager image_tag=0.14.1 image_name=kong role_name=sunbird-deploy" --vault-password-file /var/lib/jenkins/secrets/vault-pass}
[Pipeline] dir
Running in /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/APIManager/private
[Pipeline] {
[Pipeline] echo
Info: Branch override is disabled
[Pipeline] checkout
The recommended git tool is: NONE
using credential github-cred

git rev-parse --resolve-git-dir /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/APIManager/private/.git # timeout=10
Fetching changes from the remote Git repository
git config remote.origin.url https://github.com/dikshauat/Diksh-POC.git # timeout=10
Using shallow fetch with depth 1
Fetching upstream changes from https://github.com/dikshauat/Diksh-POC.git
git --version # timeout=10
git --version # 'git version 2.17.1'
using GIT_ASKPASS to set credentials
git fetch --no-tags --progress --depth=1 -- https://github.com/dikshauat/Diksh-POC.git +refs/heads/:refs/remotes/origin/ # timeout=10
git rev-parse origin/release-4.3.0^{commit} # timeout=10
Checking out Revision c9d6aed9966398f4b3105b42d8b5733969f54243 (origin/release-4.3.0)
git config core.sparsecheckout # timeout=10
git checkout -f c9d6aed9966398f4b3105b42d8b5733969f54243 # timeout=10
Commit message: "Update secrets.yml"
git rev-list --no-walk c9d6aed9966398f4b3105b42d8b5733969f54243 # timeout=10
[Pipeline] }
[Pipeline] // dir
[Pipeline] sh

• rsync -Lkr /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/APIManager/private/ansible/inventory/dev/Kubernetes/common.yml /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/APIManager/private/ansible/inventory/dev/Kubernetes/hosts /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/APIManager/private/ansible/inventory/dev/Kubernetes/secrets.yml /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/APIManager/ansible/inventory/env/
• [ -f /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/APIManager/ansible/inventory/env/kubernetes.yaml ]
• ansible-playbook -i /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/APIManager/ansible/inventory/env/hosts /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/APIManager/kubernetes/ansible/deploy_core_service.yml --extra-vars chart_path=/var/lib/jenkins/workspace/Deploy/dev/Kubernetes/APIManager/kubernetes/helm_charts/core/apimanager release_name=apimanager image_tag=0.14.1 image_name=kong role_name=sunbird-deploy --vault-password-file /var/lib/jenkins/secrets/vault-pass
  [WARNING]: Invalid characters were found in group names but not replaced, use
  -vvvv to see details

[WARNING]: * Failed to parse /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/
APIManager/ansible/inventory/env/hosts with yaml plugin: Syntax Error while
loading YAML. found unexpected ':' The error appears to be in '/var/lib/jenk
ins/workspace/Deploy/dev/Kubernetes/APIManager/ansible/inventory/env/hosts':
line 1, column 5, but may be elsewhere in the file depending on the exact
syntax problem. The offending line appears to be: [all:vars] ^ here

[WARNING]: * Failed to parse /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/
APIManager/ansible/inventory/env/hosts with ini plugin: /var/lib/jenkins/worksp
ace/Deploy/dev/Kubernetes/APIManager/ansible/inventory/env/hosts:145: Section
[core:children] includes undefined group: postgresql-slave

[WARNING]: Unable to parse /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/API
Manager/ansible/inventory/env/hosts as an inventory source

[WARNING]: No inventory was parsed, only implicit localhost is available

[WARNING]: provided hosts list is empty, only localhost is available. Note that
the implicit localhost does not match 'all'

[WARNING]: While constructing a mapping from /var/lib/jenkins/workspace/Deploy/
dev/Kubernetes/APIManager/kubernetes/ansible/roles/sunbird-
deploy/defaults/main.yml, line 3, column 1, found a duplicate dict key
(sunbird_time_zone). Using last defined value only.

[WARNING]: While constructing a mapping from /var/lib/jenkins/workspace/Deploy/
dev/Kubernetes/APIManager/kubernetes/ansible/roles/sunbird-
deploy/defaults/main.yml, line 3, column 1, found a duplicate dict key
(nginx_per_ip_connection_limit). Using last defined value only.

[WARNING]: While constructing a mapping from /var/lib/jenkins/workspace/Deploy/
dev/Kubernetes/APIManager/kubernetes/ansible/roles/sunbird-
deploy/defaults/main.yml, line 3, column 1, found a duplicate dict key
(sunbird_portal_player_cdn_enabled). Using last defined value only.

[WARNING]: While constructing a mapping from /var/lib/jenkins/workspace/Deploy/
dev/Kubernetes/APIManager/kubernetes/ansible/roles/sunbird-
deploy/defaults/main.yml, line 3, column 1, found a duplicate dict key
(dial_liveness_readiness). Using last defined value only.

[WARNING]: While constructing a mapping from /var/lib/jenkins/workspace/Deploy/
dev/Kubernetes/APIManager/kubernetes/ansible/roles/sunbird-
deploy/defaults/main.yml, line 928, column 5, found a duplicate dict key
(failureThreshold). Using last defined value only.

[WARNING]: While constructing a mapping from /var/lib/jenkins/workspace/Deploy/
dev/Kubernetes/APIManager/kubernetes/ansible/roles/sunbird-
deploy/defaults/main.yml, line 938, column 5, found a duplicate dict key
(failureThreshold). Using last defined value only.

[WARNING]: While constructing a mapping from /var/lib/jenkins/workspace/Deploy/
dev/Kubernetes/APIManager/kubernetes/ansible/roles/sunbird-
deploy/defaults/main.yml, line 951, column 5, found a duplicate dict key
(failureThreshold). Using last defined value only.

[WARNING]: While constructing a mapping from /var/lib/jenkins/workspace/Deploy/
dev/Kubernetes/APIManager/kubernetes/ansible/roles/sunbird-
deploy/defaults/main.yml, line 961, column 5, found a duplicate dict key
(failureThreshold). Using last defined value only.

PLAY [local] *******************************************************************
[WARNING]: While constructing a mapping from /var/lib/jenkins/workspace/Deploy/
dev/Kubernetes/APIManager/ansible/inventory/env/group_vars/all.yml, line 4,
column 1, found a duplicate dict key (app_es_snapshot_host). Using last defined
value only.

[WARNING]: While constructing a mapping from /var/lib/jenkins/workspace/Deploy/
dev/Kubernetes/APIManager/ansible/inventory/env/group_vars/all.yml, line 4,
column 1, found a duplicate dict key (sunbird_es_host). Using last defined
value only.

[WARNING]: While constructing a mapping from /var/lib/jenkins/workspace/Deploy/
dev/Kubernetes/APIManager/ansible/inventory/env/group_vars/all.yml, line 4,
column 1, found a duplicate dict key (log_es_snapshot_host). Using last defined
value only.

[WARNING]: While constructing a mapping from /var/lib/jenkins/workspace/Deploy/
dev/Kubernetes/APIManager/ansible/inventory/env/secrets.yml, line 3, column 1,
found a duplicate dict key (learning_content_drive_apiKey). Using last defined
value only.

TASK [check for kong db migration] *********************************************

TASK [kong-migration : check if tables are existing in kong postgres database] ***
[DEPRECATION WARNING]: Distribution Ubuntu 18.04 on host localhost should use
/usr/bin/python3, but is using /usr/bin/python for backward compatibility with
prior Ansible releases. A future Ansible release will default to using the
discovered platform python for this host. See https://docs.ansible.com/ansible/
2.8/reference_appendices/interpreter_discovery.html for more information. This
feature will be removed in version 2.12. Deprecation warnings can be disabled
by setting deprecation_warnings=False in ansible.cfg.
changed: [localhost]

TASK [kong-migration : run kong migrations for first run if tables are empty] ***
skipping: [localhost]

TASK [kong-migration : fail when migration failed] *****************************
skipping: [localhost]

TASK [sunbird-deploy : template values.yaml file] ******************************
changed: [localhost]

TASK [sunbird-deploy : copy and template env file from swarm role] *************
changed: [localhost] => (item=../../../../ansible/roles/stack-api-manager/templates/api-manager.env)

TASK [sunbird-deploy : create env configmap] ***********************************
changed: [localhost]

TASK [sunbird-deploy : create aplication configmap] ****************************
changed: [localhost]

TASK [sunbird-deploy : create xml configmap] ***********************************
changed: [localhost]

TASK [Load role to decrypt private keys, copy to helm chart and encrypt private keys] ***
skipping: [localhost] => (item=[{'key_to_register': 'access', 'values_to_pass': {'basepath': '/keys/', 'keyprefix': 'access', 'keystart': 0, 'keycount': '-1'}}, 'decrypt.yml'])
skipping: [localhost] => (item=[{'key_to_register': 'access', 'values_to_pass': {'basepath': '/keys/', 'keyprefix': 'access', 'keystart': 0, 'keycount': '-1'}}, 'generate-keys.yml'])
skipping: [localhost] => (item=[{'key_to_register': 'access', 'values_to_pass': {'basepath': '/keys/', 'keyprefix': 'access', 'keystart': 0, 'keycount': '-1'}}, 'copy-to-helm-public.yml'])

TASK [sunbird-deploy : Create the token pubic key file] ************************
skipping: [localhost]

TASK [sunbird-deploy : Replace ansible variables with values] ******************
skipping: [localhost] => (item=main.rego)
skipping: [localhost] => (item=common.rego)

TASK [sunbird-deploy : Copy the common opa policies to release folder] *********
skipping: [localhost] => (item=main.rego)
skipping: [localhost] => (item=common.rego)

TASK [sunbird-deploy : Create OPA bundle] **************************************
skipping: [localhost]

TASK [sunbird-deploy : Copy the OPA bundle to helm chart directory] ************
skipping: [localhost]

TASK [sunbird-deploy : helm upgrade] *******************************************
fatal: [localhost]: FAILED! => {
"changed": true,
"cmd": "helm upgrade --install --atomic --timeout 10m apimanager /var/lib/jenkins/workspace/Deploy/dev/Kubernetes/APIManager/kubernetes/helm_charts/core/apimanager -n dev",
"delta": "0:10:01.503537",
"end": "2022-10-07 10:03:05.579723",
"rc": 1,
"start": "2022-10-07 09:53:04.076186"
}

STDOUT:

Release "apimanager" does not exist. Installing it now.

STDERR:

Error: release apimanager failed, and has been uninstalled due to atomic being set: timed out waiting for the condition

MSG:

non-zero return code

PLAY RECAP *********************************************************************
localhost : ok=6 changed=6 unreachable=0 failed=1 skipped=8 rescued=0 ignored=0

[Pipeline] }
[Pipeline] // stage
[Pipeline] }

[Pipeline] // ansiColor
[Pipeline] }
[Pipeline] // stage
[Pipeline] ansiColor
[Pipeline] {

[Pipeline] stage
[Pipeline] { (slack_notify)
[Pipeline] sh

• git rev-parse --short HEAD
  [Pipeline] sh
• rev
• cut -d / -f1
• rev
• git name-rev --name-only HEAD
  [Pipeline] sh
• echo Deploy/dev/Kubernetes/APIManager
  [Pipeline] echo
  Could not find env specific Slack channel. Check for global slack channel..
  [Pipeline] echo
  Could not find slack environment variable. Skipping slack notification..
  [Pipeline] }
  [Pipeline] // stage
  [Pipeline] }

[Pipeline] // ansiColor
[Pipeline] ansiColor
[Pipeline] {

[Pipeline] stage
[Pipeline] { (email_notify)
[Pipeline] sh

• echo Deploy/dev/Kubernetes/APIManager
  [Pipeline] echo
  Could not find env specific email group. Check for global email group..
  [Pipeline] echo
  Could not find global email group variable. Skipping email notification..
  [Pipeline] }
  [Pipeline] // stage
  [Pipeline] }

[Pipeline] // ansiColor
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: script returned exit code 2
Finished: FAILURE

[beepdot]

Please drop the api manager schema from postgres and recreate an empty schema.
After that please rerun the deployment job.
In the deployment job, verify that this particular task is not skipped.

TASK [kong-migration : run kong migrations for first run if tables are empty] ***
skipping: [localhost]

Drop and recreate the database by running below commands in postgres (replace <database name> with your api manager database name)

DROP DATABASE <database name>;
CREATE DATABASE <database name>;

[dikshauat]

Hi @keshavprasadms , first of all thanks for all your support. I really appreciate it.

Need to confirm that if we delete the api_manager_dev database from postgres and recreate it using the below commands
DROP DATABASE api_manager_dev;
CREATE DATABASE api_manager_dev;

will it affect the present setup i.e the kubectl pods, sunbird' portal that we are getting and all that stuff. Also, please check the snapshot below and conform if we have marked the correct database or not

[beepdot]

Yes please go ahead and drop and recreate the api_manager_dev schema. There will be no impact to other services.

[dikshauat]

Thank you so much @keshavprasadms . It worked . Apimanager is up now.

TASK [kong-migration : run kong migrations for first run if tables are empty] ***
changed: [localhost]

Marking it as answered