Pods "airflow-scheduler", "statsd" and "worker" stuck in "Init:CrashLoopBackOff" - Helm install

[ChaitDevOps]

Hello - I'm installing airflow via helm on my EKS cluster in AWS. I have gitsync enabled and when i install via helm, 3 pods are erroring out. I'm unable to decipher the error. Can someone please help me out?

values.yaml

# Set the environment variable for the DAG environment
airflow:
  config:
    airflow__core__extra_pip_packages: "logger"
  env:
    dag_environment: dev


# Configure PostgreSQL database
postgresql:
  enabled: true
  postgresql.auth.username: "user"
  postgresql.auth.postgresPassword: "password"
  postgresqlDatabase: "airflow"

# Other Apache Airflow configurations can be added here if desired
namespace: airflow

dags:
  persistence:
    enabled: false
    existingClaim: "dags-volume-pvc"
  gitSync:
    enabled: true
    repo: https://<giturl>/testdag.git
    branch: master
    depth: 1
    # the number of consecutive failures allowed before aborting
    maxFailures: 0
    # subpath within the repo where dags are located
    # should be "" if dags are at repo root
    subPath: "dags"
    # if your repo needs a user name password
    # you can load them to a k8s secret like the one below
    #   ---
    #   apiVersion: v1
    #   kind: Secret
    #   metadata:
    #     name: git-credentials
    #   data:
    #     GIT_SYNC_USERNAME: <base64_encoded_git_username>
    #     GIT_SYNC_PASSWORD: <base64_encoded_git_password>
    # and specify the name of the secret below
    #
    # credentialsSecret: bitbucket-credentials
    sshKeySecret: git-sync-ssh-key

# Add plugin configurations
plugins:
  persistence:
    enabled: false
    existingClaim: "plugins-volume-pvc"
  gitSync:
    enabled: true
    repo: https://<git-url>testdag.git
    branch: master
    depth: 1
    maxFailures: 0
    subPath: "plugins"
    sshKeySecret: git-sync-ssh-key
    # credentialsSecret: bitbucket-credentials

kubectl describe pod airflow-scheduler-69b9875498-7t75d -n airflow

Name:             airflow-scheduler-69b9875498-7t75d
Namespace:        airflow
Priority:         0
Service Account:  airflow-scheduler
Node:             ip-<ipv4>ec2.internal/<ipv4>
Start Time:       Thu, 15 Jun 2023 09:18:16 -0400
Labels:           component=scheduler
                  pod-template-hash=69b9875498
                  release=airflow
                  tier=airflow
Annotations:      checksum/airflow-config: 45b3ab22ffd454cb9b2f15636d1756235656a93c1c749731e12a563eef0b5a42
                  checksum/extra-configmaps: 2e44e493035e2f6a255d08f8104087ff10d30aef6f63176f1b18f75f73295598
                  checksum/extra-secrets: bb91ef06ddc31c0c5a29973832163d8b0b597812a793ef911d33b622bc9d1655
                  checksum/metadata-secret: a5c9ef957c6afd5660a6f087e3ebf8a97991086c2f3ce203ca0c541366ddf63a
                  checksum/pgbouncer-config-secret: da52bd1edfe820f0ddfacdebb20a4cc6407d296ee45bcb500a6407e2261a5ba2
                  checksum/result-backend-secret: 74e3e99feee51248d44224665d60fab543dd6b25ba95f04e6fcb0e5758342056
                  cluster-autoscaler.kubernetes.io/safe-to-evict: true
Status:           Pending
IP:               <ipv4-addr>
IPs:
  IP:           <ipv4-addr>
Controlled By:  ReplicaSet/airflow-scheduler-69b9875498
Init Containers:
  wait-for-airflow-migrations:
    Container ID:  containerd://f35b2a04daa53c7b4fc4aceeb90c95f2a0b07faadcefe920a860a83439bb6598
    Image:         apache/airflow:2.5.3
    Image ID:      docker.io/apache/airflow@sha256:c21bc4fbc6e40a8b9ab1cded8d03d37864e91f8bdcccb619dd87b816033fd095
    Port:          <none>
    Host Port:     <none>
    Args:
      airflow
      db
      check-migrations
      --migration-wait-timeout=60
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Thu, 15 Jun 2023 09:18:18 -0400
      Finished:     Thu, 15 Jun 2023 09:19:02 -0400
    Ready:          True
    Restart Count:  0
    Environment:
      AIRFLOW__CORE__FERNET_KEY:            <set to the key 'fernet-key' in secret 'airflow-fernet-key'>                      Optional: false
      AIRFLOW__CORE__SQL_ALCHEMY_CONN:      <set to the key 'connection' in secret 'airflow-airflow-metadata'>                Optional: false
      AIRFLOW__DATABASE__SQL_ALCHEMY_CONN:  <set to the key 'connection' in secret 'airflow-airflow-metadata'>                Optional: false
      AIRFLOW_CONN_AIRFLOW_DB:              <set to the key 'connection' in secret 'airflow-airflow-metadata'>                Optional: false
      AIRFLOW__WEBSERVER__SECRET_KEY:       <set to the key 'webserver-secret-key' in secret 'airflow-webserver-secret-key'>  Optional: false
      AIRFLOW__CELERY__BROKER_URL:          <set to the key 'connection' in secret 'airflow-broker-url'>                      Optional: false
    Mounts:
      /opt/airflow/airflow.cfg from config (ro,path="airflow.cfg")
      /opt/airflow/config/airflow_local_settings.py from config (ro,path="airflow_local_settings.py")
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-xrwnh (ro)
  git-sync-init:
    Container ID:   containerd://c10f3fadd7aa61b39be9dfc8318b3b0547e443ba9a133bf72cd264e506310f73
    Image:          registry.k8s.io/git-sync/git-sync:v3.6.3
    Image ID:       registry.k8s.io/git-sync/git-sync@sha256:bc6e35fb93b8e1e67084e8b7843d17706919899900e63e92574fec530ae6fb57
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Thu, 15 Jun 2023 09:22:12 -0400
      Finished:     Thu, 15 Jun 2023 09:22:12 -0400
    Ready:          False
    Restart Count:  5
    Environment:
      GIT_SSH_KEY_FILE:            /etc/git-secret/ssh
      GIT_SYNC_SSH:                true
      GIT_KNOWN_HOSTS:             false
      GIT_SYNC_REV:                HEAD
      GIT_SYNC_BRANCH:             master
      GIT_SYNC_REPO:               https://<git-url>testdag.git
      GIT_SYNC_DEPTH:              1
      GIT_SYNC_ROOT:               /git
      GIT_SYNC_DEST:               repo
      GIT_SYNC_ADD_USER:           true
      GIT_SYNC_WAIT:               5
      GIT_SYNC_MAX_SYNC_FAILURES:  0
      GIT_SYNC_ONE_TIME:           true
    Mounts:
      /etc/git-secret/ssh from git-sync-ssh-key (ro,path="gitSshKey")
      /git from dags (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-xrwnh (ro)
Containers:
  scheduler:
    Container ID:
    Image:         apache/airflow:2.5.3
    Image ID:
    Port:          <none>
    Host Port:     <none>
    Args:
      bash
      -c
      exec airflow scheduler
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Liveness:       exec [sh -c CONNECTION_CHECK_MAX_COUNT=0 AIRFLOW__LOGGING__LOGGING_LEVEL=ERROR exec /entrypoint \
airflow jobs check --job-type SchedulerJob --local
] delay=10s timeout=20s period=60s #success=1 #failure=5
    Environment:
      AIRFLOW__CORE__FERNET_KEY:            <set to the key 'fernet-key' in secret 'airflow-fernet-key'>                      Optional: false
      AIRFLOW__CORE__SQL_ALCHEMY_CONN:      <set to the key 'connection' in secret 'airflow-airflow-metadata'>                Optional: false
      AIRFLOW__DATABASE__SQL_ALCHEMY_CONN:  <set to the key 'connection' in secret 'airflow-airflow-metadata'>                Optional: false
      AIRFLOW_CONN_AIRFLOW_DB:              <set to the key 'connection' in secret 'airflow-airflow-metadata'>                Optional: false
      AIRFLOW__WEBSERVER__SECRET_KEY:       <set to the key 'webserver-secret-key' in secret 'airflow-webserver-secret-key'>  Optional: false
      AIRFLOW__CELERY__BROKER_URL:          <set to the key 'connection' in secret 'airflow-broker-url'>                      Optional: false
    Mounts:
      /opt/airflow/airflow.cfg from config (ro,path="airflow.cfg")
      /opt/airflow/config/airflow_local_settings.py from config (ro,path="airflow_local_settings.py")
      /opt/airflow/dags from dags (ro)
      /opt/airflow/logs from logs (rw)
      /opt/airflow/pod_templates/pod_template_file.yaml from config (ro,path="pod_template_file.yaml")
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-xrwnh (ro)
  git-sync:
    Container ID:
    Image:          registry.k8s.io/git-sync/git-sync:v3.6.3
    Image ID:
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Environment:
      GIT_SSH_KEY_FILE:            /etc/git-secret/ssh
      GIT_SYNC_SSH:                true
      GIT_KNOWN_HOSTS:             false
      GIT_SYNC_REV:                HEAD
      GIT_SYNC_BRANCH:             master
      GIT_SYNC_REPO:               https://<git-url>/testdag.git
      GIT_SYNC_DEPTH:              1
      GIT_SYNC_ROOT:               /git
      GIT_SYNC_DEST:               repo
      GIT_SYNC_ADD_USER:           true
      GIT_SYNC_WAIT:               5
      GIT_SYNC_MAX_SYNC_FAILURES:  0
    Mounts:
      /etc/git-secret/ssh from git-sync-ssh-key (ro,path="gitSshKey")
      /git from dags (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-xrwnh (ro)
  scheduler-log-groomer:
    Container ID:
    Image:         apache/airflow:2.5.3
    Image ID:
    Port:          <none>
    Host Port:     <none>
    Args:
      bash
      /clean-logs
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Environment:
      AIRFLOW__LOG_RETENTION_DAYS:  15
    Mounts:
      /opt/airflow/logs from logs (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-xrwnh (ro)
Conditions:
  Type              Status
  Initialized       False
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      airflow-airflow-config
    Optional:  false
  dags:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  git-sync-ssh-key:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  git-sync-ssh-key
    Optional:    false
  logs:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  kube-api-access-xrwnh:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  6m29s                 default-scheduler  Successfully assigned airflow/airflow-scheduler-69b9875498-7t75d to ip-<ipv4>ec2.internal
  Normal   Pulled     6m28s                 kubelet            Container image "apache/airflow:2.5.3" already present on machine
  Normal   Created    6m28s                 kubelet            Created container wait-for-airflow-migrations
  Normal   Started    6m27s                 kubelet            Started container wait-for-airflow-migrations
  Normal   Pulled     4m6s (x5 over 5m43s)  kubelet            Container image "registry.k8s.io/git-sync/git-sync:v3.6.3" already present on machine
  Normal   Created    4m6s (x5 over 5m43s)  kubelet            Created container git-sync-init
  Normal   Started    4m6s (x5 over 5m42s)  kubelet            Started container git-sync-init
  Warning  BackOff    86s (x20 over 5m40s)  kubelet            Back-off restarting failed container

generate-secret
kubectl create secret generic git-sync-ssh-key --from-file=ssh-privatekey=/Users/<username>/.ssh/id_rsa -n airflow

list secret

kubectl get secret git-sync-ssh-key -n airflow                                                                                                                                            
NAME               TYPE     DATA   AGE
git-sync-ssh-key   Opaque   1      11h

scheduler pod logs

kubectl logs airflow-scheduler-69b9875498-7t75d -n airflow

Defaulted container "scheduler" out of: scheduler, git-sync, scheduler-log-groomer, wait-for-airflow-migrations (init), git-sync-init (init)
Error from server (BadRequest): container "scheduler" in pod "airflow-scheduler-69b9875498-7t75d" is waiting to start: PodInitializing

[potiuk]

You need to see the logs of the failing container. I recommend you use k9s - that's also great tool to learn and manage your k8s instance (learning how to get logs from failed pods is just the beginning, you will have to learn a lot more to manage your k8s instance).

[potiuk]

I am being honest and straightforward. It's a important to be considerate of time of others and "do your part" before asking. People here are helping more eagerly if person asking for help did any effort to solve their problem other than copy&pasting error and expecting someone solves their problem.

That's the social agreement in this forum where people like me try to help others in their free time.

[ChaitDevOps]

There's a difference between what you said now and what you said previously. There are many memebers like myself whose knowledge is very limited. you trying to gate-keep is not an encouraging note to anyone who wants to a legit questions. Either way, have a good day.

[potiuk]

I suggest next time when you copy&paste your error and expect generic help don't tag specific people. Otherwise you leave the impresion you want to ask for their exclusve time to solve your problem.

[ChaitDevOps]

I didn't tag anyone. I just replied to your comment, that's all. My suggestion to you would be to be a little kinder to people who are coming on here to ask questions. In the end this is a community and if someone is not "doing their part", we should try and inform them politely instead of being condescending.

[potiuk]

You did tag me in the message where you said "I do not know how to solve this". Right after I helped you (didn't I) with useful suggestion for the tool that you can used.

I read it as -> used tool -> got error -> tell me what to do.

If that was not it, then apologies, but this is how I read it.

Please be mindful next time the difference when you respond, vs. respond and @ChaitDevOps person. Tagging person in the response that you do is a sign that you expect them to answer in case you did not realize this is the ettiquete.

Since you used the condescending word, I will use a different one, please be mindful to how things work where you are and open to cirtique. Usually when you start in a new place it is OK to make mistakes and correct yourself when you did even if you do not know the ettiquete.

[VeMeth]

Have you ever found a solution to this? I'm stuck with a similar problem. I can't find the exact message you got. But the moment I enable git sync all my pods crash.

[ChaitDevOps]

Can you post your yaml file and logs? Yes, i did solve my issue by increasing the cpu and memory of the pods. I also added a command that send back a successful health check.

scheduler:
  resources:
    limits:
      cpu: ${cpu}
      memory: 4000Mi
    requests:
      cpu: ${cpu}
      memory: 4000Mi
  livenessProbe:
    initialDelaySeconds: 10
    timeoutSeconds: 120
    failureThreshold: 20
    periodSeconds: 60
    command:
      - sh
      - -c
      - |
        CONNECTION_CHECK_MAX_COUNT=0 AIRFLOW__LOGGING__LOGGING_LEVEL=ERROR exec /entrypoint \
        airflow jobs check
  replicas: ${replicas}`
```

[bsieraduml]

I had a similar problem today; it was related to my gitsync config layer too; looking at the initial post perhaps bad request is related to an invalid url? see values.yaml in initial post;

  GIT_SYNC_REPO:               https://**<git-url>**testdag.git