proxy_set_header does not seem to work on http-level

[anton-johansson]

What happened:

I'm trying to add global configuration for a X-Correlation-ID header. I deploy NGINX Ingress Controller with Helm into my Kubernetes cluster, and I have this configuration (non-relevant config omitted):

controller:
  config:
    log-format-escape-json: "true"
    log-format-upstream: '{"message":"$request_method $scheme://$host:$server_port$request_uri ($status)","trace":{"id":"$correlation_id""}}'
    http-snippet: |
      map $http_x_correlation_id $correlation_id {
        default   $http_x_correlation_id;
        ""        $req_id;
      }
      proxy_set_header X-Correlation-ID $correlation_id;

I would expect any upstream pod to receive the X-Correlation-ID header. But they just don't get it. They do get it if I pass it in from the outside world, but I need the fallback to work throughout my chain of services.

I've also tried with using the nginx.ingress.kubernetes.io/server-snippet annotation on the Ingress resource (which I would like to avoid, because I want a global configuration for all my services):

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/server-snippet: |
      proxy_set_header X-Correlation-ID $correlation_id;

This does not work either. What does work, however, is the nginx.ingress.kubernetes.io/configuration-snippet annotation, like this:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_set_header X-Correlation-ID $correlation_id;

As far as I can see in the generated nginx.conf, there is nothing else that should conflict with this.

What you expected to happen:

I would expect the X-Correlation-ID header to be passed in to my upstream pods, based on the fallback I've set up. Both for the http-snippet, but also for the server-snippet annotation (not preferred).

NGINX Ingress controller version (exec into the pod and run /nginx-ingress-controller --version):

-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       v1.6.4
  Build:         69e8833858fb6bda12a44990f1d5eaa7b13f4b75
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.21.6
-------------------------------------------------------------------------------

Helm chart version 4.5.2. But I've also tried with the latest Helm chart, 4.10.1, which is NGINX Ingress Controller v1.10.1.

Kubernetes version (use kubectl version):

Client Version: v1.29.5
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.29.5

Environment:

• Cloud provider or hardware configuration: Bare-metal, via Kubespray
• OS (e.g. from /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
  • Kubespray
• Basic cluster related info:

NAME             STATUS   ROLES           AGE    VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION    CONTAINER-RUNTIME
k8s-s-master01   Ready    control-plane   705d   v1.29.5   XXX.XXX.XXX.XXX <none>        Debian GNU/Linux 11 (bullseye)   5.10.0-20-amd64   containerd://1.7.16
k8s-s-master02   Ready    control-plane   705d   v1.29.5   XXX.XXX.XXX.XXX <none>        Debian GNU/Linux 11 (bullseye)   5.10.0-20-amd64   containerd://1.7.16
k8s-s-master03   Ready    control-plane   705d   v1.29.5   XXX.XXX.XXX.XXX <none>        Debian GNU/Linux 11 (bullseye)   5.10.0-20-amd64   containerd://1.7.16
k8s-s-worker04   Ready    <none>          52d    v1.29.5   XXX.XXX.XXX.XXX <none>        Debian GNU/Linux 11 (bullseye)   5.10.0-21-amd64   containerd://1.7.16
k8s-s-worker05   Ready    <none>          664d   v1.29.5   XXX.XXX.XXX.XXX <none>        Debian GNU/Linux 11 (bullseye)   5.10.0-21-amd64   containerd://1.7.16
k8s-s-worker06   Ready    <none>          664d   v1.29.5   XXX.XXX.XXX.XXX <none>        Debian GNU/Linux 11 (bullseye)   5.10.0-21-amd64   containerd://1.7.16
k8s-s-worker07   Ready    <none>          551d   v1.29.5   XXX.XXX.XXX.XXX <none>        Debian GNU/Linux 11 (bullseye)   5.10.0-23-amd64   containerd://1.7.16

• How was the ingress-nginx-controller installed:

nginx-external                   	ingress-nginx                                          	41      	2024-11-26 12:29:29.558585053 +0100 CET 	deployed	ingress-nginx-4.5.2                             	1.6.4          
nginx-internal                   	ingress-nginx                                          	22      	2024-11-26 13:25:36.828932905 +0100 CET 	deployed	ingress-nginx-4.5.2                             	1.6.4

Internal:

controller:
  admissionWebhooks:
    enabled: false
  config:
    brotli-level: "6"
    client-body-buffer-size: 64K
    enable-brotli: "true"
    enable-modsecurity: "true"
    enable-owasp-modsecurity-crs: "true"
    http-snippet: |
      map $http_x_correlation_id $correlation_id {
        default   $http_x_correlation_id;
        ""        $req_id;
      }
      proxy_set_header X-Correlation-ID $correlation_id;
    log-format-escape-json: "true"
    log-format-upstream: '{"message":"$request_method $scheme://$host:$server_port$request_uri
      ($status)","kubernetes":{"ingress":{"class":"internal","namespace":"$namespace","name":"$ingress_name","service_name":"$service_name","service_port":"$service_port"}},"url":{"scheme":"$scheme","domain":"$host","port":"$server_port","path":"$uri","query":"$query_string","username":"$remote_user"},"http":{"request":{"id":"$req_id","method":"$request_method","referrer":"$http_referer"},"response":{"status_code":"$status"},"version":"$server_protocol"},"user_agent":{"original":"$http_user_agent"},"client":{"ip":"$remote_addr"},"trace":{"id":"$correlation_id"},"nginxMsec":"$msec","requestTime":"$request_time"}}'
    modsecurity-snippet: |
      SecRuleEngine On
      SecRequestBodyAccess On
      SecAuditLog /dev/stdout
      SecAuditLogFormat JSON
      SecAuditEngine RelevantOnly

      # Include PUT, PATCH and DELETE in the allowed HTTP methods, otherwise those verbs will be rejected by rule 911100
      SecAction "id:900200,phase:1,nolog,pass,t:none,setvar:tx.allowed_methods=GET HEAD POST OPTIONS PUT PATCH DELETE"
      # Disable rule that triggers if host header is an IP-address. https://github.com/kubernetes/ingress-nginx/issues/8974
      SecRuleRemoveById 920350
    server-tokens: "false"
    use-gzip: "true"
  extraArgs:
    default-ssl-certificate: ingress-nginx/default-ssl-certificate
  ingressClass: internal
  ingressClassByName: true
  ingressClassResource:
    controllerValue: example.com/internal
    name: internal
  metrics:
    enabled: true
  podAnnotations:
    logging.example.com/ecs-logs: "true"
    prometheus.io/port: "10254"
    prometheus.io/scrape: "true"
  replicaCount: 2
  service:
    externalTrafficPolicy: Local
    loadBalancerIP: XXX.XXX.XXX.XXX

External (but lets focus on the internal for now):

controller:
  admissionWebhooks:
    enabled: false
  config:
    brotli-level: "6"
    client-body-buffer-size: 64K
    enable-brotli: "true"
    enable-modsecurity: "true"
    enable-owasp-modsecurity-crs: "true"
    hsts-include-subdomains: "false"
    http-snippet: |
      map $http_x_correlation_id $correlation_id {
        default   $http_x_correlation_id;
        ""        $req_id;
      }
    log-format-escape-json: "true"
    log-format-upstream: '{"message":"$request_method $scheme://$host:$server_port$request_uri
      ($status)","kubernetes":{"ingress":{"class":"external","namespace":"$namespace","name":"$ingress_name","service_name":"$service_name","service_port":"$service_port"}},"url":{"scheme":"$scheme","domain":"$host","port":"$server_port","path":"$uri","query":"$query_string","username":"$remote_user"},"http":{"request":{"id":"$req_id","method":"$request_method","referrer":"$http_referer"},"response":{"status_code":"$status"},"version":"$server_protocol"},"user_agent":{"original":"$http_user_agent"},"client":{"ip":"$remote_addr"},"trace":{"id":"$correlation_id"},"nginxMsec":"$msec","requestTime":"$request_time"}}'
    modsecurity-snippet: |
      SecRuleEngine On
      SecRequestBodyAccess On
      SecAuditLog /dev/stdout
      SecAuditLogFormat JSON
      SecAuditEngine RelevantOnly

      # Include PUT, PATCH and DELETE in the allowed HTTP methods, otherwise those verbs will be rejected by rule 911100
      SecAction "id:900200,phase:1,nolog,pass,t:none,setvar:tx.allowed_methods=GET HEAD POST OPTIONS PUT PATCH DELETE"
      # Disable rule that triggers if host header is an IP-address. https://github.com/kubernetes/ingress-nginx/issues/8974
      SecRuleRemoveById 920350
    server-tokens: "false"
    use-gzip: "true"
  extraArgs:
    default-ssl-certificate: ingress-nginx/default-ssl-certificate
  ingressClass: external
  ingressClassByName: true
  ingressClassResource:
    controllerValue: example.com/external
    name: external
  metrics:
    enabled: true
  podAnnotations:
    logging.example.com/ecs-logs: "true"
    prometheus.io/port: "10254"
    prometheus.io/scrape: "true"
  replicaCount: 2
  service:
    externalTrafficPolicy: Local
    loadBalancerIP: XXX.XXX.XXX.XXX

• Current State of the controller:

❯ kubectl describe ingressclasses

Name:         external
Labels:       app.kubernetes.io/component=controller
              app.kubernetes.io/instance=nginx-external
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=ingress-nginx
              app.kubernetes.io/part-of=ingress-nginx
              app.kubernetes.io/version=1.6.4
              helm.sh/chart=ingress-nginx-4.5.2
Annotations:  meta.helm.sh/release-name: nginx-external
              meta.helm.sh/release-namespace: ingress-nginx
Controller:   example.com/external
Events:       <none>


Name:         internal
Labels:       app.kubernetes.io/component=controller
              app.kubernetes.io/instance=nginx-internal
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=ingress-nginx
              app.kubernetes.io/part-of=ingress-nginx
              app.kubernetes.io/version=1.6.4
              helm.sh/chart=ingress-nginx-4.5.2
Annotations:  meta.helm.sh/release-name: nginx-internal
              meta.helm.sh/release-namespace: ingress-nginx
Controller:   example.com/internal
Events:       <none>

❯ kubectl -n ingress-nginx get all -o wide   
NAME                                                           READY   STATUS    RESTARTS   AGE   IP               NODE             NOMINATED NODE   READINESS GATES
pod/nginx-external-ingress-nginx-controller-777b6b87f8-mjx4q   1/1     Running   0          12d   XXX.XXX.XXX.XXX   k8s-s-worker04   <none>           <none>
pod/nginx-external-ingress-nginx-controller-777b6b87f8-w9tbm   1/1     Running   0          12d   XXX.XXX.XXX.XXX  k8s-s-worker07   <none>           <none>
pod/nginx-internal-ingress-nginx-controller-bfcb65794-p89gl    1/1     Running   0          23m   XXX.XXX.XXX.XXX  k8s-s-worker07   <none>           <none>
pod/nginx-internal-ingress-nginx-controller-bfcb65794-x4b24    1/1     Running   0          23m   XXX.XXX.XXX.XXX   k8s-s-worker04   <none>           <none>

NAME                                                      TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)                      AGE    SELECTOR
service/nginx-external-ingress-nginx-controller           LoadBalancer   10.233.25.193   XXX.XXX.XXX.XXX  80:30674/TCP,443:30788/TCP   697d   app.kubernetes.io/component=controller,app.kubernetes.io/instance=nginx-external,app.kubernetes.io/name=ingress-nginx
service/nginx-external-ingress-nginx-controller-metrics   ClusterIP      10.233.16.208   <none>           10254/TCP                    385d   app.kubernetes.io/component=controller,app.kubernetes.io/instance=nginx-external,app.kubernetes.io/name=ingress-nginx
service/nginx-internal-ingress-nginx-controller           LoadBalancer   10.233.34.41    XXX.XXX.XXX.XXX  80:32173/TCP,443:32690/TCP   697d   app.kubernetes.io/component=controller,app.kubernetes.io/instance=nginx-internal,app.kubernetes.io/name=ingress-nginx
service/nginx-internal-ingress-nginx-controller-metrics   ClusterIP      10.233.21.242   <none>           10254/TCP                    385d   app.kubernetes.io/component=controller,app.kubernetes.io/instance=nginx-internal,app.kubernetes.io/name=ingress-nginx

NAME                                                      READY   UP-TO-DATE   AVAILABLE   AGE    CONTAINERS   IMAGES                                                                                                                    SELECTOR
deployment.apps/nginx-external-ingress-nginx-controller   2/2     2            2           697d   controller   registry.k8s.io/ingress-nginx/controller:v1.6.4@sha256:15be4666c53052484dd2992efacf2f50ea77a78ae8aa21ccd91af6baaa7ea22f   app.kubernetes.io/component=controller,app.kubernetes.io/instance=nginx-external,app.kubernetes.io/name=ingress-nginx
deployment.apps/nginx-internal-ingress-nginx-controller   2/2     2            2           697d   controller   registry.k8s.io/ingress-nginx/controller:v1.6.4@sha256:15be4666c53052484dd2992efacf2f50ea77a78ae8aa21ccd91af6baaa7ea22f   app.kubernetes.io/component=controller,app.kubernetes.io/instance=nginx-internal,app.kubernetes.io/name=ingress-nginx

... and also a bunch of old replica sets.

• Current state of ingress object, if applicable:

Too many to name here.

• Others:

N/A.

How to reproduce this issue:

I don't have a good way of reproducing everything, I'm running with Helm in my created clusters.

Anything else we need to know:

Not that I can think off.

[k8s-ci-robot]

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[reethuj]

I can think of a couple of ways

1. Use nginx.ingress.kubernetes.io/configuration-snippet for Global Headers. If http-snippet doesn’t seem to work globally for your case, consider enforcing a global header using the Ingress annotation nginx.ingress.kubernetes.io/configuration-snippet. It is applied at the Ingress level and allows for more flexibility in inserting headers.Also you could set a default header globally with a fallback approach using Kubernetes annotations across all Ingress resources. But then you want this globally, so applying it at the Ingress controller's default configuration (via Helm) should work.

controller:
  config:
    http-snippet: |
      map $http_x_correlation_id $correlation_id {
        default   $http_x_correlation_id;
        ""        $req_id;
      }
    ingressClass: internal # If you use different ingress classes
    ingressClassByName: true

If necessary, apply nginx.ingress.kubernetes.io/configuration-snippet at the Ingress level for services where proxy_set_header is still not working.

2. Add proxy_set_header in server-snippet or location-snippet for global scope: Instead of just using http-snippet, you can try adding your proxy_set_header in a more specific context like server-snippet to ensure it gets applied correctly.

controller:
  config:
    http-snippet: |
      map $http_x_correlation_id $correlation_id {
        default   $http_x_correlation_id;
        ""        $req_id;
      }
    server-snippet: |
      # Ensures the header is set in the server context
      proxy_set_header X-Correlation-ID $correlation_id;

[anton-johansson]

Thanks for the reply @reethuj!

1. As I mentioned, I already did tried with configuration-snippet on Ingress level. It did work, but I don't want to specify it on all my Ingress resources. I want this to happen automatically for all my HTTP services.

2. As far as I know, there is no server-snippet on the actual controller. It can only be done via annotation on Ingress. And even so, it does not work on that level either, as I mentioned in my original issue.

What is most baffling to me is that the proxy_set_header directive should be supported on all levels, http, server and location. And I can clearly see it in the correct spot when i exec into the NGINX Ingress Controller pods. But yet, it does not work. :(

Update: While writing this, I Googled around some more and found the setting controller.proxySetHeaders. I tried that, and it seems to work. It seems to work because it puts them in the location block.

So from my point of view, I'm closing this issue. But it's still a bit weird that I can't get proxy_set_header to work in the http (or server) blocks?