Unable to issue certificate from lets encrypt for my domain

[shauryagoel]

I used this config to issue a proper certificate from let's encrypt-

certificate:
  type: lets-encrypt
  acme_email: <MY_EMAIL>
  # acme_server: https://acme-v02.api.letsencrypt.org/directory
  acme_server: https://acme-staging-v02.api.letsencrypt.org/director

But this is giving me the following error in traefik pod-

Unable to obtain ACME certificate for domains \"<MY_DOMAIN>\": unable to generate a certificate for the domains [<MY_DOMAIN>]: error: one or more domains had a problem:
[<MY_DOMAIN>] acme: error: 400 :: urn:ietf:params:acme:error:tls :: <IP>: remote error: tls: unrecognized name
" providerName=letsencrypt.acme ACME CA="https://acme-v02.api.letsencrypt.org/directory" routerName=dev-jupyterhub-ca277b77dd23830bef42@kubernetescrd rule="Host(`<MY_DOMAIN>`) && (Path(`/`) || PathPrefix(`/hub`) || PathPrefix(`/user`) || PathPrefix(`/services`))

I tried running it multiple times and with both staging and production server, but, still getting the same error. Can someone help me here to track the issue.

[viniciusdc]

Hey @shauryagoel just to make sure, was this working before with your cluster? In your nebari-config file, just to ensure your domain field is populated, right? Can you also check the status of the traefik pod? I just want to make sure the pod is running. You can do that by executing:

kubectl get pods -n "YOUR NAMESPACE" 

[viniciusdc]

Yes that was one of them, but also, to inspect the recurrent domain certificates, you can use https://crt.sh

[viniciusdc]

Without more context about the actual domain and other details about the cluster, it is a bit difficult to try debugging further; the last thing I can suggest that might help grant more light to this issue is to port forward trafficks dashboard for monitoring the ingress routes/middleware, etc.

kubectl port-forward -n dev nebari-traefik-ingress-aaaababababab 9000:9000

you will need to replace the pod name with the actual one from your cluster, looks like this

[shauryagoel]

Unfortunately, crt.sh is showing nothing for my nebari domain. Traefik dashboard looks fine as well (everything is success).
On deleting the traefik pod, the certificate is being re-issued. However, I am now getting Connection refused error-

time="2025-02-20T06:43:50Z" level=error msg="Unable to obtain ACME certificate for domains \"<DOMAIN>\": unable to generate a certificate for the domains [<DOMAIN>]: error: one or more domains had a problem:\n[<DOMAIN>] acme: error: 400 :: urn:ietf:params:acme:error:connection ::<IP>: Connection refused\n" routerName=dev-jupyterhub-6fbc667334a8095bdc8c@kubernetescrd rule="Host(`<DOMAIN>`) && (PathPrefix(`/home`) || PathPrefix(`/token`) || PathPrefix(`/admin`))" ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" providerName=letsencrypt.acme

I have also checked any firewalls in my GCP project, everything is open. I tried this- nmap -p80,443 <DOMAIN>. This also returns that ports are open.

[shauryagoel]

Hi, today I tried these steps-

1. I first deleted the acme.json file from the traefik pod
2. I upgraded the Nebari version to 2025.2.1 and also upgraded the K8s version in nebari-config.yaml
3. I then redeployed nebari

After these steps, I am able to issue the certificate from the production lets encrypt server.
I don't know what solved the issue though.
Thanks for your time and help.

[viniciusdc]

Hey @shauryagoel, I am glad that worked out; I don't think the upgrade/redeployment ended up affecting the issue in any capacity, if I would guess, that might've related to the removal of the acme file and its consequent certificate re-issuing (probably not rate limited). Though why that failed in the first case, I am also not entirely sure as well. I will try to include this into our FAQ in the future