500 Internal Service Error on new cert order

[michaelmedellin]

Using an identical HA Proxy configuration and curl command to the README, I can't seem to get an order to go through and either fail or succeed. Seems to fail almost immediately with a 500 error. Would appreciate some guidance—I'm fairly new to HA Proxy so I may be missing something relatively simple.

haproxy.cfg:

global
    log /dev/log local0 debug
    nbproc 1
    daemon
    lua-load config.lua
    lua-load acme.lua

defaults
    log global
    mode http
    option httplog
    timeout connect 5s
    timeout client 10s
    timeout server 10s

listen http
    bind *:80
    http-request use-service lua.acme if { path_beg /.well-known/acme-challenge/  }

listen acme
    bind 127.0.0.1:9011
    http-request use-service lua.acme

listen acme-ca
  bind 127.0.0.1:9012
  server ca acme-v02.api.letsencrypt.org:443 ssl verify required ca-file letsencrypt-x3-ca-chain.pem

CURL:

curl -XPOST -v http://127.0.0.1:9011/acme/order -F '[email protected]' \
     -F 'domain=example.net' -F '[email protected]' \
     -F 'aliases=www.example.net,example.com,www.example.com' \
     -o example.net.pem

HA Proxy Version:

HA-Proxy version 1.8.13 2018/07/30
Copyright 2000-2018 Willy Tarreau <[email protected]>

Build options :
  TARGET  = linux2628
  CPU     = generic
  CC      = gcc
  CFLAGS  = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -fno-strict-overflow -Wno-unused-label
  OPTIONS = USE_LIBCRYPT=1 USE_CRYPT_H=1 USE_ZLIB=1 USE_OPENSSL=1 USE_LUA=1 USE_PCRE=1

Curl Failure:

*   Trying 127.0.0.1...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to 127.0.0.1 (127.0.0.1) port 9011 (#0)
> POST /acme/order HTTP/1.1
> Host: 127.0.0.1:9011
> User-Agent: curl/7.47.0
> Accept: */*
> Content-Length: 5453
> Expect: 100-continue
> Content-Type: multipart/form-data; boundary=------------------------47a351a7e23f1bf9
>
< HTTP/1.1 500 Internal Server Error
< content-type: text/plain; charset=utf-8
< content-length: 0
* HTTP error before end of send, stop sending
<
  0  5453    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
* Closing connection 0

HA Proxy Debug Output:

00000000:acme-ca.accept(0007)=000a from [127.0.0.1:36338] ALPN=<none>
00000000:acme-ca.clireq[000a:ffffffff]: GET /dir HTTP/1.1
00000000:acme-ca.clihdr[000a:ffffffff]: host: 127.0.0.1:9012
00000000:acme-ca.clihdr[000a:ffffffff]: accept: */*
00000000:acme-ca.clihdr[000a:ffffffff]: user-agent: haproxy-lua-http/1.0
00000000:acme-ca.clihdr[000a:ffffffff]: connection: close
00000000:acme-ca.srvrep[000a:adfd]: HTTP/1.0 400 Bad Request
00000000:acme-ca.srvhdr[000a:adfd]: Server: AkamaiGHost
00000000:acme-ca.srvhdr[000a:adfd]: Mime-Version: 1.0
00000000:acme-ca.srvhdr[000a:adfd]: Content-Type: text/html
00000000:acme-ca.srvhdr[000a:adfd]: Content-Length: 209
00000000:acme-ca.srvhdr[000a:adfd]: Expires: Sun, 19 Aug 2018 14:04:05 GMT
00000000:acme-ca.srvhdr[000a:adfd]: Date: Sun, 19 Aug 2018 14:04:05 GMT
00000000:acme-ca.srvhdr[000a:adfd]: Connection: close

[jmagnin]

@michaelmedellin you need to add a set-header in the acme-ca listener such as the following:
http-request set-header Host acme-v02.api.letsencrypt.org

[richbayliss]

It looks like this issue is still a thing... perhaps related to ACME v2?

ingrid_1  | 0000000d:acme-ca.clireq[001a:ffffffff]: GET /acme/authz-v3/54535395 HTTP/1.1
ingrid_1  | 0000000d:acme-ca.clihdr[001a:ffffffff]: host: 127.0.0.1:9012
ingrid_1  | 0000000d:acme-ca.clihdr[001a:ffffffff]: accept: */*
ingrid_1  | 0000000d:acme-ca.clihdr[001a:ffffffff]: user-agent: haproxy-lua-http/1.0
ingrid_1  | 0000000d:acme-ca.srvrep[001a:001b]: HTTP/1.1 405 Method Not Allowed
ingrid_1  | 0000000d:acme-ca.srvhdr[001a:001b]: server: nginx
ingrid_1  | 0000000d:acme-ca.srvhdr[001a:001b]: date: Wed, 06 May 2020 16:31:58 GMT
ingrid_1  | 0000000d:acme-ca.srvhdr[001a:001b]: content-type: application/problem+json
ingrid_1  | 0000000d:acme-ca.srvhdr[001a:001b]: content-length: 103
ingrid_1  | 0000000d:acme-ca.srvhdr[001a:001b]: cache-control: public, max-age=0, no-cache
ingrid_1  | 0000000d:acme-ca.srvhdr[001a:001b]: link: <https://acme-staging-v02.api.letsencrypt.org/directory>;rel="index"
ingrid_1  | 0000000d:acme-ca.srvcls[001a:001b]
ingrid_1  | 0000000d:acme-ca.clicls[001a:001b]
ingrid_1  | 0000000d:acme-ca.closed[001a:001b]
ingrid_1  | (table) table: 0x5622f280fba0 [
ingrid_1  |     "detail": (string) "Method not allowed"
ingrid_1  |     "type": (string) "urn:ietf:params:acme:error:malformed"
ingrid_1  |     "status": (number) 405
ingrid_1  | ]

I don't think it's correct to be performing a GET on the authorization. Could someone confirm this is working with LetsEncrypt v2 staging endpoint, at least to save me trying to diagnose something which isn't broken?

[richbayliss]

Yeah, I can confirm that the errors are due to the deprecation of GET requests in v2 and switching to POST-as-GET worked in the end. I will try and PR a fix for you guys 👍

[anezirovic]

Hello Rich,

Yep, v1 API is depreciated, we're planning upgrade for a long time, but PR are always welcome :-)

[mianos]

Can you please update the README on the front page? I just spent quite a long time setting this up and working through it only to discover it does not work on account of this issue.

[delormejonathan]

I had a lot of issues to renew my certificates with this tool since ~1 month. 80% of my renewal requests was aborted.

After multiple tests, here are my results :

• POST-as-GET directive was delayed for the production environment, it's not the main issue : https://community.letsencrypt.org/t/acme-v2-scheduled-deprecation-of-unauthenticated-resource-gets/74380/5
• The current LUA script in master branch is working in ACMEv2 production environment
• Let's Encrypt infrastructure has a lot of issues, yesterday 1/3 of my requests were 503 errors
• These 503 errors are breaking the request chain in LUA script, aborting a lot of certificates
• I had an issue with my chained cert letsencrypt-x3-ca-chain.pem which was too old (it was using Let’s Encrypt Authority X3 intermediate cert which is retired)

I send a pull request today with :

• A retry mecanism to increase reliability (5 retries per request)
• Compliance with POST-as-GET directive