Specifying content type in request to application/x-www-form-urlencoded fails

[mustafaStakater]

What happened?

Specifying content type in request to application/x-www-form-urlencoded and setting the body to param1=val1&param2=val2 gives

Warning  CannotCreateExternalResource  4m7s (x28 over 24m)  managed/request.http.crossplane.io  something went wrong: unexpected token "&"

How can we reproduce it?

Deploying a request resource with following spec

apiVersion: http.crossplane.io/v1alpha1
kind: Request
metadata:
  name: example
spec:
  deletionPolicy: Delete
  forProvider:
    mappings:
      - headers:
          Content-Type:
            - application/x-www-form-urlencoded
        method: POST
        body: param1=val1&param2=val2
        url: >-
          (.payload.baseUrl + "/" +
          "some/path/")
    payload:
      baseUrl: 'https://some-url'
  managementPolicy: FullControl
  providerConfigRef:
    name: http-conf

What environment did it happen in?

Crossplane version: 1.14.0
Distribution: Openshift

[arielsepton]

Thanks for flagging this! After checking out the issue, it seems like the problem lies in how the body is set up in the request mappings.

In the provided Request resource, the body field within the mappings expects a jq filter. To fix this, just make sure to put the string in parentheses, like this:

apiVersion: http.crossplane.io/v1alpha1
kind: Request
metadata:
  name: example
spec:
  deletionPolicy: Delete
  forProvider:
    mappings:
      - headers:
          Content-Type:
            - application/x-www-form-urlencoded
        method: POST
        body: ("param1=val1&param2=val2")  # Enclose the string within parentheses
        url: >-
          (.payload.baseUrl + "/" +
          "some/path/")
    payload:
      baseUrl: 'https://some-url'
  managementPolicy: FullControl
  providerConfigRef:
    name: http-conf

That should sort out the unexpected token issue.

Oh, and I noticed you didn't include a GET mapping. Without it, the provider might struggle to validate that the resource is up to date. If you're looking to perform a one-time request, consider using the DisposableRequest resource instead. Here's a quick example:

apiVersion: http.crossplane.io/v1alpha1
kind: DesposibleRequest
metadata:
  name: example
spec:
  deletionPolicy: Orphan
  forProvider:
    url:  https://some-url/some/path
    method: POST
    body: param1=val1&param2=val2
    headers:
      Content-Type:
        - application/x-www-form-urlencoded
  providerConfigRef:
    name: http-conf

No need to worry about jq here; just specify the body without parentheses and you're good to go!

Hope this helps! Let me know if you have any further questions or concerns.

[mustafa-be]

Hello, thanks ill check it out, i m working on doing a missing configuration in keycloak (feat missing in upstream tf provider), my workflow is

• Generate a access token from keycloak api
• Use access token to perform the missing configuration against keycloak

Oh, and I noticed you didn't include a GET mapping. Without it, the provider might struggle to validate that the resource is up to date.

If i understand it correctly, i would have to include get request as well making the overall flow to be

• Generate a access token from keycloak api
• Get configuration from keycloak
• Use access token to perform the missing configuration against keycloak

[arielsepton]

Thanks for sharing your workflow! It sounds like you’re on the right track. Considering your scenario, it seems you’ll need to perform three requests:

1. Use a DisposableRequest to retrieve the access token from the Keycloak API.
2. Use another DisposableRequest to retrieve the configuration data from Keycloak.
3. For your final action, whether it’s creating, updating, or deleting resources, you can choose between a DisposableRequest and a Request:
   • If you’re performing a one-time action without managing any resources, like sending a single request to Keycloak, use a DisposableRequest.
   • If you’re managing resources and need to ensure their state, use a Request.

Just to clarify, DisposableRequest is typically used for one-time interactions, while Request is more suitable for managing resources through HTTP requests.

Let me know if you have any questions or need further clarification on implementing these steps.

[mustafa-be]

@arielsepton thanks for the comprehensive reply, it helps. Ill go with a combination of desposible request + request for performing this configuration.

[mustafa-be]

I have a request that im making from cli against keycloak api using

curl -d 'client_id=admin-cli&username=admin&password=a1b2c3&grant_type=password' https://keycloak-instance/realms/master/protocol/openid-connect/token

creating this resource as replacement for above command but it fails

apiVersion: http.crossplane.io/v1alpha1
kind: DesposibleRequest
metadata:
  name: example-disposable-request
spec:
  deletionPolicy: Orphan
  forProvider:
    waitTimeout: 5m
    rollbackRetriesLimit: 5
    url:  https://keycloak-instance/realms/master/protocol/openid-connect/token
    method: POST
    body: |
            grant_type=password&client_id=admin-cli&password=a1b2c3&username=admin
    headers:
      Content-Type:
        - application/x-www-form-urlencoded
      Accept:
        - application/json
  providerConfigRef:
    name: http-conf

it might be something with the api but your insight would be helpful if any.

[arielsepton]

Could you please share the status of the DisposableRequest resource? It contains the response from the server, which will help us troubleshoot the issue.

[mustafa-be]

Please find the error below :

  response:
    body: >-
      {"error":"invalid_grant","error_description":"Invalid user credentials"}

Edit :
API gives this erorr as well (upon changing the parameters order)

{"error":"unsupported_grant_type","error_description":"Unsupported
      grant_type"}

[arielsepton]

It seems the request was sent successfully, indicating a potential issue with the request payload.

Before proceeding, could you please provide the entire status of the DisposableRequest resource? This will help us diagnose the problem more accurately.

[mustafa-be]

Please find the status of DesposableRequest resource below,
Note that the difference among these statuses is order of parameters. But you can still confirm at your end.

status:
  conditions:
    - lastTransitionTime: '2024-03-19T14:06:23Z'
      reason: Available
      status: 'True'
      type: Ready
    - lastTransitionTime: '2024-03-19T14:07:23Z'
      reason: ReconcileSuccess
      status: 'True'
      type: Synced
  failed: 5
  requestDetails:
    body: |
      grant_type=password&client_id=admin-cli&password=admin&username=admin
    headers:
      Accept:
        - application/json
      Content-Type:
        - application/x-www-form-urlencoded
    method: POST
    url: >-
      https://keycloak-instance/realms/master/protocol/openid-connect/token
  response:
    body: '{"error":"invalid_grant","error_description":"Invalid user credentials"}'
    headers:
      Referrer-Policy:
        - no-referrer
      X-Xss-Protection:
        - 1; mode=block
      Strict-Transport-Security:
        - max-age=31536000; includeSubDomains
      X-Frame-Options:
        - SAMEORIGIN
      Content-Type:
        - application/json
      Set-Cookie:
        - >-
          63a25785f19a75e5b651f66a19c0b7aa=412610dce75fb0cc0c737424680b7a9f;
          path=/; HttpOnly; Secure; SameSite=None
      Cache-Control:
        - no-store
      X-Content-Type-Options:
        - nosniff
      Content-Length:
        - '72'
      Pragma:
        - no-cache
    statusCode: 401
  synced: true

status:
  conditions:
    - lastTransitionTime: '2024-03-19T14:52:06Z'
      reason: Available
      status: 'True'
      type: Ready
    - lastTransitionTime: '2024-03-19T14:52:36Z'
      reason: ReconcileSuccess
      status: 'True'
      type: Synced
  failed: 5
  requestDetails:
    body: |
      client_id=admin-cli&username=admin&password=admin&grant_type=password
    headers:
      Accept:
        - application/json
      Content-Type:
        - application/x-www-form-urlencoded
    method: POST
    url: >-
      https://keycloak-instance/realms/master/protocol/openid-connect/token
  response:
    body: >-
      {"error":"unsupported_grant_type","error_description":"Unsupported
      grant_type"}
    headers:
      Referrer-Policy:
        - no-referrer
      X-Xss-Protection:
        - 1; mode=block
      Strict-Transport-Security:
        - max-age=31536000; includeSubDomains
      X-Frame-Options:
        - SAMEORIGIN
      Content-Type:
        - application/json
      Set-Cookie:
        - >-
          63a25785f19a75e5b651f66a19c0b7aa=412610dce75fb0cc0c737424680b7a9f;
          path=/; HttpOnly; Secure; SameSite=None
      Cache-Control:
        - no-store
      X-Content-Type-Options:
        - nosniff
      Content-Length:
        - '79'
      Pragma:
        - no-cache
    statusCode: 400
  synced: true

[arielsepton]

Thanks for providing the status.
Looking at the requestDetails, it appears that the request sent from the provider closely resembles the curl command you mentioned.

  requestDetails:
    body: |
      client_id=admin-cli&username=admin&password=admin&grant_type=password
    headers:
      Accept:
        - application/json
      Content-Type:
        - application/x-www-form-urlencoded
    method: POST
    url: >-
      https://keycloak-instance/realms/master/protocol/openid-connect/token

The only differences I noticed are the Accept header and the password in the request body. These variances could be causing the issue. However, since the request was successfully sent, the problem likely doesn’t lie with the provider. If you have any other insights or need further assistance, feel free to let me know!

[mustafa-be]

hey @arielsepton, thanks alot for your support

I tried sending the same request with kind: Request and its working, I'm guessing that kind: DisposableRequest somehow messes the request body before sending it.

[arielsepton]

I’m glad to hear that it worked for you with the Request resource!

However, it’s unlikely that the request body is being altered differently between the two resources, especially since the same HTTP client class is used to perform the request in both cases.
That said, there might be other factors at play here. One possibility could be the lack of support for jq filters in DisposableRequest.
If you need further explanations or assistance, don’t hesitate to reach out!

[mustafa-be]

hmmm, might be possible

I did had another question, Whats a better way to provide credentials to this resource ? currently im including creds in body, but i would want it to be read from a secret as variables which are then specified in the body

The examples in this repo use injectedidentity that i dont really understand.

[arielsepton]

Currently, you can achieve this by leveraging provider-kubernetes. You can refer to this example for guidance: link.

While the feature of using secrets data directly is still in development, utilizing provider-kubernetes can fulfill this requirement in the interim.