The "formatDate" to UNIX on httpjson is not working on multiple version (7.17.0-8.11.3)

[nitzango]

Hi team,

I'm trying to pull data from an API that expect UNIX time as a query parameter (e.g. oldest event to pull).
While setting '[[ formatDate (now (parseDuration "-4h")) ]]' as the default for the first time run, it seems like the actual parameter is not transformed to UNIX as stays normal timestamp (I used ngrok to debug):

latest=Thu+Dec+21+09%3A30%3A28+UTC+2023

I tried to use the following filebeat config:

filebeat.inputs:
  - type: httpjson
    config_version: 2
    interval: 10m

    request.method: POST
    request.url: https://url.goes.here

    request.transforms:
    - set:
        target: header.Authorization
        value: "Bearer 123456"
    - set:
        target: header.Accept
        value: "application/json"
    - set:
        target: url.params.latest
        value: '[[.cursor.last_event_time]]'
        default: '[[ formatDate (now (parseDuration "-4h")) ]]'

For confirmed bugs, please report:

• Version: multiple version (7.17.0-8.11.3)
• Operating System: Docker on macOS + Kubernetes
• Steps to Reproduce: Use the config above

[botelastic]

This issue doesn't have a Team:<team> label.

[levi-burton]

Are there any updates on this issue? I am running the Elastic Agent on version '8.11.3" and also observing the same issue where I need to provide a request transform including a dynamic Unix timestamp but instead see a human readable timestamp is being passed when viewing the debug logs.

[efd6]

I have confirmed that the default behaves as intended by adding a test to the transforms test suite.

..at/input/httpjson [master|✚] $ git diff            
diff --git a/x-pack/filebeat/input/httpjson/value_tpl_test.go b/x-pack/filebeat/input/httpjson/value_tpl_test.go
index 4b642a1697..8bf8dbd2cb 100644
--- a/x-pack/filebeat/input/httpjson/value_tpl_test.go
+++ b/x-pack/filebeat/input/httpjson/value_tpl_test.go
@@ -56,6 +56,15 @@ func TestValueTpl(t *testing.T) {
                        paramDefVal: "",
                        expectedVal: "25",
                },
+               {
+                       name:        "date",
+                       setup:       func() { timeNow = func() time.Time { return time.Unix(1604582732, 0).UTC() } },
+                       teardown:    func() { timeNow = time.Now },
+                       value:       `[[ formatDate (now (parseDuration "-4h")) ]]`,
+                       paramCtx:    emptyTransformContext(),
+                       paramTr:     transformable{},
+                       expectedVal: "2020-11-05T09:25:32Z",
+               },
                {
                        name:  "can render default value if execute fails",
                        value: "[[.last_response.body.does_not_exist]]",
..at/input/httpjson [master|✚] $ go test -run TestValueTpl/date
PASS
ok  	github.com/elastic/beats/v7/x-pack/filebeat/input/httpjson	0.367s

My guess is that what is happening is that the value from the .cursor.last_event_time has not been formatted correctly and you are seeing that.