Error flatmap_filter - move value in array

[dan5e3s6ares]

Environment info:

Running with DockerFile

devopsfaith/krakend:2.8.0 as builder
...
CMD ["/usr/bin/krakend", "run", "-d", "-c", "/etc/krakend/krakend.json", "-p", "8080"]

Describe the bug
When applying :

"flatmap_filter": {
        {
            "type": "move",
            "args": [
                "collection.*.A.B.*.C.*.D.*.E.*.F",
                "collection.*.A.B.*.C.*.D.*.G"
            ]
        }
}

To the Payload:

[
    {
        "A": {
            "B": [
                {
                    "C": [
                        {
                            "D": [
                                {
                                    "F": 22,
                                }
                            ],
                        },
                        {
                            "D": [
                                {
                                    "F": 33,
                                }
                            ],
                        },
                        {
                            "D": [
                                {
                                    "F": 44,
                                }
                            ],
                        },
                    ],
                }
            ]
        }
    }
]

The result is:

[
    {
        "A": {
            "B": [
                {
                    "C": [
                        {
                            "D": [
                            ],
                        },
                        {
                            "D": [
                            ],
                        },
                        {
                            "D": [
                                    "F": 44
                            ],
                        },
                    ],
                }
            ]
        }
    }
]

Your configuration file:

We use krakend.tmpl

"$schema": "https://www.krakend.io/schema/v2.5/krakend.json",

Commands used
How did you start the software?

CMD ["/usr/bin/krakend", "run", "-d", "-c", "/etc/krakend/krakend.json", "-p", "8080"]

Expected behavior
The return should be:

[
    {
        "A": {
            "B": [
                {
                    "C": [
                        {
                            "D": [
                                    "F": 22
                            ],
                        },
                        {
                            "D": [
                                    "F": 33
                            ],
                        },
                        {
                            "D": [
                                    "F": 44
                            ],
                        },
                    ],
                }
            ]
        }
    }
]

[thedae]

Hi @dan5e3s6ares

Before digging deeper, I've noticed your expected behavior is not correct, can you please double check what are you actually expecting to see?

On the other hand, have you checked that your transformation is valid according to the documentation? https://www.krakend.io/docs/backends/flatmap/#notation-by-example

[dan5e3s6ares]

Hi, so, the expected behave is the same as this command:

  {
      "type": "move",
      "args": ["schools.*.title", "schools.*.name"]
  },

Sure, I'm trying to move to another structure, but I don`t understand why the actual result is according with the described in the result section and not according with the Expected behavior section.

[thedae]

Hi, so what you're expecting to see is this output right?

[
    {
        "A": {
            "B": [
                {
                    "C": [
                        {
                            "D": [
                                {
                                    "G": 22
                                }
                               .....
                            ]
                        }
                    ]
                }
            ]
        }
    }
]

[thedae]

In any case, and, in order to check if this is indeed a bug and needs some fixing, could you please modify this config file to reproduce your case?

{
  "$schema": "https://www.krakend.io/schema/v2.8/krakend.json",
  "version": 3,
  "endpoints": [
    {
      "endpoint": "/__internal/data",
      "backend": [
        {
          "url_pattern": "/__internal/schools"
        }
      ],
      "extra_config": {
        "proxy": {
          "static": {
            "strategy": "always",
            "data": {
              "schools": [
                {
                  "title": "School A"
                },
                {
                  "title": "School B"
                }
              ],
              "meta": []
            }
          }
        }
      }
    },
    {
      "endpoint": "/flatmap",
      "backend": [
        {
          "url_pattern": "/__internal/data",
          "host": [
            "http://localhost:8080"
          ],
          "extra_config": {
            "proxy": {
              "flatmap_filter": [
                {
                  "type": "move",
                  "args": [
                    "schools.*.title",
                    "schools.*.name"
                  ]
                }
              ]
            }
          }
        }
      ]
    }
  ]
}

[dan5e3s6ares]

Input

[{
    "A": {
        "B": [
            {
                "C": [
                    {
                        "D": [
                            {
                                "E": [
                                    {
                                        "F": 22
                                    }
                                ]
                            }
                        ]
                    },
                    {
                        "D": [
                            {
                                "E": [
                                    {
                                        "F": 33
                                    }
                                ]
                            }
                        ]
                    },
                    {
                        "D": [
                            {
                                "E": [
                                    {
                                        "F": 44
                                    }
                                ]
                            }
                        ]
                    }
                ]
            }
        ]
    }
}]

Output

[{
    "A": {
        "B": [
            {
                "C": [
                    {
                        "D": [
                            {
                                "G": 22
                            }
                        ]
                    },
                    {
                        "D": [
                            {
                                "G": 33
                            }
                        ]
                    },
                    {
                        "D": [
                            {
                                "G": 44
                            }
                        ]
                    }
                ]
            }
        ]
    }
}]

"flatmap_filter": {
        {
            "type": "move",
            "args": [
                "collection.*.A.B.*.C.*.D.*.E.*.F",
                "collection.*.A.B.*.C.*.D.*.G"
            ]
        }
}

[dan5e3s6ares]

{
    "$schema": "https://www.krakend.io/schema/v2.8/krakend.json",
    "version": 3,
    "endpoints": [
        {
            "endpoint": "/v1/version/pudate",
            "method": "POST",
            "output_encoding": "json-collection",
            "input_headers": [
                "X-Key",
                "X-ID",
                "User-Agent"
            ],
            "backend": [
                {
                    "url_pattern": "/version/update",
                    "host": [
                        "{{ .env.services.host }}"
                    ],
                    "is_collection": true,
                    "extra_config": {
                        "backend/http": {
                            "return_error_code": true
                        },
                        "flatmap_filter": {
                            {
                                "type": "move",
                                "args": [
                                    "collection.*.A.B.*.C.*.D.*.E.*.F",
                                    "collection.*.A.B.*.C.*.D.*.G"
                                ]
                            }
                        }
                    },
                    "method": "POST"
                }
            ]
        }
    ]
}

[thedae]

I cannot reproduce your problem with that config, in fact, that config does not work at all, there are some problems:

• The flatmap_filter configuration doesn't have a valid json syntax
• The /version/update response is a collection according to the is_collection flag, but the provided response mock above is not a collection

It'd be really helpful if you could provide a valid configuration aligned with the expected response of /version/update backend in order to debug the issue

Thanks

[dan5e3s6ares]

• The /version/update response is a collection according to the is_collection flag, but the provided response mock above is not a collection

Input and Output are collections, you`re right , I've added an update above.

• The flatmap_filter configuration doesn't have a valid json syntax

I'm already using it this way, but the end result is not what I expected, however it works!

[thedae]

I'm still not able to reproduce:

$ krakend run -c krakend.json 
ERROR parsing the configuration file: 'krakend.json': invalid character '{' looking for beginning of object key string, offset: 592, row: 24, col: 15

Please check your config file and provide a proper one so we can check the issue, thanks

[dan5e3s6ares]

First of all, thank you very much for your attention.

Sorry for the delay in responding, below, copied from inside the container, file krakend.json

{
    "@comment": "This file was auto generated from 'krakend.tmpl' file",
    "$schema": "https://www.krakend.io/schema/v2.5/krakend.json",
    "name": "platform-api",
    "version": 3,
    "host": [
        "http://api:8000"
    ],
    "timeout": "60s",
    "cache_ttl": "300s",
    "extra_config": {
        "router": {
            "return_error_msg": true,
            "auto_options": true
        },
        "telemetry/opentelemetry": {
            "service_name": "ABC_API",
            "service_version": "1",
            "metric_reporting_period": 1,
            "trace_sample_rate": 0.15,
            "exporters": {
                "otlp": [
                    {
                        "name": "apm",
                        "host": "apm.elastic.svc.cluster.local",
                        "port": 8200,
                        "use_http": false
                    }
                ]
            },
            "layers": {
                "global": {
                    "disable_metrics": false,
                    "disable_traces": false,
                    "disable_propagation": false
                },
                "proxy": {
                    "disable_metrics": false,
                    "disable_traces": false
                },
                "backend": {
                    "metrics": {
                        "disable_stage": false,
                        "round_trip": true,
                        "read_payload": true,
                        "detailed_connection": true
                    },
                    "traces": {
                        "disable_stage": false,
                        "round_trip": true,
                        "read_payload": true,
                        "detailed_connection": true
                    }
                }
            }
        },
        "security/cors": {
            "allow_origins": [
                "https://developer.com.br"
            ],
            "allow_methods": [
                "GET",
                "HEAD",
                "POST",
                "PATCH"
            ],
            "expose_headers": [
                "Content-Length",
                "Content-Type"
            ],
            "allow_headers": [
                "Accept-Language"
            ],
            "max_age": "12h",
            "allow_credentials": false,
            "debug": true
        },
        "auth/validator": {
            "shared_cache_duration": 3600
        },
        "telemetry/logging": {
            "level": "DEBUG",
            "prefix": "[KRAKEND]",
            "syslog": false,
            "stdout": true
        }
    },
    "plugin": {
        "pattern": ".so",
        "folder": "./plugins/"
    },
    "endpoints": [
        {
            "endpoint": "/flows",
            "method": "POST",
            "output_encoding": "json-collection",
            "input_headers": [
                "My-Key",
                "My-ID",
                "User-Agent"
            ],
            "backend": [
                {
                    "url_pattern": "/flow",
                    "host": [
                        "http://host.docker.internal:8000"
                    ],
                    "is_collection": true,
                    "extra_config": {
                        "backend/http": {
                            "return_error_code": true
                        },
                        "proxy": {
                            "flatmap_filter": [
                                {
                                    "type": "move",
                                    "args": [
                                        "collection.*.A.B.*.C.*.D.*.E.*.F",
                                        "collection.*.A.B.*.C.*.D.*.G"
                                    ]
                                }
                            ]
                        }
                    },
                    "method": "POST"
                }
            ]
        }
    ]
}

[thedae]

Hi @dan5e3s6ares

I was able to reproduce, and I'm afraid you cannot accomplish what you want with the flatmap_filter

My recommendation here would be to use the modifier/lua-backend to intercept the payload and modify it to your needs using custom code. Check the documentation about Lua modifiers here: https://www.krakend.io/docs/endpoints/lua/