What is the best way to implement a TTLV structure?

[georgePadolsey]

I'm currently trying to implement a Tag-Type-Length-Value structure (TTLV). This is very similar to the TLV structure though it allows you to represent nested structures.

For example let's say I'm trying to represent the structure:

{
    "payload": {
        "uniqueIdentifier": "FooBar",
        "ivData": "SSBsaWtlIGNha2U=",
        "date": 1564135537
    }
}

In TTLV format it looks like this:

[
  {
    "tag": "RESPONSE_PAYLOAD",
    "type": "STRUCTURE",
    "length": 30
  },
  {
    "tag": "UNIQUE_IDENTIFIER",
    "type": "STRING",
    "length": 12
  },
  {
    "value": "FooBar"
  },
  {
    "tag": "IV_DATA",
    "type": "BYTE",
    "length": 10
  },
  {
    "value": "SSBsaWtlIGNha2U="
  },
  {
    "tag": "date",
    "type": "LONG_INTEGER",
    "length": 4
  },
  {
    "value": 1564135537
  }
]

Note this structure could itself be within another structure.

I'm able to implement this in KSY as I can represent each structure tag as it's own Type with it's own Type body. I then say how only certain tag types are allowed within that body. This is what I've come up with so far:

seq:
  - id: items
    type: items
types:
  items:
    seq:
      - id: item
        repeat: eos
        type: item
  item:
    seq:
      - id: itemtag
        enum: item_tags
        type: b24
      - id: itemtype
        enum: item_types
        type: u1
      - id: itemlength
        type: u4
      - id: body_t
        size: itemlength
        type:
          switch-on: itemtag
          cases:
            item_tags::response_payload: resp_tag_body
        if: itemtag == item_tags::response_payload
      - id: body_v
        size: itemlength
        type: 
          switch-on: itemtype
          cases: 
            tag_types::structure: tags
            tag_types::long_integer: s8
            tag_types::string: string
            # bytes are parsed as is
        if: itemtag != item_tags::response_payload
  resp_tag_body:
    seq:
    - id: response_tags
      type: resp_tag
      repeat: eos
  resp_tag:
    seq:
    - id: itemtag
      type: b24
      enum: allowed_resp_tag
    - id: itemtype
      enum: item_types
      type: u1
    - id: itemlength
      type: u4
    - id: body_v
      size: itemlength
      type: 
        switch-on: itemtype
        cases: 
          tag_types::structure: tags
          tag_types::long_integer: s8
          tag_types::string: string
          # Byte strings are just represented as is
  string:
    seq: 
    - id: text
      type: str
      size-eos: true
      encoding: UTF-8

The problem with this is:

• It is very verbose (every structure and structure body requires its own Type) -- not a big issue
• I can't ensure that a structure has the required sub tags (e.g. ResponsePayload may not have the required date attribute)
• I can't ensure that each sub tag can only have one of a certain amount of types. (I.E. Date can only be a LongInteger or Integer). At least not without implementing types for every sub tag. (Update: I realise that the valid attribute may help with this from Validation of parsed data with assertions #435 but would require more complex conditionals being implemented)

Ideally I'd love a solution that allows me to have optional and required sub tags (E.g. date is required but iv_data is optional).

I'm currently looking at applying a secondary parsing with my own parser to ensure the correct attributes are available for each type. However, I would prefer to do it all in kaitai.

Apologies for the long question though any help would be appreciated :)

P.S. I had a look through many of the questions already asked on the repo and didn't seem to come up with anything that was close to my question. I'm really sorry if someones already asked something similar and I've missed it!

[KOLANICH]

Yeah, this problem is currently unsolvable in KS. But we have some related issues:
#458 #435 #434 #208 .

[dgelessus]

Also relevant would be #172, for looking up substructures by their tag.

[georgePadolsey]

Thanks for the replies. It's really annoying as I'd love to use kaitai to represent my entire binary structure. It's hard because I'm not sure how much of this is within the scope of kaitai.
However, because kaitai has no way of implementing this currently it means I need to run an additional parser on top, making it no longer language agnostic. 😢

Looking at past issues it seems there is a drag about what to implement as to not make it a full on programming language. I agree this should be something which slows you down, but when it is stopping the language from parsing any complex TLV / TTLV structures... perhaps it needs to be reconsidered?

I'll leave this issues open for a few more days in the hope that something's been missed or a new feature is about to drop! 🤞