zserio example for protocol with common header and footer (TLV)

[jbrc]

I would like to use zserio to generate code for an existing communication protocol. I think my protocol has some fairly common characteristics (a type-length-value format), but I haven't been able to find a zserio example that reaches this level of complexity. Can anyone guide me in the right direction?

Here are two example messages in this protocol:

Message 1:  [ uint8 command ] [ uint8 length ] [ uint32 payloadOne ] [ uint8 checksum ]
Message 2:  [ uint8 command ] [ uint8 length ] [  int64 payloadTwo ] [ uint8 checksum ]

Notice that the messages have common headers (command, length) and footers (checksum). Since my actual header and footer formats are more complex, I would like to share those definitions between messages. I believe that would look like this:

enum Command { COMMAND_ONE = 1, COMMAND_TWO = 2 };
struct Header { Command cmd; uint8 length; };
struct Footer { uint8 checksum; };

But I am not sure how to express these details to the zserio generator:

• Every instance of Message 1 should have cmd = 1 and length = 4, but the client will supply unique values for payloadOne.
• Similar for Message 2: cmd = 2, length = 8, unique values for payloadTwo.

After the generator runs, I would like clients to be able to write Python code like this:

m1 = tlv.MessageOne(42) # cmd is automatically set to 1, length automatically set to 4, payloadOne = 42
m1.write(writer)
m2 = tlv.MessageTwo(75) # cmd is automatically set to 2, length automatically set to 8, payloadTwo = 75
m2.write(writer)

Questions

1. What's the idiomatic way to express this message = {header, payload, footer} relationship in zserio? Are templates or choice types helpful here?
2. Rather than specifying the lengths in my interface definition file (foo.zs), can zserio calculate and initialize them for me? Something like Header.length = sizeof(payload).
3. To compute the checksum in the footer, it would be convenient to treat the message as an array of bytes rather than structured data. Is a union type the right approach here?
4. Is it possible to override or extend the write function to compute the checksum? And the read function to verify the checksum?

[fklebert]

In case you want to hardwire the lengths of the message payload based on the command, a function might come in handy:

enum uint8 Command
{
  COMMAND_1,
  COMMAND_2,
  COMMAND_3,
};

struct Message
{
  Command command;
  uint8 payload[sizeOfPayload()];
  uint8 checksum[];
  
  // returns size of payload in bytes
  function uint8 sizeOfPayload()
  {
    return (command == Command.COMMAND_1) ? 4 :
           (command == Command.COMMAND_2) ? 8 :
           (command == Command.COMMAND_3) ? 24 : 0;
  }
};

This version does not excplicitely store the length of the array in the byte stream, as it is always fixed based on the command.
The checksum field uses an so-called auto-array, which simply sets the size (it has a hidden size field in front of the checksum fiels in the final byte stream) based on the content length the user sets in the writer code. In case you want to hard-wire this to a specific length, e.g. in case you always use the same checksum generator (e.g. CRC32) you can set this for example to uint8 checksum[4]; That would then be a 4-byte array to which you will need to store the crc32.

Regarding the length you can indeed do something like that. There is a lengthof operator (see https://zserio.org/doc/ZserioLanguageOverview.html#lengthof-operator).

At the moment there is no way to extend the generated code by anything that you would put in the schema. But of course you can manually inject code after code generation to do so. We know of projects that do something similar in a post-schema-generation step.

[mikir]

To compute the checksum in the footer, it would be convenient to treat the message as an array of bytes rather than structured data. Is a union type the right approach here?

Unfortunately, unions are not designed for this. Actually, unions are auto choices.

To compute the checksum in the footer, it will be necessary to serialize message in the two steps. The first step will serialize the header and the payload, the second one will serialize everything or maybe only the checksum. It is not nice but it will work.

Example:

struct MessageOne
{
    Header header;
    PayloadOne payload;
};

struct MessageOneWithChecksum
{
    MessageOne headerAndPayload;
    Checksum checksum;
};

Templates could simplify things a little bit:

struct Message<PAYLOAD>
{
    Header header;
    PAYLOAD payload;
};

struct MessageWithChecksum<PAYLOAD>
{
    Message<PAYLOAD> headerAndPayload;
    Checksum checksum;
};