Equivalents to struct.calcsize() and struct.iter_unpack()?

[James-E-A]

I'm wondering if there's any equivalent to struct.calcsize(), to deduce the number of bits for a fmt without the roundabout process of generating fake valid values.

I need this function because in my use-case the fmt is constructed at run-time, and is not known at development-time.

Currently, the horrible "solution" I have is this:

def bitstring_calcsize(fmt):
    # FIXME https://github.com/scott-griffiths/bitstring/discussions/316
    bs = BitArray()
    b_0 = Bits(1)
    while True:
        try:
            bs.unpack(fmt)
        except bitstring.exceptions.ReadError:
            bs += b_0
        else:
            break
    return len(bs)

---

On the same note, anything for struct.iter_unpack()?

def bitstring_iter_unpack(buffer_or_file: 'Union[Buffer, BinaryIO]', fmt, *, pad_reads_to_byte_boundary=False):
    # FIXME https://github.com/scott-griffiths/bitstring/discussions/316
    bs = ConstBitStream(buffer_or_file)
    size = bitstring_calcsize(fmt)
    if pad_reads_to_byte_boundary:
        size += ((8 - size) % 8)
    assert bs.length % size == 0
    while bs.pos != bs.length:
        yield bs.read(size).unpack(fmt)

[scott-griffiths]

Hi. Those are both very good questions.

For the first part, It would be easier to use pack to create a dummy bitstring and just measure the length of that. You would need to know how many values to pack (and their types) but it's slightly less horrible?

In general however there is no way to do a calcsize as the length of some data types depends on their values - in particular you don't know how long exponential-Golomb codes are going to be until you read them. There is room for a method for when it is possible though, and I think it's a good idea. I've recently started work on a new project called bitformat where this sort of thing would be easy. Possibly I could port that new functionality into bitstring, but that's fairly long-term.

For the iter_unpack suggestion, I'm actually surprised I've not already done that, and it should be fairly easy to implement. I'll tag this as a feature request for the next version. There may be a simpler work-around to do it now but I can't think of one...

Thanks.

[James-E-A]

In general … there is no way to do a calcsize as the length of some data types depends on their values … There is room for a method for when it is possible though, and I think it's a good idea.

Yeah, a TypeError or something if the field contains a non-constant size would be ideal (in the absence of any kwargs specifically permitting them), for sure.

For the iter_unpack suggestion, I'm actually surprised I've not already done that, and it should be fairly easy to implement.

The implementation I posted depends on calcsize. Are you saying that it would be pretty straightforward to implement iter_unpack without that dependency?

[scott-griffiths]

If there are no data types with unknown lengths then the iter_unpack should be straightforward, and the ordinary unpack just becomes list(iter_unpack(...)). As well as data types that intrinsically don't have a known length there is also the concept of a 'stretchy' token when a single data type doesn't have a set length and expands to fill the gap.

For the simple case it should be fairly easy to code without doing any calcsize, but would need some internal refactoring so it's hard to give a simple workaround.

[scott-griffiths]

Having now read the docs for struct.iter_unpack (which perhaps I should have done earlier) it doesn't do what I expected. I thought it would be an iterator for each item in the struct, not just an iterator for multiple reads of the same whole struct.

Which means that the equivalent is much easier to do in bitstring, but also less interesting (to me). This was my first attempt as a new method in the Bits class, which is a bit rough but worked for what I tried it with.

    def iter_unpack(self, fmt, **kwargs):
        pos = 0
        while True:
            try:
                bs, pos = self._readlist(fmt, pos, **kwargs)
            except ValueError:
                return
            yield bs

The thing to notice is that unpack is very closely related to readlist, so you can just use a ConstBitStream and call readlist multiple times with the same format.