Compressed block headers & merkle proofs over low bandwidth communications

[remyers]

This issue by @willcl-ark was closed in the kyuupichan/eletrumx parent fork without explanation by the maintainer just before he declared that he would not support Bitcoin anymore.

Is there any interest in this idea here? For supporting low-bandwidth light clients this would be very useful.

[SomberNight]

I have written up an Electrum Protocol proposal which seeks to improve both:
(1) Reduce RPC bandwidth requirements (compressed block headers + protobuf/CBOR (de)serialisation)
(2) Reduce requirements of Clients making calls (removing block_height requirement)

I think the blockchain.transaction.get_merkle proposal (2) could be useful and it is a relatively small change anyway.
To make it even less intrusive, and to minimise my performance concern (kyuupichan/electrumx#1007 (comment)), the block_height param could be made optional: it would be used if passed (saving bitcoind RPC calls) but queried from bitcoind if absent (incurring higher DOS cost to the session).
If you contributed this change, I think we would definitely be interested in it!

---

Re (1), for Bitcoin, I am not really convinced that the savings are worth the complexity, and clearly it would be a huge change to the protocol. @JeremyRand, for example, is interested in it for Namecoin, where due to different header construction it might lead to huge savings I suppose. I take it that you yourself would want it for Bitcoin though... Have you done the math for the savings?

I am not saying we are against the change, but how much interest is there in this, and is someone willing to contribute the code?

Perhaps the two changes in (1) (change json encoding and compressed headers) should be considered separately.
I find it nice that the protocol is lightweight and hardly requires any dependency, re encoding change, introducing protobuf or something for CBOR would go against that. It's a tradeoff that should be considered.

OTOH, compressing the headers would be cheap and relatively easy to do but is that worth the savings while keeping the 2x overhead from hex as string?
Though I suppose header-compression could be introduced as an option.

[remyers]

Thanks for the feedback!

I am interested in (1) to support Electrum Bitcoin clients using a low-bandwidth (radio) transport where every byte counts.

I understand the concern about introducing dependencies/complexity and I would lean towards CBOR as a lighter weight binary alternative to JSON rather than protobufs.

Making the changes to remove block height for (2) is also intended to minimize bandwidth because it allows a low-bandwidth client to confirm a tx with a single call (assuming they have block headers sync'd). It would still be useful by itself even without (1) so that's a good suggestion for a place to start.

It looks like any alternative over-the-wire encoding would have to be introduced at the aiorpcX layer and an issue there already captures this idea.

[JeremyRand]

Re (1), for Bitcoin, I am not really convinced that the savings are worth the complexity, and clearly it would be a huge change to the protocol. @JeremyRand, for example, is interested in it for Namecoin, where due to different header construction it might lead to huge savings I suppose. I take it that you yourself would want it for Bitcoin though... Have you done the math for the savings?

@SomberNight So, I think there are two different factors: motivation, and percent savings.

In terms of motivation, Namecoin may be more highly motivated than Bitcoin to save bandwidth, because of the very stringent performance requirements that Tor Browser imposes (i.e. it needs to fully sync from scratch by the time the user finishes typing in a URL after opening Tor Browser) as well as the fact that Namecoin headers are larger than Bitcoin headers. However, in terms of percent savings in bandwidth, application-layer compressed headers are not going to be as effective in Namecoin as in Bitcoin, because the bulk of Namecoin headers consist of Bitcoin coinbase transactions, which aren't subject to application-layer compression like Bitcoin headers are. (CBOR encoding and DEFLATE compression will probably have comparable percent savings between Namecoin and Bitcoin, of course.)

I am not saying we are against the change, but how much interest is there in this, and is someone willing to contribute the code?

Namecoin is potentially willing to contribute the code for CBOR and DEFLATE (looks like NLnet may fund us to work on it, probably within the next 6 months). Note that this is mainly a change in aiorpcx, not ElectrumX. We do not currently have funding for application-layer compressed headers (which would be an ElectrumX + Electrum patch), so I'm not sure when we would get to that, but if we end up with funding for it, we would be happy to implement it.

[SomberNight]

(2) Reduce requirements of Clients making calls (removing block_height requirement)

I think the blockchain.transaction.get_merkle proposal (2) could be useful and it is a relatively small change anyway.

Note: I am working on substantial db changes that will give us this change for free. (well, a single LevelDB.get() compared to previously needing a bitcoind RPC + fs reads by bitcoind)
I will include this change in a new protocol version when I am done, which will include other stuff too.