[Draft] New Block Reconstruction Protocol Proposal #4051

walldiss · 2025-01-20T11:40:44Z

This PR introduces a draft proposal for a new block reconstruction protocol aimed at improving network efficiency and scalability.

Key improvements:

Reduces duplicate requests and bandwidth usage
Structured bitmap sharing using Roaring Bitmaps
Clear separation of client/server responsibilities
Robust failure handling with peer penalties
Optimized proof packaging and sharing

The document serves as a starting point for discussion and will be later split into CIP and ADR.

codecov-commenter · 2025-01-20T11:48:27Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 44.85%. Comparing base (2469e7a) to head (e506dc6).
Report is 433 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4051      +/-   ##
==========================================
+ Coverage   44.83%   44.85%   +0.02%     
==========================================
  Files         265      308      +43     
  Lines       14620    22488    +7868     
==========================================
+ Hits         6555    10088    +3533     
- Misses       7313    11316    +4003     
- Partials      752     1084     +332

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

musalbas · 2025-01-20T18:40:14Z

Looks generally fine to me directionally. Networking protocol should probably be in a CIP as it's relevant to other clients like lumina etc.

Wondertan · 2025-01-20T19:13:05Z

docs/adr/reconstruction.md

+
+Handshakes, Node Type Identification
+- Node type must be declared during connection handshake by each of connection nodes
+- Active connection state maintenance on full nodes. List should be maintained by node type
+- Connection state should allow subscription for updated


Whats the rationale behind sharing your node type? Is it possible to avoid sharing this information with peers?

There is a privacy concern we discussed a while ago with @musalbas and @adlerjohn. FNs ideally should be indistinguishable from LNs on p2p network, so they validators don't detect them.

If we don't think that's important anymore or sharing type is unavoidable, I also propose to cut the handshake roundtrip and rely on libp2p toolset for that.

That is indeed a goal, but the problem here is that it's implicit in this protocol that only full nodes request bitmaps from others anyway, regardless of how you identify in the handshake.

FNs pretending to be LNs (to reconstruct) would only work if the FNs spam sample requests to the block producer, which isn't something implemented yet afaik. (See https://forum.celestia.org/t/security-levels-for-data-availability-for-light-nodes/919#level-3-data-availability-sampling-without-an-honest-minority-of-light-nodes-6)

That is indeed a goal

if the FNs spam sample requests to the block producer, which isn't something implemented yet afaik.

Do you think this should be in scope of current iteration of reconstruction protocol?

I think not in this iteration, because it seems we're doing pretty good on increasing number of light nodes, so we could just focus directly on having a robust construction protocol and achieving Level 4 light nodes. It might be nice to have later as an option, though, but not high priority right now.

Good point, handshake is not necessary for initial basic implementation. It will be needed in peer scoring optimisation. When FN would need to know the type of peer for prioritisation of requests. I'll mark handshake protocol as future optimisation.

renaynay

Would this proposal also require some modifications in discovery + peer management as it currently stands? If so, can you please outline what that would look like?

renaynay · 2025-01-21T12:38:11Z

docs/adr/reconstruction.md

+    rows []peers        // Peers with row data
+    cols []peers        // Peers with column data


how is this actually going to be structured?

you keep a []peer.ID per which row they claim to have? So index 0 []peer.ID will be peers that claim to have row 0?

Yes. The idea is to have efficient look up for peers who have full row or full col.

renaynay · 2025-01-21T12:39:21Z

docs/adr/reconstruction.md

+
+Local state:
+- InProgress() -> bitmap. Tracks ongoing fetch sessions to prevent requesting duplicates
+- Have() -> bitmap.  Tracks 


Suggested change

- Have() -> bitmap. Tracks

- Have() -> bitmap. Tracks node's locally available samples

renaynay · 2025-01-21T12:40:10Z

docs/adr/reconstruction.md

+    - Pre-confirmation threshold
+    - Combination of conditions
+
+4.1.1 what and were (to request) biggest question.


what and where ?

renaynay · 2025-01-21T12:41:06Z

docs/adr/reconstruction.md

+4.2.1. Request 
+
+- Update InProgress bitmap before request initiation
+- Use shrex for data retrieval. Send bitmap for data request. Shrex has built-in support for requesting rows and samplies, however bitmap-based data retrieval is more general and would be easier to support in future, because it allows requesting multiples pieces of data with single request instead of multiple requests


Suggested change

- Use shrex for data retrieval. Send bitmap for data request. Shrex has built-in support for requesting rows and samplies, however bitmap-based data retrieval is more general and would be easier to support in future, because it allows requesting multiples pieces of data with single request instead of multiple requests

- Use shrex for data retrieval. Send bitmap for data request. Shrex has built-in support for requesting rows and samples, however bitmap-based data retrieval is more general and would be easier to support in future, because it allows requesting multiples pieces of data with single request instead of multiple requests

renaynay · 2025-01-21T12:42:20Z

docs/adr/reconstruction.md

+Optiomisations:
+- Server may split response into multiple parts
+Each part contains packed samples in Range format.
+- Each part of shoold have specified message prefix. It would allow client to identify sample container type, which would help to maintain backwards compatibility in case of breaking changes in packing algorithm


Suggested change

- Each part of shoold have specified message prefix. It would allow client to identify sample container type, which would help to maintain backwards compatibility in case of breaking changes in packing algorithm

- Each part of should have specified message prefix. It would allow client to identify sample container type, which would help to maintain backwards compatibility in case of breaking changes in packing algorithm

renaynay · 2025-01-21T12:42:41Z

docs/adr/reconstruction.md

+4.2.2. Response 
+- (base version) Server should respond with samples with proofs
+
+Optiomisations:


Suggested change

Optiomisations:

Optimisations:

renaynay · 2025-01-21T12:43:43Z

docs/adr/reconstruction.md

+### Server-Side Implementation
+
+#### 1. Storage Interface
+New storage format needs to be implemented for efficient storage of sampels proofs. The format will be used


Suggested change

New storage format needs to be implemented for efficient storage of sampels proofs. The format will be used

New storage format needs to be implemented for efficient storage of sample proofs. The format will be used

renaynay · 2025-01-21T12:43:54Z

docs/adr/reconstruction.md

+
+#### 1. Storage Interface
+New storage format needs to be implemented for efficient storage of sampels proofs. The format will be used
+initially used for storing ongoing reconstruction process and later can be used for light node storage.


Suggested change

initially used for storing ongoing reconstruction process and later can be used for light node storage.

initially for storing ongoing reconstruction process and later can be used for light node storage.

renaynay · 2025-01-21T12:46:43Z

Also generally speaking, would be nice to have a diagram at the end sketching out a reconstruction process e2e.

walldiss · 2025-01-27T14:55:48Z

Would this proposal also require some modifications in discovery + peer management as it currently stands? If so, can you please outline what that would look like?

First implementation is simplified and does need to have any additional peer management. We would still need to have at least some(5) full nodes connected, which is ensured by current implementation of discovery. Requests will be send to random peers, so the only thing that needs to be tracked is all connected peers, which is available from libp2p. It would be easier to see once I add diagrams for FN reconstruction steps.

walldiss

Feedback notes from sync call

walldiss · 2025-01-28T12:26:57Z

docs/adr/reconstruction.md

+```
+
+#### Response
+Server should respond with samples with proofs defined in shwap CIP [past link].


Server should validate returned samples matches requested bitmap.

walldiss · 2025-01-28T12:27:02Z

docs/adr/reconstruction.md

+   - Block size scaling
+   - Node count scaling
+
+## Protocol Flow 


Add general components overview from current node perspective:

daser

availability

shrex getter/ shrex seub

walldiss · 2025-01-28T12:27:18Z

docs/adr/reconstruction.md

+```
+message BitmapUpdate {
+    Bitmap bitmap = 1;
+    bool is_end = 2;


rename is_end -> completed

walldiss · 2025-01-28T12:36:36Z

docs/adr/reconstruction.md

+Remote state:
+```go
+func (s *RemoteState) GetPeersWithSample(coords []Coord) []peer.ID
+func (s *RemoteState) GetPeersWithAxis(axisIdx int, axisType AxisType) []peer.ID
+func (s *RemoteState) Available() bitmap
+```


Add comments for method usacases.

Add that some methods are not necessary for init implementation.

walldiss · 2025-01-28T12:41:28Z

docs/adr/reconstruction.md

+func (s *ProgressState) Have() bitmap
+```
+
+### 3. Bitmap Protocol


Necessary for FN-FN, can be more optimised version for FN-LN. We will start with only subsciption to have working implementation faster.

walldiss · 2025-01-28T13:03:47Z

docs/adr/reconstruction.md

+3. Fallback mechanisms
+
+
+


Add testing plan / requirements

- return handshake based on user agent. - Do not subscribe for samples from LN - Request from LN in batches

walldiss added 2 commits January 20, 2025 12:30

reconstruction draft

eba0199

reconstruction draft

c5ceeec

walldiss self-assigned this Jan 20, 2025

Wondertan reviewed Jan 20, 2025

View reviewed changes

renaynay reviewed Jan 21, 2025

View reviewed changes

refactor

f085b22

walldiss force-pushed the reconstruction-adr branch from efa49f4 to f085b22 Compare January 27, 2025 17:53

refactor 2

dc4c620

walldiss commented Jan 28, 2025

View reviewed changes

walldiss added 2 commits February 3, 2025 12:42

added:

052a31f

- return handshake based on user agent. - Do not subscribe for samples from LN - Request from LN in batches

added:

e506dc6

- return handshake based on user agent. - Do not subscribe for samples from LN - Request from LN in batches

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Draft] New Block Reconstruction Protocol Proposal #4051

[Draft] New Block Reconstruction Protocol Proposal #4051

walldiss commented Jan 20, 2025

codecov-commenter commented Jan 20, 2025 •

edited

Loading

musalbas commented Jan 20, 2025

Wondertan Jan 20, 2025

musalbas Jan 20, 2025 •

edited

Loading

Wondertan Jan 21, 2025 •

edited

Loading

musalbas Jan 21, 2025

walldiss Jan 27, 2025

renaynay left a comment

renaynay Jan 21, 2025

walldiss Jan 27, 2025

renaynay Jan 21, 2025

renaynay Jan 21, 2025

renaynay Jan 21, 2025

renaynay Jan 21, 2025

renaynay Jan 21, 2025

renaynay Jan 21, 2025

renaynay Jan 21, 2025

renaynay commented Jan 21, 2025

walldiss commented Jan 27, 2025

walldiss left a comment

walldiss Jan 28, 2025

walldiss Jan 28, 2025

walldiss Jan 28, 2025

walldiss Jan 28, 2025

walldiss Jan 28, 2025

walldiss Jan 28, 2025

		rows []peers // Peers with row data
		cols []peers // Peers with column data

	- Have() -> bitmap. Tracks
	- Have() -> bitmap. Tracks node's locally available samples

	- Use shrex for data retrieval. Send bitmap for data request. Shrex has built-in support for requesting rows and samplies, however bitmap-based data retrieval is more general and would be easier to support in future, because it allows requesting multiples pieces of data with single request instead of multiple requests
	- Use shrex for data retrieval. Send bitmap for data request. Shrex has built-in support for requesting rows and samples, however bitmap-based data retrieval is more general and would be easier to support in future, because it allows requesting multiples pieces of data with single request instead of multiple requests

	- Each part of shoold have specified message prefix. It would allow client to identify sample container type, which would help to maintain backwards compatibility in case of breaking changes in packing algorithm
	- Each part of should have specified message prefix. It would allow client to identify sample container type, which would help to maintain backwards compatibility in case of breaking changes in packing algorithm

	New storage format needs to be implemented for efficient storage of sampels proofs. The format will be used
	New storage format needs to be implemented for efficient storage of sample proofs. The format will be used

	initially used for storing ongoing reconstruction process and later can be used for light node storage.
	initially for storing ongoing reconstruction process and later can be used for light node storage.

[Draft] New Block Reconstruction Protocol Proposal #4051

Are you sure you want to change the base?

[Draft] New Block Reconstruction Protocol Proposal #4051

Conversation

walldiss commented Jan 20, 2025

codecov-commenter commented Jan 20, 2025 • edited Loading

Codecov Report

musalbas commented Jan 20, 2025

Choose a reason for hiding this comment

musalbas Jan 20, 2025 • edited Loading

Choose a reason for hiding this comment

Wondertan Jan 21, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

renaynay left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

renaynay commented Jan 21, 2025

walldiss commented Jan 27, 2025

walldiss left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov-commenter commented Jan 20, 2025 •

edited

Loading

musalbas Jan 20, 2025 •

edited

Loading

Wondertan Jan 21, 2025 •

edited

Loading