feat: support gRPC endpoints in core #1513

rach-id · 2024-10-08T09:44:17Z

This is an implementation of a streaming API for blocks in core.

Helps close celestiaorg/celestia-app#3421 but not sure it entirely closes it.

It can easily be used:

package main

import (
	"context"
	"fmt"
	coregrpc "github.com/tendermint/tendermint/rpc/grpc"
)

func main() {
	client := coregrpc.StartBlockAPIGRPCClient("tcp://localhost:9090")

	blockStreamer, err := client.BlockByHeight(context.Background(), &coregrpc.BlockByHeightRequest{Height: 2})
	if err != nil {
		panic(err)
	}
	blockMeta, err := client.BlockMetaByHeight(context.Background(), &coregrpc.BlockMetaByHeightRequest{Height: 2})
	if err != nil {
		panic(err)
	}
	parts := make([]*core.Part, 0)
	for i := 0; i < int(blockMeta.BlockMeta.BlockID.PartSetHeader.Total); i++ {
		resp, err := blockStreamer.Recv()
		if err != nil {
			panic(err)
		}
		parts = append(parts, resp.BlockPart)
		if resp.IsLast && i < int(blockMeta.BlockMeta.BlockID.PartSetHeader.Total)-1 {
			panic("couldn't get all parts")
		} else if resp.IsLast {
			break
		}
	}

	h := types.NewPartSetFromHeader(types.PartSetHeader{
		Total: blockMeta.BlockMeta.BlockID.PartSetHeader.Total,
		Hash:  blockMeta.BlockMeta.BlockID.PartSetHeader.Hash,
	})

	for _, part := range parts {
		ok, err := h.AddPart(&types.Part{
			Index: part.Index,
			Bytes: part.Bytes,
			Proof: merkle.Proof{
				Total:    part.Proof.Total,
				Index:    part.Proof.Index,
				LeafHash: part.Proof.LeafHash,
				Aunts:    part.Proof.Aunts,
			},
		})
		if err != nil {
			panic(err)
		}
		if !ok {
			panic("not okey")
		}
	}
	pbb := new(core.Block)
	bz, err := io.ReadAll(h.GetReader())
	if err != nil {
		panic(err)
	}
	err = proto.Unmarshal(bz, pbb)
	if err != nil {
		panic(err)
	}
	block, err := types.BlockFromProto(pbb)
	if err != nil {
		panic(err)
	}
	fmt.Println(block)

	// get a commit
	commit, err := client.Commit(context.Background(), &coregrpc.CommitRequest{Height: 10})
	if err != nil {
		panic(err)
	}
	fmt.Println(commit)

	// listen for new heights
	streamer, err := client.SubscribeNewHeights(context.Background(), &coregrpc.SubscribeNewHeightsRequest{})
	if err != nil {
		panic(err)
	}
	for {
		resp, err := streamer.Recv()
		if err != nil {
			panic(err)
		}
		fmt.Println(resp)
	}
}

Ps: I didn't add the tests because I didn't find a direct way of mocking the environment without polluting the rest of the repo (exporting some methods, adding new helpers, etc). And I think since the implementation is simple, just querying the block/state stores for results, it's fine to leave it untested.

rach-id · 2024-10-08T11:41:35Z

Just found out that once we enable the app grpc, these endpoints get overridden, I'll be looking into it

proto/tendermint/rpc/grpc/types.proto

rpc/grpc/api.go

@cmwaters

… by @cmwaters

This reverts commit 072b5eb.

rpc/grpc/api.go

rootulp · 2024-12-13T20:08:45Z

rpc/grpc/api.go

+// SubscriptionCapacity the maximum number of pending blocks in the subscription.
+const SubscriptionCapacity = 500
+
+func (blockAPI *BlockAPI) retryNewBlocksSubscription(ctx context.Context) (bool, error) {


[question][nit] what does the bool return argument indicate? It seems like (isSuccess bool, err error) IMO the bool argument is unnecessary b/c either this function returns:

true, no error

false, an error

So the bool argument could be removed and this could just return an error if the retry failed or nil if the retry succeeded

it also returns false, no error when the context is done. This avoids reporting a context done error unnecessarily and just stops quietly

rootulp · 2024-12-13T20:12:03Z

rpc/grpc/api.go

+	if blockAPI.heightListeners == nil {
+		// if this is nil, then there is no need to close anything
+		return
+	}


This check is unnecessary because heightListeners is always non-nil due to:

func NewBlockAPI() *BlockAPI { return &BlockAPI{ heightListeners: make(map[chan NewHeightEvent]struct{}, 1000), subscriptionID: fmt.Sprintf("block-api-subscription-%s", rand.Str(6)), subscriptionQuery: eventstypes.EventQueryNewBlock, } }

and it's never explicitly set to nil so this check will never evaluate to true. Proposal to remove it

Suggested change

if blockAPI.heightListeners == nil {

// if this is nil, then there is no need to close anything

return

}

this is just a foolproof check because the BlockAPI is exported and anyone can instantiate it and mess up initialising the fields

IMO it's unecessary to safe-guard against this b/c the default NewBlockAPI initializes fields correctly but if you want to inform developers of a code error then this should throw an error instead of silently returning.

aah sorry, meant when closing, if the field hasn't been initialized correctly or was set to nil at some level, there is no need to close anything as there is nothing to close

rootulp · 2024-12-13T20:14:20Z

rpc/grpc/api.go

+	var err error
+	// stop the events subscription
+	if blockAPI.newBlockSubscription != nil {
+		err = core.GetEnvironment().EventBus.Unsubscribe(ctx, blockAPI.subscriptionID, blockAPI.subscriptionQuery)


this error isn't explicitly handled, is that intentional? I would expect

Suggested change

err = core.GetEnvironment().EventBus.Unsubscribe(ctx, blockAPI.subscriptionID, blockAPI.subscriptionQuery)

err = core.GetEnvironment().EventBus.Unsubscribe(ctx, blockAPI.subscriptionID, blockAPI.subscriptionQuery)

if err != nil {

// handle the error

}

blockAPI.newBlockSubscription = nil

yes, it's returned at the end of the function. If you prefer the error handling to be done in place, I can change

I think it's safer to do error handling immediately after the error b/c in this case the code would log

"gRPC streaming API has been stopped"

even though there may have been an error during the unsubscribe.

yes, and would return the error that happened during the unsubscribe to be handled separately, which if I am not missing something, is also logged at a higher level.

rootulp · 2024-12-13T20:22:09Z

rpc/grpc/client_server.go

+		opts = append(opts, grpc.WithTransportCredentials(insecure.NewCredentials()))
+	}
+	opts = append(opts, grpc.WithContextDialer(dialerFunc))
+	conn, err := grpc.Dial( //nolint:staticcheck


This can be refactored to eliminate the nolint directive

Suggested change

conn, err := grpc.Dial( //nolint:staticcheck

conn, err := grpc.NewClient(

The problem with this is that it uses an address from the config which has a tcp prefix and when used with NewClient it fails to connect.

So, changing this would mean reparsing the provided address and separating it to host:port.

We can create an issue to do it along with the other gRPC endpoint for consistency instead of touching both in here

Co-authored-by: Rootul P <[email protected]>

evan-forbes

👍

rootulp

Feedback is optional not blocking

rootulp · 2025-01-14T18:55:31Z

rpc/grpc/api.go

+	"github.com/tendermint/tendermint/libs/pubsub"
+
+	"github.com/tendermint/tendermint/crypto/encoding"
+
+	"github.com/tendermint/tendermint/proto/tendermint/crypto"
+
+	"github.com/tendermint/tendermint/libs/rand"



nit: these newlines seem unintentional

Suggested change

"github.com/tendermint/tendermint/libs/pubsub"

"github.com/tendermint/tendermint/crypto/encoding"

"github.com/tendermint/tendermint/proto/tendermint/crypto"

"github.com/tendermint/tendermint/libs/rand"

"github.com/tendermint/tendermint/libs/pubsub"

"github.com/tendermint/tendermint/crypto/encoding"

"github.com/tendermint/tendermint/proto/tendermint/crypto"

"github.com/tendermint/tendermint/libs/rand"

rootulp · 2025-01-14T19:02:34Z

rpc/grpc/api.go

+	if blockAPI.heightListeners == nil {
+		// if this is nil, then there is no need to close anything
+		return
+	}


IMO it's unecessary to safe-guard against this b/c the default NewBlockAPI initializes fields correctly but if you want to inform developers of a code error then this should throw an error instead of silently returning.

rootulp · 2025-01-14T19:03:54Z

rpc/grpc/api.go

+	var err error
+	// stop the events subscription
+	if blockAPI.newBlockSubscription != nil {
+		err = core.GetEnvironment().EventBus.Unsubscribe(ctx, blockAPI.subscriptionID, blockAPI.subscriptionQuery)


I think it's safer to do error handling immediately after the error b/c in this case the code would log

"gRPC streaming API has been stopped"

even though there may have been an error during the unsubscribe.

rootulp · 2025-01-14T19:05:16Z

rpc/grpc/grpc_test.go

@@ -1,9 +1,19 @@
+//nolint:dupl


why disable this linter?

I didn't spend time refactoring the testing suite so it has multiple duplicates. I'll open a good first issue for it

## Description Addresses this #1513 (comment)

feat: working gRPC endpoints

3fa0477

rach-id requested review from walldiss and renaynay October 8, 2024 09:44

rach-id self-assigned this Oct 8, 2024

rach-id requested a review from a team as a code owner October 8, 2024 09:44

rach-id requested review from staheri14 and ninabarbakadze and removed request for a team October 8, 2024 09:44

rach-id mentioned this pull request Oct 8, 2024

Expose all celestia-node required endpoints through gRPC celestiaorg/celestia-app#3421

Closed

2 tasks

rach-id added 5 commits October 8, 2024 13:54

chore: removing buf server streaming rule

e1fd2b4

chore: gofumpt

27152f4

chore: nolint: static check

d703c13

chore: remove server streaming linter rule

e0c794d

chore: regenrate proto

d7bc23b

rach-id marked this pull request as draft October 8, 2024 11:40

feat: support streaming parts

65d936d

walldiss reviewed Oct 8, 2024

View reviewed changes

rach-id added 5 commits October 8, 2024 18:44

chore: adding context

27e3a5d

chore: correctly listening for new blocks

dc10ff9

feat: also share block heights when listening for new blocks

ba5dc9f

feat: better error handling

a490a04

feat: gofumpt

020787b

rach-id requested a review from walldiss October 9, 2024 08:09

rach-id mentioned this pull request Oct 9, 2024

feat: register the gRPC block API celestiaorg/cosmos-sdk#416

Merged

rach-id added 2 commits October 9, 2024 12:25

chore: use different subscription name

d08249a

chore: go format

56bf584

This was referenced Oct 9, 2024

feat: block by hash #1420

Closed

Add ability to request a block via its hash #1395

Open

feat: allow disabling part proofs

49befbd

rach-id added 10 commits November 21, 2024 18:53

feat: fail the server if some subscription error occurs

1ea65f7

chore: fix

f0ea3fd

feat: fail the server if some subscription error occurs as sugggested…

48f6270

… by @cmwaters

chore: fix

7aafeed

Merge remote-tracking branch 'origin/grpc-endpoints' into grpc-endpoints

5fdc9ac

chore: expose DialerFunc

072b5eb

Revert "chore: expose DialerFunc"

55b868a

This reverts commit 072b5eb.

chore: cosmetics + close the block API in case of failure

73179bb

chore: lint

6ff1472

chore: remove block meta requests

4ca8293

evan-forbes previously approved these changes Dec 12, 2024

View reviewed changes

rootulp requested review from rootulp and cmwaters and removed request for staheri14 December 13, 2024 15:54

rootulp previously approved these changes Dec 13, 2024

View reviewed changes

Update rpc/grpc/api.go

62cb1e5

Co-authored-by: Rootul P <[email protected]>

rach-id dismissed stale reviews from rootulp and evan-forbes via 62cb1e5 January 13, 2025 18:40

celestia-bot requested review from evan-forbes and rootulp January 13, 2025 18:40

rach-id mentioned this pull request Jan 13, 2025

use NewClient instead of Dial when starting a gRPC connection #1587

Open

rach-id enabled auto-merge (squash) January 14, 2025 16:28

evan-forbes approved these changes Jan 14, 2025

View reviewed changes

rootulp approved these changes Jan 14, 2025

View reviewed changes

rach-id merged commit 18a7372 into v0.34.x-celestia Jan 14, 2025
20 of 21 checks passed

rach-id deleted the grpc-endpoints branch January 14, 2025 19:06

This was referenced Jan 14, 2025

refactor the grpc tests not to require disabling the dupl linter #1590

Open

chore: remove unnecessary new import lines in grpc/api.go #1591

Merged

rach-id added a commit that referenced this pull request Jan 15, 2025

chore: remove unnecessary new import lines in grpc/api.go (#1591)

be2047e

## Description Addresses this #1513 (comment)

rach-id mentioned this pull request Jan 16, 2025

Investigate registering the core block API in app celestiaorg/celestia-app#4234

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: support gRPC endpoints in core #1513

feat: support gRPC endpoints in core #1513

rach-id commented Oct 8, 2024 •

edited

Loading

rach-id commented Oct 8, 2024

rootulp Dec 13, 2024

rach-id Jan 13, 2025

rootulp Dec 13, 2024

rach-id Jan 13, 2025

rootulp Jan 14, 2025

rach-id Jan 14, 2025

rootulp Dec 13, 2024

rach-id Jan 13, 2025

rootulp Jan 14, 2025

rach-id Jan 14, 2025

rootulp Dec 13, 2024

rach-id Jan 13, 2025

rach-id Jan 13, 2025

evan-forbes left a comment

rootulp left a comment

rootulp Jan 14, 2025

rach-id Jan 14, 2025

rootulp Jan 14, 2025

rootulp Jan 14, 2025

rootulp Jan 14, 2025

rach-id Jan 14, 2025

rach-id Jan 14, 2025

	conn, err := grpc.Dial( //nolint:staticcheck
	conn, err := grpc.NewClient(

feat: support gRPC endpoints in core #1513

feat: support gRPC endpoints in core #1513

Conversation

rach-id commented Oct 8, 2024 • edited Loading

rach-id commented Oct 8, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

evan-forbes left a comment

Choose a reason for hiding this comment

rootulp left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rach-id commented Oct 8, 2024 •

edited

Loading