Network/Actor messages visualizer

[jancionear]

NEAR network is made up of nodes which send network messages between each other, and each node is made up of actors exchanging messages between themselves. It would be nice to have a way to visualize these messages.

Visualizing network messages would be great for understanding the protocol - how blocks, chunks and receipts are distributed, who endorses what, how the data dependencies look like. It would also be useful for analyzing performance and latency - how fast messages are sent, received, seeing if any nodes are overloaded with network traffic, etc.

Visualizing actor messages would be very helpful for understanding the internals of nearcore. The flow of code can be pretty confusing, things trigger other things which do something and trigger other things, there's often no clear path to look at. The best way to understand the codebase is usually to run an example and see what sort of things get triggered. Having a visualizer which shows what actors send messages to other actors would be a great way to view things at a high level and build a better mental model of the system.

The nice thing about having nodes and actors is that they divide the code into clearly separate modules. By analyzing messages we can see how these modules interact with other modules and get a high-level overview of what's going on in the system. The problem with logs is that there's often a ton of them, too many to read through and figure out what is going on. The number of high-level messages is lower, visualizer could give a clear high-level overview that would be easier to analyze by a human. For example we could see that this node isn't sending any endorsements, or that the PartialWitnessActor never received any witness parts, etc. It might turn out that there's a ton of messages as well, but we could add some filters/aggregations to view the system in a more readable way.

The way I imagine it working is that every time a message is sent/received, some information about the message would be saved to the database. Saving the whole messages would probably be too expensive, but we could save a smaller header, which would include things like timestamp, type of message, which chunk is endorsed, block height, etc. The header would contain only information useful for debugging, without the heavy data.

There's alread a visualizer for TestLoop events written by Robin: https://github.com/near/nearcore/tree/master/tools/debug-ui#how-to-use-the-testloop-log-visualizer
It's pretty cool, but I found it hard to use. I think it would be nice to spend more time on building a more advanced visualizer.

[walnut-the-cat]

Could you add high level engineering cost?

[jancionear]

Could you add high level engineering cost?

I'd say a week to figure out a good way of dumping messages and saving them and another week to visualize them in a readable way?