Skip to content

Latest commit

 

History

History
130 lines (100 loc) · 3.79 KB

Design.md

File metadata and controls

130 lines (100 loc) · 3.79 KB

Narwhal Consensus Implementation Design Document

1. What is Narwhal Consensus?

Narwhal is a mempool protocol that enables high-throughput transaction dissemination and ordering in blockchain systems. Unlike traditional blockchain consensus mechanisms that bundle transaction dissemination with consensus, Narwhal separates these concerns:

  • Transaction Dissemination: Handled by Narwhal's DAG-based mempool
  • Consensus: Can be implemented separately on top of the DAG (e.g., using Bullshark or Tusk)

Key differences from traditional blockchain consensus:

  • Transactions are organized in a DAG (Directed Acyclic Graph) instead of a linear chain
  • Parallel processing of transactions improves throughput
  • Separation of concerns allows for better scalability
  • No immediate finality requirement during dissemination phase

2. Key Components Design

2.1 Transaction Structure

  • Unique identifier (hash-based)
  • Transaction data (payload)
  • Parent references (DAG edges)
  • Validation logic for parent existence

2.2 DAG (Directed Acyclic Graph)

  • In-memory storage using HashMap
  • Parent-child relationship tracking
  • Transaction validation
  • State query capabilities
  • Concurrent access handling using Tokio mutex

2.3 P2P Network Layer

  • Built on libp2p framework
  • Components:
    • Kademlia DHT for peer discovery
    • Request/Response protocol for transaction propagation
    • Identify protocol for peer information
    • Custom behavior implementation for message handling
  • Event-driven architecture using SwarmEvent

2.4 HTTP/Axum Endpoints

  • /transaction (POST): Submit new transactions
  • /dag/state (GET): Query current DAG state
  • Async handlers with shared state
  • JSON request/response format

2.5 Event Handling

  • Swarm event processing for P2P events
  • Transaction validation and propagation
  • Peer connection management
  • Error handling and logging

3. Implementation Challenges

3.1 Memory Safety

  • Challenge: Unsafe channel handling in P2P message processing
  • Solution: Proper ownership transfer and validation checks before processing

3.2 Concurrent Access

  • Challenge: Multiple components accessing shared state
  • Solution: Arc for safe concurrent access

3.3 Parent Validation

  • Challenge: Race conditions in parent validation
  • Solution: Atomic validation and insertion operations

3.4 P2P Message Handling

  • Challenge: Complex event handling with multiple protocols
  • Solution: Structured event handling with proper error propagation

4. Missing Components

  1. Consensus Layer

    • Bullshark/Tusk implementation
    • Block producer logic
    • Finality mechanism

  2. Persistence Layer

    • Disk-based storage for DAG
    • Transaction history maintenance
    • State recovery mechanisms

  3. Network Security

    • Peer authentication
    • Message validation
    • DoS protection

  4. Performance Optimizations

    • Transaction batching
    • Parallel validation
    • Memory pool management

5. Future Improvements

5.1 Code Quality

  • Add comprehensive test coverage
  • Implement proper error types
  • Add metrics and monitoring
  • Improve documentation

5.2 Performance

  • Optimize DAG operations
  • Implement transaction batching
  • Add connection pooling
  • Optimize memory usage

5.3 Features

  • Add configuration management
  • Implement proper logging strategy
  • Add metrics collection
  • Add health checks

5.4 Architecture

  • Modularize components further
  • Add plugin system for consensus
  • Implement proper trait boundaries
  • Add proper shutdown handling

5.5 For Contributors

  • Add contribution guidelines
  • Improve code documentation
  • Add development setup guide
  • Create issue templates

6. Getting Started

[Add instructions for building, running, and testing the implementation]

7. Contributing

[Add contribution guidelines and contact information]