From bb086236ae7dc74c6d4593742318aa044ea914a5 Mon Sep 17 00:00:00 2001 From: h5law <53987565+h5law@users.noreply.github.com> Date: Sat, 23 Dec 2023 13:17:21 +0000 Subject: [PATCH 01/32] feat: move benchmarks into docs folder, tidy readme and docs --- README.md | 254 +----------------- docs/benchmarks.md | 254 ++++++++++++++++++ docs/{KVStore.md => kvstore.md} | 20 +- docs/{MerkleSumTrie.md => merkle-sum-trie.md} | 12 +- docs/{SMT.md => smt.md} | 57 ++-- 5 files changed, 316 insertions(+), 281 deletions(-) create mode 100644 docs/benchmarks.md rename docs/{KVStore.md => kvstore.md} (90%) rename docs/{MerkleSumTrie.md => merkle-sum-trie.md} (97%) rename docs/{SMT.md => smt.md} (89%) diff --git a/README.md b/README.md index 3d9807e..a1a6128 100644 --- a/README.md +++ b/README.md @@ -2,34 +2,22 @@ [![Tag](https://img.shields.io/github/v/tag/pokt-network/smt.svg?sort=semver)](https://img.shields.io/github/v/tag/pokt-network/smt.svg?sort=semver) [![GoDoc](https://godoc.org/github.com/pokt-network/smt?status.svg)](https://godoc.org/github.com/pokt-network/smt) +![Go Version](https://img.shields.io/github/go-mod/go-version/pokt-network/smt) [![Go Report Card](https://goreportcard.com/badge/github.com/pokt-network/smt)](https://goreportcard.com/report/github.com/pokt-network/smt) [![Tests](https://github.com/pokt-network/smt/actions/workflows/test.yml/badge.svg)](https://github.com/pokt-network/smt/actions/workflows/test.yml) [![codecov](https://codecov.io/gh/pokt-network/smt/branch/main/graph/badge.svg)](https://codecov.io/gh/pokt-network/smt) -Note: **Requires Go 1.20+** - - [Overview](#overview) - [Documentation](#documentation) - [Tests](#tests) - [Benchmarks](#benchmarks) - - [Definitions](#definitions) - - [Bytes/Operation (B/op)](#bytesoperation-bop) - - [Commit](#commit) - - [Sizing](#sizing) - - [SMT](#smt) - - [Fill](#fill) - - [Operations](#operations) - - [SMST](#smst) - - [Fill](#fill-1) - - [Operations](#operations-1) - - [Proofs](#proofs) - - [SMT](#smt-1) - - [SMST](#smst-1) +**NOTE: Requires Go 1.20.12** + ## Overview This is a Go library that implements a Sparse Merkle Trie for a key-value map. @@ -41,13 +29,12 @@ mechanics. ## Documentation -Documentation for the different aspects of this library can be found in the -[docs](./docs/) directory. +Documentation for the different aspects of this library, the trie, proofs and +all its different components can be found in the [docs](./docs/) directory. ## Tests -To run all the unit tests in the repo (excluding fuzz tests and benchmarks) -simply run the following command: +To run all tests (excluding benchmarks) run the following command: ```sh make test_all @@ -55,237 +42,14 @@ make test_all ## Benchmarks -Benchmarks for the different aspects of this SMT library can be found in -[benchmarks](./benchmarks/). In order to run the entire benchmarking suite use -the following command: +To run the full suite of benchmarks simply run the following command: ```sh make benchmark_all ``` -### Definitions - -Below is a list of terms used in the benchmarks' results that may need -clarification. - -#### Bytes/Operation (B/op) - -- This refers to the number of bytes allocated for each operation. - -#### Commit - -- The `Commit` term refers to the `Commit` method of the trie. This takes all - changes (which are made in memory) to the trie and writes them to the - underlying database. - -#### Sizing - -- The tests use the following sizes: 0.1M, 0.5M, 1M, 5M, 10M. The `M` refers to - millions hence: - - 0.1M = 100,000 (One hundred thousand) - - 0.5M = 500,000 (Five hundred thousand) - - 1M = 1,000,000 (One million) - - 5M = 5,000,000 (Five million) - - 10M = 10,000,000 (Ten million) -- These sizes refer to the number of key-value pairs or key-value-sum triples - inserted into the trie either beforehand or during the benchmark depending on - which benchmark it is. - -_NOTE: Unless otherwise stated the benchmarks in this document were ran on a -2023 14-inch Macbook Pro M2 Max with 32GB of RAM. The tries tested are using the -`sha256.New()` hasher._ - -_TODO: There is an opportunity to do a fuzz test where we commit every `N` -updates, if this ever becomes a bottlneck_ - -### SMT - -In order to run the SMT benchmarks use the following command: - -```sh -make benchmark_smt -``` - -#### Fill - -The "fill" benchmarks cover the time taken to insert `N` key-value pairs into -the SMT, as well as how long it takes to do this and commit these changes to -disk. This gives us an insight into how long it takes to build a trie of a -certain size. - -In order to run the SMT filling benchmarks use the following command: - -```sh -make benchmark_smt_fill -``` - -| Benchmark | # Values | Iterations | Time (s/op) | Bytes (B/op) | Allocations (allocs/op) | -| ------------- | -------- | ---------- | ------------- | --------------- | ----------------------- | -| Fill | 0.1M | 10 | 0.162967196 | 159,479,499 | 2,371,598 | -| Fill & Commit | 0.1M | 10 | 2.877307858 | 972,961,486 | 15,992,605 | -| Fill | 0.5M | 10 | 0.926864771 | 890,408,326 | 13,021,258 | -| Fill & Commit | 0.5M | 10 | 16.043430012 | 5,640,034,396 | 82,075,720 | -| Fill | 1M | 10 | 2.033616088 | 1,860,523,968 | 27,041,639 | -| Fill & Commit | 1M | 10 | 32.617249642 | 12,655,347,004 | 166,879,661 | -| Fill | 5M | 10 | 12.502309738 | 10,229,139,731 | 146,821,675 | -| Fill & Commit | 5M | 10 | 175.421250979 | 78,981,342,709 | 870,235,579 | -| Fill | 10M | 10 | 29.718092496 | 21,255,245,031 | 303,637,210 | -| Fill & Commit | 10M | 10 | 396.142675962 | 173,053,933,624 | 1,775,304,977 | - -#### Operations - -The "operations" benchmarks cover the time taken to perform a single operation -on an SMT of a given size, and also how long doing this operation followed by a -commit would take. This gives us insight into how the SMT operates when filled -to differing degrees. - -In order to run the SMT operation benchmarks use the following command: - -```sh -make benchmark_smt_ops -``` - -| Benchmark | Prefilled Values | Iterations | Time (ns/op) | Bytes (B/op) | Allocations (allocs/op) | -| --------------- | ---------------- | ---------- | ------------ | ------------ | ----------------------- | -| Update | 0.1M | 740,618 | 1,350 | 1,753 | 25 | -| Update & Commit | 0.1M | 21,022 | 54,665 | 13,110 | 281 | -| Update | 0.5M | 605,348 | 1,682 | 1,957 | 26 | -| Update & Commit | 0.5M | 11,697 | 91,028 | 21,501 | 468 | -| Update | 1M | 545,701 | 1,890 | 2,112 | 28 | -| Update & Commit | 1M | 9,540 | 119,347 | 24,983 | 545 | -| Update | 5M | 466,688 | 2,226 | 2,453 | 31 | -| Update & Commit | 5M | 7,906 | 186,026 | 52,621 | 722 | -| Update | 10M | 284,580 | 5,263 | 2,658 | 33 | -| Update & Commit | 10M | 4,484 | 298,376 | 117,923 | 844 | -| Get | 0.1M | 3,923,601 | 303.2 | 48 | 3 | -| Get | 0.5M | 2,209,981 | 577.7 | 48 | 3 | -| Get | 1M | 1,844,431 | 661.6 | 48 | 3 | -| Get | 5M | 1,196,467 | 1,030 | 48 | 3 | -| Get | 10M | 970,195 | 2,667 | 48 | 3 | -| Prove | 0.1M | 829,801 | 1,496 | 2,177 | 17 | -| Prove | 0.5M | 610,402 | 1,835 | 2,747 | 17 | -| Prove | 1M | 605,799 | 1,905 | 2,728 | 17 | -| Prove | 5M | 566,930 | 2,129 | 2,731 | 17 | -| Prove | 10M | 458,472 | 7,113 | 2,735 | 17 | -| Delete | 0.1M | 12,081,112 | 96.18 | 50 | 3 | -| Delete & Commit | 0.1M | 26,490 | 39,568 | 7,835 | 177 | -| Delete | 0.5M | 7,253,522 | 140.3 | 64 | 3 | -| Delete & Commit | 0.5M | 12,766 | 80,518 | 16,696 | 376 | -| Delete | 1M | 1,624,569 | 629.6 | 196 | 4 | -| Delete & Commit | 1M | 9,811 | 135,606 | 20,254 | 456 | -| Delete | 5M | 856,424 | 1,400 | 443 | 6 | -| Delete & Commit | 5M | 8,431 | 151,107 | 74,133 | 626 | -| Delete | 10M | 545,876 | 4,173 | 556 | 6 | -| Delete & Commit | 10M | 3,916 | 271,332 | 108,396 | 772 | - -### SMST - -In order to run the SMST benchmarks use the following command: - -```sh -make benchmark_smst -``` - -#### Fill - -The "fill" benchmarks cover the time taken to insert `N` key-value-sum triples -into the SMST, as well as how long it takes to do this and commit these changes -to disk. This gives us an insight into how long it takes to build a trie of a -certain size. - -In order to run the SMST filling benchmarks use the following command: - -```sh -make benchmark_smst_fill -``` - -| Benchmark | # Values | Iterations | Time (s/op) | Bytes (B/op) | Allocations (allocs/op) | -| ------------- | -------- | ---------- | ------------- | --------------- | ----------------------- | -| Fill | 0.1M | 10 | 0.157951888 | 165,878,234 | 2,471,593 | -| Fill & Commit | 0.1M | 10 | 3.011097462 | 1,058,069,050 | 16,664,811 | -| Fill | 0.5M | 10 | 0.927521862 | 922,408,350 | 13,521,259 | -| Fill & Commit | 0.5M | 10 | 15.338199979 | 6,533,439,773 | 85,880,046 | -| Fill | 1M | 10 | 1.982756162 | 1,924,516,467 | 28,041,610 | -| Fill & Commit | 1M | 10 | 31.197517821 | 14,874,342,889 | 175,474,251 | -| Fill | 5M | 10 | 12.054370871 | 10,549,075,488 | 151,821,423 | -| Fill & Commit | 5M | 10 | 176.912009238 | 89,667,234,678 | 914,653,740 | -| Fill | 10M | 10 | 26.859672362 | 21,894,837,504 | 313,635,611 | -| Fill & Commit | 10M | 10 | 490.805535617 | 197,997,807,905 | 1,865,882,489 | - -#### Operations - -The "operations" benchmarks cover the time taken to perform a single operation -on an SMST of a given size, and also how long doing this operation followed by -a commit would take. This gives us insight into how the SMST operates when -filled to differing degrees. - -In order to run the SMST operation benchmarks use the following command: - -```sh -make benchmark_smst_ops -``` - -| Benchmark | Prefilled Values | Iterations | Time (ns/op) | Bytes (B/op) | Allocations (allocs/op) | -| --------------- | ---------------- | ---------- | ------------ | ------------ | ----------------------- | -| Update | 0.1M | 913,760 | 1,477 | 1,843 | 25 | -| Update & Commit | 0.1M | 20,318 | 49,705 | 13,440 | 256 | -| Update | 0.5M | 687,813 | 1,506 | 1,965 | 27 | -| Update & Commit | 0.5M | 14,526 | 83,295 | 37,604 | 428 | -| Update | 1M | 630,310 | 1,679 | 2,076 | 28 | -| Update & Commit | 1M | 11,678 | 122,568 | 25,760 | 501 | -| Update | 5M | 644,193 | 1,850 | 2,378 | 31 | -| Update & Commit | 5M | 6,214 | 184,533 | 60,755 | 723 | -| Update | 10M | 231,714 | 4,962 | 2,616 | 33 | -| Update & Commit | 10M | 4,284 | 279,893 | 77,377 | 830 | -| Get | 0.1M | 3,924,031 | 281.3 | 40 | 2 | -| Get | 0.5M | 2,080,167 | 559.6 | 40 | 2 | -| Get | 1M | 1,609,478 | 718.6 | 40 | 2 | -| Get | 5M | 1,015,630 | 1,105 | 40 | 2 | -| Get | 10M | 352,980 | 2,949 | 40 | 2 | -| Prove | 0.1M | 717,380 | 1,692 | 2,344 | 18 | -| Prove | 0.5M | 618,265 | 1,972 | 3,040 | 19 | -| Prove | 1M | 567,594 | 2,117 | 3,044 | 19 | -| Prove | 5M | 446,062 | 2,289 | 3,045 | 19 | -| Prove | 10M | 122,347 | 11,215 | 3,046 | 19 | -| Delete | 0.1M | 1,000,000 | 1,022 | 1,110 | 7 | -| Delete & Commit | 0.1M | 1,000,000 | 1,039 | 1,110 | 7 | -| Delete | 0.5M | 1,046,163 | 1,159 | 1,548 | 7 | -| Delete & Commit | 0.5M | 907,071 | 1,143 | 1,548 | 7 | -| Delete | 1M | 852,918 | 1,246 | 1,552 | 8 | -| Delete & Commit | 1M | 807,847 | 1,303 | 1,552 | 8 | -| Delete | 5M | 625,662 | 1,604 | 1,552 | 8 | -| Delete & Commit | 5M | 864,432 | 1,382 | 1,552 | 8 | -| Delete | 10M | 232,544 | 4,618 | 1,552 | 8 | -| Delete & Commit | 10M | 224,767 | 5,048 | 1,552 | 8 | - -### Proofs - -To run the tests to average the proof size for numerous prefilled tries use the -following command: - -```sh -make benchmark_proof_sizes -``` - -#### SMT - -| Prefilled Size | Average Serialised Proof Size (bytes) | Min (bytes) | Max (bytes) | Average Serialised Compacted Proof Size (bytes) | Min (bytes) | Max (bytes) | -| -------------- | ------------------------------------- | ----------- | ----------- | ----------------------------------------------- | ----------- | ----------- | -| 100,000 | 780 | 650 | 1310 | 790 | 692 | 925 | -| 500,000 | 856 | 716 | 1475 | 866 | 758 | 1024 | -| 1,000,000 | 890 | 716 | 1475 | 900 | 758 | 1057 | -| 5,000,000 | 966 | 815 | 1739 | 976 | 858 | 1156 | -| 10,000,000 | 999 | 848 | 1739 | 1010 | 891 | 1189 | - -#### SMST - -| Prefilled Size | Average Serialised Proof Size (bytes) | Min (bytes) | Max (bytes) | Average Serialised Compacted Proof Size (bytes) | Min (bytes) | Max (bytes) | -| -------------- | ------------------------------------- | ----------- | ----------- | ----------------------------------------------- | ----------- | ----------- | -| 100,000 | 935 | 780 | 1590 | 937 | 822 | 1101 | -| 500,000 | 1030 | 862 | 1795 | 1032 | 904 | 1224 | -| 1,000,000 | 1071 | 868 | 1795 | 1073 | 910 | 1265 | -| 5,000,000 | 1166 | 975 | 2123 | 1169 | 1018 | 1388 | -| 10,000,000 | 1207 | 1026 | 2123 | 1210 | 1059 | 1429 | +To view pre-ran results of the entire benchmarking suite see +[benchmarks](./docs/benchmarks.md) [jmt whitepaper]: https://developers.diem.com/papers/jellyfish-merkle-tree/2021-01-14.pdf [libra whitepaper]: https://diem-developers-components.netlify.app/papers/the-diem-blockchain/2020-05-26.pdf diff --git a/docs/benchmarks.md b/docs/benchmarks.md new file mode 100644 index 0000000..30065d1 --- /dev/null +++ b/docs/benchmarks.md @@ -0,0 +1,254 @@ +# Benchmarks + + + +- [Overview](#overview) +- [Definitions](#definitions) + * [Bytes/Operation (B/op)](#bytesoperation-bop) + * [Commit](#commit) + * [Sizing](#sizing) +- [SMT](#smt) + * [Fill](#fill) + * [Operations](#operations) +- [SMST](#smst) + * [Fill](#fill-1) + * [Operations](#operations-1) +- [Proofs](#proofs) + * [SMT](#smt-1) + * [SMST](#smst-1) + + + +## Overview + +Benchmarks for the different aspects of this SMT library can be found in +[benchmarks](../benchmarks/). In order to run the entire benchmarking suite use +the following command: + +```sh +make benchmark_all +``` + +## Definitions + +Below is a list of terms used in the benchmarks' results that may need +clarification. + +### Bytes/Operation (B/op) + +- This refers to the number of bytes allocated for each operation. + +### Commit + +- The `Commit` term refers to the `Commit` method of the trie. This takes all + changes (which are made in memory) to the trie and writes them to the + underlying database. + +### Sizing + +- The tests use the following sizes: 0.1M, 0.5M, 1M, 5M, 10M. The `M` refers to + millions hence: + - 0.1M = 100,000 (One hundred thousand) + - 0.5M = 500,000 (Five hundred thousand) + - 1M = 1,000,000 (One million) + - 5M = 5,000,000 (Five million) + - 10M = 10,000,000 (Ten million) +- These sizes refer to the number of key-value pairs or key-value-sum triples + inserted into the trie either beforehand or during the benchmark depending on + which benchmark it is. + +_NOTE: Unless otherwise stated the benchmarks in this document were ran on a +2023 14-inch Macbook Pro M2 Max with 32GB of RAM. The tries tested are using the +`sha256.New()` hasher._ + +_TODO: There is an opportunity to do a fuzz test where we commit every `N` +updates, if this ever becomes a bottlneck_ + +## SMT + +In order to run the SMT benchmarks use the following command: + +```sh +make benchmark_smt +``` + +### Fill + +The "fill" benchmarks cover the time taken to insert `N` key-value pairs into +the SMT, as well as how long it takes to do this and commit these changes to +disk. This gives us an insight into how long it takes to build a trie of a +certain size. + +In order to run the SMT filling benchmarks use the following command: + +```sh +make benchmark_smt_fill +``` + +| Benchmark | # Values | Iterations | Time (s/op) | Bytes (B/op) | Allocations (allocs/op) | +| ------------- | -------- | ---------- | ------------- | --------------- | ----------------------- | +| Fill | 0.1M | 10 | 0.162967196 | 159,479,499 | 2,371,598 | +| Fill & Commit | 0.1M | 10 | 2.877307858 | 972,961,486 | 15,992,605 | +| Fill | 0.5M | 10 | 0.926864771 | 890,408,326 | 13,021,258 | +| Fill & Commit | 0.5M | 10 | 16.043430012 | 5,640,034,396 | 82,075,720 | +| Fill | 1M | 10 | 2.033616088 | 1,860,523,968 | 27,041,639 | +| Fill & Commit | 1M | 10 | 32.617249642 | 12,655,347,004 | 166,879,661 | +| Fill | 5M | 10 | 12.502309738 | 10,229,139,731 | 146,821,675 | +| Fill & Commit | 5M | 10 | 175.421250979 | 78,981,342,709 | 870,235,579 | +| Fill | 10M | 10 | 29.718092496 | 21,255,245,031 | 303,637,210 | +| Fill & Commit | 10M | 10 | 396.142675962 | 173,053,933,624 | 1,775,304,977 | + +### Operations + +The "operations" benchmarks cover the time taken to perform a single operation +on an SMT of a given size, and also how long doing this operation followed by a +commit would take. This gives us insight into how the SMT operates when filled +to differing degrees. + +In order to run the SMT operation benchmarks use the following command: + +```sh +make benchmark_smt_ops +``` + +| Benchmark | Prefilled Values | Iterations | Time (ns/op) | Bytes (B/op) | Allocations (allocs/op) | +| --------------- | ---------------- | ---------- | ------------ | ------------ | ----------------------- | +| Update | 0.1M | 740,618 | 1,350 | 1,753 | 25 | +| Update & Commit | 0.1M | 21,022 | 54,665 | 13,110 | 281 | +| Update | 0.5M | 605,348 | 1,682 | 1,957 | 26 | +| Update & Commit | 0.5M | 11,697 | 91,028 | 21,501 | 468 | +| Update | 1M | 545,701 | 1,890 | 2,112 | 28 | +| Update & Commit | 1M | 9,540 | 119,347 | 24,983 | 545 | +| Update | 5M | 466,688 | 2,226 | 2,453 | 31 | +| Update & Commit | 5M | 7,906 | 186,026 | 52,621 | 722 | +| Update | 10M | 284,580 | 5,263 | 2,658 | 33 | +| Update & Commit | 10M | 4,484 | 298,376 | 117,923 | 844 | +| Get | 0.1M | 3,923,601 | 303.2 | 48 | 3 | +| Get | 0.5M | 2,209,981 | 577.7 | 48 | 3 | +| Get | 1M | 1,844,431 | 661.6 | 48 | 3 | +| Get | 5M | 1,196,467 | 1,030 | 48 | 3 | +| Get | 10M | 970,195 | 2,667 | 48 | 3 | +| Prove | 0.1M | 829,801 | 1,496 | 2,177 | 17 | +| Prove | 0.5M | 610,402 | 1,835 | 2,747 | 17 | +| Prove | 1M | 605,799 | 1,905 | 2,728 | 17 | +| Prove | 5M | 566,930 | 2,129 | 2,731 | 17 | +| Prove | 10M | 458,472 | 7,113 | 2,735 | 17 | +| Delete | 0.1M | 12,081,112 | 96.18 | 50 | 3 | +| Delete & Commit | 0.1M | 26,490 | 39,568 | 7,835 | 177 | +| Delete | 0.5M | 7,253,522 | 140.3 | 64 | 3 | +| Delete & Commit | 0.5M | 12,766 | 80,518 | 16,696 | 376 | +| Delete | 1M | 1,624,569 | 629.6 | 196 | 4 | +| Delete & Commit | 1M | 9,811 | 135,606 | 20,254 | 456 | +| Delete | 5M | 856,424 | 1,400 | 443 | 6 | +| Delete & Commit | 5M | 8,431 | 151,107 | 74,133 | 626 | +| Delete | 10M | 545,876 | 4,173 | 556 | 6 | +| Delete & Commit | 10M | 3,916 | 271,332 | 108,396 | 772 | + +## SMST + +In order to run the SMST benchmarks use the following command: + +```sh +make benchmark_smst +``` + +### Fill + +The "fill" benchmarks cover the time taken to insert `N` key-value-sum triples +into the SMST, as well as how long it takes to do this and commit these changes +to disk. This gives us an insight into how long it takes to build a trie of a +certain size. + +In order to run the SMST filling benchmarks use the following command: + +```sh +make benchmark_smst_fill +``` + +| Benchmark | # Values | Iterations | Time (s/op) | Bytes (B/op) | Allocations (allocs/op) | +| ------------- | -------- | ---------- | ------------- | --------------- | ----------------------- | +| Fill | 0.1M | 10 | 0.157951888 | 165,878,234 | 2,471,593 | +| Fill & Commit | 0.1M | 10 | 3.011097462 | 1,058,069,050 | 16,664,811 | +| Fill | 0.5M | 10 | 0.927521862 | 922,408,350 | 13,521,259 | +| Fill & Commit | 0.5M | 10 | 15.338199979 | 6,533,439,773 | 85,880,046 | +| Fill | 1M | 10 | 1.982756162 | 1,924,516,467 | 28,041,610 | +| Fill & Commit | 1M | 10 | 31.197517821 | 14,874,342,889 | 175,474,251 | +| Fill | 5M | 10 | 12.054370871 | 10,549,075,488 | 151,821,423 | +| Fill & Commit | 5M | 10 | 176.912009238 | 89,667,234,678 | 914,653,740 | +| Fill | 10M | 10 | 26.859672362 | 21,894,837,504 | 313,635,611 | +| Fill & Commit | 10M | 10 | 490.805535617 | 197,997,807,905 | 1,865,882,489 | + +### Operations + +The "operations" benchmarks cover the time taken to perform a single operation +on an SMST of a given size, and also how long doing this operation followed by +a commit would take. This gives us insight into how the SMST operates when +filled to differing degrees. + +In order to run the SMST operation benchmarks use the following command: + +```sh +make benchmark_smst_ops +``` + +| Benchmark | Prefilled Values | Iterations | Time (ns/op) | Bytes (B/op) | Allocations (allocs/op) | +| --------------- | ---------------- | ---------- | ------------ | ------------ | ----------------------- | +| Update | 0.1M | 913,760 | 1,477 | 1,843 | 25 | +| Update & Commit | 0.1M | 20,318 | 49,705 | 13,440 | 256 | +| Update | 0.5M | 687,813 | 1,506 | 1,965 | 27 | +| Update & Commit | 0.5M | 14,526 | 83,295 | 37,604 | 428 | +| Update | 1M | 630,310 | 1,679 | 2,076 | 28 | +| Update & Commit | 1M | 11,678 | 122,568 | 25,760 | 501 | +| Update | 5M | 644,193 | 1,850 | 2,378 | 31 | +| Update & Commit | 5M | 6,214 | 184,533 | 60,755 | 723 | +| Update | 10M | 231,714 | 4,962 | 2,616 | 33 | +| Update & Commit | 10M | 4,284 | 279,893 | 77,377 | 830 | +| Get | 0.1M | 3,924,031 | 281.3 | 40 | 2 | +| Get | 0.5M | 2,080,167 | 559.6 | 40 | 2 | +| Get | 1M | 1,609,478 | 718.6 | 40 | 2 | +| Get | 5M | 1,015,630 | 1,105 | 40 | 2 | +| Get | 10M | 352,980 | 2,949 | 40 | 2 | +| Prove | 0.1M | 717,380 | 1,692 | 2,344 | 18 | +| Prove | 0.5M | 618,265 | 1,972 | 3,040 | 19 | +| Prove | 1M | 567,594 | 2,117 | 3,044 | 19 | +| Prove | 5M | 446,062 | 2,289 | 3,045 | 19 | +| Prove | 10M | 122,347 | 11,215 | 3,046 | 19 | +| Delete | 0.1M | 1,000,000 | 1,022 | 1,110 | 7 | +| Delete & Commit | 0.1M | 1,000,000 | 1,039 | 1,110 | 7 | +| Delete | 0.5M | 1,046,163 | 1,159 | 1,548 | 7 | +| Delete & Commit | 0.5M | 907,071 | 1,143 | 1,548 | 7 | +| Delete | 1M | 852,918 | 1,246 | 1,552 | 8 | +| Delete & Commit | 1M | 807,847 | 1,303 | 1,552 | 8 | +| Delete | 5M | 625,662 | 1,604 | 1,552 | 8 | +| Delete & Commit | 5M | 864,432 | 1,382 | 1,552 | 8 | +| Delete | 10M | 232,544 | 4,618 | 1,552 | 8 | +| Delete & Commit | 10M | 224,767 | 5,048 | 1,552 | 8 | + +## Proofs + +To run the tests to average the proof size for numerous prefilled tries use the +following command: + +```sh +make benchmark_proof_sizes +``` + +### SMT + +| Prefilled Size | Average Serialised Proof Size (bytes) | Min (bytes) | Max (bytes) | Average Serialised Compacted Proof Size (bytes) | Min (bytes) | Max (bytes) | +| -------------- | ------------------------------------- | ----------- | ----------- | ----------------------------------------------- | ----------- | ----------- | +| 100,000 | 780 | 650 | 1310 | 790 | 692 | 925 | +| 500,000 | 856 | 716 | 1475 | 866 | 758 | 1024 | +| 1,000,000 | 890 | 716 | 1475 | 900 | 758 | 1057 | +| 5,000,000 | 966 | 815 | 1739 | 976 | 858 | 1156 | +| 10,000,000 | 999 | 848 | 1739 | 1010 | 891 | 1189 | + +### SMST + +| Prefilled Size | Average Serialised Proof Size (bytes) | Min (bytes) | Max (bytes) | Average Serialised Compacted Proof Size (bytes) | Min (bytes) | Max (bytes) | +| -------------- | ------------------------------------- | ----------- | ----------- | ----------------------------------------------- | ----------- | ----------- | +| 100,000 | 935 | 780 | 1590 | 937 | 822 | 1101 | +| 500,000 | 1030 | 862 | 1795 | 1032 | 904 | 1224 | +| 1,000,000 | 1071 | 868 | 1795 | 1073 | 910 | 1265 | +| 5,000,000 | 1166 | 975 | 2123 | 1169 | 1018 | 1388 | +| 10,000,000 | 1207 | 1026 | 2123 | 1210 | 1059 | 1429 | diff --git a/docs/KVStore.md b/docs/kvstore.md similarity index 90% rename from docs/KVStore.md rename to docs/kvstore.md index 3fcda9f..78c7c5a 100644 --- a/docs/KVStore.md +++ b/docs/kvstore.md @@ -4,16 +4,16 @@ - [Overview](#overview) - [Implementation](#implementation) - - [In-Memory and Persistent](#in-memory-and-persistent) - - [Store methods](#store-methods) - - [Lifecycle Methods](#lifecycle-methods) - - [Data Methods](#data-methods) - - [Backups](#backups) - - [Restorations](#restorations) - - [Accessor Methods](#accessor-methods) - - [Prefixed and Sorted Get All](#prefixed-and-sorted-get-all) - - [Clear All Key-Value Pairs](#clear-all-key-value-pairs) - - [Len](#len) + * [In-Memory and Persistent](#in-memory-and-persistent) + * [Store methods](#store-methods) + * [Lifecycle Methods](#lifecycle-methods) + * [Data Methods](#data-methods) + + [Backups](#backups) + + [Restorations](#restorations) + * [Accessor Methods](#accessor-methods) + + [Prefixed and Sorted Get All](#prefixed-and-sorted-get-all) + + [Clear All Key-Value Pairs](#clear-all-key-value-pairs) + + [Len](#len) diff --git a/docs/MerkleSumTrie.md b/docs/merkle-sum-trie.md similarity index 97% rename from docs/MerkleSumTrie.md rename to docs/merkle-sum-trie.md index a72017c..260164e 100644 --- a/docs/MerkleSumTrie.md +++ b/docs/merkle-sum-trie.md @@ -4,11 +4,11 @@ - [Overview](#overview) - [Implementation](#implementation) - - [Sum Encoding](#sum-encoding) - - [Digests](#digests) - - [Visualisations](#visualisations) - - [General Trie Structure](#general-trie-structure) - - [Binary Sum Digests](#binary-sum-digests) + * [Sum Encoding](#sum-encoding) + * [Digests](#digests) + * [Visualisations](#visualisations) + + [General Trie Structure](#general-trie-structure) + + [Binary Sum Digests](#binary-sum-digests) - [Sum](#sum) - [Nil Values](#nil-values) - [Example](#example) @@ -47,7 +47,7 @@ In practice the SMST is a wrapper around the SMT with a new field added to the `TrieSpec`: `sumTrie bool` this determines whether the SMT should follow its regular encoding of that of the sum trie. -See: the [SMT documentation](./SMT.md) for the details on how the SMT works. +See: the [SMT documentation](./smt.md) for the details on how the SMT works. The majority of the code relating to the SMST can be found in: diff --git a/docs/SMT.md b/docs/smt.md similarity index 89% rename from docs/SMT.md rename to docs/smt.md index 06ad9e2..603a710 100644 --- a/docs/SMT.md +++ b/docs/smt.md @@ -20,6 +20,7 @@ - [Proofs](#proofs) * [Verification](#verification) * [Closest Proof](#closest-proof) + + [Closest Proof Use Cases](#closest-proof-use-cases) * [Compression](#compression) * [Serialisation](#serialisation) - [Database](#database) @@ -384,21 +385,40 @@ of the trie. Since the `ClosestProof` method takes a hash as input, it is possible to place a leaf in the trie according to the hash's path, if it is known. Depending on -the use case of this function this may expose a vulnerability. _It is not -intendend to be used as a general purpose proof mechanism_. Given two parties: -the prover and the verifier, the verifier should supply the prover with a hash -after the trie has been "closed" and is no longer being updated. The prover -will then generate a `ClosestProof` for a leaf using the corresponding method. -The verifier can subsequently verify that proof for its validity. If however, -the prover were to know the hash prior to "closing" the trie, they could place -a leaf where the method would always guarantee it existence. This form of attack -can only happen due to the method's deterministic behaviour and the prover -knowing the hash before they have "closed" the trie. The intended use of this -method is that the verifier gives the hash only after the prover has closed their -trie and submitted the closed trie's root hash. This enables the verifier to -verify the integrity of the proof (if the trie was changed the root hash would -be different) and also guarantees the pseudo-random proof of inclusion was not -a maliciously placed leaf. +the use case of this function this may expose a vulnerability. **It is not +intendend to be used as a general purpose proof mechanism**, but instead as a +**Commit and Reveal** mechanism, as detailed below. + +#### Closest Proof Use Cases + +The `CloestProof` function is intended for use as a `commit & reveal` mechanism. +Where there are two actors involved, the **prover** and **verifier**. + +_NOTE: Throughout this document, `commitment` of the the trie's root hash is also +referred to as closing the trie, such that no more updates are made to it once +committed._ + +Take the following attack vector (**without** a commit prior to a reveal) into +consideration: + +1. The **prover** identifies the hash the **verifier** intends to check against +1. The **prover** then places a leaf in the tree with a common prefix to the + identified hash +1. Due to the deterministic nature of the `ClosestProof` method this leaf will + **always** be returned given the identified hash. +1. The **verifier** then verifies the revealed `ClosestProof`, in turn validating + a maliciously placed leaf. + +Take the following normal flow (**with** a commit prior to reveal) as + +1. The **prover** commits to the state of their trie and publishes their root + hash, _closing_ their trie. +1. The **verifier** provides a hash to be used in the `commit & reveal` process +1. The **prover** then utilises this hash and the `ClosestProof` method on their + _closed_ trie, producing a `ClosestProof`, thus revealing a deterministic, + yet pseudo-random leaf. +1. The **verifier** then verifies the proof, in turn verifying the commitment + made by the **prover** and the state of the **prover**'s trie. ### Compression @@ -438,7 +458,7 @@ node, **not** the leaf node's value, even when `WithValueHasher(nil)` is used. The node value can be parsed from this value, as the trie `Get` function does by removing the prefix and path bytes from the returned value. -See [KVStore.md](./KVStore.md) for the details of the implementation. +See [kvstore.md](./kvstore.md) for the details of the implementation. ### Data Loss @@ -450,7 +470,7 @@ the `Commit()` function is called and changes are persisted. ## Sparse Merkle Sum Trie This library also implements a Sparse Merkle Sum Trie (SMST), the documentation -for which can be found [here](./MerkleSumTrie.md). +for which can be found [here](./merkle-sum-trie.md). ## Example @@ -472,9 +492,6 @@ func main() { // Ensure the database connection closes defer nodeStore.Stop() - // Initialise the trie - trie := smt.NewSparseMerkleTrie(nodeStore, sha256.New()) - // Update the key "foo" with the value "bar" _ = trie.Update([]byte("foo"), []byte("bar")) From 89dee091259f8fb188a4cd749aba952226a007f5 Mon Sep 17 00:00:00 2001 From: h5law <53987565+h5law@users.noreply.github.com> Date: Sat, 23 Dec 2023 13:55:46 +0000 Subject: [PATCH 02/32] feat: update gitignore --- .gitignore | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/.gitignore b/.gitignore index 9d2d03c..aa638db 100644 --- a/.gitignore +++ b/.gitignore @@ -1,5 +1,8 @@ -# macos files +# Ignore Mac OS X system files .DS_Store -# goland -.idea/ \ No newline at end of file +# Ignore asdf local tool-versions file +.tool-versions + +# Ignore Goland and JetBrains IDE files +.idea/ From dedc8f3b574ef15311d981f3eca3187978693c52 Mon Sep 17 00:00:00 2001 From: h5law <53987565+h5law@users.noreply.github.com> Date: Sat, 23 Dec 2023 13:56:15 +0000 Subject: [PATCH 03/32] feat: cleanup and update workflow file --- .github/workflows/test.yml | 52 +++++++++++++------------------------- 1 file changed, 17 insertions(+), 35 deletions(-) diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 161589b..e981f6a 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -1,4 +1,4 @@ -name: Test & Build +name: Go Tests on: pull_request: @@ -12,7 +12,7 @@ concurrency: cancel-in-progress: true env: - TARGET_GOLANG_VERSION: "1.20.11" + TARGET_GOLANG_VERSION: "1.20.12" jobs: tests: @@ -22,7 +22,7 @@ jobs: steps: - uses: actions/checkout@v3 with: - fetch-depth: "0" # Per https://github.com/ignite/cli/issues/1674#issuecomment-1144619147 + fetch-depth: "0" - name: Setup go uses: actions/setup-go@v4 @@ -39,18 +39,27 @@ jobs: - name: Create coverage report and run tests run: | + # Run each of the tests (excluding benchmarks) outputting the JSON + # to the test_results.json file for usage in later steps. set -euo pipefail - GODEBUG=netdns=cgo go test -p 1 -json ./ -mod=readonly -timeout 8m -race -coverprofile=coverage.txt -covermode=atomic 2>&1 | tee test_results.json + go test -v -json -p 1 ./ -mod=readonly -race -coverprofile=coverage.txt -covermode=atomic 2>&1 | tee -a test_results.json - name: Sanitize test results - # We're utilizing `tee` above which can capture non-json stdout output so we need to remove non-json lines before additional parsing and submitting it to the external github action. + # We're utilizing `tee` above which can capture non-json stdout output + # so we need to remove non-json lines before additional parsing and + # submitting it to the external github action. run: cat test_results.json | jq -c -R 'fromjson? | select(type == "object")' > tmp.json && mv tmp.json test_results.json - name: Output test failures - # Makes it easier to find failed tests so no need to scroll through the whole log. if: ${{ failure() }} run: | - jq --argjson fail_tests "$(jq -c -r 'select(.Action == "fail") | select(.Test) | .Test' test_results.json | jq -R -s -c -r 'split("\n") | map(select(length>0))')" 'select(.Test as $t | ($fail_tests | arrays)[] | select($t == .)) | select(.Output) | .Output' test_results.json | jq -r | sed ':a;N;$!ba;s/\n\n/\n/g' > test_failures.json + jq --argjson fail_tests "$( \ + jq -c -r 'select(.Action == "fail") | select(.Test) | .Test' test_results.json \ + | jq -R -s -c 'split("\n") | map(select(length > 0))' \ + )" \ + 'select(.Test as $t | ($fail_tests | arrays)[] | select($t == .)) | select(.Output) | .Output' test_results.json \ + | jq -r \ + | sed ':a;N;$!ba;s/\n\n/\n/g' > test_failures.json cat test_failures.json exit 1 @@ -62,7 +71,6 @@ jobs: test_*.json - name: Annotate tests on GitHub - # Only annotate if the test failed on target version to avoid duplicated annotations on GitHub. uses: guyarb/golang-test-annotations@v0.5.1 with: test-results: test_results.json @@ -71,30 +79,4 @@ jobs: uses: codecov/codecov-action@v3 with: files: ./coverage.txt - - build: - runs-on: ubuntu-latest - needs: tests - strategy: - fail-fast: false - matrix: - goarch: ["arm64", "amd64"] - timeout-minutes: 5 - name: Build for ${{ matrix.goarch }} - steps: - - uses: actions/setup-go@v4 - with: - go-version: ${{ env.TARGET_GOLANG_VERSION }} - - - uses: actions/checkout@v3 - - - uses: technote-space/get-diff-action@v4 - with: - PATTERNS: | - **/**.go - go.mod - go.sum - - - name: Go build - run: GOOS=linux GOARCH=${{ matrix.goarch }} go build - if: env.GIT_DIFF + verbose: true From 2d215eb1c9855271f8aa2f27cdac8953a3bef0e6 Mon Sep 17 00:00:00 2001 From: h5law <53987565+h5law@users.noreply.github.com> Date: Sat, 23 Dec 2023 13:56:37 +0000 Subject: [PATCH 04/32] feat: make codecov reports informational only --- .github/codecov.yml | 9 +++++++++ 1 file changed, 9 insertions(+) create mode 100644 .github/codecov.yml diff --git a/.github/codecov.yml b/.github/codecov.yml new file mode 100644 index 0000000..2c43681 --- /dev/null +++ b/.github/codecov.yml @@ -0,0 +1,9 @@ +# Only post coverage comments and not pass/fail reports in the CI checks +coverage: + status: + project: + default: + informational: true + patch: + default: + informational: true From 63a9c3ae88ece2cc135a712c926abb0b0f56fc3c Mon Sep 17 00:00:00 2001 From: h5law <53987565+h5law@users.noreply.github.com> Date: Fri, 22 Dec 2023 14:26:04 +0000 Subject: [PATCH 05/32] feat: add badger as a submodule --- Makefile | 6 +- README.md | 7 + benchmarks/bench_leaf_test.go | 11 +- benchmarks/bench_smst_test.go | 395 ++++++++---------- benchmarks/bench_smt_test.go | 395 ++++++++---------- benchmarks/bench_utils_test.go | 30 +- benchmarks/proof_sizes_test.go | 11 +- bulk_test.go | 12 +- docs/{kvstore.md => badger-store.md} | 55 +-- docs/merkle-sum-trie.md | 6 +- docs/smt.md | 43 +- errors.go | 8 + fuzz_test.go | 26 +- go.mod | 25 +- go.sum | 109 +---- go.work | 8 + go.work.sum | 11 + godoc.go | 2 +- kvstore/badger/go.mod | 30 ++ kvstore/badger/go.sum | 124 ++++++ kvstore/badger/godoc.go | 4 + kvstore/badger/interface.go | 32 ++ kvstore.go => kvstore/badger/kvstore.go | 24 +- .../badger/kvstore_test.go | 192 ++++----- kvstore/errors.go | 12 + kvstore/simplemap.go | 75 ++++ kvstore/simplemap_test.go | 175 ++++++++ proofs.go | 3 - proofs_test.go | 8 +- smst.go | 18 +- smst_proofs_test.go | 59 +-- smst_test.go | 79 ++-- smst_utils_test.go | 19 +- smt.go | 34 +- smt_proofs_test.go | 50 +-- smt_test.go | 62 +-- smt_utils_test.go | 21 +- types.go | 25 +- 38 files changed, 1213 insertions(+), 993 deletions(-) rename docs/{kvstore.md => badger-store.md} (61%) create mode 100644 errors.go create mode 100644 go.work create mode 100644 go.work.sum create mode 100644 kvstore/badger/go.mod create mode 100644 kvstore/badger/go.sum create mode 100644 kvstore/badger/godoc.go create mode 100644 kvstore/badger/interface.go rename kvstore.go => kvstore/badger/kvstore.go (89%) rename kvstore_test.go => kvstore/badger/kvstore_test.go (62%) create mode 100644 kvstore/errors.go create mode 100644 kvstore/simplemap.go create mode 100644 kvstore/simplemap_test.go diff --git a/Makefile b/Makefile index 44e1e23..54e9722 100644 --- a/Makefile +++ b/Makefile @@ -41,7 +41,11 @@ go_docs: check_godoc ## Generate documentation for the project .PHONY: test_all test_all: ## runs the test suite - go test -v -p 1 ./ -mod=readonly -race + go test -v -p 1 ./... -mod=readonly -race + +.PHONY: test_badger +test_badger: ## runs the badger KVStore submodule's test suite + go test -v -p 1 ./kvstore/badger/... -mod=readonly -race ##################### ### Benchmark ### diff --git a/README.md b/README.md index a1a6128..0ff0d9a 100644 --- a/README.md +++ b/README.md @@ -40,6 +40,13 @@ To run all tests (excluding benchmarks) run the following command: make test_all ``` +To test the `badger` submodule that provides a more fully featured key-value +store run the following command: + +```sh +make test_badger +``` + ## Benchmarks To run the full suite of benchmarks simply run the following command: diff --git a/benchmarks/bench_leaf_test.go b/benchmarks/bench_leaf_test.go index 76f14b1..da4eb9b 100644 --- a/benchmarks/bench_leaf_test.go +++ b/benchmarks/bench_leaf_test.go @@ -1,3 +1,5 @@ +//go:build benchmarks + package smt import ( @@ -9,13 +11,13 @@ import ( "github.com/stretchr/testify/require" "github.com/pokt-network/smt" + "github.com/pokt-network/smt/kvstore" ) func BenchmarkSMTLeafSizes_Fill(b *testing.B) { trieSizes := []int{100000, 500000, 1000000, 5000000, 10000000} // number of leaves leafSizes := []int{256, 512, 1024, 2048, 4096, 8192, 16384} // number of bytes per leaf - nodes, err := smt.NewKVStore("") - require.NoError(b, err) + nodes := kvstore.NewSimpleMap() for _, trieSize := range trieSizes { for _, leafSize := range leafSizes { leaf := make([]byte, leafSize) @@ -39,14 +41,12 @@ func BenchmarkSMTLeafSizes_Fill(b *testing.B) { } } } - require.NoError(b, nodes.Stop()) } func BenchmarkSMSTLeafSizes_Fill(b *testing.B) { trieSizes := []int{100000, 500000, 1000000, 5000000, 10000000} // number of leaves leafSizes := []int{256, 512, 1024, 2048, 4096, 8192, 16384} // number of bytes per leaf - nodes, err := smt.NewKVStore("") - require.NoError(b, err) + nodes := kvstore.NewSimpleMap() for _, trieSize := range trieSizes { for _, leafSize := range leafSizes { leaf := make([]byte, leafSize) @@ -70,5 +70,4 @@ func BenchmarkSMSTLeafSizes_Fill(b *testing.B) { } } } - require.NoError(b, nodes.Stop()) } diff --git a/benchmarks/bench_smst_test.go b/benchmarks/bench_smst_test.go index 882c616..8834381 100644 --- a/benchmarks/bench_smst_test.go +++ b/benchmarks/bench_smst_test.go @@ -1,3 +1,5 @@ +//go:build benchmarks + package smt import ( @@ -11,77 +13,66 @@ import ( func BenchmarkSparseMerkleSumTrie_Fill(b *testing.B) { testCases := []struct { - name string - trieSize int - commit bool - persistent bool + name string + trieSize int + commit bool }{ { - name: "Fill (100000)", - trieSize: 100000, - commit: false, - persistent: false, + name: "Fill (100000)", + trieSize: 100000, + commit: false, }, { - name: "Fill & Commit (100000)", - trieSize: 100000, - commit: true, - persistent: false, + name: "Fill & Commit (100000)", + trieSize: 100000, + commit: true, }, { - name: "Fill (500000)", - trieSize: 500000, - commit: false, - persistent: false, + name: "Fill (500000)", + trieSize: 500000, + commit: false, }, { - name: "Fill & Commit (500000)", - trieSize: 500000, - commit: true, - persistent: false, + name: "Fill & Commit (500000)", + trieSize: 500000, + commit: true, }, { - name: "Fill (1000000)", - trieSize: 1000000, - commit: false, - persistent: false, + name: "Fill (1000000)", + trieSize: 1000000, + commit: false, }, { - name: "Fill & Commit (1000000)", - trieSize: 1000000, - commit: true, - persistent: false, + name: "Fill & Commit (1000000)", + trieSize: 1000000, + commit: true, }, { - name: "Fill (5000000)", - trieSize: 5000000, - commit: false, - persistent: false, + name: "Fill (5000000)", + trieSize: 5000000, + commit: false, }, { - name: "Fill & Commit (5000000)", - trieSize: 5000000, - commit: true, - persistent: false, + name: "Fill & Commit (5000000)", + trieSize: 5000000, + commit: true, }, { - name: "Fill (10000000)", - trieSize: 10000000, - commit: false, - persistent: false, + name: "Fill (10000000)", + trieSize: 10000000, + commit: false, }, { - name: "Fill & Commit (10000000)", - trieSize: 10000000, - commit: true, - persistent: false, + name: "Fill & Commit (10000000)", + trieSize: 10000000, + commit: true, }, } for _, tc := range testCases { b.ResetTimer() b.Run(tc.name, func(b *testing.B) { - trie := setupSMST(b, tc.persistent, tc.trieSize) + trie := setupSMST(b, tc.trieSize) b.ResetTimer() b.StartTimer() b.ReportAllocs() @@ -101,88 +92,77 @@ func BenchmarkSparseMerkleSumTrie_Fill(b *testing.B) { func BenchmarkSparseMerkleSumTrie_Update(b *testing.B) { testCases := []struct { - name string - trieSize int - commit bool - persistent bool - fn func(*smt.SMST, uint64) error + name string + trieSize int + commit bool + fn func(*smt.SMST, uint64) error }{ { - name: "Update (Prefilled: 100000)", - trieSize: 100000, - commit: false, - persistent: false, - fn: updSMST, + name: "Update (Prefilled: 100000)", + trieSize: 100000, + commit: false, + fn: updSMST, }, { - name: "Update & Commit (Prefilled: 100000)", - trieSize: 100000, - commit: true, - persistent: false, - fn: updSMST, + name: "Update & Commit (Prefilled: 100000)", + trieSize: 100000, + commit: true, + fn: updSMST, }, { - name: "Update (Prefilled: 500000)", - trieSize: 500000, - commit: false, - persistent: false, - fn: updSMST, + name: "Update (Prefilled: 500000)", + trieSize: 500000, + commit: false, + fn: updSMST, }, { - name: "Update & Commit (Prefilled: 500000)", - trieSize: 500000, - commit: true, - persistent: false, - fn: updSMST, + name: "Update & Commit (Prefilled: 500000)", + trieSize: 500000, + commit: true, + fn: updSMST, }, { - name: "Update (Prefilled: 1000000)", - trieSize: 1000000, - commit: false, - persistent: false, - fn: updSMST, + name: "Update (Prefilled: 1000000)", + trieSize: 1000000, + commit: false, + fn: updSMST, }, { - name: "Update & Commit (Prefilled: 1000000)", - trieSize: 1000000, - commit: true, - persistent: false, - fn: updSMST, + name: "Update & Commit (Prefilled: 1000000)", + trieSize: 1000000, + commit: true, + fn: updSMST, }, { - name: "Update (Prefilled: 5000000)", - trieSize: 5000000, - commit: false, - persistent: false, - fn: updSMST, + name: "Update (Prefilled: 5000000)", + trieSize: 5000000, + commit: false, + fn: updSMST, }, { - name: "Update & Commit (Prefilled: 5000000)", - trieSize: 5000000, - commit: true, - persistent: false, - fn: updSMST, + name: "Update & Commit (Prefilled: 5000000)", + trieSize: 5000000, + commit: true, + fn: updSMST, }, { - name: "Update (Prefilled: 10000000)", - trieSize: 10000000, - commit: false, - persistent: false, - fn: updSMST, + name: "Update (Prefilled: 10000000)", + trieSize: 10000000, + commit: false, + fn: updSMST, }, { - name: "Update & Commit (Prefilled: 10000000)", - trieSize: 10000000, - commit: true, - persistent: false, - fn: updSMST, + name: "Update & Commit (Prefilled: 10000000)", + trieSize: 10000000, + commit: true, + fn: updSMST, }, } for _, tc := range testCases { b.ResetTimer() b.Run(tc.name, func(b *testing.B) { - trie := setupSMST(b, tc.persistent, tc.trieSize) + trie := setupSMST(b, tc.trieSize) benchmarkSMST(b, trie, tc.commit, tc.fn) }) } @@ -190,53 +170,47 @@ func BenchmarkSparseMerkleSumTrie_Update(b *testing.B) { func BenchmarkSparseMerkleSumTrie_Get(b *testing.B) { testCases := []struct { - name string - trieSize int - commit bool - persistent bool - fn func(*smt.SMST, uint64) error + name string + trieSize int + commit bool + fn func(*smt.SMST, uint64) error }{ { - name: "Get (Prefilled: 100000)", - trieSize: 100000, - commit: false, - persistent: false, - fn: getSMST, + name: "Get (Prefilled: 100000)", + trieSize: 100000, + commit: false, + fn: getSMST, }, { - name: "Get (Prefilled: 500000)", - trieSize: 500000, - commit: false, - persistent: false, - fn: getSMST, + name: "Get (Prefilled: 500000)", + trieSize: 500000, + commit: false, + fn: getSMST, }, { - name: "Get (Prefilled: 1000000)", - trieSize: 1000000, - commit: false, - persistent: false, - fn: getSMST, + name: "Get (Prefilled: 1000000)", + trieSize: 1000000, + commit: false, + fn: getSMST, }, { - name: "Get (Prefilled: 5000000)", - trieSize: 5000000, - commit: false, - persistent: false, - fn: getSMST, + name: "Get (Prefilled: 5000000)", + trieSize: 5000000, + commit: false, + fn: getSMST, }, { - name: "Get (Prefilled: 10000000)", - trieSize: 10000000, - commit: false, - persistent: false, - fn: getSMST, + name: "Get (Prefilled: 10000000)", + trieSize: 10000000, + commit: false, + fn: getSMST, }, } for _, tc := range testCases { b.ResetTimer() b.Run(tc.name, func(b *testing.B) { - trie := setupSMST(b, tc.persistent, tc.trieSize) + trie := setupSMST(b, tc.trieSize) benchmarkSMST(b, trie, tc.commit, tc.fn) }) } @@ -244,53 +218,47 @@ func BenchmarkSparseMerkleSumTrie_Get(b *testing.B) { func BenchmarkSparseMerkleSumTrie_Prove(b *testing.B) { testCases := []struct { - name string - trieSize int - commit bool - persistent bool - fn func(*smt.SMST, uint64) error + name string + trieSize int + commit bool + fn func(*smt.SMST, uint64) error }{ { - name: "Prove (Prefilled: 100000)", - trieSize: 100000, - commit: false, - persistent: false, - fn: proSMST, + name: "Prove (Prefilled: 100000)", + trieSize: 100000, + commit: false, + fn: proSMST, }, { - name: "Prove (Prefilled: 500000)", - trieSize: 500000, - commit: false, - persistent: false, - fn: proSMST, + name: "Prove (Prefilled: 500000)", + trieSize: 500000, + commit: false, + fn: proSMST, }, { - name: "Prove (Prefilled: 1000000)", - trieSize: 1000000, - commit: false, - persistent: false, - fn: proSMST, + name: "Prove (Prefilled: 1000000)", + trieSize: 1000000, + commit: false, + fn: proSMST, }, { - name: "Prove (Prefilled: 5000000)", - trieSize: 5000000, - commit: false, - persistent: false, - fn: proSMST, + name: "Prove (Prefilled: 5000000)", + trieSize: 5000000, + commit: false, + fn: proSMST, }, { - name: "Prove (Prefilled: 10000000)", - trieSize: 10000000, - commit: false, - persistent: false, - fn: proSMST, + name: "Prove (Prefilled: 10000000)", + trieSize: 10000000, + commit: false, + fn: proSMST, }, } for _, tc := range testCases { b.ResetTimer() b.Run(tc.name, func(b *testing.B) { - trie := setupSMST(b, tc.persistent, tc.trieSize) + trie := setupSMST(b, tc.trieSize) benchmarkSMST(b, trie, tc.commit, tc.fn) }) } @@ -298,88 +266,77 @@ func BenchmarkSparseMerkleSumTrie_Prove(b *testing.B) { func BenchmarkSparseMerkleSumTrie_Delete(b *testing.B) { testCases := []struct { - name string - trieSize int - commit bool - persistent bool - fn func(*smt.SMST, uint64) error + name string + trieSize int + commit bool + fn func(*smt.SMST, uint64) error }{ { - name: "Delete (Prefilled: 100000)", - trieSize: 100000, - commit: false, - persistent: false, - fn: delSMST, + name: "Delete (Prefilled: 100000)", + trieSize: 100000, + commit: false, + fn: delSMST, }, { - name: "Delete & Commit (Prefilled: 100000)", - trieSize: 100000, - commit: true, - persistent: false, - fn: delSMST, + name: "Delete & Commit (Prefilled: 100000)", + trieSize: 100000, + commit: true, + fn: delSMST, }, { - name: "Delete (Prefilled: 500000)", - trieSize: 500000, - commit: false, - persistent: false, - fn: delSMST, + name: "Delete (Prefilled: 500000)", + trieSize: 500000, + commit: false, + fn: delSMST, }, { - name: "Delete & Commit (Prefilled: 500000)", - trieSize: 500000, - commit: true, - persistent: false, - fn: delSMST, + name: "Delete & Commit (Prefilled: 500000)", + trieSize: 500000, + commit: true, + fn: delSMST, }, { - name: "Delete (Prefilled: 1000000)", - trieSize: 1000000, - commit: false, - persistent: false, - fn: delSMST, + name: "Delete (Prefilled: 1000000)", + trieSize: 1000000, + commit: false, + fn: delSMST, }, { - name: "Delete & Commit (Prefilled: 1000000)", - trieSize: 1000000, - commit: true, - persistent: false, - fn: delSMST, + name: "Delete & Commit (Prefilled: 1000000)", + trieSize: 1000000, + commit: true, + fn: delSMST, }, { - name: "Delete (Prefilled: 5000000)", - trieSize: 5000000, - commit: false, - persistent: false, - fn: delSMST, + name: "Delete (Prefilled: 5000000)", + trieSize: 5000000, + commit: false, + fn: delSMST, }, { - name: "Delete & Commit (Prefilled: 5000000)", - trieSize: 5000000, - commit: true, - persistent: false, - fn: delSMST, + name: "Delete & Commit (Prefilled: 5000000)", + trieSize: 5000000, + commit: true, + fn: delSMST, }, { - name: "Delete (Prefilled: 10000000)", - trieSize: 10000000, - commit: false, - persistent: false, - fn: delSMST, + name: "Delete (Prefilled: 10000000)", + trieSize: 10000000, + commit: false, + fn: delSMST, }, { - name: "Delete & Commit (Prefilled: 10000000)", - trieSize: 10000000, - commit: true, - persistent: false, - fn: delSMST, + name: "Delete & Commit (Prefilled: 10000000)", + trieSize: 10000000, + commit: true, + fn: delSMST, }, } for _, tc := range testCases { b.ResetTimer() b.Run(tc.name, func(b *testing.B) { - trie := setupSMST(b, tc.persistent, tc.trieSize) + trie := setupSMST(b, tc.trieSize) benchmarkSMST(b, trie, tc.commit, tc.fn) }) } diff --git a/benchmarks/bench_smt_test.go b/benchmarks/bench_smt_test.go index 01ad070..3169068 100644 --- a/benchmarks/bench_smt_test.go +++ b/benchmarks/bench_smt_test.go @@ -1,3 +1,5 @@ +//go:build benchmarks + package smt import ( @@ -11,77 +13,66 @@ import ( func BenchmarkSparseMerkleTrie_Fill(b *testing.B) { testCases := []struct { - name string - trieSize int - commit bool - persistent bool + name string + trieSize int + commit bool }{ { - name: "Fill (100000)", - trieSize: 100000, - commit: false, - persistent: false, + name: "Fill (100000)", + trieSize: 100000, + commit: false, }, { - name: "Fill & Commit (100000)", - trieSize: 100000, - commit: true, - persistent: false, + name: "Fill & Commit (100000)", + trieSize: 100000, + commit: true, }, { - name: "Fill (500000)", - trieSize: 500000, - commit: false, - persistent: false, + name: "Fill (500000)", + trieSize: 500000, + commit: false, }, { - name: "Fill & Commit (500000)", - trieSize: 500000, - commit: true, - persistent: false, + name: "Fill & Commit (500000)", + trieSize: 500000, + commit: true, }, { - name: "Fill (1000000)", - trieSize: 1000000, - commit: false, - persistent: false, + name: "Fill (1000000)", + trieSize: 1000000, + commit: false, }, { - name: "Fill & Commit (1000000)", - trieSize: 1000000, - commit: true, - persistent: false, + name: "Fill & Commit (1000000)", + trieSize: 1000000, + commit: true, }, { - name: "Fill (5000000)", - trieSize: 5000000, - commit: false, - persistent: false, + name: "Fill (5000000)", + trieSize: 5000000, + commit: false, }, { - name: "Fill & Commit (5000000)", - trieSize: 5000000, - commit: true, - persistent: false, + name: "Fill & Commit (5000000)", + trieSize: 5000000, + commit: true, }, { - name: "Fill (10000000)", - trieSize: 10000000, - commit: false, - persistent: false, + name: "Fill (10000000)", + trieSize: 10000000, + commit: false, }, { - name: "Fill & Commit (10000000)", - trieSize: 10000000, - commit: true, - persistent: false, + name: "Fill & Commit (10000000)", + trieSize: 10000000, + commit: true, }, } for _, tc := range testCases { b.ResetTimer() b.Run(tc.name, func(b *testing.B) { - trie := setupSMT(b, tc.persistent, tc.trieSize) + trie := setupSMT(b, tc.trieSize) b.ResetTimer() b.StartTimer() b.ReportAllocs() @@ -101,88 +92,77 @@ func BenchmarkSparseMerkleTrie_Fill(b *testing.B) { func BenchmarkSparseMerkleTrie_Update(b *testing.B) { testCases := []struct { - name string - trieSize int - commit bool - persistent bool - fn func(*smt.SMT, []byte) error + name string + trieSize int + commit bool + fn func(*smt.SMT, []byte) error }{ { - name: "Update (Prefilled: 100000)", - trieSize: 100000, - commit: false, - persistent: false, - fn: updSMT, + name: "Update (Prefilled: 100000)", + trieSize: 100000, + commit: false, + fn: updSMT, }, { - name: "Update & Commit (Prefilled: 100000)", - trieSize: 100000, - commit: true, - persistent: false, - fn: updSMT, + name: "Update & Commit (Prefilled: 100000)", + trieSize: 100000, + commit: true, + fn: updSMT, }, { - name: "Update (Prefilled: 500000)", - trieSize: 500000, - commit: false, - persistent: false, - fn: updSMT, + name: "Update (Prefilled: 500000)", + trieSize: 500000, + commit: false, + fn: updSMT, }, { - name: "Update & Commit (Prefilled: 500000)", - trieSize: 500000, - commit: true, - persistent: false, - fn: updSMT, + name: "Update & Commit (Prefilled: 500000)", + trieSize: 500000, + commit: true, + fn: updSMT, }, { - name: "Update (Prefilled: 1000000)", - trieSize: 1000000, - commit: false, - persistent: false, - fn: updSMT, + name: "Update (Prefilled: 1000000)", + trieSize: 1000000, + commit: false, + fn: updSMT, }, { - name: "Update & Commit (Prefilled: 1000000)", - trieSize: 1000000, - commit: true, - persistent: false, - fn: updSMT, + name: "Update & Commit (Prefilled: 1000000)", + trieSize: 1000000, + commit: true, + fn: updSMT, }, { - name: "Update (Prefilled: 5000000)", - trieSize: 5000000, - commit: false, - persistent: false, - fn: updSMT, + name: "Update (Prefilled: 5000000)", + trieSize: 5000000, + commit: false, + fn: updSMT, }, { - name: "Update & Commit (Prefilled: 5000000)", - trieSize: 5000000, - commit: true, - persistent: false, - fn: updSMT, + name: "Update & Commit (Prefilled: 5000000)", + trieSize: 5000000, + commit: true, + fn: updSMT, }, { - name: "Update (Prefilled: 10000000)", - trieSize: 10000000, - commit: false, - persistent: false, - fn: updSMT, + name: "Update (Prefilled: 10000000)", + trieSize: 10000000, + commit: false, + fn: updSMT, }, { - name: "Update & Commit (Prefilled: 10000000)", - trieSize: 10000000, - commit: true, - persistent: false, - fn: updSMT, + name: "Update & Commit (Prefilled: 10000000)", + trieSize: 10000000, + commit: true, + fn: updSMT, }, } for _, tc := range testCases { b.ResetTimer() b.Run(tc.name, func(b *testing.B) { - trie := setupSMT(b, tc.persistent, tc.trieSize) + trie := setupSMT(b, tc.trieSize) benchmarkSMT(b, trie, tc.commit, tc.fn) }) } @@ -190,53 +170,47 @@ func BenchmarkSparseMerkleTrie_Update(b *testing.B) { func BenchmarkSparseMerkleTrie_Get(b *testing.B) { testCases := []struct { - name string - trieSize int - commit bool - persistent bool - fn func(*smt.SMT, []byte) error + name string + trieSize int + commit bool + fn func(*smt.SMT, []byte) error }{ { - name: "Get (Prefilled: 100000)", - trieSize: 100000, - commit: false, - persistent: false, - fn: getSMT, + name: "Get (Prefilled: 100000)", + trieSize: 100000, + commit: false, + fn: getSMT, }, { - name: "Get (Prefilled: 500000)", - trieSize: 500000, - commit: false, - persistent: false, - fn: getSMT, + name: "Get (Prefilled: 500000)", + trieSize: 500000, + commit: false, + fn: getSMT, }, { - name: "Get (Prefilled: 1000000)", - trieSize: 1000000, - commit: false, - persistent: false, - fn: getSMT, + name: "Get (Prefilled: 1000000)", + trieSize: 1000000, + commit: false, + fn: getSMT, }, { - name: "Get (Prefilled: 5000000)", - trieSize: 5000000, - commit: false, - persistent: false, - fn: getSMT, + name: "Get (Prefilled: 5000000)", + trieSize: 5000000, + commit: false, + fn: getSMT, }, { - name: "Get (Prefilled: 10000000)", - trieSize: 10000000, - commit: false, - persistent: false, - fn: getSMT, + name: "Get (Prefilled: 10000000)", + trieSize: 10000000, + commit: false, + fn: getSMT, }, } for _, tc := range testCases { b.ResetTimer() b.Run(tc.name, func(b *testing.B) { - trie := setupSMT(b, tc.persistent, tc.trieSize) + trie := setupSMT(b, tc.trieSize) benchmarkSMT(b, trie, tc.commit, tc.fn) }) } @@ -244,53 +218,47 @@ func BenchmarkSparseMerkleTrie_Get(b *testing.B) { func BenchmarkSparseMerkleTrie_Prove(b *testing.B) { testCases := []struct { - name string - trieSize int - commit bool - persistent bool - fn func(*smt.SMT, []byte) error + name string + trieSize int + commit bool + fn func(*smt.SMT, []byte) error }{ { - name: "Prove (Prefilled: 100000)", - trieSize: 100000, - commit: false, - persistent: false, - fn: proSMT, + name: "Prove (Prefilled: 100000)", + trieSize: 100000, + commit: false, + fn: proSMT, }, { - name: "Prove (Prefilled: 500000)", - trieSize: 500000, - commit: false, - persistent: false, - fn: proSMT, + name: "Prove (Prefilled: 500000)", + trieSize: 500000, + commit: false, + fn: proSMT, }, { - name: "Prove (Prefilled: 1000000)", - trieSize: 1000000, - commit: false, - persistent: false, - fn: proSMT, + name: "Prove (Prefilled: 1000000)", + trieSize: 1000000, + commit: false, + fn: proSMT, }, { - name: "Prove (Prefilled: 5000000)", - trieSize: 5000000, - commit: false, - persistent: false, - fn: proSMT, + name: "Prove (Prefilled: 5000000)", + trieSize: 5000000, + commit: false, + fn: proSMT, }, { - name: "Prove (Prefilled: 10000000)", - trieSize: 10000000, - commit: false, - persistent: false, - fn: proSMT, + name: "Prove (Prefilled: 10000000)", + trieSize: 10000000, + commit: false, + fn: proSMT, }, } for _, tc := range testCases { b.ResetTimer() b.Run(tc.name, func(b *testing.B) { - trie := setupSMT(b, tc.persistent, tc.trieSize) + trie := setupSMT(b, tc.trieSize) benchmarkSMT(b, trie, tc.commit, tc.fn) }) } @@ -298,88 +266,77 @@ func BenchmarkSparseMerkleTrie_Prove(b *testing.B) { func BenchmarkSparseMerkleTrie_Delete(b *testing.B) { testCases := []struct { - name string - trieSize int - commit bool - persistent bool - fn func(*smt.SMT, []byte) error + name string + trieSize int + commit bool + fn func(*smt.SMT, []byte) error }{ { - name: "Delete (Prefilled: 100000)", - trieSize: 100000, - commit: false, - persistent: false, - fn: delSMT, + name: "Delete (Prefilled: 100000)", + trieSize: 100000, + commit: false, + fn: delSMT, }, { - name: "Delete & Commit (Prefilled: 100000)", - trieSize: 100000, - commit: true, - persistent: false, - fn: delSMT, + name: "Delete & Commit (Prefilled: 100000)", + trieSize: 100000, + commit: true, + fn: delSMT, }, { - name: "Delete (Prefilled: 500000)", - trieSize: 500000, - commit: false, - persistent: false, - fn: delSMT, + name: "Delete (Prefilled: 500000)", + trieSize: 500000, + commit: false, + fn: delSMT, }, { - name: "Delete & Commit (Prefilled: 500000)", - trieSize: 500000, - commit: true, - persistent: false, - fn: delSMT, + name: "Delete & Commit (Prefilled: 500000)", + trieSize: 500000, + commit: true, + fn: delSMT, }, { - name: "Delete (Prefilled: 1000000)", - trieSize: 1000000, - commit: false, - persistent: false, - fn: delSMT, + name: "Delete (Prefilled: 1000000)", + trieSize: 1000000, + commit: false, + fn: delSMT, }, { - name: "Delete & Commit (Prefilled: 1000000)", - trieSize: 1000000, - commit: true, - persistent: false, - fn: delSMT, + name: "Delete & Commit (Prefilled: 1000000)", + trieSize: 1000000, + commit: true, + fn: delSMT, }, { - name: "Delete (Prefilled: 5000000)", - trieSize: 5000000, - commit: false, - persistent: false, - fn: delSMT, + name: "Delete (Prefilled: 5000000)", + trieSize: 5000000, + commit: false, + fn: delSMT, }, { - name: "Delete & Commit (Prefilled: 5000000)", - trieSize: 5000000, - commit: true, - persistent: false, - fn: delSMT, + name: "Delete & Commit (Prefilled: 5000000)", + trieSize: 5000000, + commit: true, + fn: delSMT, }, { - name: "Delete (Prefilled: 10000000)", - trieSize: 10000000, - commit: false, - persistent: false, - fn: delSMT, + name: "Delete (Prefilled: 10000000)", + trieSize: 10000000, + commit: false, + fn: delSMT, }, { - name: "Delete & Commit (Prefilled: 10000000)", - trieSize: 10000000, - commit: true, - persistent: false, - fn: delSMT, + name: "Delete & Commit (Prefilled: 10000000)", + trieSize: 10000000, + commit: true, + fn: delSMT, }, } for _, tc := range testCases { b.ResetTimer() b.Run(tc.name, func(b *testing.B) { - trie := setupSMT(b, tc.persistent, tc.trieSize) + trie := setupSMT(b, tc.trieSize) benchmarkSMT(b, trie, tc.commit, tc.fn) }) } diff --git a/benchmarks/bench_utils_test.go b/benchmarks/bench_utils_test.go index 0083ffa..030880c 100644 --- a/benchmarks/bench_utils_test.go +++ b/benchmarks/bench_utils_test.go @@ -1,15 +1,17 @@ +//go:build benchmarks + package smt import ( "crypto/sha256" "encoding/binary" - "os" "strconv" "testing" "github.com/stretchr/testify/require" "github.com/pokt-network/smt" + "github.com/pokt-network/smt/kvstore" ) var ( @@ -52,14 +54,9 @@ var ( } ) -func setupSMT(b *testing.B, persistent bool, numLeaves int) *smt.SMT { +func setupSMT(b *testing.B, numLeaves int) *smt.SMT { b.Helper() - path := "" - if persistent { - path = b.TempDir() - } - nodes, err := smt.NewKVStore(path) - require.NoError(b, err) + nodes := kvstore.NewSimpleMap() trie := smt.NewSparseMerkleTrie(nodes, sha256.New()) for i := 0; i < numLeaves; i++ { s := strconv.Itoa(i) @@ -68,10 +65,6 @@ func setupSMT(b *testing.B, persistent bool, numLeaves int) *smt.SMT { require.NoError(b, trie.Commit()) b.Cleanup(func() { require.NoError(b, nodes.ClearAll()) - require.NoError(b, nodes.Stop()) - if path != "" { - require.NoError(b, os.RemoveAll(path)) - } }) return trie } @@ -90,14 +83,9 @@ func benchmarkSMT(b *testing.B, trie *smt.SMT, commit bool, fn func(*smt.SMT, [] b.StopTimer() } -func setupSMST(b *testing.B, persistent bool, numLeaves int) *smt.SMST { +func setupSMST(b *testing.B, numLeaves int) *smt.SMST { b.Helper() - path := "" - if persistent { - path = b.TempDir() - } - nodes, err := smt.NewKVStore(path) - require.NoError(b, err) + nodes := kvstore.NewSimpleMap() trie := smt.NewSparseMerkleSumTrie(nodes, sha256.New()) for i := 0; i < numLeaves; i++ { s := strconv.Itoa(i) @@ -106,10 +94,6 @@ func setupSMST(b *testing.B, persistent bool, numLeaves int) *smt.SMST { require.NoError(b, trie.Commit()) b.Cleanup(func() { require.NoError(b, nodes.ClearAll()) - require.NoError(b, nodes.Stop()) - if path != "" { - require.NoError(b, os.RemoveAll(path)) - } }) return trie } diff --git a/benchmarks/proof_sizes_test.go b/benchmarks/proof_sizes_test.go index e020fad..ff2e3dc 100644 --- a/benchmarks/proof_sizes_test.go +++ b/benchmarks/proof_sizes_test.go @@ -1,3 +1,5 @@ +//go:build benchmarks + package smt import ( @@ -8,11 +10,11 @@ import ( "github.com/stretchr/testify/require" "github.com/pokt-network/smt" + "github.com/pokt-network/smt/kvstore" ) func TestSMT_ProofSizes(t *testing.T) { - nodes, err := smt.NewKVStore("") - require.NoError(t, err) + nodes := kvstore.NewSimpleMap() testCases := []struct { name string trieSize int @@ -96,12 +98,10 @@ func TestSMT_ProofSizes(t *testing.T) { }) require.NoError(t, nodes.ClearAll()) } - require.NoError(t, nodes.Stop()) } func TestSMST_ProofSizes(t *testing.T) { - nodes, err := smt.NewKVStore("") - require.NoError(t, err) + nodes := kvstore.NewSimpleMap() testCases := []struct { name string trieSize int @@ -185,5 +185,4 @@ func TestSMST_ProofSizes(t *testing.T) { }) require.NoError(t, nodes.ClearAll()) } - require.NoError(t, nodes.Stop()) } diff --git a/bulk_test.go b/bulk_test.go index e83c60e..7b58723 100644 --- a/bulk_test.go +++ b/bulk_test.go @@ -8,6 +8,8 @@ import ( "testing" "github.com/stretchr/testify/require" + + "github.com/pokt-network/smt/kvstore" ) type ( @@ -32,10 +34,8 @@ func TestBulkOperations(t *testing.T) { // Test all trie operations in bulk, with specified ratio probabilities of insert, update and delete. func bulkOperations(t *testing.T, operations int, insert int, update int, delete int) { - smn, err := NewKVStore("") - require.NoError(t, err) - smv, err := NewKVStore("") - require.NoError(t, err) + smn := kvstore.NewSimpleMap() + smv := kvstore.NewSimpleMap() smt := NewSMTWithStorage(smn, smv, sha256.New()) max := insert + update + delete @@ -82,7 +82,7 @@ func bulkOperations(t *testing.T, operations int, insert int, update int, delete ki := r.Intn(len(kv)) err := smt.Delete(kv[ki].key) - if err != nil && err != ErrKeyNotPresent { + if err != nil && err != kvstore.ErrKVStoreKeyNotFound { t.Fatalf("error: %v", err) } kv[ki].val = defaultValue @@ -90,8 +90,6 @@ func bulkOperations(t *testing.T, operations int, insert int, update int, delete } bulkCheckAll(t, smt, kv) - require.NoError(t, smn.Stop()) - require.NoError(t, smv.Stop()) } func bulkCheckAll(t *testing.T, smt *SMTWithStorage, kv []bulkop) { diff --git a/docs/kvstore.md b/docs/badger-store.md similarity index 61% rename from docs/kvstore.md rename to docs/badger-store.md index 78c7c5a..c1d9d0e 100644 --- a/docs/kvstore.md +++ b/docs/badger-store.md @@ -1,4 +1,4 @@ -# KVStore +# BadgerStore @@ -19,27 +19,30 @@ ## Overview -The `KVStore` interface is a key-value store that is used by the `SMT` and -`SMST` as its underlying database for its nodes. However, it is an independent -key-value store that can be used for any purpose. +The `bagder` submodule provides a `BadgerStore` interface that can be used with +the `SM(S)T` as both an in-memory and persistent DB backend for the trie. It also +exposes numerous methods not needed by the `SM(S)T` for use as a general purpose +key-value store. + +_NOTE: The `MapStore` interface used by the SM(S)T is implemented by the badger +`BadgerStore` interface and can be used as the database for the trie. It's +interface also has other features making it useful for uses outside of the +SM(S)T itself._ ## Implementation -The `KVStore` is implemented in [`kvstore.go`](../kvstore.go) and is a wrapper -around the [BadgerDB](https://github.com/dgraph-io/badger) key-value database. +The `BadgerStore` is implemented in [`kvstore.go`](../kvstore/badger/kvstore.go) +and is a wrapper around the [BadgerDB](https://github.com/dgraph-io/badger) v4 +key-value database. The interface defines simple key-value store accessor methods as well as other methods desired from a key-value database in general, this can be found in -[`kvstore.go`](../kvstore.go). - -_NOTE: The `KVStore` interface can be implemented by any key-value store that -satisfies the interface and used as the underlying database store for the -`SM(S)T`_ +[`kvstore.go`](../kvstore/badger/interface.go). ### In-Memory and Persistent -The `KVStore` implementation can be used as an in-memory or persistent key-value -store. The `NewKVStore` function takes a `path` argument that can be used to +The `BadgerStore` implementation can be used as an in-memory or persistent key-value +store. The `NewBadgerStore` function takes a `path` argument that can be used to specify a path to a directory to store the database files. If the `path` is an empty string, the database will be stored in-memory. @@ -48,25 +51,25 @@ and be writeable by the user running the application._ ### Store methods -As a key-value store the `KVStore` interface defines the simple `Get`, `Set` and -`Delete` methods to access and modify the underlying database. +As a key-value store the `BadgerStore` interface defines the simple `Get`, `Set` +and `Delete` methods to access and modify the underlying database. ### Lifecycle Methods -The `Stop` method **must** be called when the `KVStore` is no longer needed. +The `Stop` method **must** be called when the `BadgerStore` is no longer needed. This method closes the underlying database and frees up any resources used by -the `KVStore`. +the `BadgerStore`. For persistent databases, the `Stop` method should be called when the application no longer needs to access the database. For in-memory databases, the -`Stop` method should be called when the `KVStore` is no longer needed. +`Stop` method should be called when the `BadgerStore` is no longer needed. -_NOTE: A persistent `KVStore` that is not stopped will stop another `KVStore` -from opening the database._ +_NOTE: A persistent `BadgerStore` that is not stopped will stop another +`BadgerStore` from opening the database._ ### Data Methods -The `KVStore` interface provides two methods to allow backups and restorations. +The `BadgerStore` interface provides two methods to allow backups and restorations. #### Backups @@ -77,7 +80,7 @@ this purpose. When the `incremental` bool is `false` a full backup will be performed, otherwise an incremental backup will be performed. This is enabled by the -`KVStore` keeping the timestamp of its last backup and only backing up data that +`BadgerStore` keeping the timestamp of its last backup and only backing up data that has been modified since the last backup. #### Restorations @@ -85,10 +88,10 @@ has been modified since the last backup. The `Restore` method takes an `io.Reader` and restores the database from this reader. -The `KVStore` calling the `Restore` method is expected to be initialised and +The `BadgerStore` calling the `Restore` method is expected to be initialised and open, otherwise the restore will fail. -_NOTE: Any data contained in the `KVStore` when calling restore will be +_NOTE: Any data contained in the `BadgerStore` when calling restore will be overwritten._ ### Accessor Methods @@ -102,8 +105,8 @@ The `GetAll` method supports the retrieval of all keys and values, where the key has a specific prefix. The `descending` bool indicates whether the keys should be returned in descending order or not. -_NOTE: In order to retrieve all keys and values the empty prefix `[]byte{}` -should be used to match all keys_ +_NOTE: In order to retrieve all keys and values the empty prefix `[]byte{}` or +nil should be used to match all keys_ #### Clear All Key-Value Pairs diff --git a/docs/merkle-sum-trie.md b/docs/merkle-sum-trie.md index 260164e..498e43b 100644 --- a/docs/merkle-sum-trie.md +++ b/docs/merkle-sum-trie.md @@ -280,15 +280,13 @@ import ( "fmt" "github.com/pokt-network/smt" + "github.com/pokt-network/smt/kvstore" ) func main() { // Initialise a new in-memory key-value store to store the nodes of the trie // (Note: the trie only stores hashed values, not raw value data) - nodeStore := smt.NewKVStore("") - - // Ensure the database connection closes - defer nodeStore.Stop() + nodeStore := kvstore.NewSimpleMap() // Initialise the trie trie := smt.NewSparseMerkleSumTrie(nodeStore, sha256.New()) diff --git a/docs/smt.md b/docs/smt.md index 603a710..ca7894d 100644 --- a/docs/smt.md +++ b/docs/smt.md @@ -24,6 +24,8 @@ * [Compression](#compression) * [Serialisation](#serialisation) - [Database](#database) + * [Database Submodules](#database-submodules) + + [Badger](#badger) * [Data Loss](#data-loss) - [Sparse Merkle Sum Trie](#sparse-merkle-sum-trie) - [Example](#example) @@ -445,20 +447,31 @@ schemes. ## Database -This library defines the `KVStore` interface which by default is implemented -using [BadgerDB](https://github.com/dgraph-io/badger), however any database that -implements this interface can be used as a drop in replacement. The `KVStore` -allows for both in memory and persisted databases to be used to store the nodes -for the SMT. +By default this library provides a simple interface (`MapStore`) and a simple +in-memory key-value database found in [`simplemap.go`](../kvstore/simplemap.go). +However, any key-value store that implements this interface can be used as the +node store to back the trie. -When changes are committed to the underlying database using `Commit()` the -digests of the leaf nodes are stored at their respective paths. If retrieved -manually from the database the returned value will be the digest of the leaf -node, **not** the leaf node's value, even when `WithValueHasher(nil)` is used. -The node value can be parsed from this value, as the trie `Get` function does -by removing the prefix and path bytes from the returned value. +See: [`kvstore/simplemap.go`](../kvstore/simplemap.go) for the `MapStore` +interface and simple key-value map implementation. -See [kvstore.md](./kvstore.md) for the details of the implementation. +### Database Submodules + +In addition to providing the `MapStore` and `SimpleMap` the `smt` library also +provides wrappers around other key-value databases as submodules with more +fully featured interfaces that can be used as outside of being a node store for +the tries. These submodules can be found in the [`kvstore`](../kvstore/) +directory. + +#### Badger + +This library defines the `BadgerStore` interface which is implemented as a +wrapper around the [BadgerDB](https://github.com/dgraph-io/badger) v4 key-value +database. It's interface exposes numerous extra methods not used by the trie, +However it can still be used as a node-store with both in-memory and persistent +options. + +See [badger-store.md](./badger-store.md.md) for the details of the implementation. ### Data Loss @@ -482,15 +495,13 @@ import ( "fmt" "github.com/pokt-network/smt" + "github.com/pokt-network/smt/kvstore" ) func main() { // Initialise a new in-memory key-value store to store the nodes of the trie // (Note: the trie only stores hashed values, not raw value data) - nodeStore := smt.NewKVStore("") - - // Ensure the database connection closes - defer nodeStore.Stop() + nodeStore := kvstore.NewSimpleMap() // Update the key "foo" with the value "bar" _ = trie.Update([]byte("foo"), []byte("bar")) diff --git a/errors.go b/errors.go new file mode 100644 index 0000000..ab908fb --- /dev/null +++ b/errors.go @@ -0,0 +1,8 @@ +package smt + +import ( + "errors" +) + +// ErrBadProof is returned when an invalid Merkle proof is supplied. +var ErrBadProof = errors.New("bad proof") diff --git a/fuzz_test.go b/fuzz_test.go index 347abeb..aed858f 100644 --- a/fuzz_test.go +++ b/fuzz_test.go @@ -8,11 +8,13 @@ import ( "testing" "github.com/stretchr/testify/require" + + "github.com/pokt-network/smt/kvstore" ) // FuzzSMT uses fuzzing to attempt to break the SMT implementation // in its current state. This fuzzing test does not confirm the SMT -// functions correctly, it only trys to detect when it fails unexpectedly +// functions correctly, it only tries to detect when it fails unexpectedly func FuzzSMT_DetectUnexpectedFailures(f *testing.F) { seeds := [][]byte{ []byte(""), @@ -25,8 +27,7 @@ func FuzzSMT_DetectUnexpectedFailures(f *testing.F) { f.Add(s) } f.Fuzz(func(t *testing.T, input []byte) { - smn, err := NewKVStore("") - require.NoError(t, err) + smn := kvstore.NewSimpleMap() trie := NewSparseMerkleTrie(smn, sha256.New()) r := bytes.NewReader(input) @@ -66,7 +67,10 @@ func FuzzSMT_DetectUnexpectedFailures(f *testing.F) { case Get: _, err := trie.Get(key()) if err != nil { - require.ErrorIsf(t, err, ErrKeyNotPresent, "unknown error occured while getting") + require.ErrorIsf( + t, err, kvstore.ErrKVStoreKeyNotFound, + "unknown error occurred while getting", + ) } newRoot := trie.Root() require.Equal(t, originalRoot, newRoot, "root changed while getting") @@ -74,13 +78,16 @@ func FuzzSMT_DetectUnexpectedFailures(f *testing.F) { value := make([]byte, 32) binary.BigEndian.PutUint64(value, uint64(i)) err := trie.Update(key(), value) - require.NoErrorf(t, err, "unknown error occured while updating") + require.NoErrorf(t, err, "unknown error occurred while updating") newRoot := trie.Root() require.NotEqual(t, originalRoot, newRoot, "root unchanged while updating") case Delete: err := trie.Delete(key()) if err != nil { - require.ErrorIsf(t, err, ErrKeyNotPresent, "unknown error occured while deleting") + require.ErrorIsf( + t, err, kvstore.ErrKVStoreKeyNotFound, + "unknown error occurred while deleting", + ) continue } // If the key was present check root has changed @@ -89,7 +96,10 @@ func FuzzSMT_DetectUnexpectedFailures(f *testing.F) { case Prove: _, err := trie.Prove(key()) if err != nil { - require.ErrorIsf(t, err, ErrKeyNotPresent, "unknown error occured while proving") + require.ErrorIsf( + t, err, kvstore.ErrKVStoreKeyNotFound, + "unknown error occurred while proving", + ) } newRoot := trie.Root() require.Equal(t, originalRoot, newRoot, "root changed while proving") @@ -100,8 +110,6 @@ func FuzzSMT_DetectUnexpectedFailures(f *testing.F) { newRoot := trie.Root() require.Greater(t, len(newRoot), 0, "new root is empty while err is nil") } - - require.NoError(t, smn.Stop()) }) } diff --git a/go.mod b/go.mod index aa402d0..31c721b 100644 --- a/go.mod +++ b/go.mod @@ -1,31 +1,14 @@ module github.com/pokt-network/smt -go 1.20 +go 1.21.5 -require ( - github.com/dgraph-io/badger/v4 v4.2.0 - github.com/stretchr/testify v1.7.1 -) +require github.com/stretchr/testify v1.7.1 require ( - github.com/cespare/xxhash/v2 v2.2.0 // indirect github.com/davecgh/go-spew v1.1.1 // indirect - github.com/dgraph-io/ristretto v0.1.1 // indirect - github.com/dustin/go-humanize v1.0.0 // indirect - github.com/gogo/protobuf v1.3.2 // indirect - github.com/golang/glog v1.0.0 // indirect - github.com/golang/groupcache v0.0.0-20190702054246-869f871628b6 // indirect - github.com/golang/protobuf v1.5.2 // indirect - github.com/golang/snappy v0.0.3 // indirect - github.com/google/flatbuffers v1.12.1 // indirect - github.com/google/go-cmp v0.5.8 // indirect - github.com/klauspost/compress v1.12.3 // indirect + github.com/kr/pretty v0.1.0 // indirect github.com/kr/text v0.2.0 // indirect - github.com/pkg/errors v0.9.1 // indirect github.com/pmezard/go-difflib v1.0.0 // indirect - go.opencensus.io v0.22.5 // indirect - golang.org/x/net v0.7.0 // indirect - golang.org/x/sys v0.11.0 // indirect - google.golang.org/protobuf v1.28.1 // indirect + gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15 // indirect gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c // indirect ) diff --git a/go.sum b/go.sum index a3b8661..492d451 100644 --- a/go.sum +++ b/go.sum @@ -1,121 +1,20 @@ -cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw= -github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU= -github.com/cespare/xxhash/v2 v2.1.1/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs= -github.com/cespare/xxhash/v2 v2.2.0 h1:DC2CZ1Ep5Y4k3ZQ899DldepgrayRUGE6BBZ/cd9Cj44= -github.com/cespare/xxhash/v2 v2.2.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs= -github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw= github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E= github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= -github.com/dgraph-io/badger/v4 v4.2.0 h1:kJrlajbXXL9DFTNuhhu9yCx7JJa4qpYWxtE8BzuWsEs= -github.com/dgraph-io/badger/v4 v4.2.0/go.mod h1:qfCqhPoWDFJRx1gp5QwwyGo8xk1lbHUxvK9nK0OGAak= -github.com/dgraph-io/ristretto v0.1.1 h1:6CWw5tJNgpegArSHpNHJKldNeq03FQCwYvfMVWajOK8= -github.com/dgraph-io/ristretto v0.1.1/go.mod h1:S1GPSBCYCIhmVNfcth17y2zZtQT6wzkzgwUve0VDWWA= -github.com/dgryski/go-farm v0.0.0-20190423205320-6a90982ecee2 h1:tdlZCpZ/P9DhczCTSixgIKmwPv6+wP5DGjqLYw5SUiA= -github.com/dgryski/go-farm v0.0.0-20190423205320-6a90982ecee2/go.mod h1:SqUrOPUnsFjfmXRMNPybcSiG0BgUW2AuFH8PAnS2iTw= -github.com/dustin/go-humanize v1.0.0 h1:VSnTsYCnlFHaM2/igO1h6X3HA71jcobQuxemgkq4zYo= -github.com/dustin/go-humanize v1.0.0/go.mod h1:HtrtbFcZ19U5GC7JDqmcUSB87Iq5E25KnS6fMYU6eOk= -github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q= -github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q= -github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q= -github.com/golang/glog v1.0.0 h1:nfP3RFugxnNRyKgeWd4oI1nYvXpxrx8ck8ZrcizshdQ= -github.com/golang/glog v1.0.0/go.mod h1:EWib/APOK0SL3dFbYqvxE3UYd8E6s1ouQ7iEp/0LWV4= -github.com/golang/groupcache v0.0.0-20190702054246-869f871628b6 h1:ZgQEtGgCBiWRM39fZuwSd1LwSqqSW0hOdXCYYDX0R3I= -github.com/golang/groupcache v0.0.0-20190702054246-869f871628b6/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc= -github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A= -github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= -github.com/golang/protobuf v1.3.1/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= -github.com/golang/protobuf v1.5.0/go.mod h1:FsONVRAS9T7sI+LIUmWTfcYkHO4aIWwzhcaSAoJOfIk= -github.com/golang/protobuf v1.5.2 h1:ROPKBNFfQgOUMifHyP+KYbvpjbdoFNs+aK7DXlji0Tw= -github.com/golang/protobuf v1.5.2/go.mod h1:XVQd3VNwM+JqD3oG2Ue2ip4fOMUkwXdXDdiuN0vRsmY= -github.com/golang/snappy v0.0.3 h1:fHPg5GQYlCeLIPB9BZqMVR5nR9A+IM5zcgeTdjMYmLA= -github.com/golang/snappy v0.0.3/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q= -github.com/google/flatbuffers v1.12.1 h1:MVlul7pQNoDzWRLTw5imwYsl+usrS1TXG2H4jg6ImGw= -github.com/google/flatbuffers v1.12.1/go.mod h1:1AeVuKshWv4vARoZatz6mlQ0JxURH0Kv5+zNeJKJCa8= -github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU= -github.com/google/go-cmp v0.5.5/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= -github.com/google/go-cmp v0.5.8 h1:e6P7q2lk1O+qJJb4BtCQXlK8vWEO8V1ZeuEdJNOqZyg= -github.com/google/go-cmp v0.5.8/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY= -github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8= -github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck= -github.com/klauspost/compress v1.12.3 h1:G5AfA94pHPysR56qqrkO2pxEexdDzrpFJ6yt/VqWxVU= -github.com/klauspost/compress v1.12.3/go.mod h1:8dP1Hq4DHOhN9w426knH3Rhby4rFm6D8eO+e+Dq5Gzg= github.com/kr/pretty v0.1.0 h1:L/CwN0zerZDmRFUapSPitk6f+Q3+0za1rQkzVuMiMFI= +github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo= +github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ= +github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI= github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY= github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE= -github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4= -github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= -github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4= github.com/stretchr/testify v1.7.1 h1:5TQK59W5E3v0r2duFAb7P95B6hEeOyEnHRa8MjYSMTY= github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= -github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= -github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= -go.opencensus.io v0.22.5 h1:dntmOdLpSpHlVqbW5Eay97DelsZHe+55D+xC6i0dDS0= -go.opencensus.io v0.22.5/go.mod h1:5pWMHQbX5EPX2/62yrJeAkowc+lfs/XD7Uxpq3pI6kk= -golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= -golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= -golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= -golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= -golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE= -golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU= -golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc= -golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= -golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= -golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= -golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= -golang.org/x/net v0.0.0-20190213061140-3a22650c66bd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= -golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= -golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= -golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= -golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= -golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU= -golang.org/x/net v0.7.0 h1:rJrUqqhjsgNp7KqAIc25s9pZnjU7TUcSY7HcVZjdn1g= -golang.org/x/net v0.7.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs= -golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U= -golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= -golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= -golang.org/x/sync v0.0.0-20190227155943-e225da77a7e6/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= -golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= -golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= -golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= -golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= -golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= -golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20190502145724-3ef323f4f1fd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20221010170243-090e33056c14/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.11.0 h1:eG7RXZHdqOJ1i+0lgLgCpSXAp6M3LYlAo6osgSi0xOM= -golang.org/x/sys v0.11.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= -golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= -golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= -golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= -golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY= -golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs= -golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= -golang.org/x/tools v0.0.0-20200619180055-7c47624df98f/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE= -golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA= -golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= -golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= -golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= -golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= -google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM= -google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4= -google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc= -google.golang.org/genproto v0.0.0-20190425155659-357c62f0e4bb/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE= -google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c= -google.golang.org/grpc v1.20.1/go.mod h1:10oTOabMzJvdu6/UiuZezV6QK5dSlG84ov/aaiqXj38= -google.golang.org/protobuf v1.26.0-rc.1/go.mod h1:jlhhOSvTdKEhbULTjvd4ARK9grFBp09yW+WbY/TyQbw= -google.golang.org/protobuf v1.26.0/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc= -google.golang.org/protobuf v1.28.1 h1:d0NfwRgPtno5B1Wa6L2DAG+KivqkdutMf1UhdNx175w= -google.golang.org/protobuf v1.28.1/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqwMG9pJV4I= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15 h1:YR8cESwS4TdDjEe65xsg0ogRM/Nc3DYOhEAlW+xobZo= -gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= +gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c h1:dUUwHk2QECo/6vqA44rthZ8ie2QXMNeKRTHCNY2nXvo= gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= -honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= diff --git a/go.work b/go.work new file mode 100644 index 0000000..7b9600a --- /dev/null +++ b/go.work @@ -0,0 +1,8 @@ +go 1.21.5 + +use ( + // Include the main SMT module + . + // Include the badger KVStore submodule + ./kvstore/badger +) diff --git a/go.work.sum b/go.work.sum new file mode 100644 index 0000000..d82afb6 --- /dev/null +++ b/go.work.sum @@ -0,0 +1,11 @@ +github.com/inconshreveable/mousetrap v1.0.0/go.mod h1:PxqpIevigyE2G7u3NXJIT2ANytuPF1OarO4DADm73n8= +github.com/spf13/cobra v0.0.5/go.mod h1:3K3wKZymM7VvHMDS9+Akkh4K60UwM26emMESw8tLCHU= +github.com/spf13/pflag v1.0.3/go.mod h1:DYY7MBk1bdzusC3SYhjObp+wFpr4gzcvqqNjLnInEg4= +github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw= +github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo= +github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU= +golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k= +golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8= +google.golang.org/genproto v0.0.0-20230110181048-76db0878b65f/go.mod h1:RGgjbofJ8xD9Sq1VVhDM1Vok1vRONV+rg+CjzG4SZKM= +google.golang.org/grpc v1.53.0/go.mod h1:OnIrk0ipVdj4N5d9IUoFUx72/VlD7+jUsHwZgwSMQpw= +gopkg.in/yaml.v2 v2.2.8/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= diff --git a/godoc.go b/godoc.go index ef4c5ff..c795203 100644 --- a/godoc.go +++ b/godoc.go @@ -2,7 +2,7 @@ // key-value map. The trie implements the same optimisations specified in the // Libra whitepaper to reduce the number of hash operations required per trie // operation to O(k) where k is the number of non-empty elements in the trie. -// And is implemente in a similar way to the JMT whitepaper, with additional +// And is implement in a similar way to the JMT whitepaper, with additional // features and proof mechanics, such as a Sparse Merkle Sum Trie and new // ClosestProof mechanics. package smt diff --git a/kvstore/badger/go.mod b/kvstore/badger/go.mod new file mode 100644 index 0000000..74f5917 --- /dev/null +++ b/kvstore/badger/go.mod @@ -0,0 +1,30 @@ +module github.com/pokt-network/smt/badger + +go 1.21.5 + +require ( + github.com/dgraph-io/badger/v4 v4.2.0 + github.com/pokt-network/smt v0.7.1 + github.com/stretchr/testify v1.8.4 +) + +require ( + github.com/cespare/xxhash/v2 v2.2.0 // indirect + github.com/davecgh/go-spew v1.1.1 // indirect + github.com/dgraph-io/ristretto v0.1.1 // indirect + github.com/dustin/go-humanize v1.0.0 // indirect + github.com/gogo/protobuf v1.3.2 // indirect + github.com/golang/glog v1.0.0 // indirect + github.com/golang/groupcache v0.0.0-20190702054246-869f871628b6 // indirect + github.com/golang/protobuf v1.5.2 // indirect + github.com/golang/snappy v0.0.3 // indirect + github.com/google/flatbuffers v1.12.1 // indirect + github.com/klauspost/compress v1.12.3 // indirect + github.com/pkg/errors v0.9.1 // indirect + github.com/pmezard/go-difflib v1.0.0 // indirect + go.opencensus.io v0.22.5 // indirect + golang.org/x/net v0.7.0 // indirect + golang.org/x/sys v0.11.0 // indirect + google.golang.org/protobuf v1.28.1 // indirect + gopkg.in/yaml.v3 v3.0.1 // indirect +) diff --git a/kvstore/badger/go.sum b/kvstore/badger/go.sum new file mode 100644 index 0000000..72e0a04 --- /dev/null +++ b/kvstore/badger/go.sum @@ -0,0 +1,124 @@ +cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw= +github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU= +github.com/cespare/xxhash/v2 v2.1.1/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs= +github.com/cespare/xxhash/v2 v2.2.0 h1:DC2CZ1Ep5Y4k3ZQ899DldepgrayRUGE6BBZ/cd9Cj44= +github.com/cespare/xxhash/v2 v2.2.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs= +github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw= +github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= +github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/dgraph-io/badger/v4 v4.2.0 h1:kJrlajbXXL9DFTNuhhu9yCx7JJa4qpYWxtE8BzuWsEs= +github.com/dgraph-io/badger/v4 v4.2.0/go.mod h1:qfCqhPoWDFJRx1gp5QwwyGo8xk1lbHUxvK9nK0OGAak= +github.com/dgraph-io/ristretto v0.1.1 h1:6CWw5tJNgpegArSHpNHJKldNeq03FQCwYvfMVWajOK8= +github.com/dgraph-io/ristretto v0.1.1/go.mod h1:S1GPSBCYCIhmVNfcth17y2zZtQT6wzkzgwUve0VDWWA= +github.com/dgryski/go-farm v0.0.0-20190423205320-6a90982ecee2 h1:tdlZCpZ/P9DhczCTSixgIKmwPv6+wP5DGjqLYw5SUiA= +github.com/dgryski/go-farm v0.0.0-20190423205320-6a90982ecee2/go.mod h1:SqUrOPUnsFjfmXRMNPybcSiG0BgUW2AuFH8PAnS2iTw= +github.com/dustin/go-humanize v1.0.0 h1:VSnTsYCnlFHaM2/igO1h6X3HA71jcobQuxemgkq4zYo= +github.com/dustin/go-humanize v1.0.0/go.mod h1:HtrtbFcZ19U5GC7JDqmcUSB87Iq5E25KnS6fMYU6eOk= +github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q= +github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q= +github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q= +github.com/golang/glog v1.0.0 h1:nfP3RFugxnNRyKgeWd4oI1nYvXpxrx8ck8ZrcizshdQ= +github.com/golang/glog v1.0.0/go.mod h1:EWib/APOK0SL3dFbYqvxE3UYd8E6s1ouQ7iEp/0LWV4= +github.com/golang/groupcache v0.0.0-20190702054246-869f871628b6 h1:ZgQEtGgCBiWRM39fZuwSd1LwSqqSW0hOdXCYYDX0R3I= +github.com/golang/groupcache v0.0.0-20190702054246-869f871628b6/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc= +github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A= +github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= +github.com/golang/protobuf v1.3.1/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= +github.com/golang/protobuf v1.5.0/go.mod h1:FsONVRAS9T7sI+LIUmWTfcYkHO4aIWwzhcaSAoJOfIk= +github.com/golang/protobuf v1.5.2 h1:ROPKBNFfQgOUMifHyP+KYbvpjbdoFNs+aK7DXlji0Tw= +github.com/golang/protobuf v1.5.2/go.mod h1:XVQd3VNwM+JqD3oG2Ue2ip4fOMUkwXdXDdiuN0vRsmY= +github.com/golang/snappy v0.0.3 h1:fHPg5GQYlCeLIPB9BZqMVR5nR9A+IM5zcgeTdjMYmLA= +github.com/golang/snappy v0.0.3/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q= +github.com/google/flatbuffers v1.12.1 h1:MVlul7pQNoDzWRLTw5imwYsl+usrS1TXG2H4jg6ImGw= +github.com/google/flatbuffers v1.12.1/go.mod h1:1AeVuKshWv4vARoZatz6mlQ0JxURH0Kv5+zNeJKJCa8= +github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU= +github.com/google/go-cmp v0.5.5/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= +github.com/google/go-cmp v0.5.8 h1:e6P7q2lk1O+qJJb4BtCQXlK8vWEO8V1ZeuEdJNOqZyg= +github.com/google/go-cmp v0.5.8/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY= +github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8= +github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck= +github.com/klauspost/compress v1.12.3 h1:G5AfA94pHPysR56qqrkO2pxEexdDzrpFJ6yt/VqWxVU= +github.com/klauspost/compress v1.12.3/go.mod h1:8dP1Hq4DHOhN9w426knH3Rhby4rFm6D8eO+e+Dq5Gzg= +github.com/kr/pretty v0.1.0 h1:L/CwN0zerZDmRFUapSPitk6f+Q3+0za1rQkzVuMiMFI= +github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo= +github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY= +github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE= +github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4= +github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= +github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= +github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= +github.com/pokt-network/smt v0.7.1 h1:WHcZeMLe+9U1/kCAhdbssdyTYzYxxb74sf8MCvG34M8= +github.com/pokt-network/smt v0.7.1/go.mod h1:K7BLEOWoZGZmY5USQuYvTkZ3qXjE6m39BMufBvVo3U8= +github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= +github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4= +github.com/stretchr/testify v1.8.4 h1:CcVxjf3Q8PM0mHUKJCdn+eZZtm5yQwehR5yeSVQQcUk= +github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo= +github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= +github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= +go.opencensus.io v0.22.5 h1:dntmOdLpSpHlVqbW5Eay97DelsZHe+55D+xC6i0dDS0= +go.opencensus.io v0.22.5/go.mod h1:5pWMHQbX5EPX2/62yrJeAkowc+lfs/XD7Uxpq3pI6kk= +golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= +golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= +golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= +golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= +golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE= +golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU= +golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc= +golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= +golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= +golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= +golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= +golang.org/x/net v0.0.0-20190213061140-3a22650c66bd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= +golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= +golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= +golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= +golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= +golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU= +golang.org/x/net v0.7.0 h1:rJrUqqhjsgNp7KqAIc25s9pZnjU7TUcSY7HcVZjdn1g= +golang.org/x/net v0.7.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs= +golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U= +golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.0.0-20190227155943-e225da77a7e6/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= +golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= +golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20190502145724-3ef323f4f1fd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20221010170243-090e33056c14/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.11.0 h1:eG7RXZHdqOJ1i+0lgLgCpSXAp6M3LYlAo6osgSi0xOM= +golang.org/x/sys v0.11.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= +golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= +golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= +golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= +golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY= +golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs= +golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= +golang.org/x/tools v0.0.0-20200619180055-7c47624df98f/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE= +golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA= +golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= +golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= +golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= +golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= +google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM= +google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4= +google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc= +google.golang.org/genproto v0.0.0-20190425155659-357c62f0e4bb/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE= +google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c= +google.golang.org/grpc v1.20.1/go.mod h1:10oTOabMzJvdu6/UiuZezV6QK5dSlG84ov/aaiqXj38= +google.golang.org/protobuf v1.26.0-rc.1/go.mod h1:jlhhOSvTdKEhbULTjvd4ARK9grFBp09yW+WbY/TyQbw= +google.golang.org/protobuf v1.26.0/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc= +google.golang.org/protobuf v1.28.1 h1:d0NfwRgPtno5B1Wa6L2DAG+KivqkdutMf1UhdNx175w= +google.golang.org/protobuf v1.28.1/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqwMG9pJV4I= +gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15 h1:YR8cESwS4TdDjEe65xsg0ogRM/Nc3DYOhEAlW+xobZo= +gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= +gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= +gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= +honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= diff --git a/kvstore/badger/godoc.go b/kvstore/badger/godoc.go new file mode 100644 index 0000000..e66c0f3 --- /dev/null +++ b/kvstore/badger/godoc.go @@ -0,0 +1,4 @@ +// Package badger is a wrapper around the BadgerDB key-value store for use in the +// SM(S)T implementation and as a general purpose key-value store for both +// in-memory and persistent use cases. +package badger diff --git a/kvstore/badger/interface.go b/kvstore/badger/interface.go new file mode 100644 index 0000000..6780a5f --- /dev/null +++ b/kvstore/badger/interface.go @@ -0,0 +1,32 @@ +package badger + +import ( + "io" + + "github.com/pokt-network/smt/kvstore" +) + +// Ensure the KVStore can be used as an SMT node store +var _ kvstore.MapStore = (KVStore)(nil) + +// KVStore is an interface that defines a key-value store +// that can be used standalone or as the node store for an SMT. +type KVStore interface { + // Store methods + Get(key []byte) ([]byte, error) + Set(key, value []byte) error + Delete(key []byte) error + + // Lifecycle methods + Stop() error + + // Data methods + Backup(writer io.Writer, incremental bool) error + Restore(io.Reader) error + + // Accessors + GetAll(prefixKey []byte, descending bool) (keys, values [][]byte, err error) + Exists(key []byte) (bool, error) + ClearAll() error + Len() int +} diff --git a/kvstore.go b/kvstore/badger/kvstore.go similarity index 89% rename from kvstore.go rename to kvstore/badger/kvstore.go index 27bf2e7..6e6c5af 100644 --- a/kvstore.go +++ b/kvstore/badger/kvstore.go @@ -1,4 +1,4 @@ -package smt +package badger import ( "errors" @@ -8,28 +8,6 @@ import ( badger "github.com/dgraph-io/badger/v4" ) -// KVStore is an interface that defines a key-value store -// that can be used standalone or as the node store for an SMT. -type KVStore interface { - // Store methods - Get(key []byte) ([]byte, error) - Set(key, value []byte) error - Delete(key []byte) error - - // Lifecycle methods - Stop() error - - // Data methods - Backup(writer io.Writer, incremental bool) error - Restore(io.Reader) error - - // Accessors - GetAll(prefixKey []byte, descending bool) (keys, values [][]byte, err error) - Exists(key []byte) (bool, error) - ClearAll() error - Len() int -} - const ( maxPendingWrites = 16 // used in backup restoration ) diff --git a/kvstore_test.go b/kvstore/badger/kvstore_test.go similarity index 62% rename from kvstore_test.go rename to kvstore/badger/kvstore_test.go index 6c37d61..7bbbcfd 100644 --- a/kvstore_test.go +++ b/kvstore/badger/kvstore_test.go @@ -1,4 +1,4 @@ -package smt +package badger_test import ( "bytes" @@ -7,116 +7,118 @@ import ( "strings" "testing" - badger "github.com/dgraph-io/badger/v4" + v4 "github.com/dgraph-io/badger/v4" "github.com/stretchr/testify/require" + + "github.com/pokt-network/smt/badger" ) -func TestKVStore_BasicOperations(t *testing.T) { - store, err := NewKVStore("") +func TestBadger_KVStore_BasicOperations(t *testing.T) { + store, err := badger.NewKVStore("") require.NoError(t, err) require.NotNil(t, store) invalidKey := [65001]byte{} testCases := []struct { - name string - op string - key []byte - value []byte - fail bool - expected error + desc string + op string + key []byte + value []byte + fail bool + expectedErr error }{ { - name: "Successfully sets a value in the store", - op: "set", - key: []byte("testKey"), - value: []byte("testValue"), - fail: false, - expected: nil, + desc: "Successfully sets a value in the store", + op: "set", + key: []byte("testKey"), + value: []byte("testValue"), + fail: false, + expectedErr: nil, }, { - name: "Successfully updates a value in the store", - op: "set", - key: []byte("foo"), - value: []byte("new value"), - fail: false, - expected: nil, + desc: "Successfully updates a value in the store", + op: "set", + key: []byte("foo"), + value: []byte("new value"), + fail: false, + expectedErr: nil, }, { - name: "Fails to set value to nil key", - op: "set", - key: nil, - value: []byte("bar"), - fail: true, - expected: badger.ErrEmptyKey, + desc: "Fails to set value to nil key", + op: "set", + key: nil, + value: []byte("bar"), + fail: true, + expectedErr: v4.ErrEmptyKey, }, { - name: "Fails to set a value to a key that is too large", - op: "set", - key: invalidKey[:], - value: []byte("bar"), - fail: true, - expected: fmt.Errorf("Key with size 65001 exceeded 65000 limit. Key:\n%s", hex.Dump(invalidKey[:1<<10])), + desc: "Fails to set a value to a key that is too large", + op: "set", + key: invalidKey[:], + value: []byte("bar"), + fail: true, + expectedErr: fmt.Errorf("Key with size 65001 exceeded 65000 limit. Key:\n%s", hex.Dump(invalidKey[:1<<10])), }, { - name: "Successfully retrieve a value from the store", - op: "get", - key: []byte("foo"), - value: []byte("bar"), - fail: false, - expected: nil, + desc: "Successfully retrieve a value from the store", + op: "get", + key: []byte("foo"), + value: []byte("bar"), + fail: false, + expectedErr: nil, }, { - name: "Fails to get a value that is not stored", - op: "get", - key: []byte("bar"), - value: nil, - fail: true, - expected: badger.ErrKeyNotFound, + desc: "Fails to get a value that is not stored", + op: "get", + key: []byte("bar"), + value: nil, + fail: true, + expectedErr: v4.ErrKeyNotFound, }, { - name: "Fails when the key is empty", - op: "get", - key: nil, - value: nil, - fail: true, - expected: badger.ErrEmptyKey, + desc: "Fails when the key is empty", + op: "get", + key: nil, + value: nil, + fail: true, + expectedErr: v4.ErrEmptyKey, }, { - name: "Successfully deletes a value in the store", - op: "delete", - key: []byte("foo"), - value: nil, - fail: false, - expected: nil, + desc: "Successfully deletes a value in the store", + op: "delete", + key: []byte("foo"), + value: nil, + fail: false, + expectedErr: nil, }, { - name: "Fails to delete a value not in the store", - op: "delete", - key: []byte("bar"), - value: nil, - fail: false, - expected: nil, + desc: "Fails to delete a value not in the store", + op: "delete", + key: []byte("bar"), + value: nil, + fail: false, + expectedErr: nil, }, { - name: "Fails to set value to nil key", - op: "delete", - key: nil, - value: nil, - fail: true, - expected: badger.ErrEmptyKey, + desc: "Fails to set value to nil key", + op: "delete", + key: nil, + value: nil, + fail: true, + expectedErr: v4.ErrEmptyKey, }, { - name: "Fails to set a value to a key that is too large", - op: "delete", - key: invalidKey[:], - value: nil, - fail: true, - expected: fmt.Errorf("Key with size 65001 exceeded 65000 limit. Key:\n%s", hex.Dump(invalidKey[:1<<10])), + desc: "Fails to set a value to a key that is too large", + op: "delete", + key: invalidKey[:], + value: nil, + fail: true, + expectedErr: fmt.Errorf("Key with size 65001 exceeded 65000 limit. Key:\n%s", hex.Dump(invalidKey[:1<<10])), }, } for _, tc := range testCases { - t.Run(tc.name, func(t *testing.T) { + t.Run(tc.desc, func(t *testing.T) { err := store.ClearAll() require.NoError(t, err) setupStore(t, store) @@ -125,7 +127,7 @@ func TestKVStore_BasicOperations(t *testing.T) { err := store.Set(tc.key, tc.value) if tc.fail { require.Error(t, err) - require.EqualError(t, tc.expected, err.Error()) + require.EqualError(t, tc.expectedErr, err.Error()) } else { require.NoError(t, err) got, err := store.Get(tc.key) @@ -136,7 +138,7 @@ func TestKVStore_BasicOperations(t *testing.T) { got, err := store.Get(tc.key) if tc.fail { require.Error(t, err) - require.EqualError(t, tc.expected, err.Error()) + require.EqualError(t, tc.expectedErr, err.Error()) } else { require.NoError(t, err) require.Equal(t, tc.value, got) @@ -145,11 +147,11 @@ func TestKVStore_BasicOperations(t *testing.T) { err := store.Delete(tc.key) if tc.fail { require.Error(t, err) - require.EqualError(t, tc.expected, err.Error()) + require.EqualError(t, tc.expectedErr, err.Error()) } else { require.NoError(t, err) _, err := store.Get(tc.key) - require.EqualError(t, err, badger.ErrKeyNotFound.Error()) + require.EqualError(t, err, v4.ErrKeyNotFound.Error()) } } }) @@ -159,8 +161,8 @@ func TestKVStore_BasicOperations(t *testing.T) { require.NoError(t, err) } -func TestKVStore_GetAllBasic(t *testing.T) { - store, err := NewKVStore("") +func TestBadger_KVStore_GetAllBasic(t *testing.T) { + store, err := badger.NewKVStore("") require.NoError(t, err) require.NotNil(t, store) @@ -196,8 +198,8 @@ func TestKVStore_GetAllBasic(t *testing.T) { require.NoError(t, err) } -func TestKVStore_GetAllPrefixed(t *testing.T) { - store, err := NewKVStore("") +func TestBadger_KVStore_GetAllPrefixed(t *testing.T) { + store, err := badger.NewKVStore("") require.NoError(t, err) require.NotNil(t, store) @@ -246,8 +248,8 @@ func TestKVStore_GetAllPrefixed(t *testing.T) { require.NoError(t, err) } -func TestKVStore_Exists(t *testing.T) { - store, err := NewKVStore("") +func TestBadger_KVStore_Exists(t *testing.T) { + store, err := badger.NewKVStore("") require.NoError(t, err) require.NotNil(t, store) @@ -281,15 +283,15 @@ func TestKVStore_Exists(t *testing.T) { // Key does not exist exists, err = store.Exists([]byte("oof")) - require.EqualError(t, err, badger.ErrKeyNotFound.Error()) + require.EqualError(t, err, v4.ErrKeyNotFound.Error()) require.False(t, exists) err = store.Stop() require.NoError(t, err) } -func TestKVStore_ClearAll(t *testing.T) { - store, err := NewKVStore("") +func TestBadger_KVStore_ClearAll(t *testing.T) { + store, err := badger.NewKVStore("") require.NoError(t, err) require.NotNil(t, store) @@ -336,8 +338,8 @@ func TestKVStore_ClearAll(t *testing.T) { require.NoError(t, err) } -func TestKVStore_BackupAndRestore(t *testing.T) { - store, err := NewKVStore("") +func TestBadger_KVStore_BackupAndRestore(t *testing.T) { + store, err := badger.NewKVStore("") require.NoError(t, err) require.NotNil(t, store) @@ -361,8 +363,8 @@ func TestKVStore_BackupAndRestore(t *testing.T) { require.Equal(t, values, newValues) } -func TestKVStore_Len(t *testing.T) { - store, err := NewKVStore("") +func TestBadger_KVStore_Len(t *testing.T) { + store, err := badger.NewKVStore("") require.NoError(t, err) require.NotNil(t, store) @@ -394,7 +396,7 @@ func TestKVStore_Len(t *testing.T) { } } -func setupStore(t *testing.T, store KVStore) { +func setupStore(t *testing.T, store badger.KVStore) { t.Helper() err := store.Set([]byte("foo"), []byte("bar")) require.NoError(t, err) diff --git a/kvstore/errors.go b/kvstore/errors.go new file mode 100644 index 0000000..5542bec --- /dev/null +++ b/kvstore/errors.go @@ -0,0 +1,12 @@ +package kvstore + +import ( + "errors" +) + +var ( + // ErrKVStoreKeyNotFound is returned when a key is not present in the trie. + ErrKVStoreKeyNotFound = errors.New("key already empty") + // ErrKVStoreEmptyKey is returned when the given key is empty. + ErrKVStoreEmptyKey = errors.New("key is empty") +) diff --git a/kvstore/simplemap.go b/kvstore/simplemap.go new file mode 100644 index 0000000..7e3325a --- /dev/null +++ b/kvstore/simplemap.go @@ -0,0 +1,75 @@ +package kvstore + +// MapStore defines an interface that represents a key-value store that backs +// the SM(S)T. It is a subset of the full functionality of a key-value store +// needed for the SM(S)T to function. By using a simplified interface any +// key-value store that implements these methods can be used with the SM(S)T. +type MapStore interface { + // Accessors + Get(key []byte) ([]byte, error) + Set(key, value []byte) error + Delete(key []byte) error + Len() int + + // Debug + ClearAll() error +} + +// SimpleMap is a simple in-memory map. +type SimpleMap struct { + m map[string][]byte +} + +// NewSimpleMap creates a new SimpleMap instance. +func NewSimpleMap() MapStore { + return &SimpleMap{ + m: make(map[string][]byte), + } +} + +// Get gets the value for a key. +func (sm *SimpleMap) Get(key []byte) ([]byte, error) { + if len(key) == 0 { + return nil, ErrKVStoreEmptyKey + } + + if value, ok := sm.m[string(key)]; ok { + return value, nil + } + + return nil, ErrKVStoreKeyNotFound +} + +// Set updates the value for a key. +func (sm *SimpleMap) Set(key []byte, value []byte) error { + if len(key) == 0 { + return ErrKVStoreEmptyKey + } + sm.m[string(key)] = value + return nil +} + +// Delete deletes a key. +func (sm *SimpleMap) Delete(key []byte) error { + if len(key) == 0 { + return ErrKVStoreEmptyKey + } + _, ok := sm.m[string(key)] + if ok { + delete(sm.m, string(key)) + return nil + } + return nil +} + +// Len returns the number of key-value pairs in the store. +func (sm *SimpleMap) Len() int { + return len(sm.m) +} + +// ClearAll clears all key-value pairs +// NB: This should only be used for testing purposes. +func (sm *SimpleMap) ClearAll() error { + sm.m = make(map[string][]byte) + return nil +} diff --git a/kvstore/simplemap_test.go b/kvstore/simplemap_test.go new file mode 100644 index 0000000..313f41b --- /dev/null +++ b/kvstore/simplemap_test.go @@ -0,0 +1,175 @@ +package kvstore + +import ( + "testing" + + "github.com/stretchr/testify/require" +) + +func TestSimpleMap_Get(t *testing.T) { + store := NewSimpleMap() + key := []byte("key1") + value := []byte("value1") + + // Set a value to retrieve + require.NoError(t, store.Set(key, value)) + + tests := []struct { + desc string + key []byte + want []byte + expectedErr error + }{ + { + desc: "Get existing key", + key: key, + want: value, + expectedErr: nil, + }, + { + desc: "Get non-existing key", + key: []byte("nonexistent"), + want: nil, + expectedErr: ErrKVStoreKeyNotFound, + }, + { + desc: "Get with empty key", + key: []byte(""), + want: nil, + expectedErr: ErrKVStoreEmptyKey, + }, + } + + for _, tt := range tests { + t.Run(tt.desc, func(t *testing.T) { + got, err := store.Get(tt.key) + if tt.expectedErr != nil { + require.ErrorIs(t, err, tt.expectedErr) + } else { + require.NoError(t, err) + require.Equal(t, tt.want, got) + } + }) + } +} + +func TestSimpleMap_Set(t *testing.T) { + store := NewSimpleMap() + + tests := []struct { + desc string + key []byte + value []byte + expectedErr error + }{ + { + desc: "Set valid key-value", + key: []byte("key1"), + value: []byte("value1"), + expectedErr: nil, + }, + { + desc: "Set with empty key", + key: []byte(""), + value: []byte("value1"), + expectedErr: ErrKVStoreEmptyKey, + }, + } + + for _, tt := range tests { + t.Run(tt.desc, func(t *testing.T) { + err := store.Set(tt.key, tt.value) + if tt.expectedErr != nil { + require.ErrorIs(t, err, tt.expectedErr) + } else { + require.NoError(t, err) + } + }) + } +} + +func TestSimpleMap_Delete(t *testing.T) { + store := NewSimpleMap() + key := []byte("key1") + value := []byte("value1") + + // Set a value to delete + require.NoError(t, store.Set(key, value)) + + tests := []struct { + desc string + key []byte + expectedErr error + }{ + { + desc: "Delete existing key", + key: key, + expectedErr: nil, + }, + { + desc: "Delete non-existing key", + key: []byte("nonexistent"), + expectedErr: nil, + }, + { + desc: "Delete with empty key", + key: []byte(""), + expectedErr: ErrKVStoreEmptyKey, + }, + } + + for _, tt := range tests { + t.Run(tt.desc, func(t *testing.T) { + err := store.Delete(tt.key) + if tt.expectedErr != nil { + require.ErrorIs(t, err, tt.expectedErr) + } else { + require.NoError(t, err) + } + }) + } +} + +func TestSimpleMap_Len(t *testing.T) { + store := NewSimpleMap() + + tests := []struct { + desc string + setup func(MapStore) + expectedLen int + }{ + { + desc: "Length of empty map", + setup: func(store MapStore) {}, + expectedLen: 0, + }, + { + desc: "Length after adding items", + setup: func(sm MapStore) { + require.NoError(t, sm.Set([]byte("key1"), []byte("value1"))) + require.NoError(t, sm.Set([]byte("key2"), []byte("value2"))) + }, + expectedLen: 2, + }, + } + + for _, tt := range tests { + t.Run(tt.desc, func(t *testing.T) { + tt.setup(store) + require.Equal(t, tt.expectedLen, store.Len()) + }) + } +} + +func TestSimpleMap_ClearAll(t *testing.T) { + store := NewSimpleMap() + + // Add some elements + require.NoError(t, store.Set([]byte("key1"), []byte("value1"))) + require.NoError(t, store.Set([]byte("key2"), []byte("value2"))) + + // Clear all elements + require.NoError(t, store.ClearAll()) + + require.Equal(t, 0, store.Len()) +} diff --git a/proofs.go b/proofs.go index 32983a9..64ce171 100644 --- a/proofs.go +++ b/proofs.go @@ -16,9 +16,6 @@ func init() { gob.Register(SparseCompactMerkleClosestProof{}) } -// ErrBadProof is returned when an invalid Merkle proof is supplied. -var ErrBadProof = errors.New("bad proof") - // SparseMerkleProof is a Merkle proof for an element in a SparseMerkleTrie. // TODO: Look into whether the SiblingData is required and remove it if not type SparseMerkleProof struct { diff --git a/proofs_test.go b/proofs_test.go index af2157b..74cdf7e 100644 --- a/proofs_test.go +++ b/proofs_test.go @@ -6,6 +6,8 @@ import ( "testing" "github.com/stretchr/testify/require" + + "github.com/pokt-network/smt/kvstore" ) func TestSparseMerkleProof_Marshal(t *testing.T) { @@ -150,11 +152,7 @@ func TestSparseCompactMerkleProof_Unmarshal(t *testing.T) { func setupTrie(t *testing.T) *SMT { t.Helper() - db, err := NewKVStore("") - require.NoError(t, err) - t.Cleanup(func() { - require.NoError(t, db.Stop()) - }) + db := kvstore.NewSimpleMap() trie := NewSparseMerkleTrie(db, sha256.New()) require.NoError(t, trie.Update([]byte("key"), []byte("value"))) diff --git a/smst.go b/smst.go index 9f571cc..d1c4368 100644 --- a/smst.go +++ b/smst.go @@ -4,6 +4,8 @@ import ( "bytes" "encoding/binary" "hash" + + "github.com/pokt-network/smt/kvstore" ) var _ SparseMerkleSumTrie = (*SMST)(nil) @@ -15,7 +17,11 @@ type SMST struct { } // NewSparseMerkleSumTrie returns a pointer to an SMST struct -func NewSparseMerkleSumTrie(nodes KVStore, hasher hash.Hash, options ...Option) *SMST { +func NewSparseMerkleSumTrie( + nodes kvstore.MapStore, + hasher hash.Hash, + options ...Option, +) *SMST { smt := &SMT{ TrieSpec: newTrieSpec(hasher, true), nodes: nodes, @@ -36,7 +42,12 @@ func NewSparseMerkleSumTrie(nodes KVStore, hasher hash.Hash, options ...Option) } // ImportSparseMerkleSumTrie returns a pointer to an SMST struct with the root hash provided -func ImportSparseMerkleSumTrie(nodes KVStore, hasher hash.Hash, root []byte, options ...Option) *SMST { +func ImportSparseMerkleSumTrie( + nodes kvstore.MapStore, + hasher hash.Hash, + root []byte, + options ...Option, +) *SMST { smst := NewSparseMerkleSumTrie(nodes, hasher, options...) smst.trie = &lazyNode{root} smst.savedRoot = root @@ -48,7 +59,8 @@ func (smst *SMST) Spec() *TrieSpec { return &smst.TrieSpec } -// Get returns the digest of the value stored at the given key and the weight of the leaf node +// Get returns the digest of the value stored at the given key and the weight +// of the leaf node func (smst *SMST) Get(key []byte) ([]byte, uint64, error) { valueHash, err := smst.SMT.Get(key) if err != nil { diff --git a/smst_proofs_test.go b/smst_proofs_test.go index 84c7cf5..ae038f5 100644 --- a/smst_proofs_test.go +++ b/smst_proofs_test.go @@ -8,20 +8,22 @@ import ( "testing" "github.com/stretchr/testify/require" + + "github.com/pokt-network/smt/kvstore" ) // Test base case Merkle proof operations. func TestSMST_Proof_Operations(t *testing.T) { - var smn, smv KVStore + var smn, smv kvstore.MapStore var smst *SMSTWithStorage var proof *SparseMerkleProof var result bool var root []byte var err error - smn, err = NewKVStore("") + smn = kvstore.NewSimpleMap() require.NoError(t, err) - smv, err = NewKVStore("") + smv = kvstore.NewSimpleMap() require.NoError(t, err) smst = NewSMSTWithStorage(smn, smv, sha256.New()) base := smst.Spec() @@ -129,21 +131,16 @@ func TestSMST_Proof_Operations(t *testing.T) { result, err = VerifySumProof(randomiseSumProof(proof), root, []byte("testKey3"), defaultValue, 0, base) // invalid proof require.NoError(t, err) require.False(t, result) - - require.NoError(t, smn.Stop()) - require.NoError(t, smv.Stop()) } // Test sanity check cases for non-compact proofs. func TestSMST_Proof_ValidateBasic(t *testing.T) { - smn, err := NewKVStore("") - require.NoError(t, err) - smv, err := NewKVStore("") - require.NoError(t, err) + smn := kvstore.NewSimpleMap() + smv := kvstore.NewSimpleMap() smst := NewSMSTWithStorage(smn, smv, sha256.New()) base := smst.Spec() - err = smst.Update([]byte("testKey1"), []byte("testValue1"), 1) + err := smst.Update([]byte("testKey1"), []byte("testValue1"), 1) require.NoError(t, err) err = smst.Update([]byte("testKey2"), []byte("testValue2"), 2) require.NoError(t, err) @@ -201,14 +198,10 @@ func TestSMST_Proof_ValidateBasic(t *testing.T) { require.False(t, result) _, err = CompactProof(proof, base) require.Error(t, err) - - require.NoError(t, smn.Stop()) - require.NoError(t, smv.Stop()) } func TestSMST_ClosestProof_ValidateBasic(t *testing.T) { - smn, err := NewKVStore("") - require.NoError(t, err) + smn := kvstore.NewSimpleMap() smst := NewSparseMerkleSumTrie(smn, sha256.New()) np := NoPrehashSpec(sha256.New(), true) base := smst.Spec() @@ -281,7 +274,7 @@ func TestSMST_ClosestProof_ValidateBasic(t *testing.T) { // ProveClosest test against a visual representation of the trie // See: https://github.com/pokt-network/smt/assets/53987565/2a2f33e0-f81f-41c5-bd76-af0cd1cd8f15 func TestSMST_ProveClosest(t *testing.T) { - var smn KVStore + var smn kvstore.MapStore var smst *SMST var proof *SparseMerkleClosestProof var result bool @@ -289,7 +282,7 @@ func TestSMST_ProveClosest(t *testing.T) { var err error var sumBz [sumSize]byte - smn, err = NewKVStore("") + smn = kvstore.NewSimpleMap() require.NoError(t, err) smst = NewSparseMerkleSumTrie(smn, sha256.New(), WithValueHasher(nil)) @@ -307,11 +300,11 @@ func TestSMST_ProveClosest(t *testing.T) { root = smst.Root() - // `testKey2` is the child of an inner node, which is the child of an extension node. - // The extension node has the path bounds of [3, 7]. This means any bits between - // 3-6 can be flipped, and the resulting path would still traverse through the same - // extension node and lead to testKey2 - the closest key. However, flipping bit 7 - // will lead to testKey4. + // `testKey2` is the child of an inner node, which is the child of an + // extension node. The extension node has the path bounds of [3, 7]. This + // means any bits between 3-6 can be flipped, and the resulting path would + // still traverse through the same extension node and lead to testKey2 - + // the closest key. However, flipping bit 7 will lead to testKey4. path := sha256.Sum256([]byte("testKey2")) flipPathBit(path[:], 3) flipPathBit(path[:], 6) @@ -361,17 +354,15 @@ func TestSMST_ProveClosest(t *testing.T) { result, err = VerifyClosestProof(proof, root, NoPrehashSpec(sha256.New(), true)) require.NoError(t, err) require.True(t, result) - - require.NoError(t, smn.Stop()) } func TestSMST_ProveClosest_Empty(t *testing.T) { - var smn KVStore + var smn kvstore.MapStore var smst *SMST var proof *SparseMerkleClosestProof var err error - smn, err = NewKVStore("") + smn = kvstore.NewSimpleMap() require.NoError(t, err) smst = NewSparseMerkleSumTrie(smn, sha256.New(), WithValueHasher(nil)) @@ -392,17 +383,15 @@ func TestSMST_ProveClosest_Empty(t *testing.T) { result, err := VerifyClosestProof(proof, smst.Root(), NoPrehashSpec(sha256.New(), true)) require.NoError(t, err) require.True(t, result) - - require.NoError(t, smn.Stop()) } func TestSMST_ProveClosest_OneNode(t *testing.T) { - var smn KVStore + var smn kvstore.MapStore var smst *SMST var proof *SparseMerkleClosestProof var err error - smn, err = NewKVStore("") + smn = kvstore.NewSimpleMap() require.NoError(t, err) smst = NewSparseMerkleSumTrie(smn, sha256.New(), WithValueHasher(nil)) @@ -432,12 +421,10 @@ func TestSMST_ProveClosest_OneNode(t *testing.T) { result, err := VerifyClosestProof(proof, smst.Root(), NoPrehashSpec(sha256.New(), true)) require.NoError(t, err) require.True(t, result) - - require.NoError(t, smn.Stop()) } func TestSMST_ProveClosest_Proof(t *testing.T) { - var smn KVStore + var smn kvstore.MapStore var smst256 *SMST var smst512 *SMST var proof256 *SparseMerkleClosestProof @@ -445,7 +432,7 @@ func TestSMST_ProveClosest_Proof(t *testing.T) { var err error // setup trie (256+512 path hasher) and nodestore - smn, err = NewKVStore("") + smn = kvstore.NewSimpleMap() require.NoError(t, err) smst256 = NewSparseMerkleSumTrie(smn, sha256.New()) smst512 = NewSparseMerkleSumTrie(smn, sha512.New()) @@ -466,6 +453,4 @@ func TestSMST_ProveClosest_Proof(t *testing.T) { checkClosestCompactEquivalence(t, proof256, smst256.Spec()) checkClosestCompactEquivalence(t, proof512, smst512.Spec()) } - - require.NoError(t, smn.Stop()) } diff --git a/smst_test.go b/smst_test.go index ca40dfa..2c7a05e 100644 --- a/smst_test.go +++ b/smst_test.go @@ -9,9 +9,15 @@ import ( "testing" "github.com/stretchr/testify/require" + + "github.com/pokt-network/smt/kvstore" ) -func NewSMSTWithStorage(nodes, preimages KVStore, hasher hash.Hash, options ...Option) *SMSTWithStorage { +func NewSMSTWithStorage( + nodes, preimages kvstore.MapStore, + hasher hash.Hash, + options ...Option, +) *SMSTWithStorage { return &SMSTWithStorage{ SMST: NewSparseMerkleSumTrie(nodes, hasher, options...), preimages: preimages, @@ -19,15 +25,14 @@ func NewSMSTWithStorage(nodes, preimages KVStore, hasher hash.Hash, options ...O } func TestSMST_TrieUpdateBasic(t *testing.T) { - smn, err := NewKVStore("") - require.NoError(t, err) - smv, err := NewKVStore("") - require.NoError(t, err) + smn := kvstore.NewSimpleMap() + smv := kvstore.NewSimpleMap() lazy := NewSparseMerkleSumTrie(smn, sha256.New()) smst := &SMSTWithStorage{SMST: lazy, preimages: smv} var value []byte var sum uint64 var has bool + var err error // Test getting an empty key. value, sum, err = smst.GetValueSum([]byte("testKey")) @@ -106,23 +111,18 @@ func TestSMST_TrieUpdateBasic(t *testing.T) { require.NoError(t, err) require.Equal(t, []byte("testValue3"), value) require.Equal(t, uint64(5), sum) - - require.NoError(t, smn.Stop()) - require.NoError(t, smv.Stop()) } // Test base case trie delete operations with a few keys. func TestSMST_TrieDeleteBasic(t *testing.T) { - smn, err := NewKVStore("") - require.NoError(t, err) - smv, err := NewKVStore("") - require.NoError(t, err) + smn := kvstore.NewSimpleMap() + smv := kvstore.NewSimpleMap() lazy := NewSparseMerkleSumTrie(smn, sha256.New()) smst := &SMSTWithStorage{SMST: lazy, preimages: smv} rootEmpty := smst.Root() // Testing inserting, deleting a key, and inserting it again. - err = smst.Update([]byte("testKey"), []byte("testValue"), 5) + err := smst.Update([]byte("testKey"), []byte("testValue"), 5) require.NoError(t, err) root1 := smst.Root() @@ -217,18 +217,13 @@ func TestSMST_TrieDeleteBasic(t *testing.T) { require.Equal(t, []byte("testValue"), value) require.Equal(t, uint64(5), sum) require.Equal(t, root1, smst.Root(), "re-inserting key after deletion") - - require.NoError(t, smn.Stop()) - require.NoError(t, smv.Stop()) } // Test trie ops with known paths func TestSMST_TrieKnownPath(t *testing.T) { ph := dummyPathHasher{32} - smn, err := NewKVStore("") - require.NoError(t, err) - smv, err := NewKVStore("") - require.NoError(t, err) + smn := kvstore.NewSimpleMap() + smv := kvstore.NewSimpleMap() smst := NewSMSTWithStorage(smn, smv, sha256.New(), WithPathHasher(ph)) var value []byte var sum uint64 @@ -247,7 +242,7 @@ func TestSMST_TrieKnownPath(t *testing.T) { keys[5][0] = byte(0b11100000) keys[6][0] = byte(0b11110000) - err = smst.Update(keys[0], []byte("testValue1"), 1) + err := smst.Update(keys[0], []byte("testValue1"), 1) require.NoError(t, err) err = smst.Update(keys[1], []byte("testValue2"), 2) require.NoError(t, err) @@ -301,18 +296,13 @@ func TestSMST_TrieKnownPath(t *testing.T) { require.NoError(t, err) require.Equal(t, []byte("testValue6"), value) require.Equal(t, uint64(6), sum) - - require.NoError(t, smn.Stop()) - require.NoError(t, smv.Stop()) } // Test trie operations when two leafs are immediate neighbors. func TestSMST_TrieMaxHeightCase(t *testing.T) { ph := dummyPathHasher{32} - smn, err := NewKVStore("") - require.NoError(t, err) - smv, err := NewKVStore("") - require.NoError(t, err) + smn := kvstore.NewSimpleMap() + smv := kvstore.NewSimpleMap() smst := NewSMSTWithStorage(smn, smv, sha256.New(), WithPathHasher(ph)) var value []byte var sum uint64 @@ -321,7 +311,7 @@ func TestSMST_TrieMaxHeightCase(t *testing.T) { // The dummy hash function will return the preimage itself as the digest. key1 := make([]byte, ph.PathSize()) key2 := make([]byte, ph.PathSize()) - _, err = rand.Read(key1) + _, err := rand.Read(key1) require.NoError(t, err) copy(key2, key1) // We make key2's least significant bit different than key1's @@ -347,13 +337,10 @@ func TestSMST_TrieMaxHeightCase(t *testing.T) { proof, err := smst.Prove(key1) require.NoError(t, err) require.Equal(t, 256, len(proof.SideNodes), "unexpected proof size") - - require.NoError(t, smn.Stop()) - require.NoError(t, smv.Stop()) } func TestSMST_OrphanRemoval(t *testing.T) { - var smn, smv KVStore + var smn, smv kvstore.MapStore var impl *SMST var smst *SMSTWithStorage var err error @@ -363,10 +350,8 @@ func TestSMST_OrphanRemoval(t *testing.T) { return smn.Len() } setup := func() { - smn, err = NewKVStore("") - require.NoError(t, err) - smv, err = NewKVStore("") - require.NoError(t, err) + smn = kvstore.NewSimpleMap() + smv = kvstore.NewSimpleMap() impl = NewSparseMerkleSumTrie(smn, sha256.New()) smst = &SMSTWithStorage{SMST: impl, preimages: smv} @@ -441,16 +426,12 @@ func TestSMST_OrphanRemoval(t *testing.T) { require.Equal(t, 1, nodeCount(t), tci) } }) - - require.NoError(t, smn.Stop()) - require.NoError(t, smv.Stop()) } func TestSMST_TotalSum(t *testing.T) { - snm, err := NewKVStore("") - require.NoError(t, err) + snm := kvstore.NewSimpleMap() smst := NewSparseMerkleSumTrie(snm, sha256.New()) - err = smst.Update([]byte("key1"), []byte("value1"), 5) + err := smst.Update([]byte("key1"), []byte("value1"), 5) require.NoError(t, err) err = smst.Update([]byte("key2"), []byte("value2"), 5) require.NoError(t, err) @@ -490,8 +471,7 @@ func TestSMST_TotalSum(t *testing.T) { require.Equal(t, sum, uint64(10)) // Calculate the total sum of a larger trie - snm, err = NewKVStore("") - require.NoError(t, err) + snm = kvstore.NewSimpleMap() smst = NewSparseMerkleSumTrie(snm, sha256.New()) for i := 1; i < 10000; i++ { err := smst.Update([]byte(fmt.Sprintf("testKey%d", i)), []byte(fmt.Sprintf("testValue%d", i)), uint64(i)) @@ -500,16 +480,13 @@ func TestSMST_TotalSum(t *testing.T) { require.NoError(t, smst.Commit()) sum = smst.Sum() require.Equal(t, sum, uint64(49995000)) - - require.NoError(t, snm.Stop()) } func TestSMST_Retrieval(t *testing.T) { - snm, err := NewKVStore("") - require.NoError(t, err) + snm := kvstore.NewSimpleMap() smst := NewSparseMerkleSumTrie(snm, sha256.New(), WithValueHasher(nil)) - err = smst.Update([]byte("key1"), []byte("value1"), 5) + err := smst.Update([]byte("key1"), []byte("value1"), 5) require.NoError(t, err) err = smst.Update([]byte("key2"), []byte("value2"), 5) require.NoError(t, err) @@ -571,6 +548,4 @@ func TestSMST_Retrieval(t *testing.T) { sum = lazy.Sum() require.Equal(t, sum, uint64(15)) - - require.NoError(t, snm.Stop()) } diff --git a/smst_utils_test.go b/smst_utils_test.go index da26d01..882f764 100644 --- a/smst_utils_test.go +++ b/smst_utils_test.go @@ -6,17 +6,20 @@ import ( "errors" "fmt" - "github.com/dgraph-io/badger/v4" + "github.com/pokt-network/smt/kvstore" ) -// SMSTWithStorage wraps an SMST with a mapping of value hashes to values with sums (preimages), for use in tests. -// Note: this doesn't delete from preimages (inputs to hashing functions), since there could be duplicate stored values. +// SMSTWithStorage wraps an SMST with a mapping of value hashes to values with +// sums (preimages), for use in tests. +// Note: this doesn't delete from preimages (inputs to hashing functions), since +// there could be duplicate stored values. type SMSTWithStorage struct { *SMST - preimages KVStore + preimages kvstore.MapStore } -// Update updates a key with a new value in the trie and adds the value to the preimages KVStore +// Update updates a key with a new value in the trie and adds the value to the +// preimages KVStore func (smst *SMSTWithStorage) Update(key, value []byte, sum uint64) error { if err := smst.SMST.Update(key, value, sum); err != nil { return err @@ -33,8 +36,8 @@ func (smst *SMSTWithStorage) Delete(key []byte) error { return smst.SMST.Delete(key) } -// GetValueSum returns the value and sum of the key stored in the trie, by looking up -// the value hash in the preimages KVStore and extracting the sum +// GetValueSum returns the value and sum of the key stored in the trie, by +// looking up the value hash in the preimages KVStore and extracting the sum func (smst *SMSTWithStorage) GetValueSum(key []byte) ([]byte, uint64, error) { valueHash, sum, err := smst.Get(key) if err != nil { @@ -45,7 +48,7 @@ func (smst *SMSTWithStorage) GetValueSum(key []byte) ([]byte, uint64, error) { } value, err := smst.preimages.Get(valueHash) if err != nil { - if errors.Is(err, badger.ErrKeyNotFound) { + if errors.Is(err, kvstore.ErrKVStoreKeyNotFound) { // If key isn't found, return default value and sum return defaultValue, 0, nil } diff --git a/smt.go b/smt.go index 01184ba..2bdcd96 100644 --- a/smt.go +++ b/smt.go @@ -3,6 +3,8 @@ package smt import ( "bytes" "hash" + + "github.com/pokt-network/smt/kvstore" ) var ( @@ -12,7 +14,8 @@ var ( ) type trieNode interface { - Persisted() bool // when a node is being commited to disk, if already persisted it is skipped + // when committing a node to disk, if already persisted it is skipped + Persisted() bool CachedDigest() []byte } @@ -52,7 +55,7 @@ type lazyNode struct { // SMT is a Sparse Merkle Trie object that implements the SparseMerkleTrie interface type SMT struct { TrieSpec - nodes KVStore + nodes kvstore.MapStore // Last persisted root hash savedRoot []byte // Current state of trie @@ -64,8 +67,13 @@ type SMT struct { // Hashes of persisted nodes deleted from trie type orphanNodes = [][]byte -// NewSparseMerkleTrie returns a new pointer to an SMT struct, and applys any options provided -func NewSparseMerkleTrie(nodes KVStore, hasher hash.Hash, options ...Option) *SMT { +// NewSparseMerkleTrie returns a new pointer to an SMT struct, and applies any +// options provided +func NewSparseMerkleTrie( + nodes kvstore.MapStore, + hasher hash.Hash, + options ...Option, +) *SMT { smt := SMT{ TrieSpec: newTrieSpec(hasher, false), nodes: nodes, @@ -76,8 +84,14 @@ func NewSparseMerkleTrie(nodes KVStore, hasher hash.Hash, options ...Option) *SM return &smt } -// ImportSparseMerkleTrie returns a pointer to an SMT struct with the provided root hash -func ImportSparseMerkleTrie(nodes KVStore, hasher hash.Hash, root []byte, options ...Option) *SMT { +// ImportSparseMerkleTrie returns a pointer to an SMT struct with the provided +// root hash +func ImportSparseMerkleTrie( + nodes kvstore.MapStore, + hasher hash.Hash, + root []byte, + options ...Option, +) *SMT { smt := NewSparseMerkleTrie(nodes, hasher, options...) smt.trie = &lazyNode{root} smt.savedRoot = root @@ -232,11 +246,11 @@ func (smt *SMT) delete(node trieNode, depth int, path []byte, orphans *orphanNod } if node == nil { - return node, ErrKeyNotPresent + return node, kvstore.ErrKVStoreKeyNotFound } if leaf, ok := node.(*leafNode); ok { if !bytes.Equal(path, leaf.path) { - return node, ErrKeyNotPresent + return node, kvstore.ErrKVStoreKeyNotFound } smt.addOrphan(orphans, node) return nil, nil @@ -246,7 +260,7 @@ func (smt *SMT) delete(node trieNode, depth int, path []byte, orphans *orphanNod if ext, ok := node.(*extensionNode); ok { if _, match := ext.match(path, depth); !match { - return node, ErrKeyNotPresent + return node, kvstore.ErrKVStoreKeyNotFound } ext.child, err = smt.delete(ext.child, depth+ext.length(), path, orphans) if err != nil { @@ -386,7 +400,7 @@ func (smt *SMT) Prove(key []byte) (proof *SparseMerkleProof, err error) { // the key (and return the key-value internal pair) as they share a common // prefix. If however, during the trie traversal according to the path, a nil // node is encountered, the traversal backsteps and flips the path bit for that -// depth (ie tries left if it tried right and vice versa). This guarentees that +// depth (ie tries left if it tried right and vice versa). This guarantees that // a proof of inclusion is found that has the most common bits with the path // provided, biased to the longest common prefix func (smt *SMT) ProveClosest(path []byte) ( diff --git a/smt_proofs_test.go b/smt_proofs_test.go index cefed7a..400d6aa 100644 --- a/smt_proofs_test.go +++ b/smt_proofs_test.go @@ -7,20 +7,22 @@ import ( "testing" "github.com/stretchr/testify/require" + + "github.com/pokt-network/smt/kvstore" ) // Test base case Merkle proof operations. func TestSMT_Proof_Operations(t *testing.T) { - var smn, smv KVStore + var smn, smv kvstore.MapStore var smt *SMTWithStorage var proof *SparseMerkleProof var result bool var root []byte var err error - smn, err = NewKVStore("") + smn = kvstore.NewSimpleMap() require.NoError(t, err) - smv, err = NewKVStore("") + smv = kvstore.NewSimpleMap() require.NoError(t, err) smt = NewSMTWithStorage(smn, smv, sha256.New()) base := smt.Spec() @@ -103,21 +105,16 @@ func TestSMT_Proof_Operations(t *testing.T) { result, err = VerifyProof(randomiseProof(proof), root, []byte("testKey3"), defaultValue, base) require.NoError(t, err) require.False(t, result) - - require.NoError(t, smn.Stop()) - require.NoError(t, smv.Stop()) } // Test sanity check cases for non-compact proofs. func TestSMT_Proof_ValidateBasic(t *testing.T) { - smn, err := NewKVStore("") - require.NoError(t, err) - smv, err := NewKVStore("") - require.NoError(t, err) + smn := kvstore.NewSimpleMap() + smv := kvstore.NewSimpleMap() smt := NewSMTWithStorage(smn, smv, sha256.New()) base := smt.Spec() - err = smt.Update([]byte("testKey1"), []byte("testValue1")) + err := smt.Update([]byte("testKey1"), []byte("testValue1")) require.NoError(t, err) err = smt.Update([]byte("testKey2"), []byte("testValue2")) require.NoError(t, err) @@ -175,14 +172,10 @@ func TestSMT_Proof_ValidateBasic(t *testing.T) { require.False(t, result) _, err = CompactProof(proof, base) require.Error(t, err) - - require.NoError(t, smn.Stop()) - require.NoError(t, smv.Stop()) } func TestSMT_ClosestProof_ValidateBasic(t *testing.T) { - smn, err := NewKVStore("") - require.NoError(t, err) + smn := kvstore.NewSimpleMap() smt := NewSparseMerkleTrie(smn, sha256.New()) np := NoPrehashSpec(sha256.New(), false) base := smt.Spec() @@ -255,14 +248,14 @@ func TestSMT_ClosestProof_ValidateBasic(t *testing.T) { // ProveClosest test against a visual representation of the trie // See: https://github.com/pokt-network/smt/assets/53987565/2c2ea530-a2e8-49d7-89c2-ca9c615b0c79 func TestSMT_ProveClosest(t *testing.T) { - var smn KVStore + var smn kvstore.MapStore var smt *SMT var proof *SparseMerkleClosestProof var result bool var root []byte var err error - smn, err = NewKVStore("") + smn = kvstore.NewSimpleMap() require.NoError(t, err) smt = NewSparseMerkleTrie(smn, sha256.New(), WithValueHasher(nil)) @@ -316,17 +309,15 @@ func TestSMT_ProveClosest(t *testing.T) { closestPath = sha256.Sum256([]byte("testKey4")) require.Equal(t, closestPath[:], proof.ClosestPath) require.Equal(t, []byte("testValue4"), proof.ClosestValueHash) - - require.NoError(t, smn.Stop()) } func TestSMT_ProveClosest_Empty(t *testing.T) { - var smn KVStore + var smn kvstore.MapStore var smt *SMT var proof *SparseMerkleClosestProof var err error - smn, err = NewKVStore("") + smn = kvstore.NewSimpleMap() require.NoError(t, err) smt = NewSparseMerkleTrie(smn, sha256.New(), WithValueHasher(nil)) @@ -347,18 +338,15 @@ func TestSMT_ProveClosest_Empty(t *testing.T) { result, err := VerifyClosestProof(proof, smt.Root(), NoPrehashSpec(sha256.New(), false)) require.NoError(t, err) require.True(t, result) - - require.NoError(t, smn.Stop()) } func TestSMT_ProveClosest_OneNode(t *testing.T) { - var smn KVStore + var smn kvstore.MapStore var smt *SMT var proof *SparseMerkleClosestProof var err error - smn, err = NewKVStore("") - require.NoError(t, err) + smn = kvstore.NewSimpleMap() smt = NewSparseMerkleTrie(smn, sha256.New(), WithValueHasher(nil)) require.NoError(t, smt.Update([]byte("foo"), []byte("bar"))) @@ -382,12 +370,10 @@ func TestSMT_ProveClosest_OneNode(t *testing.T) { result, err := VerifyClosestProof(proof, smt.Root(), NoPrehashSpec(sha256.New(), false)) require.NoError(t, err) require.True(t, result) - - require.NoError(t, smn.Stop()) } func TestSMT_ProveClosest_Proof(t *testing.T) { - var smn KVStore + var smn kvstore.MapStore var smt256 *SMT var smt512 *SMT var proof256 *SparseMerkleClosestProof @@ -395,7 +381,7 @@ func TestSMT_ProveClosest_Proof(t *testing.T) { var err error // setup trie (256+512 path hasher) and nodestore - smn, err = NewKVStore("") + smn = kvstore.NewSimpleMap() require.NoError(t, err) smt256 = NewSparseMerkleTrie(smn, sha256.New()) smt512 = NewSparseMerkleTrie(smn, sha512.New()) @@ -416,6 +402,4 @@ func TestSMT_ProveClosest_Proof(t *testing.T) { checkClosestCompactEquivalence(t, proof256, smt256.Spec()) checkClosestCompactEquivalence(t, proof512, smt512.Spec()) } - - require.NoError(t, smn.Stop()) } diff --git a/smt_test.go b/smt_test.go index c9545a2..f09682a 100644 --- a/smt_test.go +++ b/smt_test.go @@ -7,9 +7,15 @@ import ( "testing" "github.com/stretchr/testify/require" + + "github.com/pokt-network/smt/kvstore" ) -func NewSMTWithStorage(nodes, preimages KVStore, hasher hash.Hash, options ...Option) *SMTWithStorage { +func NewSMTWithStorage( + nodes, preimages kvstore.MapStore, + hasher hash.Hash, + options ...Option, +) *SMTWithStorage { return &SMTWithStorage{ SMT: NewSparseMerkleTrie(nodes, hasher, options...), preimages: preimages, @@ -17,17 +23,15 @@ func NewSMTWithStorage(nodes, preimages KVStore, hasher hash.Hash, options ...Op } func TestSMT_TrieUpdateBasic(t *testing.T) { - smn, err := NewKVStore("") - require.NoError(t, err) - smv, err := NewKVStore("") - require.NoError(t, err) + smn := kvstore.NewSimpleMap() + smv := kvstore.NewSimpleMap() lazy := NewSparseMerkleTrie(smn, sha256.New()) smt := &SMTWithStorage{SMT: lazy, preimages: smv} var value []byte var has bool // Test getting an empty key. - value, err = smt.GetValue([]byte("testKey")) + value, err := smt.GetValue([]byte("testKey")) require.NoError(t, err) require.Equal(t, defaultValue, value) @@ -94,23 +98,18 @@ func TestSMT_TrieUpdateBasic(t *testing.T) { value, err = smt.GetValue([]byte("testKey2")) require.NoError(t, err) require.Equal(t, []byte("testValue"), value) - - require.NoError(t, smn.Stop()) - require.NoError(t, smv.Stop()) } // Test base case trie delete operations with a few keys. func TestSMT_TrieDeleteBasic(t *testing.T) { - smn, err := NewKVStore("") - require.NoError(t, err) - smv, err := NewKVStore("") - require.NoError(t, err) + smn := kvstore.NewSimpleMap() + smv := kvstore.NewSimpleMap() lazy := NewSparseMerkleTrie(smn, sha256.New()) smt := &SMTWithStorage{SMT: lazy, preimages: smv} rootEmpty := smt.Root() // Testing inserting, deleting a key, and inserting it again. - err = smt.Update([]byte("testKey"), []byte("testValue")) + err := smt.Update([]byte("testKey"), []byte("testValue")) require.NoError(t, err) root1 := smt.Root() @@ -197,18 +196,13 @@ func TestSMT_TrieDeleteBasic(t *testing.T) { require.NoError(t, err) require.Equal(t, []byte("testValue"), value) require.Equal(t, root1, smt.Root(), "re-inserting key after deletion") - - require.NoError(t, smn.Stop()) - require.NoError(t, smv.Stop()) } // Test trie ops with known paths func TestSMT_TrieKnownPath(t *testing.T) { ph := dummyPathHasher{32} - smn, err := NewKVStore("") - require.NoError(t, err) - smv, err := NewKVStore("") - require.NoError(t, err) + smn := kvstore.NewSimpleMap() + smv := kvstore.NewSimpleMap() smt := NewSMTWithStorage(smn, smv, sha256.New(), WithPathHasher(ph)) var value []byte @@ -226,7 +220,7 @@ func TestSMT_TrieKnownPath(t *testing.T) { keys[5][0] = byte(0b11100000) keys[6][0] = byte(0b11110000) - err = smt.Update(keys[0], []byte("testValue1")) + err := smt.Update(keys[0], []byte("testValue1")) require.NoError(t, err) err = smt.Update(keys[1], []byte("testValue2")) require.NoError(t, err) @@ -273,18 +267,13 @@ func TestSMT_TrieKnownPath(t *testing.T) { value, err = smt.GetValue(keys[5]) require.NoError(t, err) require.Equal(t, []byte("testValue6"), value) - - require.NoError(t, smn.Stop()) - require.NoError(t, smv.Stop()) } // Test trie operations when two leafs are immediate neighbors. func TestSMT_TrieMaxHeightCase(t *testing.T) { ph := dummyPathHasher{32} - smn, err := NewKVStore("") - require.NoError(t, err) - smv, err := NewKVStore("") - require.NoError(t, err) + smn := kvstore.NewSimpleMap() + smv := kvstore.NewSimpleMap() smt := NewSMTWithStorage(smn, smv, sha256.New(), WithPathHasher(ph)) var value []byte @@ -292,7 +281,7 @@ func TestSMT_TrieMaxHeightCase(t *testing.T) { // The dummy hash function will return the preimage itself as the digest. key1 := make([]byte, ph.PathSize()) key2 := make([]byte, ph.PathSize()) - _, err = rand.Read(key1) + _, err := rand.Read(key1) require.NoError(t, err) copy(key2, key1) // We make key2's least significant bit different than key1's @@ -316,13 +305,10 @@ func TestSMT_TrieMaxHeightCase(t *testing.T) { proof, err := smt.Prove(key1) require.NoError(t, err) require.Equal(t, 256, len(proof.SideNodes), "unexpected proof size") - - require.NoError(t, smn.Stop()) - require.NoError(t, smv.Stop()) } func TestSMT_OrphanRemoval(t *testing.T) { - var smn, smv KVStore + var smn, smv kvstore.MapStore var impl *SMT var smt *SMTWithStorage var err error @@ -332,9 +318,8 @@ func TestSMT_OrphanRemoval(t *testing.T) { return smn.Len() } setup := func() { - smn, err = NewKVStore("") - require.NoError(t, err) - smv, err = NewKVStore("") + smn = kvstore.NewSimpleMap() + smv = kvstore.NewSimpleMap() require.NoError(t, err) impl = NewSparseMerkleTrie(smn, sha256.New()) smt = &SMTWithStorage{SMT: impl, preimages: smv} @@ -410,7 +395,4 @@ func TestSMT_OrphanRemoval(t *testing.T) { require.Equal(t, 1, nodeCount(t), tci) } }) - - require.NoError(t, smn.Stop()) - require.NoError(t, smv.Stop()) } diff --git a/smt_utils_test.go b/smt_utils_test.go index 70e9db6..89ea986 100644 --- a/smt_utils_test.go +++ b/smt_utils_test.go @@ -4,17 +4,20 @@ import ( "bytes" "errors" - badger "github.com/dgraph-io/badger/v4" + "github.com/pokt-network/smt/kvstore" ) -// SMTWithStorage wraps an SMT with a mapping of value hashes to values (preimages), for use in tests. -// Note: this doesn't delete from preimages (inputs to hashing functions), since there could be duplicate stored values. +// SMTWithStorage wraps an SMT with a mapping of value hashes to values +// (preimages), for use in tests. +// Note: this doesn't delete from preimages (inputs to hashing functions), +// since there could be duplicate stored values. type SMTWithStorage struct { *SMT - preimages KVStore + preimages kvstore.MapStore } -// Update updates a key with a new value in the trie and adds the value to the preimages KVStore +// Update updates a key with a new value in the trie and adds the value to +// the preimages KVStore func (smt *SMTWithStorage) Update(key, value []byte) error { if err := smt.SMT.Update(key, value); err != nil { return err @@ -39,7 +42,7 @@ func (smt *SMTWithStorage) GetValue(key []byte) ([]byte, error) { } value, err := smt.preimages.Get(valueHash) if err != nil { - if errors.Is(err, badger.ErrKeyNotFound) { + if errors.Is(err, kvstore.ErrKVStoreKeyNotFound) { // If key isn't found, return default value value = defaultValue } else { @@ -57,7 +60,8 @@ func (smt *SMTWithStorage) Has(key []byte) (bool, error) { return !bytes.Equal(defaultValue, val), err } -// ProveCompact generates a compacted Merkle proof for a key against the current root. +// ProveCompact generates a compacted Merkle proof for a key against the +// current root. func ProveCompact(key []byte, smt SparseMerkleTrie) (*SparseCompactMerkleProof, error) { proof, err := smt.Prove(key) if err != nil { @@ -66,7 +70,8 @@ func ProveCompact(key []byte, smt SparseMerkleTrie) (*SparseCompactMerkleProof, return CompactProof(proof, smt.Spec()) } -// dummyHasher is a dummy hasher for tests, where the digest of keys is equivalent to the preimage. +// dummyHasher is a dummy hasher for tests, where the digest of keys is +// equivalent to the preimage. type dummyPathHasher struct { size int } diff --git a/types.go b/types.go index 0431d25..ace6cad 100644 --- a/types.go +++ b/types.go @@ -1,7 +1,6 @@ package smt import ( - "errors" "hash" ) @@ -13,9 +12,6 @@ const ( var ( defaultValue []byte defaultSum [sumSize]byte - - // ErrKeyNotPresent is returned when a key is not present in the trie. - ErrKeyNotPresent = errors.New("key already empty") ) // SparseMerkleTrie represents a Sparse Merkle Trie. @@ -30,10 +26,11 @@ type SparseMerkleTrie interface { Root() []byte // Prove computes a Merkle proof of inclusion or exclusion of a key. Prove(key []byte) (*SparseMerkleProof, error) - // ProveClosest computes a Merkle proof of inclusion for a key in the trie which is - // closest to the path provided. It will search for the key with the longest common - // prefix before finding the key with the most common bits as the path provided. - ProveClosest([]byte) (proof *SparseMerkleClosestProof, err error) + // ProveClosest computes a Merkle proof of inclusion for a key in the trie + // which is closest to the path provided. It will search for the key with + // the longest common prefix before finding the key with the most common + // bits as the path provided. + ProveClosest([]byte) (*SparseMerkleClosestProof, error) // Commit saves the trie's state to its persistent storage. Commit() error // Spec returns the TrieSpec for the trie @@ -54,10 +51,11 @@ type SparseMerkleSumTrie interface { Sum() uint64 // Prove computes a Merkle proof of inclusion or exclusion of a key. Prove(key []byte) (*SparseMerkleProof, error) - // ProveClosest computes a Merkle proof of inclusion for a key in the trie which is - // closest to the path provided. It will search for the key with the longest common - // prefix before finding the key with the most common bits as the path provided. - ProveClosest([]byte) (proof *SparseMerkleClosestProof, err error) + // ProveClosest computes a Merkle proof of inclusion for a key in the trie + // which is closest to the path provided. It will search for the key with + // the longest common prefix before finding the key with the most common + // bits as the path provided. + ProveClosest([]byte) (*SparseMerkleClosestProof, error) // Commit saves the trie's state to its persistent storage. Commit() error // Spec returns the TrieSpec for the trie @@ -133,7 +131,8 @@ func (spec *TrieSpec) hashNode(node trieNode) []byte { return *cache } -// sumSerialize serializes a node returning the preimage hash, its sum and any errors encountered +// sumSerialize serializes a node returning the preimage hash, its sum and any +// errors encountered func (spec *TrieSpec) sumSerialize(node trieNode) (preimage []byte) { switch n := node.(type) { case *lazyNode: From 83a994a44f8eadaa0d2f94a1979977e341e207a3 Mon Sep 17 00:00:00 2001 From: h5law <53987565+h5law@users.noreply.github.com> Date: Fri, 22 Dec 2023 14:44:43 +0000 Subject: [PATCH 06/32] chore: makefile changes --- Makefile | 5 ++--- benchmarks/bench_leaf_test.go | 2 -- benchmarks/bench_smst_test.go | 2 -- benchmarks/bench_smt_test.go | 2 -- benchmarks/bench_utils_test.go | 2 -- benchmarks/proof_sizes_test.go | 2 -- 6 files changed, 2 insertions(+), 13 deletions(-) diff --git a/Makefile b/Makefile index 54e9722..dc1e1d5 100644 --- a/Makefile +++ b/Makefile @@ -1,5 +1,3 @@ -SHELL := /bin/bash - .SILENT: ##################### @@ -41,7 +39,8 @@ go_docs: check_godoc ## Generate documentation for the project .PHONY: test_all test_all: ## runs the test suite - go test -v -p 1 ./... -mod=readonly -race + go test -v -p 1 ./ -mod=readonly -race + go test -v -p 1 ./kvstore/... -race .PHONY: test_badger test_badger: ## runs the badger KVStore submodule's test suite diff --git a/benchmarks/bench_leaf_test.go b/benchmarks/bench_leaf_test.go index da4eb9b..2c0fb31 100644 --- a/benchmarks/bench_leaf_test.go +++ b/benchmarks/bench_leaf_test.go @@ -1,5 +1,3 @@ -//go:build benchmarks - package smt import ( diff --git a/benchmarks/bench_smst_test.go b/benchmarks/bench_smst_test.go index 8834381..5c9bfa1 100644 --- a/benchmarks/bench_smst_test.go +++ b/benchmarks/bench_smst_test.go @@ -1,5 +1,3 @@ -//go:build benchmarks - package smt import ( diff --git a/benchmarks/bench_smt_test.go b/benchmarks/bench_smt_test.go index 3169068..6a3e67f 100644 --- a/benchmarks/bench_smt_test.go +++ b/benchmarks/bench_smt_test.go @@ -1,5 +1,3 @@ -//go:build benchmarks - package smt import ( diff --git a/benchmarks/bench_utils_test.go b/benchmarks/bench_utils_test.go index 030880c..947e366 100644 --- a/benchmarks/bench_utils_test.go +++ b/benchmarks/bench_utils_test.go @@ -1,5 +1,3 @@ -//go:build benchmarks - package smt import ( diff --git a/benchmarks/proof_sizes_test.go b/benchmarks/proof_sizes_test.go index ff2e3dc..609d991 100644 --- a/benchmarks/proof_sizes_test.go +++ b/benchmarks/proof_sizes_test.go @@ -1,5 +1,3 @@ -//go:build benchmarks - package smt import ( From bcc569f61436ea94005103a37d7010b6018fc056 Mon Sep 17 00:00:00 2001 From: h5law <53987565+h5law@users.noreply.github.com> Date: Fri, 22 Dec 2023 14:51:53 +0000 Subject: [PATCH 07/32] chore: update workflow to cover submodule tests --- .github/workflows/test.yml | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index e981f6a..3e91f8b 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -37,12 +37,19 @@ jobs: skip-cache: true only-new-issues: true + - name: Install gocovmerge + run: go install github.com/wadey/gocovmerge@latest + - name: Create coverage report and run tests run: | # Run each of the tests (excluding benchmarks) outputting the JSON # to the test_results.json file for usage in later steps. set -euo pipefail - go test -v -json -p 1 ./ -mod=readonly -race -coverprofile=coverage.txt -covermode=atomic 2>&1 | tee -a test_results.json + go test -v -json -p 1 ./ -mod=readonly -race -coverprofile=coverage1.txt -covermode=atomic 2>&1 | tee -a test_results.json + go test -v -json -p 1 ./kvstore/... -race -coverprofile=coverage2.txt -covermode=atomic 2>&1 | tee -a test_results.json + go test -v -json -p 1 ./kvstore/badger/... -mod=readonly -race -coverprofile=coverage3.txt -covermode=atomic 2>&1 | tee -a test_results.json + # Combine coverage reports + gocovmerge coverage1.txt coverage2.txt coverage3.txt > coverage.txt - name: Sanitize test results # We're utilizing `tee` above which can capture non-json stdout output From 428098623d51dd655ac7f0e1c5768d92ca8e775d Mon Sep 17 00:00:00 2001 From: h5law <53987565+h5law@users.noreply.github.com> Date: Sat, 23 Dec 2023 02:08:35 +0000 Subject: [PATCH 08/32] chore: revert go version to 1.20.12 --- go.mod | 2 +- go.work | 2 +- kvstore/badger/go.mod | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/go.mod b/go.mod index 31c721b..f901632 100644 --- a/go.mod +++ b/go.mod @@ -1,6 +1,6 @@ module github.com/pokt-network/smt -go 1.21.5 +go 1.20 require github.com/stretchr/testify v1.7.1 diff --git a/go.work b/go.work index 7b9600a..a3758d3 100644 --- a/go.work +++ b/go.work @@ -1,4 +1,4 @@ -go 1.21.5 +go 1.20 use ( // Include the main SMT module diff --git a/kvstore/badger/go.mod b/kvstore/badger/go.mod index 74f5917..d1e71cd 100644 --- a/kvstore/badger/go.mod +++ b/kvstore/badger/go.mod @@ -1,6 +1,6 @@ module github.com/pokt-network/smt/badger -go 1.21.5 +go 1.20 require ( github.com/dgraph-io/badger/v4 v4.2.0 From 0ce48aab4d9e46841cae523888cc5febfe17baf5 Mon Sep 17 00:00:00 2001 From: h5law <53987565+h5law@users.noreply.github.com> Date: Sat, 23 Dec 2023 02:09:20 +0000 Subject: [PATCH 09/32] chore: unexport simpleMap MapStore implementation --- kvstore/simplemap.go | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/kvstore/simplemap.go b/kvstore/simplemap.go index 7e3325a..41e0446 100644 --- a/kvstore/simplemap.go +++ b/kvstore/simplemap.go @@ -15,20 +15,20 @@ type MapStore interface { ClearAll() error } -// SimpleMap is a simple in-memory map. -type SimpleMap struct { +// simpleMap is a simple in-memory map. +type simpleMap struct { m map[string][]byte } // NewSimpleMap creates a new SimpleMap instance. func NewSimpleMap() MapStore { - return &SimpleMap{ + return &simpleMap{ m: make(map[string][]byte), } } // Get gets the value for a key. -func (sm *SimpleMap) Get(key []byte) ([]byte, error) { +func (sm *simpleMap) Get(key []byte) ([]byte, error) { if len(key) == 0 { return nil, ErrKVStoreEmptyKey } @@ -41,7 +41,7 @@ func (sm *SimpleMap) Get(key []byte) ([]byte, error) { } // Set updates the value for a key. -func (sm *SimpleMap) Set(key []byte, value []byte) error { +func (sm *simpleMap) Set(key []byte, value []byte) error { if len(key) == 0 { return ErrKVStoreEmptyKey } @@ -50,7 +50,7 @@ func (sm *SimpleMap) Set(key []byte, value []byte) error { } // Delete deletes a key. -func (sm *SimpleMap) Delete(key []byte) error { +func (sm *simpleMap) Delete(key []byte) error { if len(key) == 0 { return ErrKVStoreEmptyKey } @@ -63,13 +63,13 @@ func (sm *SimpleMap) Delete(key []byte) error { } // Len returns the number of key-value pairs in the store. -func (sm *SimpleMap) Len() int { +func (sm *simpleMap) Len() int { return len(sm.m) } // ClearAll clears all key-value pairs // NB: This should only be used for testing purposes. -func (sm *SimpleMap) ClearAll() error { +func (sm *simpleMap) ClearAll() error { sm.m = make(map[string][]byte) return nil } From 27dd854bb8cb98be86a4a2afc5130b7c8042dcff Mon Sep 17 00:00:00 2001 From: h5law <53987565+h5law@users.noreply.github.com> Date: Sat, 23 Dec 2023 02:09:42 +0000 Subject: [PATCH 10/32] chore: add kvstore package godoc --- kvstore/godoc.go | 12 ++++++++++++ 1 file changed, 12 insertions(+) create mode 100644 kvstore/godoc.go diff --git a/kvstore/godoc.go b/kvstore/godoc.go new file mode 100644 index 0000000..a40e6e3 --- /dev/null +++ b/kvstore/godoc.go @@ -0,0 +1,12 @@ +// Package kvstore provides a simple interface for a key-value store, the +// `MapStore` and a simple implementation of this interface. These are provided +// for convenience and for simple in-memory use cases. The KVStore package also +// contains submodules with more complex implementations of key-value stores +// that can be used independently, or as the backend for the SM(S)T. +// These submodules satisfy the simple `MapStore` interface as well as also +// providing their own more complex interfaces, which can provide more features +// and better performance - such as persistence, backups and restores for example. +// These are included as submodules as they are not required for the SM(S)T and +// any key-value store that satisfies the `MapStore` interface can be used with +// the SM(S)T. +package kvstore From 66beac15fb3e18bdf5da4d727c6df0a660568b84 Mon Sep 17 00:00:00 2001 From: h5law <53987565+h5law@users.noreply.github.com> Date: Sat, 23 Dec 2023 02:10:12 +0000 Subject: [PATCH 11/32] chore: s/badger/v4/g --- kvstore/badger/kvstore.go | 75 ++++++++++++++++++++++----------------- 1 file changed, 43 insertions(+), 32 deletions(-) diff --git a/kvstore/badger/kvstore.go b/kvstore/badger/kvstore.go index 6e6c5af..d818887 100644 --- a/kvstore/badger/kvstore.go +++ b/kvstore/badger/kvstore.go @@ -2,10 +2,9 @@ package badger import ( "errors" - "fmt" "io" - badger "github.com/dgraph-io/badger/v4" + v4 "github.com/dgraph-io/badger/v4" ) const ( @@ -14,43 +13,42 @@ const ( var _ KVStore = &badgerKVStore{} -var ( - ErrKVStoreExists = errors.New("kvstore already exists") - ErrKVStoreNotExists = errors.New("kvstore does not exist") -) - type badgerKVStore struct { - db *badger.DB - last_backup uint64 // timestamp of the most recent backup + db *v4.DB + lastBackup uint64 // timestamp of the most recent backup } // NewKVStore creates a new KVStore using badger as the underlying database -// if no path for a peristence database is provided it will create one in-memory +// if no path for a persistence database is provided it will create one in-memory func NewKVStore(path string) (KVStore, error) { - var db *badger.DB + var db *v4.DB var err error if path == "" { - db, err = badger.Open(badgerOptions("").WithInMemory(true)) + db, err = v4.Open(badgerOptions("").WithInMemory(true)) } else { - db, err = badger.Open(badgerOptions(path)) + db, err = v4.Open(badgerOptions(path)) } if err != nil { - return nil, err + return nil, errors.Join(ErrBadgerOpeningStore, err) } return &badgerKVStore{db: db}, nil } // Set sets/updates the value for a given key func (store *badgerKVStore) Set(key, value []byte) error { - return store.db.Update(func(tx *badger.Txn) error { + err := store.db.Update(func(tx *v4.Txn) error { return tx.Set(key, value) }) + if err != nil { + return errors.Join(ErrBadgerUnableToSetValue, err) + } + return nil } // Get returns the value for a given key func (store *badgerKVStore) Get(key []byte) ([]byte, error) { var val []byte - if err := store.db.View(func(tx *badger.Txn) error { + if err := store.db.View(func(tx *v4.Txn) error { item, err := tx.Get(key) if err != nil { return err @@ -61,23 +59,27 @@ func (store *badgerKVStore) Get(key []byte) ([]byte, error) { } return nil }); err != nil { - return nil, err + return nil, errors.Join(ErrBadgerUnableToGetValue, err) } return val, nil } // Delete removes a key and its value from the store func (store *badgerKVStore) Delete(key []byte) error { - return store.db.Update(func(tx *badger.Txn) error { + err := store.db.Update(func(tx *v4.Txn) error { return tx.Delete(key) }) + if err != nil { + return errors.Join(ErrBadgerUnableToDeleteValue, err) + } + return nil } // GetAll returns all keys and values with the given prefix in the specified order // if the prefix []byte{} is given then all key-value pairs are returned func (store *badgerKVStore) GetAll(prefix []byte, descending bool) (keys, values [][]byte, err error) { - if err := store.db.View(func(tx *badger.Txn) error { - opt := badger.DefaultIteratorOptions + if err := store.db.View(func(tx *v4.Txn) error { + opt := v4.DefaultIteratorOptions opt.Prefix = prefix opt.Reverse = descending if descending { @@ -102,7 +104,7 @@ func (store *badgerKVStore) GetAll(prefix []byte, descending bool) (keys, values } return nil }); err != nil { - return nil, nil, err + return nil, nil, errors.Join(ErrBadgerIteratingStore, err) } return keys, values, nil } @@ -118,7 +120,10 @@ func (store *badgerKVStore) Exists(key []byte) (bool, error) { // ClearAll deletes all key-value pairs in the store func (store *badgerKVStore) ClearAll() error { - return store.db.DropAll() + if err := store.db.DropAll(); err != nil { + return errors.Join(ErrBadgerClearingStore, err) + } + return nil } // Backup creates a full backup of the store written to the provided writer @@ -126,32 +131,38 @@ func (store *badgerKVStore) ClearAll() error { func (store *badgerKVStore) Backup(w io.Writer, incremental bool) error { version := uint64(0) if incremental { - version = store.last_backup + version = store.lastBackup } timestamp, err := store.db.Backup(w, version) if err != nil { - return err + return errors.Join(ErrBadgerUnableToBackup, err) } - store.last_backup = timestamp + store.lastBackup = timestamp return nil } // Restore loads the store from a backup in the reader provided // NOTE: Do not call on a database that is running other concurrent transactions func (store *badgerKVStore) Restore(r io.Reader) error { - return store.db.Load(r, maxPendingWrites) + if err := store.db.Load(r, maxPendingWrites); err != nil { + return errors.Join(ErrBadgerUnableToRestore, err) + } + return nil } // Stop closes the database connection, disabling any access to the store func (store *badgerKVStore) Stop() error { - return store.db.Close() + if err := store.db.Close(); err != nil { + return errors.Join(ErrBadgerClosingStore, err) + } + return nil } // Len gives the number of keys in the store func (store *badgerKVStore) Len() int { count := 0 - if err := store.db.View(func(tx *badger.Txn) error { - opt := badger.DefaultIteratorOptions + if err := store.db.View(func(tx *v4.Txn) error { + opt := v4.DefaultIteratorOptions opt.Prefix = []byte{} opt.Reverse = false it := tx.NewIterator(opt) @@ -161,7 +172,7 @@ func (store *badgerKVStore) Len() int { } return nil }); err != nil { - panic(fmt.Sprintf("error getting key count: %v", err)) + panic(errors.Join(ErrBadgerGettingStoreLength, err)) } return count } @@ -182,8 +193,8 @@ func prefixEndBytes(prefix []byte) []byte { } // badgerOptions returns the badger options for the store being created -func badgerOptions(path string) badger.Options { - opts := badger.DefaultOptions(path) +func badgerOptions(path string) v4.Options { + opts := v4.DefaultOptions(path) opts.Logger = nil // disable badger's logger since it's very noisy return opts } From a0ebc3fd20898883f08706bc4bd9d9be25402057 Mon Sep 17 00:00:00 2001 From: h5law <53987565+h5law@users.noreply.github.com> Date: Sat, 23 Dec 2023 02:10:42 +0000 Subject: [PATCH 12/32] chore: use custom errors and wrap badgerv4 errors --- kvstore/badger/errors.go | 38 ++++++++++++++++++++++++++++++++++ kvstore/badger/kvstore_test.go | 29 ++++++++++++-------------- 2 files changed, 51 insertions(+), 16 deletions(-) create mode 100644 kvstore/badger/errors.go diff --git a/kvstore/badger/errors.go b/kvstore/badger/errors.go new file mode 100644 index 0000000..6759b30 --- /dev/null +++ b/kvstore/badger/errors.go @@ -0,0 +1,38 @@ +package badger + +import ( + "errors" +) + +var ( + // ErrBadgerOpeningStore is returned when the badger store cannot be opened + // or an error occurs while opening/creating the KVStore + ErrBadgerOpeningStore = errors.New("error opening the store") + // ErrBadgerUnableToSetValue is returned when the badger store fails to + // set a value + ErrBadgerUnableToSetValue = errors.New("unable to set value") + // ErrBadgerUnableToGetValue is returned when the badger store fails to + // retrieve a value + ErrBadgerUnableToGetValue = errors.New("unable to get value") + // ErrBadgerUnableToDeleteValue is returned when the badger store fails to + // delete a value + ErrBadgerUnableToDeleteValue = errors.New("unable to delete value") + // ErrBadgerIteratingStore is returned when the badger store fails to + // iterate over the database + ErrBadgerIteratingStore = errors.New("unable to iterate over database") + // ErrBadgerClearingStore is returned when the badger store fails to + // clear all values + ErrBadgerClearingStore = errors.New("unable to clear store") + // ErrBadgerUnableToBackup is returned when the badger store fails to + // backup the database + ErrBadgerUnableToBackup = errors.New("unable to backup database") + // ErrBadgerUnableToRestore is returned when the badger store fails to + // restore the database + ErrBadgerUnableToRestore = errors.New("unable to restore database") + // ErrBadgerClosingStore is returned when the badger store fails to + // close the database + ErrBadgerClosingStore = errors.New("unable to close database") + // ErrBadgerGettingStoreLength is returned when the badger store fails to + // get the length of the database + ErrBadgerGettingStoreLength = errors.New("unable to get database length") +) diff --git a/kvstore/badger/kvstore_test.go b/kvstore/badger/kvstore_test.go index 7bbbcfd..bac9d3d 100644 --- a/kvstore/badger/kvstore_test.go +++ b/kvstore/badger/kvstore_test.go @@ -2,12 +2,9 @@ package badger_test import ( "bytes" - "encoding/hex" - "fmt" "strings" "testing" - v4 "github.com/dgraph-io/badger/v4" "github.com/stretchr/testify/require" "github.com/pokt-network/smt/badger" @@ -49,7 +46,7 @@ func TestBadger_KVStore_BasicOperations(t *testing.T) { key: nil, value: []byte("bar"), fail: true, - expectedErr: v4.ErrEmptyKey, + expectedErr: badger.ErrBadgerUnableToSetValue, }, { desc: "Fails to set a value to a key that is too large", @@ -57,7 +54,7 @@ func TestBadger_KVStore_BasicOperations(t *testing.T) { key: invalidKey[:], value: []byte("bar"), fail: true, - expectedErr: fmt.Errorf("Key with size 65001 exceeded 65000 limit. Key:\n%s", hex.Dump(invalidKey[:1<<10])), + expectedErr: badger.ErrBadgerUnableToSetValue, }, { desc: "Successfully retrieve a value from the store", @@ -73,7 +70,7 @@ func TestBadger_KVStore_BasicOperations(t *testing.T) { key: []byte("bar"), value: nil, fail: true, - expectedErr: v4.ErrKeyNotFound, + expectedErr: badger.ErrBadgerUnableToGetValue, }, { desc: "Fails when the key is empty", @@ -81,7 +78,7 @@ func TestBadger_KVStore_BasicOperations(t *testing.T) { key: nil, value: nil, fail: true, - expectedErr: v4.ErrEmptyKey, + expectedErr: badger.ErrBadgerUnableToGetValue, }, { desc: "Successfully deletes a value in the store", @@ -100,20 +97,20 @@ func TestBadger_KVStore_BasicOperations(t *testing.T) { expectedErr: nil, }, { - desc: "Fails to set value to nil key", + desc: "Fails to delete a nil key", op: "delete", key: nil, value: nil, fail: true, - expectedErr: v4.ErrEmptyKey, + expectedErr: badger.ErrBadgerUnableToDeleteValue, }, { - desc: "Fails to set a value to a key that is too large", + desc: "Fails to delete a value for a key that is too large", op: "delete", key: invalidKey[:], value: nil, fail: true, - expectedErr: fmt.Errorf("Key with size 65001 exceeded 65000 limit. Key:\n%s", hex.Dump(invalidKey[:1<<10])), + expectedErr: badger.ErrBadgerUnableToDeleteValue, }, } @@ -127,7 +124,7 @@ func TestBadger_KVStore_BasicOperations(t *testing.T) { err := store.Set(tc.key, tc.value) if tc.fail { require.Error(t, err) - require.EqualError(t, tc.expectedErr, err.Error()) + require.ErrorIs(t, err, tc.expectedErr) } else { require.NoError(t, err) got, err := store.Get(tc.key) @@ -138,7 +135,7 @@ func TestBadger_KVStore_BasicOperations(t *testing.T) { got, err := store.Get(tc.key) if tc.fail { require.Error(t, err) - require.EqualError(t, tc.expectedErr, err.Error()) + require.ErrorIs(t, err, tc.expectedErr) } else { require.NoError(t, err) require.Equal(t, tc.value, got) @@ -147,11 +144,11 @@ func TestBadger_KVStore_BasicOperations(t *testing.T) { err := store.Delete(tc.key) if tc.fail { require.Error(t, err) - require.EqualError(t, tc.expectedErr, err.Error()) + require.ErrorIs(t, err, tc.expectedErr) } else { require.NoError(t, err) _, err := store.Get(tc.key) - require.EqualError(t, err, v4.ErrKeyNotFound.Error()) + require.ErrorIs(t, err, badger.ErrBadgerUnableToGetValue) } } }) @@ -283,7 +280,7 @@ func TestBadger_KVStore_Exists(t *testing.T) { // Key does not exist exists, err = store.Exists([]byte("oof")) - require.EqualError(t, err, v4.ErrKeyNotFound.Error()) + require.ErrorIs(t, err, badger.ErrBadgerUnableToGetValue) require.False(t, exists) err = store.Stop() From 719ea5559a9e08e291e0860fe7fca964337d71d7 Mon Sep 17 00:00:00 2001 From: h5law <53987565+h5law@users.noreply.github.com> Date: Wed, 3 Jan 2024 14:07:08 +0000 Subject: [PATCH 13/32] squash: merged feat/doc-changes commit 179f61687f2e98d83931eb526b363f409d45d2b6 Author: h5law <53987565+h5law@users.noreply.github.com> Date: Wed Jan 3 14:06:26 2024 +0000 chore: address comments --- README.md | 2 +- docs/smt.md | 42 ++++++++++++++++++++++++------------------ 2 files changed, 25 insertions(+), 19 deletions(-) diff --git a/README.md b/README.md index 0ff0d9a..9368f9f 100644 --- a/README.md +++ b/README.md @@ -16,7 +16,7 @@ -**NOTE: Requires Go 1.20.12** +**NOTE: Requires Go 1.20.12+** ## Overview diff --git a/docs/smt.md b/docs/smt.md index ca7894d..29eb03c 100644 --- a/docs/smt.md +++ b/docs/smt.md @@ -400,27 +400,33 @@ _NOTE: Throughout this document, `commitment` of the the trie's root hash is als referred to as closing the trie, such that no more updates are made to it once committed._ -Take the following attack vector (**without** a commit prior to a reveal) into -consideration: +Consider the following attack vector (**without** a commit prior to a reveal) +into consideration: -1. The **prover** identifies the hash the **verifier** intends to check against -1. The **prover** then places a leaf in the tree with a common prefix to the - identified hash -1. Due to the deterministic nature of the `ClosestProof` method this leaf will +1. The **verifier** picks the hash (i.e. a single branch) they intend to check +1. The **prover** inserts a leaf (i.e. a value) whose key (determined via the + hasher) has a longer common prefix than any other leaf in the trie. +1. Due to the deterministic nature of the `ClosestProof`, method this leaf will **always** be returned given the identified hash. -1. The **verifier** then verifies the revealed `ClosestProof`, in turn validating - a maliciously placed leaf. - -Take the following normal flow (**with** a commit prior to reveal) as - -1. The **prover** commits to the state of their trie and publishes their root - hash, _closing_ their trie. -1. The **verifier** provides a hash to be used in the `commit & reveal` process -1. The **prover** then utilises this hash and the `ClosestProof` method on their +1. The **verifier** then verifies the revealed `ClosestProof`, which returns a + branch the **prover** inserted after knowing which leaf was going to be + checked. + +Consider the following normal flow (**with** a commit prior to reveal) as + +1. The **prover** commits to the state of their trie by publishes their root + hash, thereby _closing_ their trie and not being able to make further + changes. +1. The **verifier** selects a hash to be used in the `commit & reveal` process + that the **prover** must provide a closest proof for. +1. The **prover** utilises this hash and computes the `ClosestProof` on their _closed_ trie, producing a `ClosestProof`, thus revealing a deterministic, - yet pseudo-random leaf. -1. The **verifier** then verifies the proof, in turn verifying the commitment - made by the **prover** and the state of the **prover**'s trie. + pseudo-random leaf that existed in the tree prior to commitment, yet +1. The **verifier** verifies the proof, in turn, verifying the commitment + made by the **prover** to the state of the trie in the first step. +1. The **prover** had no opportunity to insert a new leaf into the trie + after learning which hash the **verifier** was going to require a + `ClosestProof` for. ### Compression From 7a0f8fbb33da723212fac470ab85b3c368d499b7 Mon Sep 17 00:00:00 2001 From: h5law <53987565+h5law@users.noreply.github.com> Date: Wed, 3 Jan 2024 14:57:42 +0000 Subject: [PATCH 14/32] chore: add executable code examples --- docs/merkle-sum-trie.md | 46 ------------------------------------ docs/smt.md | 52 ++++++----------------------------------- smst_example_test.go | 43 ++++++++++++++++++++++++++++++++++ smt_example_test.go | 36 ++++++++++++++++++++++++++++ 4 files changed, 86 insertions(+), 91 deletions(-) create mode 100644 smst_example_test.go create mode 100644 smt_example_test.go diff --git a/docs/merkle-sum-trie.md b/docs/merkle-sum-trie.md index 498e43b..757536e 100644 --- a/docs/merkle-sum-trie.md +++ b/docs/merkle-sum-trie.md @@ -11,7 +11,6 @@ + [Binary Sum Digests](#binary-sum-digests) - [Sum](#sum) - [Nil Values](#nil-values) -- [Example](#example) @@ -270,49 +269,4 @@ Assume `(key, value, weight)` groupings as follows: - `(key, value, weight)` -> DOES modify the `root` hash - Proving this `key` is in the trie will succeed -## Example - -```go -package main - -import ( - "crypto/sha256" - "fmt" - - "github.com/pokt-network/smt" - "github.com/pokt-network/smt/kvstore" -) - -func main() { - // Initialise a new in-memory key-value store to store the nodes of the trie - // (Note: the trie only stores hashed values, not raw value data) - nodeStore := kvstore.NewSimpleMap() - - // Initialise the trie - trie := smt.NewSparseMerkleSumTrie(nodeStore, sha256.New()) - - // Update trie with keys, values and their sums - _ = trie.Update([]byte("foo"), []byte("oof"), 10) - _ = trie.Update([]byte("baz"), []byte("zab"), 7) - _ = trie.Update([]byte("bin"), []byte("nib"), 3) - - // Commit the changes to the nodeStore - _ = trie.Commit() - - sum := trie.Sum() - fmt.Println(sum == 20) // true - - // Generate a Merkle proof for "foo" - proof, _ := trie.Prove([]byte("foo")) - root := trie.Root() // We also need the current trie root for the proof - - // Verify the Merkle proof for "foo"="oof" where "foo" has a sum of 10 - if valid := smt.VerifySumProof(proof, root, []byte("foo"), []byte("oof"), 10, trie.Spec()); valid { - fmt.Println("Proof verification succeeded.") - } else { - fmt.Println("Proof verification failed.") - } -} -``` - [plasma core docs]: https://plasma-core.readthedocs.io/en/latest/specs/sum-tree.html diff --git a/docs/smt.md b/docs/smt.md index 29eb03c..f12b3d7 100644 --- a/docs/smt.md +++ b/docs/smt.md @@ -28,7 +28,6 @@ + [Badger](#badger) * [Data Loss](#data-loss) - [Sparse Merkle Sum Trie](#sparse-merkle-sum-trie) -- [Example](#example) @@ -453,9 +452,9 @@ schemes. ## Database -By default this library provides a simple interface (`MapStore`) and a simple +By default, this library provides a simple interface (`MapStore`) and a simple in-memory key-value database found in [`simplemap.go`](../kvstore/simplemap.go). -However, any key-value store that implements this interface can be used as the +However, any key-value store implementing this interface can be used as the node store to back the trie. See: [`kvstore/simplemap.go`](../kvstore/simplemap.go) for the `MapStore` @@ -463,11 +462,11 @@ interface and simple key-value map implementation. ### Database Submodules -In addition to providing the `MapStore` and `SimpleMap` the `smt` library also -provides wrappers around other key-value databases as submodules with more -fully featured interfaces that can be used as outside of being a node store for -the tries. These submodules can be found in the [`kvstore`](../kvstore/) -directory. +In addition to providing the `MapStore` and `SimpleMap` interfaces and +implementations, the `smt` library also provides wrappers around other key-value +databases as submodules with more fully-featured interfaces that can be used +outside of backing key-value engines for tries. These submodules can be found in +the [`kvstore`](../kvstore/) directory. #### Badger @@ -490,40 +489,3 @@ the `Commit()` function is called and changes are persisted. This library also implements a Sparse Merkle Sum Trie (SMST), the documentation for which can be found [here](./merkle-sum-trie.md). - -## Example - -```go -package main - -import ( - "crypto/sha256" - "fmt" - - "github.com/pokt-network/smt" - "github.com/pokt-network/smt/kvstore" -) - -func main() { - // Initialise a new in-memory key-value store to store the nodes of the trie - // (Note: the trie only stores hashed values, not raw value data) - nodeStore := kvstore.NewSimpleMap() - - // Update the key "foo" with the value "bar" - _ = trie.Update([]byte("foo"), []byte("bar")) - - // Commit the changes to the node store - _ = trie.Commit() - - // Generate a Merkle proof for "foo" - proof, _ := trie.Prove([]byte("foo")) - root := trie.Root() // We also need the current trie root for the proof - - // Verify the Merkle proof for "foo"="bar" - if smt.VerifyProof(proof, root, []byte("foo"), []byte("bar"), trie.Spec()) { - fmt.Println("Proof verification succeeded.") - } else { - fmt.Println("Proof verification failed.") - } -} -``` diff --git a/smst_example_test.go b/smst_example_test.go new file mode 100644 index 0000000..b390af4 --- /dev/null +++ b/smst_example_test.go @@ -0,0 +1,43 @@ +package smt_test + +import ( + "crypto/sha256" + "fmt" + + "github.com/pokt-network/smt" + "github.com/pokt-network/smt/kvstore" +) + +func ExampleSMST() { + // Initialise a new in-memory key-value store to store the nodes of the trie + // (Note: the trie only stores hashed values, not raw value data) + nodeStore := kvstore.NewSimpleMap() + + // Initialise the trie + trie := smt.NewSparseMerkleSumTrie(nodeStore, sha256.New()) + + // Update trie with keys, values and their sums + _ = trie.Update([]byte("foo"), []byte("oof"), 10) + _ = trie.Update([]byte("baz"), []byte("zab"), 7) + _ = trie.Update([]byte("bin"), []byte("nib"), 3) + + // Commit the changes to the nodeStore + _ = trie.Commit() + + sum := trie.Sum() + fmt.Println(sum == 20) // true + + // Generate a Merkle proof for "foo" + proof, _ := trie.Prove([]byte("foo")) + root := trie.Root() // We also need the current trie root for the proof + // Verify the Merkle proof for "foo"="oof" where "foo" has a sum of 10 + valid_true1, _ := smt.VerifySumProof(proof, root, []byte("foo"), []byte("oof"), 10, trie.Spec()) + // Verify the Merkle proof for "baz"="zab" where "baz" has a sum of 7 + valid_true2, _ := smt.VerifySumProof(proof, root, []byte("baz"), []byte("zab"), 7, trie.Spec()) + // Verify the Merkle proof for "bin"="nib" where "bin" has a sum of 3 + valid_true3, _ := smt.VerifySumProof(proof, root, []byte("bin"), []byte("nib"), 3, trie.Spec()) + // Fail to verify the Merkle proof for "foo"="oof" where "foo" has a sum of 11 + valid_false1, _ := smt.VerifySumProof(proof, root, []byte("foo"), []byte("oof"), 11, trie.Spec()) + fmt.Println(valid_true1, valid_true2, valid_true3, valid_false1) + // Output: true true true false +} diff --git a/smt_example_test.go b/smt_example_test.go new file mode 100644 index 0000000..b91ba5e --- /dev/null +++ b/smt_example_test.go @@ -0,0 +1,36 @@ +package smt_test + +import ( + "crypto/sha256" + "fmt" + + "github.com/pokt-network/smt" + "github.com/pokt-network/smt/kvstore" +) + +func ExampleSMT() { + // Initialise a new in-memory key-value store to store the nodes of the trie + // (Note: the trie only stores hashed values, not raw value data) + nodeStore := kvstore.NewSimpleMap() + + // Initialise the trie + trie := smt.NewSparseMerkleTrie(nodeStore, sha256.New()) + + // Update the key "foo" with the value "bar" + _ = trie.Update([]byte("foo"), []byte("bar")) + + // Commit the changes to the node store + _ = trie.Commit() + + // Generate a Merkle proof for "foo" + proof, _ := trie.Prove([]byte("foo")) + root := trie.Root() // We also need the current trie root for the proof + + // Verify the Merkle proof for "foo"="bar" + valid, _ := smt.VerifyProof(proof, root, []byte("foo"), []byte("bar"), trie.Spec()) + fmt.Println(valid) + // Attempt to verify the Merkle proof for "foo"="baz" + false, _ := smt.VerifyProof(proof, root, []byte("foo"), []byte("baz"), trie.Spec()) + fmt.Println(valid, false) + // Output: true false +} From 17ccf6809cba0ae056954814ebe35f401ad8ba7e Mon Sep 17 00:00:00 2001 From: h5law <53987565+h5law@users.noreply.github.com> Date: Wed, 3 Jan 2024 15:43:48 +0000 Subject: [PATCH 15/32] feat: move interface out of submodule declarations and add simplemap as a submodule --- kvstore/badger/kvstore.go | 28 +++++++++++------------ kvstore/interfaces.go | 15 ++++++++++++ kvstore/{ => simplemap}/errors.go | 2 +- kvstore/{ => simplemap}/godoc.go | 2 +- kvstore/{ => simplemap}/simplemap.go | 21 ++++------------- kvstore/{ => simplemap}/simplemap_test.go | 10 ++++---- 6 files changed, 42 insertions(+), 36 deletions(-) create mode 100644 kvstore/interfaces.go rename kvstore/{ => simplemap}/errors.go (93%) rename kvstore/{ => simplemap}/godoc.go (97%) rename kvstore/{ => simplemap}/simplemap.go (68%) rename kvstore/{ => simplemap}/simplemap_test.go (94%) diff --git a/kvstore/badger/kvstore.go b/kvstore/badger/kvstore.go index d818887..2d8edd4 100644 --- a/kvstore/badger/kvstore.go +++ b/kvstore/badger/kvstore.go @@ -4,7 +4,7 @@ import ( "errors" "io" - v4 "github.com/dgraph-io/badger/v4" + badgerv4 "github.com/dgraph-io/badger/v4" ) const ( @@ -14,19 +14,19 @@ const ( var _ KVStore = &badgerKVStore{} type badgerKVStore struct { - db *v4.DB + db *badgerv4.DB lastBackup uint64 // timestamp of the most recent backup } // NewKVStore creates a new KVStore using badger as the underlying database // if no path for a persistence database is provided it will create one in-memory func NewKVStore(path string) (KVStore, error) { - var db *v4.DB + var db *badgerv4.DB var err error if path == "" { - db, err = v4.Open(badgerOptions("").WithInMemory(true)) + db, err = badgerv4.Open(badgerOptions("").WithInMemory(true)) } else { - db, err = v4.Open(badgerOptions(path)) + db, err = badgerv4.Open(badgerOptions(path)) } if err != nil { return nil, errors.Join(ErrBadgerOpeningStore, err) @@ -36,7 +36,7 @@ func NewKVStore(path string) (KVStore, error) { // Set sets/updates the value for a given key func (store *badgerKVStore) Set(key, value []byte) error { - err := store.db.Update(func(tx *v4.Txn) error { + err := store.db.Update(func(tx *badgerv4.Txn) error { return tx.Set(key, value) }) if err != nil { @@ -48,7 +48,7 @@ func (store *badgerKVStore) Set(key, value []byte) error { // Get returns the value for a given key func (store *badgerKVStore) Get(key []byte) ([]byte, error) { var val []byte - if err := store.db.View(func(tx *v4.Txn) error { + if err := store.db.View(func(tx *badgerv4.Txn) error { item, err := tx.Get(key) if err != nil { return err @@ -66,7 +66,7 @@ func (store *badgerKVStore) Get(key []byte) ([]byte, error) { // Delete removes a key and its value from the store func (store *badgerKVStore) Delete(key []byte) error { - err := store.db.Update(func(tx *v4.Txn) error { + err := store.db.Update(func(tx *badgerv4.Txn) error { return tx.Delete(key) }) if err != nil { @@ -78,8 +78,8 @@ func (store *badgerKVStore) Delete(key []byte) error { // GetAll returns all keys and values with the given prefix in the specified order // if the prefix []byte{} is given then all key-value pairs are returned func (store *badgerKVStore) GetAll(prefix []byte, descending bool) (keys, values [][]byte, err error) { - if err := store.db.View(func(tx *v4.Txn) error { - opt := v4.DefaultIteratorOptions + if err := store.db.View(func(tx *badgerv4.Txn) error { + opt := badgerv4.DefaultIteratorOptions opt.Prefix = prefix opt.Reverse = descending if descending { @@ -161,8 +161,8 @@ func (store *badgerKVStore) Stop() error { // Len gives the number of keys in the store func (store *badgerKVStore) Len() int { count := 0 - if err := store.db.View(func(tx *v4.Txn) error { - opt := v4.DefaultIteratorOptions + if err := store.db.View(func(tx *badgerv4.Txn) error { + opt := badgerv4.DefaultIteratorOptions opt.Prefix = []byte{} opt.Reverse = false it := tx.NewIterator(opt) @@ -193,8 +193,8 @@ func prefixEndBytes(prefix []byte) []byte { } // badgerOptions returns the badger options for the store being created -func badgerOptions(path string) v4.Options { - opts := v4.DefaultOptions(path) +func badgerOptions(path string) badgerv4.Options { + opts := badgerv4.DefaultOptions(path) opts.Logger = nil // disable badger's logger since it's very noisy return opts } diff --git a/kvstore/interfaces.go b/kvstore/interfaces.go new file mode 100644 index 0000000..0d0ac53 --- /dev/null +++ b/kvstore/interfaces.go @@ -0,0 +1,15 @@ +package kvstore + +// MapStore defines an interface that represents a key-value store that backs +// the SM(S)T. It is the minimum viable subset of functionality a key-value +// store requires in order to back an SM(S)T. +type MapStore interface { + // --- Accessors --- + Get(key []byte) ([]byte, error) + Set(key, value []byte) error + Delete(key []byte) error + Len() int + + // --- Debug --- + ClearAll() error +} diff --git a/kvstore/errors.go b/kvstore/simplemap/errors.go similarity index 93% rename from kvstore/errors.go rename to kvstore/simplemap/errors.go index 5542bec..4fbead6 100644 --- a/kvstore/errors.go +++ b/kvstore/simplemap/errors.go @@ -1,4 +1,4 @@ -package kvstore +package simplemap import ( "errors" diff --git a/kvstore/godoc.go b/kvstore/simplemap/godoc.go similarity index 97% rename from kvstore/godoc.go rename to kvstore/simplemap/godoc.go index a40e6e3..738dd27 100644 --- a/kvstore/godoc.go +++ b/kvstore/simplemap/godoc.go @@ -9,4 +9,4 @@ // These are included as submodules as they are not required for the SM(S)T and // any key-value store that satisfies the `MapStore` interface can be used with // the SM(S)T. -package kvstore +package simplemap diff --git a/kvstore/simplemap.go b/kvstore/simplemap/simplemap.go similarity index 68% rename from kvstore/simplemap.go rename to kvstore/simplemap/simplemap.go index 41e0446..d7b066d 100644 --- a/kvstore/simplemap.go +++ b/kvstore/simplemap/simplemap.go @@ -1,19 +1,8 @@ -package kvstore +package simplemap -// MapStore defines an interface that represents a key-value store that backs -// the SM(S)T. It is a subset of the full functionality of a key-value store -// needed for the SM(S)T to function. By using a simplified interface any -// key-value store that implements these methods can be used with the SM(S)T. -type MapStore interface { - // Accessors - Get(key []byte) ([]byte, error) - Set(key, value []byte) error - Delete(key []byte) error - Len() int - - // Debug - ClearAll() error -} +import ( + "github.com/pokt-network/smt/kvstore" +) // simpleMap is a simple in-memory map. type simpleMap struct { @@ -21,7 +10,7 @@ type simpleMap struct { } // NewSimpleMap creates a new SimpleMap instance. -func NewSimpleMap() MapStore { +func NewSimpleMap() kvstore.MapStore { return &simpleMap{ m: make(map[string][]byte), } diff --git a/kvstore/simplemap_test.go b/kvstore/simplemap/simplemap_test.go similarity index 94% rename from kvstore/simplemap_test.go rename to kvstore/simplemap/simplemap_test.go index 313f41b..54437a8 100644 --- a/kvstore/simplemap_test.go +++ b/kvstore/simplemap/simplemap_test.go @@ -1,9 +1,11 @@ -package kvstore +package simplemap import ( "testing" "github.com/stretchr/testify/require" + + "github.com/pokt-network/smt/kvstore" ) func TestSimpleMap_Get(t *testing.T) { @@ -135,17 +137,17 @@ func TestSimpleMap_Len(t *testing.T) { tests := []struct { desc string - setup func(MapStore) + setup func(kvstore.MapStore) expectedLen int }{ { desc: "Length of empty map", - setup: func(store MapStore) {}, + setup: func(store kvstore.MapStore) {}, expectedLen: 0, }, { desc: "Length after adding items", - setup: func(sm MapStore) { + setup: func(sm kvstore.MapStore) { require.NoError(t, sm.Set([]byte("key1"), []byte("value1"))) require.NoError(t, sm.Set([]byte("key2"), []byte("value2"))) }, From e4670f17eaa6e270a91226362ad55877dc2f8841 Mon Sep 17 00:00:00 2001 From: h5law <53987565+h5law@users.noreply.github.com> Date: Wed, 3 Jan 2024 15:44:51 +0000 Subject: [PATCH 16/32] chore: better submodule support --- Makefile | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/Makefile b/Makefile index dc1e1d5..032f2da 100644 --- a/Makefile +++ b/Makefile @@ -40,12 +40,22 @@ go_docs: check_godoc ## Generate documentation for the project .PHONY: test_all test_all: ## runs the test suite go test -v -p 1 ./ -mod=readonly -race - go test -v -p 1 ./kvstore/... -race + go test -v -p 1 ./kvstore/*/... -race .PHONY: test_badger test_badger: ## runs the badger KVStore submodule's test suite go test -v -p 1 ./kvstore/badger/... -mod=readonly -race + +##################### +### go mod ### +##################### +.PHONY: mod_tidy +mod_tidy: ## runs go mod tidy for all (sub)modules + go mod tidy + cd kvstore/simplemap && go mod tidy + cd kvstore/badger && go mod tidy + ##################### ### Benchmark ### ##################### From 45183879f0f0b0094467365fcb4252e22c6b5c13 Mon Sep 17 00:00:00 2001 From: h5law <53987565+h5law@users.noreply.github.com> Date: Wed, 3 Jan 2024 15:46:17 +0000 Subject: [PATCH 17/32] chore: grammar --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 9368f9f..e608206 100644 --- a/README.md +++ b/README.md @@ -41,7 +41,7 @@ make test_all ``` To test the `badger` submodule that provides a more fully featured key-value -store run the following command: +store, run the following command: ```sh make test_badger From afeb02ef0c9ce1874416c6d3d9eeef891acc65e5 Mon Sep 17 00:00:00 2001 From: h5law <53987565+h5law@users.noreply.github.com> Date: Wed, 3 Jan 2024 15:46:59 +0000 Subject: [PATCH 18/32] chore: fix example tests --- smst_example_test.go | 25 +++++++++++++++---------- smt_example_test.go | 9 ++++----- 2 files changed, 19 insertions(+), 15 deletions(-) diff --git a/smst_example_test.go b/smst_example_test.go index b390af4..f1cdbca 100644 --- a/smst_example_test.go +++ b/smst_example_test.go @@ -5,13 +5,13 @@ import ( "fmt" "github.com/pokt-network/smt" - "github.com/pokt-network/smt/kvstore" + "github.com/pokt-network/smt/kvstore/simplemap" ) func ExampleSMST() { // Initialise a new in-memory key-value store to store the nodes of the trie // (Note: the trie only stores hashed values, not raw value data) - nodeStore := kvstore.NewSimpleMap() + nodeStore := simplemap.NewSimpleMap() // Initialise the trie trie := smt.NewSparseMerkleSumTrie(nodeStore, sha256.New()) @@ -24,20 +24,25 @@ func ExampleSMST() { // Commit the changes to the nodeStore _ = trie.Commit() - sum := trie.Sum() - fmt.Println(sum == 20) // true + // Calculate the total sum of the trie + _ = trie.Sum() // 20 // Generate a Merkle proof for "foo" - proof, _ := trie.Prove([]byte("foo")) - root := trie.Root() // We also need the current trie root for the proof + proof1, _ := trie.Prove([]byte("foo")) + proof2, _ := trie.Prove([]byte("baz")) + proof3, _ := trie.Prove([]byte("bin")) + + // We also need the current trie root for the proof + root := trie.Root() + // Verify the Merkle proof for "foo"="oof" where "foo" has a sum of 10 - valid_true1, _ := smt.VerifySumProof(proof, root, []byte("foo"), []byte("oof"), 10, trie.Spec()) + valid_true1, _ := smt.VerifySumProof(proof1, root, []byte("foo"), []byte("oof"), 10, trie.Spec()) // Verify the Merkle proof for "baz"="zab" where "baz" has a sum of 7 - valid_true2, _ := smt.VerifySumProof(proof, root, []byte("baz"), []byte("zab"), 7, trie.Spec()) + valid_true2, _ := smt.VerifySumProof(proof2, root, []byte("baz"), []byte("zab"), 7, trie.Spec()) // Verify the Merkle proof for "bin"="nib" where "bin" has a sum of 3 - valid_true3, _ := smt.VerifySumProof(proof, root, []byte("bin"), []byte("nib"), 3, trie.Spec()) + valid_true3, _ := smt.VerifySumProof(proof3, root, []byte("bin"), []byte("nib"), 3, trie.Spec()) // Fail to verify the Merkle proof for "foo"="oof" where "foo" has a sum of 11 - valid_false1, _ := smt.VerifySumProof(proof, root, []byte("foo"), []byte("oof"), 11, trie.Spec()) + valid_false1, _ := smt.VerifySumProof(proof1, root, []byte("foo"), []byte("oof"), 11, trie.Spec()) fmt.Println(valid_true1, valid_true2, valid_true3, valid_false1) // Output: true true true false } diff --git a/smt_example_test.go b/smt_example_test.go index b91ba5e..6d74980 100644 --- a/smt_example_test.go +++ b/smt_example_test.go @@ -5,13 +5,13 @@ import ( "fmt" "github.com/pokt-network/smt" - "github.com/pokt-network/smt/kvstore" + "github.com/pokt-network/smt/kvstore/simplemap" ) func ExampleSMT() { // Initialise a new in-memory key-value store to store the nodes of the trie // (Note: the trie only stores hashed values, not raw value data) - nodeStore := kvstore.NewSimpleMap() + nodeStore := simplemap.NewSimpleMap() // Initialise the trie trie := smt.NewSparseMerkleTrie(nodeStore, sha256.New()) @@ -28,9 +28,8 @@ func ExampleSMT() { // Verify the Merkle proof for "foo"="bar" valid, _ := smt.VerifyProof(proof, root, []byte("foo"), []byte("bar"), trie.Spec()) - fmt.Println(valid) // Attempt to verify the Merkle proof for "foo"="baz" - false, _ := smt.VerifyProof(proof, root, []byte("foo"), []byte("baz"), trie.Spec()) - fmt.Println(valid, false) + invalid, _ := smt.VerifyProof(proof, root, []byte("foo"), []byte("baz"), trie.Spec()) + fmt.Println(valid, invalid) // Output: true false } From a64495a6b5dbcb5a1b71fac09c52ea7887fd6d5b Mon Sep 17 00:00:00 2001 From: h5law <53987565+h5law@users.noreply.github.com> Date: Wed, 3 Jan 2024 15:47:57 +0000 Subject: [PATCH 19/32] s/name/desc/g --- benchmarks/bench_leaf_test.go | 6 +-- benchmarks/bench_smst_test.go | 70 +++++++++++++++++----------------- benchmarks/bench_smt_test.go | 68 ++++++++++++++++----------------- benchmarks/bench_utils_test.go | 6 +-- benchmarks/proof_sizes_test.go | 34 ++++++++--------- 5 files changed, 92 insertions(+), 92 deletions(-) diff --git a/benchmarks/bench_leaf_test.go b/benchmarks/bench_leaf_test.go index 2c0fb31..3ae12d5 100644 --- a/benchmarks/bench_leaf_test.go +++ b/benchmarks/bench_leaf_test.go @@ -9,13 +9,13 @@ import ( "github.com/stretchr/testify/require" "github.com/pokt-network/smt" - "github.com/pokt-network/smt/kvstore" + "github.com/pokt-network/smt/kvstore/simplemap" ) func BenchmarkSMTLeafSizes_Fill(b *testing.B) { trieSizes := []int{100000, 500000, 1000000, 5000000, 10000000} // number of leaves leafSizes := []int{256, 512, 1024, 2048, 4096, 8192, 16384} // number of bytes per leaf - nodes := kvstore.NewSimpleMap() + nodes := simplemap.NewSimpleMap() for _, trieSize := range trieSizes { for _, leafSize := range leafSizes { leaf := make([]byte, leafSize) @@ -44,7 +44,7 @@ func BenchmarkSMTLeafSizes_Fill(b *testing.B) { func BenchmarkSMSTLeafSizes_Fill(b *testing.B) { trieSizes := []int{100000, 500000, 1000000, 5000000, 10000000} // number of leaves leafSizes := []int{256, 512, 1024, 2048, 4096, 8192, 16384} // number of bytes per leaf - nodes := kvstore.NewSimpleMap() + nodes := simplemap.NewSimpleMap() for _, trieSize := range trieSizes { for _, leafSize := range leafSizes { leaf := make([]byte, leafSize) diff --git a/benchmarks/bench_smst_test.go b/benchmarks/bench_smst_test.go index 5c9bfa1..6b68fd3 100644 --- a/benchmarks/bench_smst_test.go +++ b/benchmarks/bench_smst_test.go @@ -11,57 +11,57 @@ import ( func BenchmarkSparseMerkleSumTrie_Fill(b *testing.B) { testCases := []struct { - name string + desc string trieSize int commit bool }{ { - name: "Fill (100000)", + desc: "Fill (100000)", trieSize: 100000, commit: false, }, { - name: "Fill & Commit (100000)", + desc: "Fill & Commit (100000)", trieSize: 100000, commit: true, }, { - name: "Fill (500000)", + desc: "Fill (500000)", trieSize: 500000, commit: false, }, { - name: "Fill & Commit (500000)", + desc: "Fill & Commit (500000)", trieSize: 500000, commit: true, }, { - name: "Fill (1000000)", + desc: "Fill (1000000)", trieSize: 1000000, commit: false, }, { - name: "Fill & Commit (1000000)", + desc: "Fill & Commit (1000000)", trieSize: 1000000, commit: true, }, { - name: "Fill (5000000)", + desc: "Fill (5000000)", trieSize: 5000000, commit: false, }, { - name: "Fill & Commit (5000000)", + desc: "Fill & Commit (5000000)", trieSize: 5000000, commit: true, }, { - name: "Fill (10000000)", + desc: "Fill (10000000)", trieSize: 10000000, commit: false, }, { - name: "Fill & Commit (10000000)", + desc: "Fill & Commit (10000000)", trieSize: 10000000, commit: true, }, @@ -69,7 +69,7 @@ func BenchmarkSparseMerkleSumTrie_Fill(b *testing.B) { for _, tc := range testCases { b.ResetTimer() - b.Run(tc.name, func(b *testing.B) { + b.Run(tc.desc, func(b *testing.B) { trie := setupSMST(b, tc.trieSize) b.ResetTimer() b.StartTimer() @@ -90,67 +90,67 @@ func BenchmarkSparseMerkleSumTrie_Fill(b *testing.B) { func BenchmarkSparseMerkleSumTrie_Update(b *testing.B) { testCases := []struct { - name string + desc string trieSize int commit bool fn func(*smt.SMST, uint64) error }{ { - name: "Update (Prefilled: 100000)", + desc: "Update (Prefilled: 100000)", trieSize: 100000, commit: false, fn: updSMST, }, { - name: "Update & Commit (Prefilled: 100000)", + desc: "Update & Commit (Prefilled: 100000)", trieSize: 100000, commit: true, fn: updSMST, }, { - name: "Update (Prefilled: 500000)", + desc: "Update (Prefilled: 500000)", trieSize: 500000, commit: false, fn: updSMST, }, { - name: "Update & Commit (Prefilled: 500000)", + desc: "Update & Commit (Prefilled: 500000)", trieSize: 500000, commit: true, fn: updSMST, }, { - name: "Update (Prefilled: 1000000)", + desc: "Update (Prefilled: 1000000)", trieSize: 1000000, commit: false, fn: updSMST, }, { - name: "Update & Commit (Prefilled: 1000000)", + desc: "Update & Commit (Prefilled: 1000000)", trieSize: 1000000, commit: true, fn: updSMST, }, { - name: "Update (Prefilled: 5000000)", + desc: "Update (Prefilled: 5000000)", trieSize: 5000000, commit: false, fn: updSMST, }, { - name: "Update & Commit (Prefilled: 5000000)", + desc: "Update & Commit (Prefilled: 5000000)", trieSize: 5000000, commit: true, fn: updSMST, }, { - name: "Update (Prefilled: 10000000)", + desc: "Update (Prefilled: 10000000)", trieSize: 10000000, commit: false, fn: updSMST, }, { - name: "Update & Commit (Prefilled: 10000000)", + desc: "Update & Commit (Prefilled: 10000000)", trieSize: 10000000, commit: true, fn: updSMST, @@ -159,7 +159,7 @@ func BenchmarkSparseMerkleSumTrie_Update(b *testing.B) { for _, tc := range testCases { b.ResetTimer() - b.Run(tc.name, func(b *testing.B) { + b.Run(tc.desc, func(b *testing.B) { trie := setupSMST(b, tc.trieSize) benchmarkSMST(b, trie, tc.commit, tc.fn) }) @@ -168,37 +168,37 @@ func BenchmarkSparseMerkleSumTrie_Update(b *testing.B) { func BenchmarkSparseMerkleSumTrie_Get(b *testing.B) { testCases := []struct { - name string + desc string trieSize int commit bool fn func(*smt.SMST, uint64) error }{ { - name: "Get (Prefilled: 100000)", + desc: "Get (Prefilled: 100000)", trieSize: 100000, commit: false, fn: getSMST, }, { - name: "Get (Prefilled: 500000)", + desc: "Get (Prefilled: 500000)", trieSize: 500000, commit: false, fn: getSMST, }, { - name: "Get (Prefilled: 1000000)", + desc: "Get (Prefilled: 1000000)", trieSize: 1000000, commit: false, fn: getSMST, }, { - name: "Get (Prefilled: 5000000)", + desc: "Get (Prefilled: 5000000)", trieSize: 5000000, commit: false, fn: getSMST, }, { - name: "Get (Prefilled: 10000000)", + desc: "Get (Prefilled: 10000000)", trieSize: 10000000, commit: false, fn: getSMST, @@ -207,7 +207,7 @@ func BenchmarkSparseMerkleSumTrie_Get(b *testing.B) { for _, tc := range testCases { b.ResetTimer() - b.Run(tc.name, func(b *testing.B) { + b.Run(tc.desc, func(b *testing.B) { trie := setupSMST(b, tc.trieSize) benchmarkSMST(b, trie, tc.commit, tc.fn) }) @@ -216,25 +216,25 @@ func BenchmarkSparseMerkleSumTrie_Get(b *testing.B) { func BenchmarkSparseMerkleSumTrie_Prove(b *testing.B) { testCases := []struct { - name string + desc string trieSize int commit bool fn func(*smt.SMST, uint64) error }{ { - name: "Prove (Prefilled: 100000)", + desc: "Prove (Prefilled: 100000)", trieSize: 100000, commit: false, fn: proSMST, }, { - name: "Prove (Prefilled: 500000)", + desc: "Prove (Prefilled: 500000)", trieSize: 500000, commit: false, fn: proSMST, }, { - name: "Prove (Prefilled: 1000000)", + desc: "Prove (Prefilled: 1000000)", trieSize: 1000000, commit: false, fn: proSMST, diff --git a/benchmarks/bench_smt_test.go b/benchmarks/bench_smt_test.go index 6a3e67f..57f3e16 100644 --- a/benchmarks/bench_smt_test.go +++ b/benchmarks/bench_smt_test.go @@ -11,57 +11,57 @@ import ( func BenchmarkSparseMerkleTrie_Fill(b *testing.B) { testCases := []struct { - name string + desc string trieSize int commit bool }{ { - name: "Fill (100000)", + desc: "Fill (100000)", trieSize: 100000, commit: false, }, { - name: "Fill & Commit (100000)", + desc: "Fill & Commit (100000)", trieSize: 100000, commit: true, }, { - name: "Fill (500000)", + desc: "Fill (500000)", trieSize: 500000, commit: false, }, { - name: "Fill & Commit (500000)", + desc: "Fill & Commit (500000)", trieSize: 500000, commit: true, }, { - name: "Fill (1000000)", + desc: "Fill (1000000)", trieSize: 1000000, commit: false, }, { - name: "Fill & Commit (1000000)", + desc: "Fill & Commit (1000000)", trieSize: 1000000, commit: true, }, { - name: "Fill (5000000)", + desc: "Fill (5000000)", trieSize: 5000000, commit: false, }, { - name: "Fill & Commit (5000000)", + desc: "Fill & Commit (5000000)", trieSize: 5000000, commit: true, }, { - name: "Fill (10000000)", + desc: "Fill (10000000)", trieSize: 10000000, commit: false, }, { - name: "Fill & Commit (10000000)", + desc: "Fill & Commit (10000000)", trieSize: 10000000, commit: true, }, @@ -69,7 +69,7 @@ func BenchmarkSparseMerkleTrie_Fill(b *testing.B) { for _, tc := range testCases { b.ResetTimer() - b.Run(tc.name, func(b *testing.B) { + b.Run(tc.desc, func(b *testing.B) { trie := setupSMT(b, tc.trieSize) b.ResetTimer() b.StartTimer() @@ -90,67 +90,67 @@ func BenchmarkSparseMerkleTrie_Fill(b *testing.B) { func BenchmarkSparseMerkleTrie_Update(b *testing.B) { testCases := []struct { - name string + desc string trieSize int commit bool fn func(*smt.SMT, []byte) error }{ { - name: "Update (Prefilled: 100000)", + desc: "Update (Prefilled: 100000)", trieSize: 100000, commit: false, fn: updSMT, }, { - name: "Update & Commit (Prefilled: 100000)", + desc: "Update & Commit (Prefilled: 100000)", trieSize: 100000, commit: true, fn: updSMT, }, { - name: "Update (Prefilled: 500000)", + desc: "Update (Prefilled: 500000)", trieSize: 500000, commit: false, fn: updSMT, }, { - name: "Update & Commit (Prefilled: 500000)", + desc: "Update & Commit (Prefilled: 500000)", trieSize: 500000, commit: true, fn: updSMT, }, { - name: "Update (Prefilled: 1000000)", + desc: "Update (Prefilled: 1000000)", trieSize: 1000000, commit: false, fn: updSMT, }, { - name: "Update & Commit (Prefilled: 1000000)", + desc: "Update & Commit (Prefilled: 1000000)", trieSize: 1000000, commit: true, fn: updSMT, }, { - name: "Update (Prefilled: 5000000)", + desc: "Update (Prefilled: 5000000)", trieSize: 5000000, commit: false, fn: updSMT, }, { - name: "Update & Commit (Prefilled: 5000000)", + desc: "Update & Commit (Prefilled: 5000000)", trieSize: 5000000, commit: true, fn: updSMT, }, { - name: "Update (Prefilled: 10000000)", + desc: "Update (Prefilled: 10000000)", trieSize: 10000000, commit: false, fn: updSMT, }, { - name: "Update & Commit (Prefilled: 10000000)", + desc: "Update & Commit (Prefilled: 10000000)", trieSize: 10000000, commit: true, fn: updSMT, @@ -159,7 +159,7 @@ func BenchmarkSparseMerkleTrie_Update(b *testing.B) { for _, tc := range testCases { b.ResetTimer() - b.Run(tc.name, func(b *testing.B) { + b.Run(tc.desc, func(b *testing.B) { trie := setupSMT(b, tc.trieSize) benchmarkSMT(b, trie, tc.commit, tc.fn) }) @@ -168,37 +168,37 @@ func BenchmarkSparseMerkleTrie_Update(b *testing.B) { func BenchmarkSparseMerkleTrie_Get(b *testing.B) { testCases := []struct { - name string + desc string trieSize int commit bool fn func(*smt.SMT, []byte) error }{ { - name: "Get (Prefilled: 100000)", + desc: "Get (Prefilled: 100000)", trieSize: 100000, commit: false, fn: getSMT, }, { - name: "Get (Prefilled: 500000)", + desc: "Get (Prefilled: 500000)", trieSize: 500000, commit: false, fn: getSMT, }, { - name: "Get (Prefilled: 1000000)", + desc: "Get (Prefilled: 1000000)", trieSize: 1000000, commit: false, fn: getSMT, }, { - name: "Get (Prefilled: 5000000)", + desc: "Get (Prefilled: 5000000)", trieSize: 5000000, commit: false, fn: getSMT, }, { - name: "Get (Prefilled: 10000000)", + desc: "Get (Prefilled: 10000000)", trieSize: 10000000, commit: false, fn: getSMT, @@ -207,7 +207,7 @@ func BenchmarkSparseMerkleTrie_Get(b *testing.B) { for _, tc := range testCases { b.ResetTimer() - b.Run(tc.name, func(b *testing.B) { + b.Run(tc.desc, func(b *testing.B) { trie := setupSMT(b, tc.trieSize) benchmarkSMT(b, trie, tc.commit, tc.fn) }) @@ -216,19 +216,19 @@ func BenchmarkSparseMerkleTrie_Get(b *testing.B) { func BenchmarkSparseMerkleTrie_Prove(b *testing.B) { testCases := []struct { - name string + desc string trieSize int commit bool fn func(*smt.SMT, []byte) error }{ { - name: "Prove (Prefilled: 100000)", + desc: "Prove (Prefilled: 100000)", trieSize: 100000, commit: false, fn: proSMT, }, { - name: "Prove (Prefilled: 500000)", + desc: "Prove (Prefilled: 500000)", trieSize: 500000, commit: false, fn: proSMT, diff --git a/benchmarks/bench_utils_test.go b/benchmarks/bench_utils_test.go index 947e366..791a0e7 100644 --- a/benchmarks/bench_utils_test.go +++ b/benchmarks/bench_utils_test.go @@ -9,7 +9,7 @@ import ( "github.com/stretchr/testify/require" "github.com/pokt-network/smt" - "github.com/pokt-network/smt/kvstore" + "github.com/pokt-network/smt/kvstore/simplemap" ) var ( @@ -54,7 +54,7 @@ var ( func setupSMT(b *testing.B, numLeaves int) *smt.SMT { b.Helper() - nodes := kvstore.NewSimpleMap() + nodes := simplemap.NewSimpleMap() trie := smt.NewSparseMerkleTrie(nodes, sha256.New()) for i := 0; i < numLeaves; i++ { s := strconv.Itoa(i) @@ -83,7 +83,7 @@ func benchmarkSMT(b *testing.B, trie *smt.SMT, commit bool, fn func(*smt.SMT, [] func setupSMST(b *testing.B, numLeaves int) *smt.SMST { b.Helper() - nodes := kvstore.NewSimpleMap() + nodes := simplemap.NewSimpleMap() trie := smt.NewSparseMerkleSumTrie(nodes, sha256.New()) for i := 0; i < numLeaves; i++ { s := strconv.Itoa(i) diff --git a/benchmarks/proof_sizes_test.go b/benchmarks/proof_sizes_test.go index 609d991..b3573ea 100644 --- a/benchmarks/proof_sizes_test.go +++ b/benchmarks/proof_sizes_test.go @@ -8,39 +8,39 @@ import ( "github.com/stretchr/testify/require" "github.com/pokt-network/smt" - "github.com/pokt-network/smt/kvstore" + "github.com/pokt-network/smt/kvstore/simplemap" ) func TestSMT_ProofSizes(t *testing.T) { - nodes := kvstore.NewSimpleMap() + nodes := simplemap.NewSimpleMap() testCases := []struct { - name string + desc string trieSize int }{ { - name: "Proof Size (Prefilled: 100000)", + desc: "Proof Size (Prefilled: 100000)", trieSize: 100000, }, { - name: "Proof Size (Prefilled: 500000)", + desc: "Proof Size (Prefilled: 500000)", trieSize: 500000, }, { - name: "Proof Size (Prefilled: 1000000)", + desc: "Proof Size (Prefilled: 1000000)", trieSize: 1000000, }, { - name: "Proof Size (Prefilled: 5000000)", + desc: "Proof Size (Prefilled: 5000000)", trieSize: 5000000, }, { - name: "Proof Size (Prefilled: 10000000)", + desc: "Proof Size (Prefilled: 10000000)", trieSize: 10000000, }, } for _, tc := range testCases { trie := smt.NewSparseMerkleTrie(nodes, sha256.New()) - t.Run(tc.name, func(t *testing.T) { + t.Run(tc.desc, func(t *testing.T) { for i := 0; i < tc.trieSize; i++ { b := make([]byte, 8) binary.BigEndian.PutUint64(b, uint64(i)) @@ -99,35 +99,35 @@ func TestSMT_ProofSizes(t *testing.T) { } func TestSMST_ProofSizes(t *testing.T) { - nodes := kvstore.NewSimpleMap() + nodes := simplemap.NewSimpleMap() testCases := []struct { - name string + desc string trieSize int }{ { - name: "Proof Size (Prefilled: 100000)", + desc: "Proof Size (Prefilled: 100000)", trieSize: 100000, }, { - name: "Proof Size (Prefilled: 500000)", + desc: "Proof Size (Prefilled: 500000)", trieSize: 500000, }, { - name: "Proof Size (Prefilled: 1000000)", + desc: "Proof Size (Prefilled: 1000000)", trieSize: 1000000, }, { - name: "Proof Size (Prefilled: 5000000)", + desc: "Proof Size (Prefilled: 5000000)", trieSize: 5000000, }, { - name: "Proof Size (Prefilled: 10000000)", + desc: "Proof Size (Prefilled: 10000000)", trieSize: 10000000, }, } for _, tc := range testCases { trie := smt.NewSparseMerkleSumTrie(nodes, sha256.New()) - t.Run(tc.name, func(t *testing.T) { + t.Run(tc.desc, func(t *testing.T) { for i := 0; i < tc.trieSize; i++ { b := make([]byte, 8) binary.BigEndian.PutUint64(b, uint64(i)) From f88a178dd704b666e4535136e6753de71375de42 Mon Sep 17 00:00:00 2001 From: h5law <53987565+h5law@users.noreply.github.com> Date: Wed, 3 Jan 2024 15:48:50 +0000 Subject: [PATCH 20/32] chore: clarify godoc --- godoc.go | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-) diff --git a/godoc.go b/godoc.go index c795203..7354c89 100644 --- a/godoc.go +++ b/godoc.go @@ -1,8 +1,11 @@ -// Package smt provides an implementation of a Sparse Merkle trie for a -// key-value map. The trie implements the same optimisations specified in the -// Libra whitepaper to reduce the number of hash operations required per trie -// operation to O(k) where k is the number of non-empty elements in the trie. -// And is implement in a similar way to the JMT whitepaper, with additional -// features and proof mechanics, such as a Sparse Merkle Sum Trie and new -// ClosestProof mechanics. +// Package smt provides an implementation of a Sparse Merkle Trie for a +// key-value map. +// +// The trie implements the same optimizations specified in the JMT +// whitepaper to account for empty and single-node subtrees. Unlike the +// JMT, it only supports binary trees and does not optimise for RockDB +// on-disk storage. +// +// This package implements novel features that include native in-node +// weight sums, as well as support for ClosestProof mechanics. package smt From 81ff1b8f1fff371e69a4487590b3587a6c667087 Mon Sep 17 00:00:00 2001 From: h5law <53987565+h5law@users.noreply.github.com> Date: Wed, 3 Jan 2024 15:50:19 +0000 Subject: [PATCH 21/32] chore: refactor to use simplemap submodules --- bulk_test.go | 8 ++++---- fuzz_test.go | 10 +++++----- go.work.sum | 28 ++++++++++++++++++++++++++++ proofs_test.go | 8 ++++---- smst_proofs_test.go | 19 ++++++++++--------- smst_test.go | 27 ++++++++++++++------------- smst_utils_test.go | 3 ++- smt.go | 9 +++++---- smt_proofs_test.go | 19 ++++++++++--------- smt_test.go | 21 +++++++++++---------- smt_utils_test.go | 3 ++- 11 files changed, 95 insertions(+), 60 deletions(-) diff --git a/bulk_test.go b/bulk_test.go index 7b58723..eb01e86 100644 --- a/bulk_test.go +++ b/bulk_test.go @@ -9,7 +9,7 @@ import ( "github.com/stretchr/testify/require" - "github.com/pokt-network/smt/kvstore" + "github.com/pokt-network/smt/kvstore/simplemap" ) type ( @@ -34,8 +34,8 @@ func TestBulkOperations(t *testing.T) { // Test all trie operations in bulk, with specified ratio probabilities of insert, update and delete. func bulkOperations(t *testing.T, operations int, insert int, update int, delete int) { - smn := kvstore.NewSimpleMap() - smv := kvstore.NewSimpleMap() + smn := simplemap.NewSimpleMap() + smv := simplemap.NewSimpleMap() smt := NewSMTWithStorage(smn, smv, sha256.New()) max := insert + update + delete @@ -82,7 +82,7 @@ func bulkOperations(t *testing.T, operations int, insert int, update int, delete ki := r.Intn(len(kv)) err := smt.Delete(kv[ki].key) - if err != nil && err != kvstore.ErrKVStoreKeyNotFound { + if err != nil && err != simplemap.ErrKVStoreKeyNotFound { t.Fatalf("error: %v", err) } kv[ki].val = defaultValue diff --git a/fuzz_test.go b/fuzz_test.go index aed858f..37f16a4 100644 --- a/fuzz_test.go +++ b/fuzz_test.go @@ -9,7 +9,7 @@ import ( "github.com/stretchr/testify/require" - "github.com/pokt-network/smt/kvstore" + "github.com/pokt-network/smt/kvstore/simplemap" ) // FuzzSMT uses fuzzing to attempt to break the SMT implementation @@ -27,7 +27,7 @@ func FuzzSMT_DetectUnexpectedFailures(f *testing.F) { f.Add(s) } f.Fuzz(func(t *testing.T, input []byte) { - smn := kvstore.NewSimpleMap() + smn := simplemap.NewSimpleMap() trie := NewSparseMerkleTrie(smn, sha256.New()) r := bytes.NewReader(input) @@ -68,7 +68,7 @@ func FuzzSMT_DetectUnexpectedFailures(f *testing.F) { _, err := trie.Get(key()) if err != nil { require.ErrorIsf( - t, err, kvstore.ErrKVStoreKeyNotFound, + t, err, simplemap.ErrKVStoreKeyNotFound, "unknown error occurred while getting", ) } @@ -85,7 +85,7 @@ func FuzzSMT_DetectUnexpectedFailures(f *testing.F) { err := trie.Delete(key()) if err != nil { require.ErrorIsf( - t, err, kvstore.ErrKVStoreKeyNotFound, + t, err, simplemap.ErrKVStoreKeyNotFound, "unknown error occurred while deleting", ) continue @@ -97,7 +97,7 @@ func FuzzSMT_DetectUnexpectedFailures(f *testing.F) { _, err := trie.Prove(key()) if err != nil { require.ErrorIsf( - t, err, kvstore.ErrKVStoreKeyNotFound, + t, err, simplemap.ErrKVStoreKeyNotFound, "unknown error occurred while proving", ) } diff --git a/go.work.sum b/go.work.sum index d82afb6..b7d7c0d 100644 --- a/go.work.sum +++ b/go.work.sum @@ -1,11 +1,39 @@ +cloud.google.com/go v0.26.0 h1:e0WKqKTd5BnrG8aKH3J3h+QvEIQtSUcf2n5UZ5ZgLtQ= +github.com/BurntSushi/toml v0.3.1 h1:WXkYYl6Yr3qBf1K79EBnL4mak0OimBfB0XUf9Vl28OQ= +github.com/client9/misspell v0.3.4 h1:ta993UF76GwbvJcIo3Y68y/M3WxlpEHPWIGDkJYwzJI= +github.com/creack/pty v1.1.9 h1:uDmaGzcdjhF4i/plgjmEsriH11Y0o7RKapEf/LDaM3w= +github.com/golang/mock v1.1.1 h1:G5FRp8JnTd7RQH5kemVNlMeyXQAztQ3mOWV95KxsXH8= +github.com/inconshreveable/mousetrap v1.0.0 h1:Z8tu5sraLXCXIcARxBp/8cbvlwVa7Z1NHg9XEKhtSvM= github.com/inconshreveable/mousetrap v1.0.0/go.mod h1:PxqpIevigyE2G7u3NXJIT2ANytuPF1OarO4DADm73n8= +github.com/kisielk/errcheck v1.5.0 h1:e8esj/e4R+SAOwFwN+n3zr0nYeCyeweozKfO23MvHzY= +github.com/kisielk/gotool v1.0.0 h1:AV2c/EiW3KqPNT9ZKl07ehoAGi4C5/01Cfbblndcapg= +github.com/kr/pty v1.1.1 h1:VkoXIwSboBpnk99O/KFauAEILuNHv5DVFKZMBN/gUgw= +github.com/spf13/cobra v0.0.5 h1:f0B+LkLX6DtmRH1isoNA9VTtNUK9K8xYd28JNNfOv/s= github.com/spf13/cobra v0.0.5/go.mod h1:3K3wKZymM7VvHMDS9+Akkh4K60UwM26emMESw8tLCHU= +github.com/spf13/pflag v1.0.3 h1:zPAT6CGy6wXeQ7NtTnaTerfKOsV6V6F8agHXFiazDkg= github.com/spf13/pflag v1.0.3/go.mod h1:DYY7MBk1bdzusC3SYhjObp+wFpr4gzcvqqNjLnInEg4= github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw= +github.com/stretchr/objx v0.5.0 h1:1zr/of2m5FGMsad5YfcqgdqdWrIhu+EBEJRhR1U7z/c= github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo= github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU= +github.com/yuin/goldmark v1.2.1 h1:ruQGxdhGHe7FWOJPT0mKs5+pD2Xs1Bm/kdGlHO04FmM= +golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9 h1:psW17arqaxU48Z5kZ0CQnkZWQJsqcURM6tKiBApRjXI= +golang.org/x/exp v0.0.0-20190121172915-509febef88a4 h1:c2HOrn5iMezYjSlGPncknSEr/8x5LELb/ilJbXi9DEA= +golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3 h1:XQyxROzUlZH+WIQwySDgnISgOivlhjIEwaQaJEJrrN0= +golang.org/x/mod v0.3.0 h1:RM4zey1++hCTbCVQfnWeKs9/IEsaBLA8vTkd0WVtmH4= +golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be h1:vEDujvNQGv4jgYKudGeI/+DAX4Jffq6hpD55MmoEvKs= +golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9 h1:SQFwaSi55rU7vdNs9Yr0Z324VNlrF+0wMqRXT4St8ck= +golang.org/x/term v0.5.0 h1:n2a8QNdAb0sZNpU9R1ALUXBbY+w51fCQDN+7EdxNBsY= golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k= +golang.org/x/text v0.7.0 h1:4BRB4x83lYWy72KwLD/qYDuTu7q9PjSagHvijDw7cLo= golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8= +golang.org/x/tools v0.0.0-20210106214847-113979e3529a h1:CB3a9Nez8M13wwlr/E2YtwoU+qYHKfC+JrDa45RXXoQ= +golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 h1:go1bK/D/BFZV2I8cIQd1NKEZ+0owSTG1fDTci4IqFcE= +google.golang.org/appengine v1.4.0 h1:/wp5JvzpHIxhs/dumFmF7BXTf3Z+dd4uXta4kVyO508= +google.golang.org/genproto v0.0.0-20230110181048-76db0878b65f h1:BWUVssLB0HVOSY78gIdvk1dTVYtT1y8SBWtPYuTJ/6w= google.golang.org/genproto v0.0.0-20230110181048-76db0878b65f/go.mod h1:RGgjbofJ8xD9Sq1VVhDM1Vok1vRONV+rg+CjzG4SZKM= +google.golang.org/grpc v1.53.0 h1:LAv2ds7cmFV/XTS3XG1NneeENYrXGmorPxsBbptIjNc= google.golang.org/grpc v1.53.0/go.mod h1:OnIrk0ipVdj4N5d9IUoFUx72/VlD7+jUsHwZgwSMQpw= +gopkg.in/yaml.v2 v2.2.8 h1:obN1ZagJSUGI0Ek/LBmuj4SNLPfIny3KsKFopxRdj10= gopkg.in/yaml.v2 v2.2.8/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= +honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099 h1:XJP7lxbSxWLOMNdBE4B/STaqVy6L73o0knwj2vIlxnw= diff --git a/proofs_test.go b/proofs_test.go index 74cdf7e..6248e5c 100644 --- a/proofs_test.go +++ b/proofs_test.go @@ -7,7 +7,7 @@ import ( "github.com/stretchr/testify/require" - "github.com/pokt-network/smt/kvstore" + "github.com/pokt-network/smt/kvstore/simplemap" ) func TestSparseMerkleProof_Marshal(t *testing.T) { @@ -152,7 +152,7 @@ func TestSparseCompactMerkleProof_Unmarshal(t *testing.T) { func setupTrie(t *testing.T) *SMT { t.Helper() - db := kvstore.NewSimpleMap() + db := simplemap.NewSimpleMap() trie := NewSparseMerkleTrie(db, sha256.New()) require.NoError(t, trie.Update([]byte("key"), []byte("value"))) @@ -166,7 +166,7 @@ func randomiseProof(proof *SparseMerkleProof) *SparseMerkleProof { sideNodes := make([][]byte, len(proof.SideNodes)) for i := range sideNodes { sideNodes[i] = make([]byte, len(proof.SideNodes[i])) - rand.Read(sideNodes[i]) //nolint: errcheck + rand.Read(sideNodes[i]) // nolint: errcheck } return &SparseMerkleProof{ SideNodes: sideNodes, @@ -178,7 +178,7 @@ func randomiseSumProof(proof *SparseMerkleProof) *SparseMerkleProof { sideNodes := make([][]byte, len(proof.SideNodes)) for i := range sideNodes { sideNodes[i] = make([]byte, len(proof.SideNodes[i])-sumSize) - rand.Read(sideNodes[i]) //nolint: errcheck + rand.Read(sideNodes[i]) // nolint: errcheck sideNodes[i] = append(sideNodes[i], proof.SideNodes[i][len(proof.SideNodes[i])-sumSize:]...) } return &SparseMerkleProof{ diff --git a/smst_proofs_test.go b/smst_proofs_test.go index ae038f5..21fd454 100644 --- a/smst_proofs_test.go +++ b/smst_proofs_test.go @@ -10,6 +10,7 @@ import ( "github.com/stretchr/testify/require" "github.com/pokt-network/smt/kvstore" + "github.com/pokt-network/smt/kvstore/simplemap" ) // Test base case Merkle proof operations. @@ -21,9 +22,9 @@ func TestSMST_Proof_Operations(t *testing.T) { var root []byte var err error - smn = kvstore.NewSimpleMap() + smn = simplemap.NewSimpleMap() require.NoError(t, err) - smv = kvstore.NewSimpleMap() + smv = simplemap.NewSimpleMap() require.NoError(t, err) smst = NewSMSTWithStorage(smn, smv, sha256.New()) base := smst.Spec() @@ -135,8 +136,8 @@ func TestSMST_Proof_Operations(t *testing.T) { // Test sanity check cases for non-compact proofs. func TestSMST_Proof_ValidateBasic(t *testing.T) { - smn := kvstore.NewSimpleMap() - smv := kvstore.NewSimpleMap() + smn := simplemap.NewSimpleMap() + smv := simplemap.NewSimpleMap() smst := NewSMSTWithStorage(smn, smv, sha256.New()) base := smst.Spec() @@ -201,7 +202,7 @@ func TestSMST_Proof_ValidateBasic(t *testing.T) { } func TestSMST_ClosestProof_ValidateBasic(t *testing.T) { - smn := kvstore.NewSimpleMap() + smn := simplemap.NewSimpleMap() smst := NewSparseMerkleSumTrie(smn, sha256.New()) np := NoPrehashSpec(sha256.New(), true) base := smst.Spec() @@ -282,7 +283,7 @@ func TestSMST_ProveClosest(t *testing.T) { var err error var sumBz [sumSize]byte - smn = kvstore.NewSimpleMap() + smn = simplemap.NewSimpleMap() require.NoError(t, err) smst = NewSparseMerkleSumTrie(smn, sha256.New(), WithValueHasher(nil)) @@ -362,7 +363,7 @@ func TestSMST_ProveClosest_Empty(t *testing.T) { var proof *SparseMerkleClosestProof var err error - smn = kvstore.NewSimpleMap() + smn = simplemap.NewSimpleMap() require.NoError(t, err) smst = NewSparseMerkleSumTrie(smn, sha256.New(), WithValueHasher(nil)) @@ -391,7 +392,7 @@ func TestSMST_ProveClosest_OneNode(t *testing.T) { var proof *SparseMerkleClosestProof var err error - smn = kvstore.NewSimpleMap() + smn = simplemap.NewSimpleMap() require.NoError(t, err) smst = NewSparseMerkleSumTrie(smn, sha256.New(), WithValueHasher(nil)) @@ -432,7 +433,7 @@ func TestSMST_ProveClosest_Proof(t *testing.T) { var err error // setup trie (256+512 path hasher) and nodestore - smn = kvstore.NewSimpleMap() + smn = simplemap.NewSimpleMap() require.NoError(t, err) smst256 = NewSparseMerkleSumTrie(smn, sha256.New()) smst512 = NewSparseMerkleSumTrie(smn, sha512.New()) diff --git a/smst_test.go b/smst_test.go index 2c7a05e..d8e0b3c 100644 --- a/smst_test.go +++ b/smst_test.go @@ -11,6 +11,7 @@ import ( "github.com/stretchr/testify/require" "github.com/pokt-network/smt/kvstore" + "github.com/pokt-network/smt/kvstore/simplemap" ) func NewSMSTWithStorage( @@ -25,8 +26,8 @@ func NewSMSTWithStorage( } func TestSMST_TrieUpdateBasic(t *testing.T) { - smn := kvstore.NewSimpleMap() - smv := kvstore.NewSimpleMap() + smn := simplemap.NewSimpleMap() + smv := simplemap.NewSimpleMap() lazy := NewSparseMerkleSumTrie(smn, sha256.New()) smst := &SMSTWithStorage{SMST: lazy, preimages: smv} var value []byte @@ -115,8 +116,8 @@ func TestSMST_TrieUpdateBasic(t *testing.T) { // Test base case trie delete operations with a few keys. func TestSMST_TrieDeleteBasic(t *testing.T) { - smn := kvstore.NewSimpleMap() - smv := kvstore.NewSimpleMap() + smn := simplemap.NewSimpleMap() + smv := simplemap.NewSimpleMap() lazy := NewSparseMerkleSumTrie(smn, sha256.New()) smst := &SMSTWithStorage{SMST: lazy, preimages: smv} rootEmpty := smst.Root() @@ -222,8 +223,8 @@ func TestSMST_TrieDeleteBasic(t *testing.T) { // Test trie ops with known paths func TestSMST_TrieKnownPath(t *testing.T) { ph := dummyPathHasher{32} - smn := kvstore.NewSimpleMap() - smv := kvstore.NewSimpleMap() + smn := simplemap.NewSimpleMap() + smv := simplemap.NewSimpleMap() smst := NewSMSTWithStorage(smn, smv, sha256.New(), WithPathHasher(ph)) var value []byte var sum uint64 @@ -301,8 +302,8 @@ func TestSMST_TrieKnownPath(t *testing.T) { // Test trie operations when two leafs are immediate neighbors. func TestSMST_TrieMaxHeightCase(t *testing.T) { ph := dummyPathHasher{32} - smn := kvstore.NewSimpleMap() - smv := kvstore.NewSimpleMap() + smn := simplemap.NewSimpleMap() + smv := simplemap.NewSimpleMap() smst := NewSMSTWithStorage(smn, smv, sha256.New(), WithPathHasher(ph)) var value []byte var sum uint64 @@ -350,8 +351,8 @@ func TestSMST_OrphanRemoval(t *testing.T) { return smn.Len() } setup := func() { - smn = kvstore.NewSimpleMap() - smv = kvstore.NewSimpleMap() + smn = simplemap.NewSimpleMap() + smv = simplemap.NewSimpleMap() impl = NewSparseMerkleSumTrie(smn, sha256.New()) smst = &SMSTWithStorage{SMST: impl, preimages: smv} @@ -429,7 +430,7 @@ func TestSMST_OrphanRemoval(t *testing.T) { } func TestSMST_TotalSum(t *testing.T) { - snm := kvstore.NewSimpleMap() + snm := simplemap.NewSimpleMap() smst := NewSparseMerkleSumTrie(snm, sha256.New()) err := smst.Update([]byte("key1"), []byte("value1"), 5) require.NoError(t, err) @@ -471,7 +472,7 @@ func TestSMST_TotalSum(t *testing.T) { require.Equal(t, sum, uint64(10)) // Calculate the total sum of a larger trie - snm = kvstore.NewSimpleMap() + snm = simplemap.NewSimpleMap() smst = NewSparseMerkleSumTrie(snm, sha256.New()) for i := 1; i < 10000; i++ { err := smst.Update([]byte(fmt.Sprintf("testKey%d", i)), []byte(fmt.Sprintf("testValue%d", i)), uint64(i)) @@ -483,7 +484,7 @@ func TestSMST_TotalSum(t *testing.T) { } func TestSMST_Retrieval(t *testing.T) { - snm := kvstore.NewSimpleMap() + snm := simplemap.NewSimpleMap() smst := NewSparseMerkleSumTrie(snm, sha256.New(), WithValueHasher(nil)) err := smst.Update([]byte("key1"), []byte("value1"), 5) diff --git a/smst_utils_test.go b/smst_utils_test.go index 882f764..47c006f 100644 --- a/smst_utils_test.go +++ b/smst_utils_test.go @@ -7,6 +7,7 @@ import ( "fmt" "github.com/pokt-network/smt/kvstore" + "github.com/pokt-network/smt/kvstore/simplemap" ) // SMSTWithStorage wraps an SMST with a mapping of value hashes to values with @@ -48,7 +49,7 @@ func (smst *SMSTWithStorage) GetValueSum(key []byte) ([]byte, uint64, error) { } value, err := smst.preimages.Get(valueHash) if err != nil { - if errors.Is(err, kvstore.ErrKVStoreKeyNotFound) { + if errors.Is(err, simplemap.ErrKVStoreKeyNotFound) { // If key isn't found, return default value and sum return defaultValue, 0, nil } diff --git a/smt.go b/smt.go index 2bdcd96..fe3e8b7 100644 --- a/smt.go +++ b/smt.go @@ -5,6 +5,7 @@ import ( "hash" "github.com/pokt-network/smt/kvstore" + "github.com/pokt-network/smt/kvstore/simplemap" ) var ( @@ -14,7 +15,7 @@ var ( ) type trieNode interface { - // when committing a node to disk, if already persisted it is skipped + // when committing a node to disk, skip if already persisted Persisted() bool CachedDigest() []byte } @@ -246,11 +247,11 @@ func (smt *SMT) delete(node trieNode, depth int, path []byte, orphans *orphanNod } if node == nil { - return node, kvstore.ErrKVStoreKeyNotFound + return node, simplemap.ErrKVStoreKeyNotFound } if leaf, ok := node.(*leafNode); ok { if !bytes.Equal(path, leaf.path) { - return node, kvstore.ErrKVStoreKeyNotFound + return node, simplemap.ErrKVStoreKeyNotFound } smt.addOrphan(orphans, node) return nil, nil @@ -260,7 +261,7 @@ func (smt *SMT) delete(node trieNode, depth int, path []byte, orphans *orphanNod if ext, ok := node.(*extensionNode); ok { if _, match := ext.match(path, depth); !match { - return node, kvstore.ErrKVStoreKeyNotFound + return node, simplemap.ErrKVStoreKeyNotFound } ext.child, err = smt.delete(ext.child, depth+ext.length(), path, orphans) if err != nil { diff --git a/smt_proofs_test.go b/smt_proofs_test.go index 400d6aa..2cf70c8 100644 --- a/smt_proofs_test.go +++ b/smt_proofs_test.go @@ -9,6 +9,7 @@ import ( "github.com/stretchr/testify/require" "github.com/pokt-network/smt/kvstore" + "github.com/pokt-network/smt/kvstore/simplemap" ) // Test base case Merkle proof operations. @@ -20,9 +21,9 @@ func TestSMT_Proof_Operations(t *testing.T) { var root []byte var err error - smn = kvstore.NewSimpleMap() + smn = simplemap.NewSimpleMap() require.NoError(t, err) - smv = kvstore.NewSimpleMap() + smv = simplemap.NewSimpleMap() require.NoError(t, err) smt = NewSMTWithStorage(smn, smv, sha256.New()) base := smt.Spec() @@ -109,8 +110,8 @@ func TestSMT_Proof_Operations(t *testing.T) { // Test sanity check cases for non-compact proofs. func TestSMT_Proof_ValidateBasic(t *testing.T) { - smn := kvstore.NewSimpleMap() - smv := kvstore.NewSimpleMap() + smn := simplemap.NewSimpleMap() + smv := simplemap.NewSimpleMap() smt := NewSMTWithStorage(smn, smv, sha256.New()) base := smt.Spec() @@ -175,7 +176,7 @@ func TestSMT_Proof_ValidateBasic(t *testing.T) { } func TestSMT_ClosestProof_ValidateBasic(t *testing.T) { - smn := kvstore.NewSimpleMap() + smn := simplemap.NewSimpleMap() smt := NewSparseMerkleTrie(smn, sha256.New()) np := NoPrehashSpec(sha256.New(), false) base := smt.Spec() @@ -255,7 +256,7 @@ func TestSMT_ProveClosest(t *testing.T) { var root []byte var err error - smn = kvstore.NewSimpleMap() + smn = simplemap.NewSimpleMap() require.NoError(t, err) smt = NewSparseMerkleTrie(smn, sha256.New(), WithValueHasher(nil)) @@ -317,7 +318,7 @@ func TestSMT_ProveClosest_Empty(t *testing.T) { var proof *SparseMerkleClosestProof var err error - smn = kvstore.NewSimpleMap() + smn = simplemap.NewSimpleMap() require.NoError(t, err) smt = NewSparseMerkleTrie(smn, sha256.New(), WithValueHasher(nil)) @@ -346,7 +347,7 @@ func TestSMT_ProveClosest_OneNode(t *testing.T) { var proof *SparseMerkleClosestProof var err error - smn = kvstore.NewSimpleMap() + smn = simplemap.NewSimpleMap() smt = NewSparseMerkleTrie(smn, sha256.New(), WithValueHasher(nil)) require.NoError(t, smt.Update([]byte("foo"), []byte("bar"))) @@ -381,7 +382,7 @@ func TestSMT_ProveClosest_Proof(t *testing.T) { var err error // setup trie (256+512 path hasher) and nodestore - smn = kvstore.NewSimpleMap() + smn = simplemap.NewSimpleMap() require.NoError(t, err) smt256 = NewSparseMerkleTrie(smn, sha256.New()) smt512 = NewSparseMerkleTrie(smn, sha512.New()) diff --git a/smt_test.go b/smt_test.go index f09682a..fd1adac 100644 --- a/smt_test.go +++ b/smt_test.go @@ -9,6 +9,7 @@ import ( "github.com/stretchr/testify/require" "github.com/pokt-network/smt/kvstore" + "github.com/pokt-network/smt/kvstore/simplemap" ) func NewSMTWithStorage( @@ -23,8 +24,8 @@ func NewSMTWithStorage( } func TestSMT_TrieUpdateBasic(t *testing.T) { - smn := kvstore.NewSimpleMap() - smv := kvstore.NewSimpleMap() + smn := simplemap.NewSimpleMap() + smv := simplemap.NewSimpleMap() lazy := NewSparseMerkleTrie(smn, sha256.New()) smt := &SMTWithStorage{SMT: lazy, preimages: smv} var value []byte @@ -102,8 +103,8 @@ func TestSMT_TrieUpdateBasic(t *testing.T) { // Test base case trie delete operations with a few keys. func TestSMT_TrieDeleteBasic(t *testing.T) { - smn := kvstore.NewSimpleMap() - smv := kvstore.NewSimpleMap() + smn := simplemap.NewSimpleMap() + smv := simplemap.NewSimpleMap() lazy := NewSparseMerkleTrie(smn, sha256.New()) smt := &SMTWithStorage{SMT: lazy, preimages: smv} rootEmpty := smt.Root() @@ -201,8 +202,8 @@ func TestSMT_TrieDeleteBasic(t *testing.T) { // Test trie ops with known paths func TestSMT_TrieKnownPath(t *testing.T) { ph := dummyPathHasher{32} - smn := kvstore.NewSimpleMap() - smv := kvstore.NewSimpleMap() + smn := simplemap.NewSimpleMap() + smv := simplemap.NewSimpleMap() smt := NewSMTWithStorage(smn, smv, sha256.New(), WithPathHasher(ph)) var value []byte @@ -272,8 +273,8 @@ func TestSMT_TrieKnownPath(t *testing.T) { // Test trie operations when two leafs are immediate neighbors. func TestSMT_TrieMaxHeightCase(t *testing.T) { ph := dummyPathHasher{32} - smn := kvstore.NewSimpleMap() - smv := kvstore.NewSimpleMap() + smn := simplemap.NewSimpleMap() + smv := simplemap.NewSimpleMap() smt := NewSMTWithStorage(smn, smv, sha256.New(), WithPathHasher(ph)) var value []byte @@ -318,8 +319,8 @@ func TestSMT_OrphanRemoval(t *testing.T) { return smn.Len() } setup := func() { - smn = kvstore.NewSimpleMap() - smv = kvstore.NewSimpleMap() + smn = simplemap.NewSimpleMap() + smv = simplemap.NewSimpleMap() require.NoError(t, err) impl = NewSparseMerkleTrie(smn, sha256.New()) smt = &SMTWithStorage{SMT: impl, preimages: smv} diff --git a/smt_utils_test.go b/smt_utils_test.go index 89ea986..4418862 100644 --- a/smt_utils_test.go +++ b/smt_utils_test.go @@ -5,6 +5,7 @@ import ( "errors" "github.com/pokt-network/smt/kvstore" + "github.com/pokt-network/smt/kvstore/simplemap" ) // SMTWithStorage wraps an SMT with a mapping of value hashes to values @@ -42,7 +43,7 @@ func (smt *SMTWithStorage) GetValue(key []byte) ([]byte, error) { } value, err := smt.preimages.Get(valueHash) if err != nil { - if errors.Is(err, kvstore.ErrKVStoreKeyNotFound) { + if errors.Is(err, simplemap.ErrKVStoreKeyNotFound) { // If key isn't found, return default value value = defaultValue } else { From e9892b1544d1aa5601e3e4d1b25a00c4dbe0a828 Mon Sep 17 00:00:00 2001 From: h5law <53987565+h5law@users.noreply.github.com> Date: Wed, 3 Jan 2024 16:04:18 +0000 Subject: [PATCH 22/32] chore: fix tests to cover simplemap submodule --- .github/workflows/test.yml | 4 ++-- Makefile | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 3e91f8b..74e706d 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -45,8 +45,8 @@ jobs: # Run each of the tests (excluding benchmarks) outputting the JSON # to the test_results.json file for usage in later steps. set -euo pipefail - go test -v -json -p 1 ./ -mod=readonly -race -coverprofile=coverage1.txt -covermode=atomic 2>&1 | tee -a test_results.json - go test -v -json -p 1 ./kvstore/... -race -coverprofile=coverage2.txt -covermode=atomic 2>&1 | tee -a test_results.json + go test -v -json -p 1 ./ -mod=readonly -race -coverprofile=coverage1.txt -covermode=atomic 2>&1 | tee test_results.json + go test -v -json -p 1 ./kvstore/simplemap/... -mod=readonly -race -coverprofile=coverage2.txt -covermode=atomic 2>&1 | tee -a test_results.json go test -v -json -p 1 ./kvstore/badger/... -mod=readonly -race -coverprofile=coverage3.txt -covermode=atomic 2>&1 | tee -a test_results.json # Combine coverage reports gocovmerge coverage1.txt coverage2.txt coverage3.txt > coverage.txt diff --git a/Makefile b/Makefile index 032f2da..970d4bc 100644 --- a/Makefile +++ b/Makefile @@ -40,7 +40,7 @@ go_docs: check_godoc ## Generate documentation for the project .PHONY: test_all test_all: ## runs the test suite go test -v -p 1 ./ -mod=readonly -race - go test -v -p 1 ./kvstore/*/... -race + go test -v -p 1 ./kvstore/simplemap/... -mod=readonly -race .PHONY: test_badger test_badger: ## runs the badger KVStore submodule's test suite From 2d70eb0fff51dbc18ecbe97ebbdfd7353176a056 Mon Sep 17 00:00:00 2001 From: h5law <53987565+h5law@users.noreply.github.com> Date: Wed, 3 Jan 2024 16:05:05 +0000 Subject: [PATCH 23/32] chore: update godocs --- kvstore/godoc.go | 6 ++++++ kvstore/simplemap/godoc.go | 13 ++----------- 2 files changed, 8 insertions(+), 11 deletions(-) create mode 100644 kvstore/godoc.go diff --git a/kvstore/godoc.go b/kvstore/godoc.go new file mode 100644 index 0000000..b7a5dd7 --- /dev/null +++ b/kvstore/godoc.go @@ -0,0 +1,6 @@ +// Package kvstore provides a series of sub-modules that (if they comply with +// the MapStore interface) can be used as the underlying nodestore for the trie. +// Ultimatetly the trie's reqeuire only the methods exposed by the MapStore, +// and as such any more advanced wrappers or DBs can be used IFF they implement +// this interface. +package kvstore diff --git a/kvstore/simplemap/godoc.go b/kvstore/simplemap/godoc.go index 738dd27..590b545 100644 --- a/kvstore/simplemap/godoc.go +++ b/kvstore/simplemap/godoc.go @@ -1,12 +1,3 @@ -// Package kvstore provides a simple interface for a key-value store, the -// `MapStore` and a simple implementation of this interface. These are provided -// for convenience and for simple in-memory use cases. The KVStore package also -// contains submodules with more complex implementations of key-value stores -// that can be used independently, or as the backend for the SM(S)T. -// These submodules satisfy the simple `MapStore` interface as well as also -// providing their own more complex interfaces, which can provide more features -// and better performance - such as persistence, backups and restores for example. -// These are included as submodules as they are not required for the SM(S)T and -// any key-value store that satisfies the `MapStore` interface can be used with -// the SM(S)T. +// Package simplemap provides a simple in-memory map. This can be used as the +// nodestore for both the Sparse Merkle Trie and the Sparse Merkle Tree. package simplemap From 71ec219769a8eb2a7a2b5d73581c5cbd2545b244 Mon Sep 17 00:00:00 2001 From: h5law <53987565+h5law@users.noreply.github.com> Date: Wed, 3 Jan 2024 16:05:25 +0000 Subject: [PATCH 24/32] feat: add SMT specific errors --- bulk_test.go | 2 +- errors.go | 8 ++++++-- fuzz_test.go | 6 +++--- smst_utils_test.go | 3 +-- smt.go | 7 +++---- smt_utils_test.go | 3 +-- 6 files changed, 15 insertions(+), 14 deletions(-) diff --git a/bulk_test.go b/bulk_test.go index eb01e86..0f378d1 100644 --- a/bulk_test.go +++ b/bulk_test.go @@ -82,7 +82,7 @@ func bulkOperations(t *testing.T, operations int, insert int, update int, delete ki := r.Intn(len(kv)) err := smt.Delete(kv[ki].key) - if err != nil && err != simplemap.ErrKVStoreKeyNotFound { + if err != nil && err != ErrKeyNotFound { t.Fatalf("error: %v", err) } kv[ki].val = defaultValue diff --git a/errors.go b/errors.go index ab908fb..b8d6bbe 100644 --- a/errors.go +++ b/errors.go @@ -4,5 +4,9 @@ import ( "errors" ) -// ErrBadProof is returned when an invalid Merkle proof is supplied. -var ErrBadProof = errors.New("bad proof") +var ( + // ErrBadProof is returned when an invalid Merkle proof is supplied. + ErrBadProof = errors.New("bad proof") + // ErrKeyNotFound is returned when a key is not found in the tree. + ErrKeyNotFound = errors.New("key not found") +) diff --git a/fuzz_test.go b/fuzz_test.go index 37f16a4..9c72466 100644 --- a/fuzz_test.go +++ b/fuzz_test.go @@ -68,7 +68,7 @@ func FuzzSMT_DetectUnexpectedFailures(f *testing.F) { _, err := trie.Get(key()) if err != nil { require.ErrorIsf( - t, err, simplemap.ErrKVStoreKeyNotFound, + t, err, ErrKeyNotFound, "unknown error occurred while getting", ) } @@ -85,7 +85,7 @@ func FuzzSMT_DetectUnexpectedFailures(f *testing.F) { err := trie.Delete(key()) if err != nil { require.ErrorIsf( - t, err, simplemap.ErrKVStoreKeyNotFound, + t, err, ErrKeyNotFound, "unknown error occurred while deleting", ) continue @@ -97,7 +97,7 @@ func FuzzSMT_DetectUnexpectedFailures(f *testing.F) { _, err := trie.Prove(key()) if err != nil { require.ErrorIsf( - t, err, simplemap.ErrKVStoreKeyNotFound, + t, err, ErrKeyNotFound, "unknown error occurred while proving", ) } diff --git a/smst_utils_test.go b/smst_utils_test.go index 47c006f..623de52 100644 --- a/smst_utils_test.go +++ b/smst_utils_test.go @@ -7,7 +7,6 @@ import ( "fmt" "github.com/pokt-network/smt/kvstore" - "github.com/pokt-network/smt/kvstore/simplemap" ) // SMSTWithStorage wraps an SMST with a mapping of value hashes to values with @@ -49,7 +48,7 @@ func (smst *SMSTWithStorage) GetValueSum(key []byte) ([]byte, uint64, error) { } value, err := smst.preimages.Get(valueHash) if err != nil { - if errors.Is(err, simplemap.ErrKVStoreKeyNotFound) { + if errors.Is(err, ErrKeyNotFound) { // If key isn't found, return default value and sum return defaultValue, 0, nil } diff --git a/smt.go b/smt.go index fe3e8b7..f4ceacd 100644 --- a/smt.go +++ b/smt.go @@ -5,7 +5,6 @@ import ( "hash" "github.com/pokt-network/smt/kvstore" - "github.com/pokt-network/smt/kvstore/simplemap" ) var ( @@ -247,11 +246,11 @@ func (smt *SMT) delete(node trieNode, depth int, path []byte, orphans *orphanNod } if node == nil { - return node, simplemap.ErrKVStoreKeyNotFound + return node, ErrKeyNotFound } if leaf, ok := node.(*leafNode); ok { if !bytes.Equal(path, leaf.path) { - return node, simplemap.ErrKVStoreKeyNotFound + return node, ErrKeyNotFound } smt.addOrphan(orphans, node) return nil, nil @@ -261,7 +260,7 @@ func (smt *SMT) delete(node trieNode, depth int, path []byte, orphans *orphanNod if ext, ok := node.(*extensionNode); ok { if _, match := ext.match(path, depth); !match { - return node, simplemap.ErrKVStoreKeyNotFound + return node, ErrKeyNotFound } ext.child, err = smt.delete(ext.child, depth+ext.length(), path, orphans) if err != nil { diff --git a/smt_utils_test.go b/smt_utils_test.go index 4418862..e8f84d2 100644 --- a/smt_utils_test.go +++ b/smt_utils_test.go @@ -5,7 +5,6 @@ import ( "errors" "github.com/pokt-network/smt/kvstore" - "github.com/pokt-network/smt/kvstore/simplemap" ) // SMTWithStorage wraps an SMT with a mapping of value hashes to values @@ -43,7 +42,7 @@ func (smt *SMTWithStorage) GetValue(key []byte) ([]byte, error) { } value, err := smt.preimages.Get(valueHash) if err != nil { - if errors.Is(err, simplemap.ErrKVStoreKeyNotFound) { + if errors.Is(err, ErrKeyNotFound) { // If key isn't found, return default value value = defaultValue } else { From 9bcd021223cefa81563c21892b3a36323aca1aa2 Mon Sep 17 00:00:00 2001 From: h5law <53987565+h5law@users.noreply.github.com> Date: Wed, 3 Jan 2024 16:22:12 +0000 Subject: [PATCH 25/32] chore: add simplestore MapStore complaiance check --- kvstore/simplemap/simplemap.go | 3 +++ 1 file changed, 3 insertions(+) diff --git a/kvstore/simplemap/simplemap.go b/kvstore/simplemap/simplemap.go index d7b066d..9b548ab 100644 --- a/kvstore/simplemap/simplemap.go +++ b/kvstore/simplemap/simplemap.go @@ -4,6 +4,9 @@ import ( "github.com/pokt-network/smt/kvstore" ) +// Ensure that the SimpleMap can be used as an SMT node store +var _ kvstore.MapStore = (*simpleMap)(nil) + // simpleMap is a simple in-memory map. type simpleMap struct { m map[string][]byte From f927cf234a40030b1fa7f6a18524df4921673d78 Mon Sep 17 00:00:00 2001 From: h5law <53987565+h5law@users.noreply.github.com> Date: Wed, 3 Jan 2024 18:13:09 +0000 Subject: [PATCH 26/32] chore: address comments --- .github/workflows/test.yml | 7 +++---- Makefile | 17 ++++++++--------- benchmarks/proof_sizes_test.go | 2 ++ smst_utils_test.go | 2 ++ smt_utils_test.go | 2 ++ 5 files changed, 17 insertions(+), 13 deletions(-) diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 74e706d..9b1e03c 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -45,11 +45,10 @@ jobs: # Run each of the tests (excluding benchmarks) outputting the JSON # to the test_results.json file for usage in later steps. set -euo pipefail - go test -v -json -p 1 ./ -mod=readonly -race -coverprofile=coverage1.txt -covermode=atomic 2>&1 | tee test_results.json - go test -v -json -p 1 ./kvstore/simplemap/... -mod=readonly -race -coverprofile=coverage2.txt -covermode=atomic 2>&1 | tee -a test_results.json - go test -v -json -p 1 ./kvstore/badger/... -mod=readonly -race -coverprofile=coverage3.txt -covermode=atomic 2>&1 | tee -a test_results.json + go test -v -json -p 1 ./... -mod=readonly -race -coverprofile=coverage1.txt -covermode=atomic 2>&1 | tee test_results.json + go test -v -json -p 1 ./kvstore/badger/... -mod=readonly -race -coverprofile=coverage2.txt -covermode=atomic 2>&1 | tee -a test_results.json # Combine coverage reports - gocovmerge coverage1.txt coverage2.txt coverage3.txt > coverage.txt + gocovmerge coverage1.txt coverage2.txt > coverage.txt - name: Sanitize test results # We're utilizing `tee` above which can capture non-json stdout output diff --git a/Makefile b/Makefile index 970d4bc..2899b51 100644 --- a/Makefile +++ b/Makefile @@ -28,19 +28,13 @@ check_godoc: ### Documentation ### ##################### -.PHONY: go_docs -go_docs: check_godoc ## Generate documentation for the project - echo "Visit http://localhost:6060/pkg/github.com/pokt-network/smt/" - godoc -http=:6060 - ##################### #### Testing #### ##################### .PHONY: test_all test_all: ## runs the test suite - go test -v -p 1 ./ -mod=readonly -race - go test -v -p 1 ./kvstore/simplemap/... -mod=readonly -race + go test -v -p 1 ./... -mod=readonly -race .PHONY: test_badger test_badger: ## runs the badger KVStore submodule's test suite @@ -48,14 +42,19 @@ test_badger: ## runs the badger KVStore submodule's test suite ##################### -### go mod ### +### go helpers ### ##################### .PHONY: mod_tidy -mod_tidy: ## runs go mod tidy for all (sub)modules +mod_tidy: check_godoc ## runs go mod tidy for all (sub)modules go mod tidy cd kvstore/simplemap && go mod tidy cd kvstore/badger && go mod tidy +.PHONY: go_docs +go_docs: check_godoc ## Generate documentation for the project + echo "Visit http://localhost:6060/pkg/github.com/pokt-network/smt/" + godoc -http=:6060 + ##################### ### Benchmark ### ##################### diff --git a/benchmarks/proof_sizes_test.go b/benchmarks/proof_sizes_test.go index b3573ea..b1f2cf5 100644 --- a/benchmarks/proof_sizes_test.go +++ b/benchmarks/proof_sizes_test.go @@ -1,3 +1,5 @@ +//go:build bechmark + package smt import ( diff --git a/smst_utils_test.go b/smst_utils_test.go index 623de52..21a9ac7 100644 --- a/smst_utils_test.go +++ b/smst_utils_test.go @@ -20,6 +20,8 @@ type SMSTWithStorage struct { // Update updates a key with a new value in the trie and adds the value to the // preimages KVStore +// Preimages are the values prior to them being hashed - they are used to +// confirm the values are in the trie func (smst *SMSTWithStorage) Update(key, value []byte, sum uint64) error { if err := smst.SMST.Update(key, value, sum); err != nil { return err diff --git a/smt_utils_test.go b/smt_utils_test.go index e8f84d2..2912e26 100644 --- a/smt_utils_test.go +++ b/smt_utils_test.go @@ -18,6 +18,8 @@ type SMTWithStorage struct { // Update updates a key with a new value in the trie and adds the value to // the preimages KVStore +// Preimages are the values prior to them being hashed - they are used to +// confirm the values are in the trie func (smt *SMTWithStorage) Update(key, value []byte) error { if err := smt.SMT.Update(key, value); err != nil { return err From 6850a8abb0a125ffa6da78070e3ae8f3b7d97579 Mon Sep 17 00:00:00 2001 From: h5law <53987565+h5law@users.noreply.github.com> Date: Wed, 3 Jan 2024 18:21:51 +0000 Subject: [PATCH 27/32] chore: interface commennts --- kvstore/badger/errors.go | 2 +- kvstore/badger/interface.go | 37 +++++++++++++++++++++++++--------- kvstore/badger/kvstore.go | 6 +++--- kvstore/badger/kvstore_test.go | 2 +- kvstore/interfaces.go | 7 +++++++ 5 files changed, 40 insertions(+), 14 deletions(-) diff --git a/kvstore/badger/errors.go b/kvstore/badger/errors.go index 6759b30..6bdb083 100644 --- a/kvstore/badger/errors.go +++ b/kvstore/badger/errors.go @@ -6,7 +6,7 @@ import ( var ( // ErrBadgerOpeningStore is returned when the badger store cannot be opened - // or an error occurs while opening/creating the KVStore + // or an error occurs while opening/creating the BadgerKVStore ErrBadgerOpeningStore = errors.New("error opening the store") // ErrBadgerUnableToSetValue is returned when the badger store fails to // set a value diff --git a/kvstore/badger/interface.go b/kvstore/badger/interface.go index 6780a5f..5691868 100644 --- a/kvstore/badger/interface.go +++ b/kvstore/badger/interface.go @@ -6,27 +6,46 @@ import ( "github.com/pokt-network/smt/kvstore" ) -// Ensure the KVStore can be used as an SMT node store -var _ kvstore.MapStore = (KVStore)(nil) +// Ensure the BadgerKVStore can be used as an SMT node store +var _ kvstore.MapStore = (BadgerKVStore)(nil) -// KVStore is an interface that defines a key-value store +// BadgerKVStore is an interface that defines a key-value store // that can be used standalone or as the node store for an SMT. -type KVStore interface { - // Store methods +// This is a superset of the MapStore interface that offers more +// features and can be used as a standalone key-value store. +type BadgerKVStore interface { + // --- Store methods --- + + // Get returns the value for a given key Get(key []byte) ([]byte, error) + // Set sets/updates the value for a given key Set(key, value []byte) error + // Delete removes a key Delete(key []byte) error - // Lifecycle methods + // --- Lifecycle methods --- + + // Stop closes the database connection, disabling any access to the store Stop() error - // Data methods + // --- Data methods --- + + // Backup creates a full backup of the store written to the provided writer Backup(writer io.Writer, incremental bool) error + // Restore loads the store from a backup in the reader provided Restore(io.Reader) error - // Accessors + // --- Accessors --- + + // GetAll returns all keys and values with the given prefix in the specified order GetAll(prefixKey []byte, descending bool) (keys, values [][]byte, err error) + // Exists returns true if the key exists Exists(key []byte) (bool, error) - ClearAll() error + // Len returns the number of key-value pairs in the store Len() int + + // --- Data management --- + + // ClearAll deletes all key-value pairs in the store + ClearAll() error } diff --git a/kvstore/badger/kvstore.go b/kvstore/badger/kvstore.go index 2d8edd4..07e2caf 100644 --- a/kvstore/badger/kvstore.go +++ b/kvstore/badger/kvstore.go @@ -11,16 +11,16 @@ const ( maxPendingWrites = 16 // used in backup restoration ) -var _ KVStore = &badgerKVStore{} +var _ BadgerKVStore = &badgerKVStore{} type badgerKVStore struct { db *badgerv4.DB lastBackup uint64 // timestamp of the most recent backup } -// NewKVStore creates a new KVStore using badger as the underlying database +// NewKVStore creates a new BadgerKVStore using badger as the underlying database // if no path for a persistence database is provided it will create one in-memory -func NewKVStore(path string) (KVStore, error) { +func NewKVStore(path string) (BadgerKVStore, error) { var db *badgerv4.DB var err error if path == "" { diff --git a/kvstore/badger/kvstore_test.go b/kvstore/badger/kvstore_test.go index bac9d3d..0f41b6d 100644 --- a/kvstore/badger/kvstore_test.go +++ b/kvstore/badger/kvstore_test.go @@ -393,7 +393,7 @@ func TestBadger_KVStore_Len(t *testing.T) { } } -func setupStore(t *testing.T, store badger.KVStore) { +func setupStore(t *testing.T, store badger.BadgerKVStore) { t.Helper() err := store.Set([]byte("foo"), []byte("bar")) require.NoError(t, err) diff --git a/kvstore/interfaces.go b/kvstore/interfaces.go index 0d0ac53..e0c92a4 100644 --- a/kvstore/interfaces.go +++ b/kvstore/interfaces.go @@ -5,11 +5,18 @@ package kvstore // store requires in order to back an SM(S)T. type MapStore interface { // --- Accessors --- + + // Get returns the value for a given key Get(key []byte) ([]byte, error) + // Set sets/updates the value for a given key Set(key, value []byte) error + // Delete removes a key Delete(key []byte) error + // Len returns the number of key-value pairs in the store Len() int // --- Debug --- + + // ClearAll deletes all key-value pairs in the store ClearAll() error } From fee3061f82bf349d32dd19b648d484fffba3a754 Mon Sep 17 00:00:00 2001 From: h5law <53987565+h5law@users.noreply.github.com> Date: Wed, 3 Jan 2024 18:47:32 +0000 Subject: [PATCH 28/32] chore: s/name/desc/g --- benchmarks/bench_smst_test.go | 30 +++++++++++++++--------------- benchmarks/bench_smt_test.go | 32 ++++++++++++++++---------------- 2 files changed, 31 insertions(+), 31 deletions(-) diff --git a/benchmarks/bench_smst_test.go b/benchmarks/bench_smst_test.go index 6b68fd3..ea38890 100644 --- a/benchmarks/bench_smst_test.go +++ b/benchmarks/bench_smst_test.go @@ -240,13 +240,13 @@ func BenchmarkSparseMerkleSumTrie_Prove(b *testing.B) { fn: proSMST, }, { - name: "Prove (Prefilled: 5000000)", + desc: "Prove (Prefilled: 5000000)", trieSize: 5000000, commit: false, fn: proSMST, }, { - name: "Prove (Prefilled: 10000000)", + desc: "Prove (Prefilled: 10000000)", trieSize: 10000000, commit: false, fn: proSMST, @@ -255,7 +255,7 @@ func BenchmarkSparseMerkleSumTrie_Prove(b *testing.B) { for _, tc := range testCases { b.ResetTimer() - b.Run(tc.name, func(b *testing.B) { + b.Run(tc.desc, func(b *testing.B) { trie := setupSMST(b, tc.trieSize) benchmarkSMST(b, trie, tc.commit, tc.fn) }) @@ -264,67 +264,67 @@ func BenchmarkSparseMerkleSumTrie_Prove(b *testing.B) { func BenchmarkSparseMerkleSumTrie_Delete(b *testing.B) { testCases := []struct { - name string + desc string trieSize int commit bool fn func(*smt.SMST, uint64) error }{ { - name: "Delete (Prefilled: 100000)", + desc: "Delete (Prefilled: 100000)", trieSize: 100000, commit: false, fn: delSMST, }, { - name: "Delete & Commit (Prefilled: 100000)", + desc: "Delete & Commit (Prefilled: 100000)", trieSize: 100000, commit: true, fn: delSMST, }, { - name: "Delete (Prefilled: 500000)", + desc: "Delete (Prefilled: 500000)", trieSize: 500000, commit: false, fn: delSMST, }, { - name: "Delete & Commit (Prefilled: 500000)", + desc: "Delete & Commit (Prefilled: 500000)", trieSize: 500000, commit: true, fn: delSMST, }, { - name: "Delete (Prefilled: 1000000)", + desc: "Delete (Prefilled: 1000000)", trieSize: 1000000, commit: false, fn: delSMST, }, { - name: "Delete & Commit (Prefilled: 1000000)", + desc: "Delete & Commit (Prefilled: 1000000)", trieSize: 1000000, commit: true, fn: delSMST, }, { - name: "Delete (Prefilled: 5000000)", + desc: "Delete (Prefilled: 5000000)", trieSize: 5000000, commit: false, fn: delSMST, }, { - name: "Delete & Commit (Prefilled: 5000000)", + desc: "Delete & Commit (Prefilled: 5000000)", trieSize: 5000000, commit: true, fn: delSMST, }, { - name: "Delete (Prefilled: 10000000)", + desc: "Delete (Prefilled: 10000000)", trieSize: 10000000, commit: false, fn: delSMST, }, { - name: "Delete & Commit (Prefilled: 10000000)", + desc: "Delete & Commit (Prefilled: 10000000)", trieSize: 10000000, commit: true, fn: delSMST, @@ -333,7 +333,7 @@ func BenchmarkSparseMerkleSumTrie_Delete(b *testing.B) { for _, tc := range testCases { b.ResetTimer() - b.Run(tc.name, func(b *testing.B) { + b.Run(tc.desc, func(b *testing.B) { trie := setupSMST(b, tc.trieSize) benchmarkSMST(b, trie, tc.commit, tc.fn) }) diff --git a/benchmarks/bench_smt_test.go b/benchmarks/bench_smt_test.go index 57f3e16..b857b5d 100644 --- a/benchmarks/bench_smt_test.go +++ b/benchmarks/bench_smt_test.go @@ -234,19 +234,19 @@ func BenchmarkSparseMerkleTrie_Prove(b *testing.B) { fn: proSMT, }, { - name: "Prove (Prefilled: 1000000)", + desc: "Prove (Prefilled: 1000000)", trieSize: 1000000, commit: false, fn: proSMT, }, { - name: "Prove (Prefilled: 5000000)", + desc: "Prove (Prefilled: 5000000)", trieSize: 5000000, commit: false, fn: proSMT, }, { - name: "Prove (Prefilled: 10000000)", + desc: "Prove (Prefilled: 10000000)", trieSize: 10000000, commit: false, fn: proSMT, @@ -255,7 +255,7 @@ func BenchmarkSparseMerkleTrie_Prove(b *testing.B) { for _, tc := range testCases { b.ResetTimer() - b.Run(tc.name, func(b *testing.B) { + b.Run(tc.desc, func(b *testing.B) { trie := setupSMT(b, tc.trieSize) benchmarkSMT(b, trie, tc.commit, tc.fn) }) @@ -264,67 +264,67 @@ func BenchmarkSparseMerkleTrie_Prove(b *testing.B) { func BenchmarkSparseMerkleTrie_Delete(b *testing.B) { testCases := []struct { - name string + desc string trieSize int commit bool fn func(*smt.SMT, []byte) error }{ { - name: "Delete (Prefilled: 100000)", + desc: "Delete (Prefilled: 100000)", trieSize: 100000, commit: false, fn: delSMT, }, { - name: "Delete & Commit (Prefilled: 100000)", + desc: "Delete & Commit (Prefilled: 100000)", trieSize: 100000, commit: true, fn: delSMT, }, { - name: "Delete (Prefilled: 500000)", + desc: "Delete (Prefilled: 500000)", trieSize: 500000, commit: false, fn: delSMT, }, { - name: "Delete & Commit (Prefilled: 500000)", + desc: "Delete & Commit (Prefilled: 500000)", trieSize: 500000, commit: true, fn: delSMT, }, { - name: "Delete (Prefilled: 1000000)", + desc: "Delete (Prefilled: 1000000)", trieSize: 1000000, commit: false, fn: delSMT, }, { - name: "Delete & Commit (Prefilled: 1000000)", + desc: "Delete & Commit (Prefilled: 1000000)", trieSize: 1000000, commit: true, fn: delSMT, }, { - name: "Delete (Prefilled: 5000000)", + desc: "Delete (Prefilled: 5000000)", trieSize: 5000000, commit: false, fn: delSMT, }, { - name: "Delete & Commit (Prefilled: 5000000)", + desc: "Delete & Commit (Prefilled: 5000000)", trieSize: 5000000, commit: true, fn: delSMT, }, { - name: "Delete (Prefilled: 10000000)", + desc: "Delete (Prefilled: 10000000)", trieSize: 10000000, commit: false, fn: delSMT, }, { - name: "Delete & Commit (Prefilled: 10000000)", + desc: "Delete & Commit (Prefilled: 10000000)", trieSize: 10000000, commit: true, fn: delSMT, @@ -333,7 +333,7 @@ func BenchmarkSparseMerkleTrie_Delete(b *testing.B) { for _, tc := range testCases { b.ResetTimer() - b.Run(tc.name, func(b *testing.B) { + b.Run(tc.desc, func(b *testing.B) { trie := setupSMT(b, tc.trieSize) benchmarkSMT(b, trie, tc.commit, tc.fn) }) From a7c793cd72eb4f9cd585c0fe6e69d35bfb3bef68 Mon Sep 17 00:00:00 2001 From: h5law <53987565+h5law@users.noreply.github.com> Date: Wed, 3 Jan 2024 20:32:58 +0000 Subject: [PATCH 29/32] chore: address comments --- Makefile | 42 +++++++++++++++++----------------- benchmarks/bench_leaf_test.go | 2 ++ benchmarks/bench_smst_test.go | 2 ++ benchmarks/bench_smt_test.go | 2 ++ benchmarks/bench_utils_test.go | 2 ++ benchmarks/proof_sizes_test.go | 2 +- 6 files changed, 30 insertions(+), 22 deletions(-) diff --git a/Makefile b/Makefile index 2899b51..0f6e4b1 100644 --- a/Makefile +++ b/Makefile @@ -13,20 +13,20 @@ help: ## Prints all the targets in all the Makefiles list: ## List all make targets @${MAKE} -pRrn : -f $(MAKEFILE_LIST) 2>/dev/null | awk -v RS= -F: '/^# File/,/^# Finished Make data base/ {if ($$1 !~ "^[#.]") {print $$1}}' | egrep -v -e '^[^[:alnum:]]' -e '^$@$$' | sort +##################### +### Documentation ### +##################### + ## Ensure godoc is installed .PHONY: check_godoc -# Internal helper target - check if godoc is installed + # Internal helper target - check if godoc is installed check_godoc: { \ - if ( ! ( command -v godoc >/dev/null )); then \ + if ( ! ( command -v godoc >/dev/null )); then \ echo "Seems like you don't have godoc installed. Make sure you install it via 'go install golang.org/x/tools/cmd/godoc@latest' before continuing"; \ exit 1; \ - fi; \ - } - -##################### -### Documentation ### -##################### + fi; \ + } ##################### #### Testing #### @@ -45,7 +45,7 @@ test_badger: ## runs the badger KVStore submodule's test suite ### go helpers ### ##################### .PHONY: mod_tidy -mod_tidy: check_godoc ## runs go mod tidy for all (sub)modules +mod_tidy: ## runs go mod tidy for all (sub)modules go mod tidy cd kvstore/simplemap && go mod tidy cd kvstore/badger && go mod tidy @@ -60,33 +60,33 @@ go_docs: check_godoc ## Generate documentation for the project ##################### .PHONY: benchmark_all -bechmark_all: ## runs all benchmarks - go test -benchmem -run=^$ -bench Benchmark ./benchmarks -timeout 0 +benchmark_all: ## runs all benchmarks + go test -tags=benchmark -benchmem -run=^$ -bench Benchmark ./benchmarks -timeout 0 .PHONY: benchmark_smt benchmark_smt: ## runs all benchmarks for the SMT - go test -benchmem -run=^$ -bench=BenchmarkSparseMerkleTrie ./benchmarks -timeout 0 + go test -tags=benchmark -benchmem -run=^$ -bench=BenchmarkSparseMerkleTrie ./benchmarks -timeout 0 .PHONY: benchmark_smt_fill benchmark_smt_fill: ## runs a benchmark on filling the SMT with different amounts of values - go test -benchmem -run=^$ -bench=BenchmarkSparseMerkleTrie_Fill ./benchmarks -timeout 0 -benchtime 10x + go test -tags=benchmark -benchmem -run=^$ -bench=BenchmarkSparseMerkleTrie_Fill ./benchmarks -timeout 0 -benchtime 10x .PHONY: benchmark_smt_ops benchmark_smt_ops: ## runs the benchmarks testing different operations on the SMT against different sized tries - go test -benchmem -run=^$ -bench='BenchmarkSparseMerkleTrie_(Update|Get|Prove|Delete)' ./benchmarks -timeout 0 + go test -tags=benchmark -benchmem -run=^$ -bench='BenchmarkSparseMerkleTrie_(Update|Get|Prove|Delete)' ./benchmarks -timeout 0 .PHONY: benchmark_smst benchmark_smst: ## runs all benchmarks for the SMST - go test -benchmem -run=^$ -bench=BenchmarkSparseMerkleSumTrie ./benchmarks -timeout 0 + go test -tags=benchmark -benchmem -run=^$ -bench=BenchmarkSparseMerkleSumTrie ./benchmarks -timeout 0 .PHONY: benchmark_smst_fill benchmark_smst_fill: ## runs a benchmark on filling the SMST with different amounts of values - go test -benchmem -run=^$ -bench=BenchmarkSparseMerkleSumTrie_Fill ./benchmarks -timeout 0 -benchtime 10x + go test -tags=benchmark -benchmem -run=^$ -bench=BenchmarkSparseMerkleSumTrie_Fill ./benchmarks -timeout 0 -benchtime 10x .PHONY: benchmark_smst_ops -benchmark_smst_ops: ## runs the benchmarks testing different operations on the SMST against different sized tries - go test -benchmem -run=^$ -bench='BenchmarkSparseMerkleSumTrie_(Update|Get|Prove|Delete)' ./benchmarks -timeout 0 +benchmark_smst_ops: ## runs the benchmarks test different operations on the SMST against different sized tries + go test -tags=benchmark -benchmem -run=^$ -bench='BenchmarkSparseMerkleSumTrie_(Update|Get|Prove|Delete)' ./benchmarks -timeout 0 -.PHONY: bechmark_proof_sizes -benchmark_proof_sizes: ## runs the benchmarks testing the proof sizes for different sized tries - go test -v ./benchmarks -run ProofSizes +.PHONY: benchmark_proof_sizes +benchmark_proof_sizes: ## runs the benchmarks test the proof sizes for different sized tries + go test -tags=benchmark -v ./benchmarks -run ProofSizes diff --git a/benchmarks/bench_leaf_test.go b/benchmarks/bench_leaf_test.go index 3ae12d5..7ccc46e 100644 --- a/benchmarks/bench_leaf_test.go +++ b/benchmarks/bench_leaf_test.go @@ -1,3 +1,5 @@ +//go:build benchmark + package smt import ( diff --git a/benchmarks/bench_smst_test.go b/benchmarks/bench_smst_test.go index ea38890..ea15dda 100644 --- a/benchmarks/bench_smst_test.go +++ b/benchmarks/bench_smst_test.go @@ -1,3 +1,5 @@ +//go:build benchmark + package smt import ( diff --git a/benchmarks/bench_smt_test.go b/benchmarks/bench_smt_test.go index b857b5d..d1362a2 100644 --- a/benchmarks/bench_smt_test.go +++ b/benchmarks/bench_smt_test.go @@ -1,3 +1,5 @@ +//go:build benchmark + package smt import ( diff --git a/benchmarks/bench_utils_test.go b/benchmarks/bench_utils_test.go index 791a0e7..6694086 100644 --- a/benchmarks/bench_utils_test.go +++ b/benchmarks/bench_utils_test.go @@ -1,3 +1,5 @@ +//go:build benchmark + package smt import ( diff --git a/benchmarks/proof_sizes_test.go b/benchmarks/proof_sizes_test.go index b1f2cf5..e7a169a 100644 --- a/benchmarks/proof_sizes_test.go +++ b/benchmarks/proof_sizes_test.go @@ -1,4 +1,4 @@ -//go:build bechmark +//go:build benchmark package smt From 9b1371ccb2ad19681ceabe81f2ae8eafafbf38c8 Mon Sep 17 00:00:00 2001 From: h5law <53987565+h5law@users.noreply.github.com> Date: Wed, 3 Jan 2024 21:48:43 +0000 Subject: [PATCH 30/32] chore: indentation --- Makefile | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/Makefile b/Makefile index 0f6e4b1..9022383 100644 --- a/Makefile +++ b/Makefile @@ -18,15 +18,15 @@ list: ## List all make targets ##################### ## Ensure godoc is installed -.PHONY: check_godoc - # Internal helper target - check if godoc is installed +.PHONY: +# Internal helper target - check if godoc is installed check_godoc: { \ if ( ! ( command -v godoc >/dev/null )); then \ echo "Seems like you don't have godoc installed. Make sure you install it via 'go install golang.org/x/tools/cmd/godoc@latest' before continuing"; \ exit 1; \ fi; \ - } + } ##################### #### Testing #### From 538dcb5e8e9d3f1784ba1451738197dec71ad19d Mon Sep 17 00:00:00 2001 From: h5law <53987565+h5law@users.noreply.github.com> Date: Wed, 3 Jan 2024 22:18:24 +0000 Subject: [PATCH 31/32] chore: add simple to simplemap description --- docs/kvstore.md | 4 +++- docs/smt.md | 7 ++++--- 2 files changed, 7 insertions(+), 4 deletions(-) diff --git a/docs/kvstore.md b/docs/kvstore.md index bd447ca..d2de87c 100644 --- a/docs/kvstore.md +++ b/docs/kvstore.md @@ -25,7 +25,9 @@ submodule the library (the SMT) can be used without it. As long as the node store provided to the SMT adheres to the `MapStore` interface **anything** can be used. -This is recommemded for testing purposes or non-production usage. +This is recommemded for simple, testing purposes or non-production usage. + +See [simplemap.go](../kvstore/simplemap/simplemap.go) for more details. ### BadgerV4 diff --git a/docs/smt.md b/docs/smt.md index c3e2c14..5cf1938 100644 --- a/docs/smt.md +++ b/docs/smt.md @@ -470,9 +470,10 @@ the [`kvstore`](../kvstore/) directory. #### SimpleMap This library defines the `SimpleMap` interface which is implemented as an -extremely simple in-memory key-value store. Although it is is a submodule, -it is ideal for testing and non-production use cases, and is used in the tests -throughout the library. +extremely simple in-memory key-value store. + +Although it is is a submodule, it is ideal for simple, testing or non-production +use cases. It is used in the tests throughout the library. See [simplemap.go](../kvstore/simplemap/simplemap.go) for the implementation details. From 58cc54115380ec00617e739396c5f39d77ed3212 Mon Sep 17 00:00:00 2001 From: h5law <53987565+h5law@users.noreply.github.com> Date: Wed, 3 Jan 2024 22:48:57 +0000 Subject: [PATCH 32/32] chore: address final comments --- docs/kvstore.md | 39 --------------------------------------- docs/mapstore.md | 41 +++++++++++++++++++++++++++++++++++++++++ docs/smt.md | 2 +- 3 files changed, 42 insertions(+), 40 deletions(-) delete mode 100644 docs/kvstore.md create mode 100644 docs/mapstore.md diff --git a/docs/kvstore.md b/docs/kvstore.md deleted file mode 100644 index d2de87c..0000000 --- a/docs/kvstore.md +++ /dev/null @@ -1,39 +0,0 @@ -# MapStore - - - -- [Implementations](#implementations) - * [SimpleMap](#simplemap) - * [BadgerV4](#badgerv4) - - - -The `MapStore` is a simple interface used by the SM(S)T to store, delete and -retrieve key-value pairs. Due to its simplicity and more complex key-value -databases that implement it can be used in its place as the node store for a -trie. - -See: [the interface](../kvstore/interfaces.go) for a more detailed description -of the simple interface required by the SM(S)T. - -## Implementations - -### SimpleMap - -The `simplemap` is a simple kv-store shipped with the SM(S)T. But as a -submodule the library (the SMT) can be used without it. As long as the node -store provided to the SMT adheres to the `MapStore` interface **anything** can -be used. - -This is recommemded for simple, testing purposes or non-production usage. - -See [simplemap.go](../kvstore/simplemap/simplemap.go) for more details. - -### BadgerV4 - -As documented in the [badger](./badger-store.md) documentation the interface -implements the `MapStore` interface as well as many other methods, and offers -persistence. - -See: [badger](../kvstore/badger/) for more details on the implementation of -this submodule. diff --git a/docs/mapstore.md b/docs/mapstore.md new file mode 100644 index 0000000..a47c670 --- /dev/null +++ b/docs/mapstore.md @@ -0,0 +1,41 @@ +# MapStore + + + +- [Implementations](#implementations) + * [SimpleMap](#simplemap) + * [BadgerV4](#badgerv4) + + + +The `MapStore` is a simple interface used by the SM(S)T to store, delete and +retrieve key-value pairs. It is intentionally simple and minimalistic so as to +enable different key-value engines to implement and back the trie database. + +See: [the interface](../kvstore/interfaces.go) for a more detailed description +of the simple interface required by the SM(S)T. + +## Implementations + +### SimpleMap + +`simplemap` is a simple kv-store shipped with the SM(S)T. The SMT library that +can be used without it as long as the selected node store adheres to the +`Mapstore` interface. + +This library is recommended for development, testing and exploration purposes. + +This library **SHOULD NOT** be used in production. + +See [simplemap.go](../kvstore/simplemap/simplemap.go) for more details. + +### BadgerV4 + +This library provides a wrapper around [dgraph-io/badger][badgerv4] to adhere +to the `MapStore` interface. See the [full documentation](./badger-store.md) +for additional functionality and implementation details. + +See: [badger](../kvstore/badger/) for more details on the implementation of +this submodule. + +[badgerv4]: https://github.com/dgraph-io/badger diff --git a/docs/smt.md b/docs/smt.md index 5cf1938..31a0509 100644 --- a/docs/smt.md +++ b/docs/smt.md @@ -472,7 +472,7 @@ the [`kvstore`](../kvstore/) directory. This library defines the `SimpleMap` interface which is implemented as an extremely simple in-memory key-value store. -Although it is is a submodule, it is ideal for simple, testing or non-production +Although it is a submodule, it is ideal for simple, testing or non-production use cases. It is used in the tests throughout the library. See [simplemap.go](../kvstore/simplemap/simplemap.go) for the implementation