Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Encode/Decode to polynomials in monomial basis #2959

Merged
merged 4 commits into from
Jan 23, 2025

Conversation

martyall
Copy link
Contributor

@martyall martyall commented Jan 22, 2025

Changes

  • encode/decode to polynomials in coefficient form (previously used only field elems)
  • add large randomized test to CI

Example

I ran the test script locally on a 500mb json file and got the following output

2025-01-22T18:53:21.081Z DEBUG saffron: Encoding file domain_size=65536 input_file="/home/martyall/tmp/fork_config.json"
2025-01-22T18:53:25.159Z DEBUG encode: saffron::serialization: Encoded 528127993 bytes into 260 polynomials
2025-01-22T18:53:25.206Z  INFO encode: saffron::serialization: close time.busy=3.77s time.idle=5.37µs
2025-01-22T18:53:25.206Z DEBUG saffron: Writing encoded blob to file output_file="/home/martyall/tmp/fork_config.bin"
2025-01-22T18:53:26.415Z DEBUG saffron: Decoding file domain_size=65536 input_file="/home/martyall/tmp/fork_config.bin"
2025-01-22T18:53:31.173Z  INFO decode: saffron::serialization: close time.busy=3.52s time.idle=3.97µs
2025-01-22T18:53:31.173Z DEBUG saffron: Writing decoded blob to file output_file="/home/martyall/tmp/fork_config-decodedjson"

So both encode/decode for 500MB takes about 3.5 seconds (not including file I/O). This seems like a reasonable speed for getting this demo off the ground.

Copy link

codecov bot commented Jan 22, 2025

Codecov Report

Attention: Patch coverage is 49.01961% with 26 lines in your changes missing coverage. Please review.

Project coverage is 76.85%. Comparing base (70732bc) to head (fe10025).
Report is 6 commits behind head on master.

Files with missing lines Patch % Lines
saffron/src/main.rs 0.00% 14 Missing ⚠️
saffron/src/serialization.rs 73.52% 9 Missing ⚠️
saffron/src/cli.rs 0.00% 3 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #2959      +/-   ##
==========================================
+ Coverage   76.79%   76.85%   +0.06%     
==========================================
  Files         258      258              
  Lines       61166    61202      +36     
==========================================
+ Hits        46971    47036      +65     
+ Misses      14195    14166      -29     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@martyall martyall force-pushed the martin/saffron-blob-chunks branch 4 times, most recently from ccd6dae to 16fab13 Compare January 22, 2025 05:49
Base automatically changed from add-saffron-ci to master January 22, 2025 17:39
@martyall martyall marked this pull request as ready for review January 22, 2025 18:38
@martyall martyall marked this pull request as draft January 22, 2025 19:10
@martyall martyall force-pushed the martin/saffron-blob-chunks branch from 34c9590 to 1ed55b0 Compare January 22, 2025 19:28
@martyall martyall changed the base branch from master to martin/add-tracing-to-saffron January 22, 2025 19:29
@martyall martyall mentioned this pull request Jan 22, 2025
@martyall martyall force-pushed the martin/saffron-blob-chunks branch from 1ed55b0 to 5cbdff5 Compare January 22, 2025 19:35
@martyall martyall marked this pull request as ready for review January 22, 2025 19:42
Base automatically changed from martin/add-tracing-to-saffron to master January 23, 2025 00:27
@martyall martyall force-pushed the martin/saffron-blob-chunks branch from 5cbdff5 to fe10025 Compare January 23, 2025 02:29
@martyall martyall changed the title Encode/Decode to polynomials in coefficient form Encode/Decode to polynomials in monomial basis Jan 23, 2025
@martyall martyall merged commit 48a8ebc into master Jan 23, 2025
13 checks passed
@martyall martyall deleted the martin/saffron-blob-chunks branch January 23, 2025 19:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants