feat: binary storage db #153

nuts-rice · 2025-01-07T14:03:53Z

Feature

addresses #141
provides option to store objects as binary instead of hex strings for no-code projects

How?

config field for PostgresDetails , wrapping objects using sql_wrapper types

vercel · 2025-01-07T14:03:57Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
rindexer-documentation	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jan 15, 2025 10:23pm

joshstevens19 · 2025-01-07T14:18:08Z

great thanks for this its not a hard PR if you wanted to do it:

https://github.com/joshstevens19/rindexer/blob/master/core/src/database/postgres/sql_type_wrapper.rs#L944 this method is the one which maps the ABI items to the EthereumSqlTypeWrapper now https://github.com/joshstevens19/rindexer/blob/master/core/src/database/postgres/sql_type_wrapper.rs#L151 we already started making these BlahBytes types so we would just need to make sure every single variant which can be stored as bytea in postgres has its own type here. To make this even less changes on surface we can add binary_mode to map_log_params_to_ethereum_wrapper

pub fn map_log_params_to_ethereum_wrapper(
    abi_inputs: &[ABIInput],
    params: &[LogParam],
    binary_mode: bool,
) -> Vec<EthereumSqlTypeWrapper> {

we can then create a new method which is called map_ethereum_wrapper_to_byte which maps the type to the byte type which can be done when binary_mode is true

pub fn map_log_params_to_ethereum_wrapper(
    abi_inputs: &[ABIInput],
    params: &[LogParam],
    binary_mode: bool,
) -> Vec<EthereumSqlTypeWrapper> {
    let mut wrappers = vec![];

    for (index, param) in params.iter().enumerate() {
        if let Some(abi_input) = abi_inputs.get(index) {
            match &param.value {
                Token::Tuple(tuple) => {
                
                   // TODO: CHECK IF binary_mode IS TRUE AND IF SO WRAP process_tuple RESULT IN map_ethereum_wrapper_to_byte
                 
                    wrappers.extend(process_tuple(
                        abi_input
                            .components
                            .as_ref()
                            .expect("tuple should have a component ABI on"),
                        tuple,
                    ));
                }
                _ => {
                    // TODO: CHECK IF binary_mode IS TRUE AND IF SO WRAP map_log_token_to_ethereum_wrapper RESULT IN map_ethereum_wrapper_to_byte
                    
                    wrappers.push(map_log_token_to_ethereum_wrapper(abi_input, &param.value));
                }
            }
        } else {
            panic!("No ABI input found for log param at index: {}", index)
        }
    }

    wrappers
}

we then just need to change https://github.com/joshstevens19/rindexer/blob/master/core/src/database/postgres/generate.rs#L42

extending it to have a binary_mode: bool, and if thats the case we can change the generate_columns_with_data_types to consider the types possible to be binary and if so use BYTEA instead

then thats it bingo - we bring in the config setting you put above and add to docs @nuts-rice fancy it?

nuts-rice · 2025-01-07T14:24:42Z

Yup ill get to it 😎

joshstevens19

ahh sorry i think you went a bit crazy with the bytes

the only thing which can be bytes are address (handle array), bytes (handle array), + main tables tx hash, block hash, contract address so this is why i suggest above we turn this to max_optimisation we can then make:

uint256 NUMERIC(80) as well (fixing the VARCHAR(78)) we use

and then pass in optimisations on the solidity_type_to_db_type alongside fix the other bits

joshstevens19 · 2025-01-09T18:19:34Z

cli/src/commands/new.rs

@@ -170,6 +170,7 @@ pub fn handle_new_command(
            postgres: if postgres_enabled {
                Some(PostgresDetails {
                    enabled: true,
+                    binary_storage: None,


add to the docs https://rindexer.xyz/docs/start-building/yaml-config/storage#postgres

i think this should actually be called "max_optimisation"

joshstevens19 · 2025-01-09T18:21:00Z

core/src/database/postgres/generate.rs

@@ -53,7 +54,15 @@ fn generate_event_table_sql_with_comments(
            let event_columns = if event_info.inputs.is_empty() {
                "".to_string()
            } else {
-                generate_columns_with_data_types(&event_info.inputs).join(", ") + ","
+                if binary_mode {
+                    generate_columns_with_data_types(&event_info.inputs)


not everything can be stored in bytes like booleans etc we only want to store hex stuff as bytes so i think you need to filter out based on the solidity type which ones to do?

should we use a .filter() or .filter_map() for this? I'm not sure exactly

joshstevens19 · 2025-01-09T18:21:31Z

core/src/database/postgres/sql_type_wrapper.rs

+                                .expect("tuple should have a component ABI on"),
+                            tuple,
+                        ));
+                        map_ethereum_wrapper_to_byte(


this should be in wrappers.extend

joshstevens19 · 2025-01-09T18:21:48Z

core/src/database/postgres/sql_type_wrapper.rs

@@ -960,6 +980,20 @@ pub fn map_log_params_to_ethereum_wrapper(
                    ));
                }
                _ => {
+                    if binary_mode {
+                        wrappers.push(map_log_token_to_ethereum_wrapper(abi_input, &param.value));
+                        map_ethereum_wrapper_to_byte(


this should be in wrappers.extend

joshstevens19 · 2025-01-09T18:22:27Z

core/src/database/postgres/generate.rs

+                if binary_mode {
+                    generate_columns_with_data_types(&event_info.inputs)
+                        .iter()
+                        .map(|column| format!("{} BYTEA", column))


for arrays it has to be BYTEA[]

joshstevens19 · 2025-01-09T18:23:13Z

core/src/database/postgres/sql_type_wrapper.rs

+                        let bytes = bytes::Bytes::from(bytes);
+                        bytes
+                    }
+                    EthereumSqlTypeWrapper::String(s) |


we wouldnt convert strings to bytea i dont think - just hex string values because it wont be about to handle UTF8 and graphql wont be able to get the correct values back

joshstevens19 · 2025-01-09T18:24:20Z

core/src/database/postgres/sql_type_wrapper.rs

+                        let bytes = bytes::Bytes::from(buf);
+                        bytes
+                    }
+                    EthereumSqlTypeWrapper::DateTime(date_time) => {


we should not turn datetime to bytea

joshstevens19 · 2025-01-09T18:24:48Z

core/src/database/postgres/sql_type_wrapper.rs

+                        let bytes = bytes::Bytes::from(date_time.to_rfc3339().as_bytes().to_vec());
+                        bytes
+                    }
+                    EthereumSqlTypeWrapper::JSONB(json) => {


we should not convert JSONB to bytea

joshstevens19 · 2025-01-10T17:53:28Z

core/src/database/postgres/sql_type_wrapper.rs

@@ -971,6 +1005,373 @@ pub fn map_log_params_to_ethereum_wrapper(
    wrappers
 }

+pub fn map_ethereum_wrapper_to_byte(


the only thing which can be bytes are address (handle array), bytes, + main tables tx hash, block hash, contract address so this is why i suggest above we turn this to max_optimisation we can then make:

uint256 NUMBER(80)

and then pass in optimisations on the solidity_type_to_db_type

so should solidity_type_to_db_type have the max_optimisation field too?

🔧 feat: binary db config check

1fbebe9

nuts-rice mentioned this pull request Jan 7, 2025

No-code advanced db storage = smaller db - binary #141

Open

vercel bot deployed to Preview January 7, 2025 14:06 View deployment

🚧 wip: binary_mode check + wrapping, fmt

32f7731

vercel bot deployed to Preview January 7, 2025 16:13 View deployment

🚧 wip: map types to bytes

8dceec6

vercel bot deployed to Preview January 7, 2025 21:03 View deployment

🏷️ feat: map ethereum wrapper type to bytes

6d9fe73

vercel bot deployed to Preview January 7, 2025 22:14 View deployment

✨ feat: binary_mode param for no-code db

e6dfe4a

vercel bot deployed to Preview January 9, 2025 00:14 View deployment

nuts-rice marked this pull request as ready for review January 9, 2025 13:32

joshstevens19 requested changes Jan 10, 2025

View reviewed changes

nuts-rice marked this pull request as draft January 11, 2025 00:12

🩹 feat(max_optimisation): types and naming

651aafc

vercel bot deployed to Preview January 11, 2025 21:54 View deployment

🚧 wip: filtering types

2f59b25

vercel bot deployed to Preview January 15, 2025 15:44 View deployment

🏷️ feat(binary storage db): sql table types

ad375ad

vercel bot deployed to Preview January 15, 2025 22:23 View deployment

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: binary storage db #153

feat: binary storage db #153

nuts-rice commented Jan 7, 2025

vercel bot commented Jan 7, 2025 •

edited

Loading

joshstevens19 commented Jan 7, 2025 •

edited

Loading

nuts-rice commented Jan 7, 2025

joshstevens19 left a comment •

edited

Loading

joshstevens19 Jan 9, 2025

joshstevens19 Jan 10, 2025

joshstevens19 Jan 9, 2025

nuts-rice Jan 12, 2025

joshstevens19 Jan 9, 2025

joshstevens19 Jan 9, 2025

joshstevens19 Jan 9, 2025

joshstevens19 Jan 9, 2025

joshstevens19 Jan 9, 2025

joshstevens19 Jan 9, 2025

joshstevens19 Jan 10, 2025

nuts-rice Jan 12, 2025

feat: binary storage db #153

Are you sure you want to change the base?

feat: binary storage db #153

Conversation

nuts-rice commented Jan 7, 2025

Feature

How?

vercel bot commented Jan 7, 2025 • edited Loading

joshstevens19 commented Jan 7, 2025 • edited Loading

nuts-rice commented Jan 7, 2025

joshstevens19 left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vercel bot commented Jan 7, 2025 •

edited

Loading

joshstevens19 commented Jan 7, 2025 •

edited

Loading

joshstevens19 left a comment •

edited

Loading