chore: setup migrations for indexer #18
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
LGTM, as discussed change the merge target to sepolia-db branch |
cwkang1998
added a commit
that referenced
this pull request
Feb 11, 2025
* feat: Add Sepolia deployment workflow - Add separate GitHub Actions workflow for Sepolia testnet - Configure deployment to trigger on sepolia branch pushes - Update git commands to pull from sepolia branch - Maintain separate deployment pipeline from mainnet * ci fix sepolia deploy * fix: Add environment variables to Sepolia deployment - Add env section to pass GitHub secrets to SSH action - Create .env file on remote server during deployment - Pass required environment variables for PostgreSQL and services - Include all necessary Sepolia-specific variables * fix: Align environment variables in docker-compose - Fix PostgreSQL environment variable names to match secrets - Ensure consistent variable naming between GitHub Actions and docker-compose - Update healthcheck to use correct PostgreSQL user variable * fix docker containers clash * feat: update Sepolia deployment workflow for isolated environment setup - Updated `sepolia.deploy` workflow to ensure proper isolation using `--project-name sepolia`. - Integrated `.env.sepolia` file for environment-specific variables. - Added commands to stop existing Sepolia services before deployment for a clean setup. - Ensured Docker Compose builds and deploys Sepolia services with no conflicts. - Included verification step to list running services after deployment. This change ensures Sepolia services run independently without interfering with other environments (e.g., Mainnet). * fix: update deploy scripts to use explicit docker-compose files - Updated deploy.mainnet to use docker-compose.mainnet.yml with --env-file and --project-name flags. - Updated sepolia.deploy to use docker-compose.sepolia.yml with --env-file and --project-name flags. - Ensures proper isolation of Mainnet and Sepolia environments during deployment. - Avoids reliance on default docker-compose.yml naming convention. * chore: setup migrations for indexer (#18) * ci: add cargo caching to improve CI performance * chore: add clippy configuration in .cargo/config * refactor: address clippy warnings - Convert match statements to if let where appropriate - Replace unwrap/expect with proper error handling * added test docker compose + env * swapped eprintln with tracing error * fix: remove redundant rustflags section in cargo config - Remove duplicate [rustflags] section - Clean up .cargo/config.toml structure - Keep clippy linting rules unchanged * chore(deps): update dependencies - Update hashbrown to 0.15.2 - Update tokio to 1.42.0 - Update axum to 0.7.9 - Update reqwest to 0.12.9 - Update other dependencies to latest compatible versions Notable updates: - serde to 1.0.216 - tracing to 0.1.41 - hyper to 1.5.1 - wasm-bindgen to 0.2.99 * fix: handle rpc conn loss in indexer (#24) Handles the rpc connection loss in the indexer via adding a simple retry mechanism. Directly using the retry mechanisms already implemented for the db with some changes. Closes #21 * feat: impl quick indexing (#23) * chore: rename type utils to utils * feat: impl quick indexer poc * feat: initial impl of the db and repo * feat: add batch handling for the quick indexer * fix: sql string not generated correctly - fix issue where attempting to return the bounded sql statement fails as it loses the bind values. Done via passing tx context. Does make it less testable without sqlx. - add debug config for vscode * chore: fix clippy formatting * chore: relax tracing for now to allow easier debug * chore: fix clippy lints * feat: impl batch indexing (#25) * feat: impl batch indexing * chore: fix linting and formatting * fix: propagate rpc error encountered during index * chore: fix formatting * chore: remove redundant utils * chore: fix formatting * chore: setup ci for testing with docker compose (#29) * tests: add unit tests for components (#26) * tests: add unit tests for components * chore: add coverage lib and script (#27) * chore: remove redundant scripts * chore: fix formatting * chore: add migration steps * chore: fix broken migration ci step * chore: fix missing env vars for migrations * feat: impl only index header mode by default (#31) The indexer now defaults to only indexing header, and allows switching on indexing transactions when the tx flags is set. * tests: add integration tests (#28) * tests: add basic indexer creation unit test * feat: add modular rpc for easier testing * chore: further slim down the rpc abstractions * test: add full tests for indexing * chore: extract indexing services into lib for integration * feat: introduce tx enum variant for block rpc * feat: setup docker & local docker compose * tests: add basic int tests * chore: remove coverage ci Remove coverage ci since its erroring out. * fix: use tokio instead of std threads (#34) After investigation it seems that there should be no difference between using tokio and std threads for our current indexer and using tokio should further reduce our memory overhead. As such this PR switches to using tokio instead. However it should be noted that CPU bounded tasks such as heavy calculation will stale the async tokio runtime and those should still use either std thread or `spawn_blocking`. As of right now the current indexer does not seem to have such issue. Closes #32 --------- Co-authored-by: Alex Metelli <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Setup proper migrations implementation for the indexers.