chore: Merge Orchestrator history #471

notlesh · 2025-01-21T23:15:46Z

This PR pulls in the entire madara-orchestrator git history into this project.

Because the entire history was pulled in (using git merge --allow-unrelated-histories) the PR is absolutely massive (and probably difficult to review). My process was basically this:

git merge --allow-unrelated-histories (that is, pull in the entire codebase history)
Move files (crates under crates/orchestrator and project files to a temp dir)
Update Cargo.toml to build (note that orchestrator is not a member of default-members, disagree if you want!)
Fight dependency conflicts

Status 🚧

So far this builds and madara appears to be unaffected. The project files for madara-orchestrator were kept, but were moved (temporarily) to old_orchestrator_files/. This means that things like CI will not currently work.

Repository forks 🍴

This required forking several repositories in order to avoid version conflicts. Each fork uses the same madara-monorepo branch name. These repos were forked:

Init

Change the external_id field to be an enum instead of a regular String

refactor to workspace

* initialised atlantic service * feat: added l1 and l2 proving apis * add header * feat:added testcases for atlantic service * chore: change to atlantic proving service * fix: change layout to 'starknet_with_keccak' * change layout to all_cairo for testing * update gps verifier * chore: add program hash log * fix: change program hash * ran formatter * refactor init commit * broken http client * refactor: http_client and prover services refactored * fix: build issue fixed with temp snos * chore: linting and formatting * chore: todo added for later to change snos * refactor: atlantic test updated as pwe the new changes * chore: linting and removed print statements * refactor: prover-service renamed to prover-clients * changelog: updated * chore: formatting and linting * fix: changing prover service to sharp * fix: adding proof layout to env file * fix: updated the prvoving zip * fix: program hash fixed and fact of the block registered * fix: sleep added after setting fact * refactor: toml files cleaning and http mock version update * update: duplicate SNOS_PROOD_LAYOUT removed * changelog: updated * chore: linting * fix: snos layout updated to dynamic * fix: linting and proof layout * refactor: added different layout name for snos and proving * fix: comments resolved, docker-compose deleted, request-builder updated * changelog: updated * resolved: comments and path updated in yaml for s3 download * refactor: replaced mock server with dummy url to create http client * fix: atlantic endpoint updated * refactor: atlantic test now use mock server * fix: files added * fix: env variable's name updated * chores: removed extra print statement --------- Co-authored-by: byteZorvin <[email protected]> Co-authored-by: apoorvsadana <[email protected]> Co-authored-by: mohiiit <[email protected]>

* refactor: rust_log filtering * hardcoding localstack sha version on e2e

* fix: atlantic added in clap, get status code updated and snos branch updated as well * fix: atlantic tests and url update * update: starknet-os version * verification values udpated for proving * verification logic changed * dockerfile updated * atlantic query response updated * nack->ack and max 10 blocks in update state * fix: aws event bridge setup * update: delete storage setup * update: fixed serialization of trigger message * update: fix worker trigger for aws * update: re-fix CI issue * random commit to re-trigger CI * update: added more metric * update: Bucket Region from ENV * update: reworked location constraint --------- Co-authored-by: mohiiit <[email protected]> Co-authored-by: apoorvsadana <[email protected]>

update: remove dynamic entries from attributes

* update: readme and example env

* fix worker enabled check --------- Co-authored-by: Heemank Verma <[email protected]>

* update: added verification status to verify job * update: added verification time * update: JOB_METADATA_PROCESSING_COMPLETED_AT

* update: added operation_job_status

* chore: readme udpated: * chore: readme updated * chore(readme): re-added a few sections that were missed earlier * chore(readme): linting * chore: linting and formatting * chore: linting and formatting * chore: linting and formatting * chore: script added for dummy contract deployment, readme udpated to use dummy sharp service * update: aws_event_bridge rework * update: e2e * chore: linting-formatting and readme updated * chore: readme misinformation fixed * update: verification job timeout and polling changed to work for 25 hours * chore: comment added for older python version needed * refactor: event bridge type added * chore: linting and formatting of toml files * update: time delay for txn finality in state update increased to 30 mins --------- Co-authored-by: mohiiit <[email protected]> Co-authored-by: Heemank Verma <[email protected]>

* update: fixed block_state collection

* refactor: error handling added for the storage functions * chore: removed expect * chore: changelog * refactor: http-client refactored to handle panic better --------- Co-authored-by: mohiiit <[email protected]>

Trantorian1

Given we are moving to a monorepo, I feel like we need to rethink what files we have at the root of the project. We probably want to move any configuration files and READMEs to either crates/madara or crates/orchestrator and keep only common logic at the root.

We also need to think about how we want to run the CI for each project so that this doesn't take for ever on simple changes.

Trantorian1 · 2025-01-22T08:23:11Z

.env.example

Shouldn't this be moved to crates/orchestrator if it is related to that?

Trantorian1 · 2025-01-22T08:23:38Z

.env.test

Trantorian1 · 2025-01-22T08:24:44Z

pathfinder

Why is this here?

Trantorian1 · 2025-01-22T08:25:38Z

scripts/artifacts/README.md

Are we sure we want to strore artefacts under scripts?

Trantorian1 · 2025-01-22T08:29:31Z

.github/workflows/rust-build.yml

We probably want to figure out a way to compile only SNOS or Madara if we change only one of those no? At least I feel like testing Madara against SNOS should be a separate and last step in the CI if all other steps pass. And make it so that if changes are made to SNOS we do not run e2e on Madara.

Trantorian1 · 2025-01-22T08:30:09Z

.github/workflows/e2e-test.yml

I feel like we should tests Madara and SNOS in different CI actions

Trantorian1 · 2025-01-22T08:33:38Z

bootstrap_input.json

Imo this should also be under crates/ortchestrator. Given we are making this a monorepo we probably also want to move some Madara-specific files such as the README under crates/madara

Trantorian1 · 2025-01-22T08:36:17Z

Cargo.toml

@@ -139,7 +153,7 @@ mc-devnet = { path = "crates/madara/client/devnet" }
 m-cairo-test-contracts = { path = "crates/madara/cairo-test-contracts" }

 # Starknet dependencies
-cairo-vm = "=1.0.1"
+cairo-vm = { git = "https://github.com/Moonsong-Labs/cairo-vm", branch = "madara-monorepo" }


Instead of using forks I feel like we should be importing two separate versions of the same crate under an alias so that madara and the orchestrator can both have their own version. This feels more maintainable long term as far as supporting new features is concerned, and should also make it easier to merge both into a single dependency in the future.

apoorvsadana and others added 30 commits March 5, 2024 10:56

Initial commit

5490788

init

e320c80

add workflows

3e9b41c

cliipy fix

06e7964

more clippy fixes

6e14ef0

more clippy fixes

c81d279

remove .idea

1ee102b

fix ci

9062d21

add .DS_store to ignore

b04bfdd

remove coverage

eaf571f

Merge pull request #1 from karnotxyz/init

eea4f83

Init

change the external_id field to be an enum instead of a regular string

4a948da

Merge pull request #3 from nils-mathieu/free_external_id

b420e1d

Change the external_id field to be an enum instead of a regular String

refactor to workspace

1768f47

taplo

5dd26f8

fix typo

119e5bb

Merge pull request #4 from karnotxyz/orchestrator_refactor

3fe3733

refactor to workspace

initial commit for review

5051bec

move tests to dev dependencies

9dcd974

set up modules for testing each crate

e8eaeda

add previous unti tests

8c4a854

restructure common fixtures and helper function, add rstest cases

0c34cbd

write tests for config, and try out da, database and starknetclient

fb815f7

fix database test, and write test to check config init multiple times

f6c4c10

write job tests, create new fixtures

5fa41a8

write da_job tests, process_job remaining

f6c73c8

implement mocks for config

99b6f9d

mock starknet_client

8b3874e

Try out more approaches for mock_client

b2ca7e9

temp commit only for starknet-provider

52ec7fe

Mohiiit and others added 22 commits November 20, 2024 21:36

Refactor: RUST_LOG filtering (#185)

4b0fce4

* refactor: rust_log filtering * hardcoding localstack sha version on e2e

Fix/telemetry (#189)

f79c7e1

update: remove dynamic entries from attributes

update: readme and example env (#191)

d970d22

* update: readme and example env

fix worker enabled check (#187)

c2e00d1

* fix worker enabled check --------- Co-authored-by: Heemank Verma <[email protected]>

Fix/instrumentations (#192)

9e67649

* update: added verification status to verify job * update: added verification time * update: JOB_METADATA_PROCESSING_COMPLETED_AT

update: added operation_job_status (#195)

07c38a3

* update: added operation_job_status

added: grafana dashboard (#196)

302542a

Fix/instrumentation (#200)

15b321b

* update: fixed block_state collection

Expect removed from various places (#197)

cb8d82f

* refactor: error handling added for the storage functions * chore: removed expect * chore: changelog * refactor: http-client refactored to handle panic better --------- Co-authored-by: mohiiit <[email protected]>

fixed: instrumentation (#203)

3ec2873

Merge madara-orchestrator history into this repo

b9f0947

Move orchestrator crates into crates/orchestrator/

8e6bfe2

Incremental: merge orchestrator Cargo.toml into ours

8c7fb53

Use nightly rust compiler channel to work around SNOS issue

59df910

Incremental: bump Cargo.lock

3d453a2

Use consistent zaun

45b1a75

Bump Cargo.lock

1e2eede

Update dependencies to use git instead of local paths

b7c683f

Magic Cargo.lock

0d7df60

Trantorian1 assigned notlesh Jan 22, 2025

Trantorian1 requested changes Jan 22, 2025

View reviewed changes

Trantorian1 added docs Documentation changes or improvements feature Request for new feature or enhancement infrastructure CI/CD, deployment and infrastructure changes labels Jan 22, 2025

Trantorian1 requested review from HermanObst, antiyro and jbcaron January 22, 2025 08:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore: Merge Orchestrator history #471

chore: Merge Orchestrator history #471

notlesh commented Jan 21, 2025

Trantorian1 left a comment

Trantorian1 Jan 22, 2025

Trantorian1 Jan 22, 2025

Trantorian1 Jan 22, 2025

Trantorian1 Jan 22, 2025

Trantorian1 Jan 22, 2025 •

edited

Loading

Trantorian1 Jan 22, 2025

Trantorian1 Jan 22, 2025

Trantorian1 Jan 22, 2025

chore: Merge Orchestrator history #471

Are you sure you want to change the base?

chore: Merge Orchestrator history #471

Conversation

notlesh commented Jan 21, 2025

Status 🚧

Repository forks 🍴

Trantorian1 left a comment

Choose a reason for hiding this comment

Trantorian1 Jan 22, 2025

Choose a reason for hiding this comment

Trantorian1 Jan 22, 2025

Choose a reason for hiding this comment

Trantorian1 Jan 22, 2025

Choose a reason for hiding this comment

Trantorian1 Jan 22, 2025

Choose a reason for hiding this comment

Trantorian1 Jan 22, 2025 • edited Loading

Choose a reason for hiding this comment

Trantorian1 Jan 22, 2025

Choose a reason for hiding this comment

Trantorian1 Jan 22, 2025

Choose a reason for hiding this comment

Trantorian1 Jan 22, 2025

Choose a reason for hiding this comment

Trantorian1 Jan 22, 2025 •

edited

Loading