Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: Merge Orchestrator history #471

Open
wants to merge 140 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
140 commits
Select commit Hold shift + click to select a range
5490788
Initial commit
apoorvsadana Mar 5, 2024
e320c80
init
apoorvsadana Mar 7, 2024
3e9b41c
add workflows
apoorvsadana Mar 8, 2024
06e7964
cliipy fix
apoorvsadana Mar 8, 2024
6e14ef0
more clippy fixes
apoorvsadana Mar 8, 2024
c81d279
more clippy fixes
apoorvsadana Mar 8, 2024
1ee102b
remove .idea
apoorvsadana Mar 8, 2024
9062d21
fix ci
apoorvsadana Mar 8, 2024
b04bfdd
add .DS_store to ignore
apoorvsadana Mar 8, 2024
eaf571f
remove coverage
apoorvsadana Mar 8, 2024
eea4f83
Merge pull request #1 from karnotxyz/init
apoorvsadana Mar 16, 2024
4a948da
change the external_id field to be an enum instead of a regular string
nils-mathieu Mar 16, 2024
b420e1d
Merge pull request #3 from nils-mathieu/free_external_id
apoorvsadana Mar 26, 2024
1768f47
refactor to workspace
apoorvsadana Apr 5, 2024
5dd26f8
taplo
apoorvsadana Apr 5, 2024
119e5bb
fix typo
apoorvsadana Apr 8, 2024
3fe3733
Merge pull request #4 from karnotxyz/orchestrator_refactor
apoorvsadana Apr 8, 2024
5051bec
initial commit for review
AryanGodara Apr 15, 2024
9dcd974
move tests to dev dependencies
AryanGodara Apr 15, 2024
e8eaeda
set up modules for testing each crate
AryanGodara Apr 15, 2024
8c4a854
add previous unti tests
AryanGodara Apr 15, 2024
0c34cbd
restructure common fixtures and helper function, add rstest cases
AryanGodara Apr 17, 2024
fb815f7
write tests for config, and try out da, database and starknetclient
AryanGodara Apr 17, 2024
f6c4c10
fix database test, and write test to check config init multiple times
AryanGodara Apr 17, 2024
5fa41a8
write job tests, create new fixtures
AryanGodara Apr 17, 2024
f6c73c8
write da_job tests, process_job remaining
AryanGodara Apr 17, 2024
99b6f9d
implement mocks for config
AryanGodara Apr 18, 2024
8b3874e
mock starknet_client
AryanGodara Apr 18, 2024
b2ca7e9
Try out more approaches for mock_client
AryanGodara Apr 18, 2024
52ec7fe
temp commit only for starknet-provider
AryanGodara May 3, 2024
3af2334
temp commit: add mock implementation
AryanGodara May 3, 2024
267fff6
fix tests
apoorvsadana May 24, 2024
602eb9a
fix clippy
apoorvsadana May 24, 2024
8384cb0
fix toml
apoorvsadana May 24, 2024
41bc907
add coverage to workflow
apoorvsadana May 24, 2024
859bacf
prettier
apoorvsadana May 24, 2024
0f3c6c3
fmt
apoorvsadana May 24, 2024
7a311cc
prettier
apoorvsadana May 24, 2024
c2cd2df
fix workflow and make config fields private
apoorvsadana May 24, 2024
b8a9748
Merge pull request #5 from AryanGodara/e2e-tests
apoorvsadana May 24, 2024
4e4b401
new docs design
apoorvsadana Jun 1, 2024
ef7790b
fix markdown lint
apoorvsadana Jun 1, 2024
9d5cdd7
Merge pull request #8 from karnotxyz/docs_new_flow
apoorvsadana Jun 1, 2024
2a9d77b
boilerplate code (#9)
apoorvsadana Jun 1, 2024
6174f9d
Orthograph error (#14)
L0GYKAL Jun 14, 2024
4343a41
Publish State Diff to Eth (#7)
Mohiiit Jun 17, 2024
b625f90
feat : added snos worker implementation and unit tests (#16)
ocdbytes Jun 17, 2024
cb74280
Prover service trait, SHARP client, fact checker, Stone stubs (#6)
unstark Jun 19, 2024
5c96c8e
Feature: Implement proving worker (#22)
ocdbytes Jun 24, 2024
bd7854c
ci: add .deepsource.toml
deepsource-io[bot] Jun 25, 2024
a342034
ci: update .deepsource.toml
deepsource-io[bot] Jun 26, 2024
a4e6cba
feat : added aws s3 adapter and data storage trait (#27)
ocdbytes Jun 27, 2024
00fc6f5
feat: state update job (#23)
akhercha Jul 2, 2024
33a12b5
feat : added state update worker (#36)
ocdbytes Jul 17, 2024
7baf190
fix: change snos crate repo (#33)
0x5ea000000 Jul 18, 2024
7f53cf5
Build Client inside DaConfig (#46)
heemankv Jul 20, 2024
f1f1e4c
Chore/changelog mgmt (#49)
heemankv Jul 22, 2024
6803b91
feat : kzg proof calculation during state update & state update trans…
ocdbytes Jul 22, 2024
d3e57f9
chore: rust toolchain (#57)
heemankv Jul 30, 2024
43db46e
feat: Data Submission Worker Integration. (#51)
heemankv Aug 3, 2024
c155210
Feat : add testing fixtures and a basic test (#52)
ocdbytes Aug 7, 2024
b905890
Feat : job handler tests (#67)
ocdbytes Aug 7, 2024
36e7890
Tests/data availability job (#74)
heemankv Aug 8, 2024
9e2369d
move mongodb serde behind feature flag (#80)
apoorvsadana Aug 12, 2024
371d08d
Test : state update job (#72)
ocdbytes Aug 12, 2024
73355ca
Feature: data letter queue (#73)
heemankv Aug 13, 2024
cfa43b0
add coveralls (#77)
apoorvsadana Aug 13, 2024
5cfe98e
Feat : added cherry picked changes from database tests branch (#82)
ocdbytes Aug 13, 2024
59d50ad
Fix : fixed the worker logic and updated the tests (#76)
ocdbytes Aug 14, 2024
7e771e7
Feature : JobError using thisError (#78)
heemankv Aug 16, 2024
ae8fc95
Feat : removed init_config function from tests (#87)
ocdbytes Aug 22, 2024
fa59cdf
update: Introduced Rust Cache (#93)
heemankv Aug 22, 2024
7926588
Tests : proving client tests (#85)
ocdbytes Aug 22, 2024
5728df2
feat : added worker queues and listeners to consume the message (#92)
ocdbytes Aug 23, 2024
11dc132
Test: Ethereum Settlement Client (#81)
heemankv Aug 23, 2024
0ad9bfb
show logs (#100)
apoorvsadana Aug 24, 2024
77bf9c6
refactor aws and clean .env files (#98)
apoorvsadana Aug 26, 2024
f84ba41
Feat : alerts module (#95)
ocdbytes Aug 30, 2024
21ec2cb
update: better alert impl. (#111)
heemankv Sep 5, 2024
c3f0a50
Revert "update: better alert impl. (#111)" (#113)
heemankv Sep 5, 2024
f3e20b6
workspace oriented dependencies (#115)
heemankv Sep 6, 2024
145df8e
Fix : generate merkle root function during fact calculation (#103)
ocdbytes Sep 6, 2024
6bd19cb
Feat : Database/timestamp (#109)
ocdbytes Sep 9, 2024
764c4fa
Enhancement : Refactor Settings (#106)
ocdbytes Sep 9, 2024
369ccbe
Tests : e2e test (#101)
ocdbytes Sep 9, 2024
bbcf7b0
Feature: Drilled Config (#107)
heemankv Sep 11, 2024
05e8acb
Refactor : da/ethereum client (#121)
ocdbytes Sep 11, 2024
ab66b83
update: SharpClient shifted to new_with_settings from new (#132)
heemankv Sep 28, 2024
7b85453
Instrumentation using OTEL (#131)
heemankv Oct 5, 2024
60768cc
move starknet settlement to zaun (#125)
byteZorvin Oct 5, 2024
705b119
snos refactor (#136)
apoorvsadana Oct 5, 2024
f65e11c
update: simplified otel logic (#143)
heemankv Oct 9, 2024
86b8d9a
feat : added docker build and updated docker compose (#141)
apoorvsadana Oct 9, 2024
ba2aeb8
update snos to latest commit (#139)
apoorvsadana Oct 9, 2024
63e4fa1
Betterment/update job (#134)
heemankv Oct 9, 2024
5922af6
refactor env (#145)
apoorvsadana Oct 9, 2024
ccb213b
handle new jobs in new tokio tasks (#150)
apoorvsadana Oct 10, 2024
b77250d
Mongo db migrations js (#149)
apoorvsadana Oct 10, 2024
e5727c5
handle jobs as tasks (#151)
apoorvsadana Oct 11, 2024
1c090fb
Adding Logs (#147)
heemankv Oct 11, 2024
f5d986d
remove all unwraps (#153)
byteZorvin Oct 11, 2024
5fb2de6
atomic create job (#152)
apoorvsadana Oct 11, 2024
64b375d
move failed jobs to the failed state (#159)
apoorvsadana Oct 15, 2024
1b21d4d
Add multiple queue based on job type (#158)
byteZorvin Oct 16, 2024
82024c7
snos release patches (#162)
apoorvsadana Oct 18, 2024
e7a51d9
enable RUST_LOG=<module_name>=<level> (#163)
heemankv Oct 21, 2024
b58fb2b
Endpoints to trigger Process and Verify Jobs (#164)
heemankv Oct 28, 2024
e14ee33
fix: otel (#167)
heemankv Oct 28, 2024
85aacf6
added makefile and fixes for testnet launch (#165)
apoorvsadana Nov 2, 2024
ce5219d
Fix l1 bridge upgrade (#172)
apoorvsadana Nov 4, 2024
2c4c047
fix: indexing for get_jobs_without_successor (#175)
heemankv Nov 4, 2024
296c7a9
update: otel refactor (#174)
heemankv Nov 4, 2024
4d61e3a
Fix : create job error logs (#173)
ocdbytes Nov 4, 2024
ac131a5
Refactor: Makefile (#177)
Mohiiit Nov 6, 2024
803be5f
Fix : snos orch blob data (#176)
ocdbytes Nov 6, 2024
c66c0f3
Add panic recovery handling for process_job with test coverage (#178)
Mohiiit Nov 9, 2024
a9d0e30
Feat : setup scripts (#181)
ocdbytes Nov 12, 2024
58cca2c
Feature/clap (#179)
heemankv Nov 20, 2024
2ab2291
Integration of Atlantic Service and HTTP Client Infrastructure (#182)
Mohiiit Nov 20, 2024
4b0fce4
Refactor: RUST_LOG filtering (#185)
heemankv Nov 28, 2024
9d5ace8
Fix/test aws (#186)
heemankv Dec 10, 2024
f79c7e1
Fix/telemetry (#189)
heemankv Dec 10, 2024
d970d22
update: readme and example env (#191)
heemankv Dec 10, 2024
c2e00d1
fix worker enabled check (#187)
apoorvsadana Dec 10, 2024
9e67649
Fix/instrumentations (#192)
heemankv Dec 15, 2024
07c38a3
update: added operation_job_status (#195)
heemankv Dec 19, 2024
302542a
added: grafana dashboard (#196)
heemankv Dec 23, 2024
a008642
Readme Updated (#193)
Mohiiit Dec 24, 2024
15b321b
Fix/instrumentation (#200)
heemankv Dec 27, 2024
cb8d82f
Expect removed from various places (#197)
Mohiiit Jan 7, 2025
3ec2873
fixed: instrumentation (#203)
heemankv Jan 7, 2025
b9f0947
Merge madara-orchestrator history into this repo
notlesh Jan 13, 2025
8e6bfe2
Move orchestrator crates into crates/orchestrator/
notlesh Jan 14, 2025
8c7fb53
Incremental: merge orchestrator Cargo.toml into ours
notlesh Jan 14, 2025
59df910
Use nightly rust compiler channel to work around SNOS issue
notlesh Jan 14, 2025
3d453a2
Incremental: bump Cargo.lock
notlesh Jan 14, 2025
45b1a75
Use consistent zaun
notlesh Jan 21, 2025
1e2eede
Bump Cargo.lock
notlesh Jan 21, 2025
b7c683f
Update dependencies to use git instead of local paths
notlesh Jan 21, 2025
0d7df60
Magic Cargo.lock
notlesh Jan 21, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
89 changes: 89 additions & 0 deletions .env.example
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't this be moved to crates/orchestrator if it is related to that?

Original file line number Diff line number Diff line change
@@ -0,0 +1,89 @@
#### AWS CONFIG ####
AWS_ACCESS_KEY_ID= # Your AWS access key ID
AWS_SECRET_ACCESS_KEY= # Your AWS secret access key
AWS_REGION= # AWS region
# For AWS SDK
AWS_ENDPOINT_URL= # AWS endpoint URL
# For Omniqueue
AWS_DEFAULT_REGION= # AWS default region

# For EventBridge
MADARA_ORCHESTRATOR_EVENT_BRIDGE_TYPE= # EventBridge type (rule/schedule)
MADARA_ORCHESTRATOR_EVENT_BRIDGE_TRIGGER_RULE_NAME= # EventBridge rule name
MADARA_ORCHESTRATOR_EVENT_BRIDGE_TRIGGER_ROLE_NAME= # EventBridge role name
MADARA_ORCHESTRATOR_EVENT_BRIDGE_TRIGGER_POLICY_NAME= # EventBridge policy name

#### ALERTS ####
MADARA_ORCHESTRATOR_AWS_SNS_ARN= # SNS ARN for alerts

#### DATA AVAILABILITY ####
## ETHEREUM ##
MADARA_ORCHESTRATOR_ETHEREUM_DA_RPC_URL= # Ethereum RPC URL for data availability

#### DATABASE ####
## MONGODB ##
MADARA_ORCHESTRATOR_MONGODB_CONNECTION_URL= # MongoDB connection URL
MADARA_ORCHESTRATOR_DATABASE_NAME= # MongoDB database name

#### PROVER ####
## SHARP ##
MADARA_ORCHESTRATOR_SHARP_CUSTOMER_ID= # SHARP customer ID
MADARA_ORCHESTRATOR_SHARP_URL= # SHARP service URL
MADARA_ORCHESTRATOR_SHARP_USER_CRT= # SHARP user certificate
MADARA_ORCHESTRATOR_SHARP_USER_KEY= # SHARP user private key
MADARA_ORCHESTRATOR_SHARP_SERVER_CRT= # SHARP server certificate
MADARA_ORCHESTRATOR_SHARP_RPC_NODE_URL= # SHARP RPC node URL
MADARA_ORCHESTRATOR_SHARP_PROOF_LAYOUT= # SHARP proof layout
MADARA_ORCHESTRATOR_GPS_VERIFIER_CONTRACT_ADDRESS= # GPS verifier contract address

## ATLANTIC ##
MADARA_ORCHESTRATOR_ATLANTIC_API_KEY= # Atlantic API key
MADARA_ORCHESTRATOR_ATLANTIC_SERVICE_URL= # Atlantic service URL
MADARA_ORCHESTRATOR_ATLANTIC_MOCK_FACT_HASH= # Whether to use mock fact registry (true/false)
MADARA_ORCHESTRATOR_ATLANTIC_PROVER_TYPE= # Prover type (herodotus/starkware)
MADARA_ORCHESTRATOR_ATLANTIC_SETTLEMENT_LAYER= # Settlement layer (ethereum/starknet)
MADARA_ORCHESTRATOR_ATLANTIC_VERIFIER_CONTRACT_ADDRESS= # Atlantic verifier contract address
MADARA_ORCHESTRATOR_ATLANTIC_RPC_NODE_URL= # Atlantic RPC node URL

#### QUEUE ####
## AWS SQS ##
MADARA_ORCHESTRATOR_SQS_PREFIX= # SQS queue prefix
MADARA_ORCHESTRATOR_SQS_SUFFIX= # SQS queue suffix
MADARA_ORCHESTRATOR_EVENT_BRIDGE_TARGET_QUEUE_NAME= # EventBridge target queue name
MADARA_ORCHESTRATOR_SQS_BASE_QUEUE_URL= # SQS base queue URL

#### SETTLEMENT ####
## ETHEREUM ##
MADARA_ORCHESTRATOR_ETHEREUM_SETTLEMENT_RPC_URL= # Ethereum settlement RPC URL
MADARA_ORCHESTRATOR_ETHEREUM_PRIVATE_KEY= # Ethereum private key
MADARA_ORCHESTRATOR_L1_CORE_CONTRACT_ADDRESS= # L1 core contract address
MADARA_ORCHESTRATOR_STARKNET_OPERATOR_ADDRESS= # Starknet operator address

## STARKNET ##
MADARA_ORCHESTRATOR_STARKNET_SETTLEMENT_RPC_URL= # Starknet settlement RPC URL
MADARA_ORCHESTRATOR_STARKNET_PRIVATE_KEY= # Starknet private key
MADARA_ORCHESTRATOR_STARKNET_ACCOUNT_ADDRESS= # Starknet account address
MADARA_ORCHESTRATOR_STARKNET_CAIRO_CORE_CONTRACT_ADDRESS= # Starknet Cairo core contract address
MADARA_ORCHESTRATOR_STARKNET_FINALITY_RETRY_WAIT_IN_SECS= # Retry wait time in seconds
# MADARA_ORCHESTRATOR_MADARA_BINARY_PATH= # Optional: Madara binary path

#### STORAGE ####
## AWS S3 ##
MADARA_ORCHESTRATOR_AWS_S3_BUCKET_NAME= # S3 bucket name

#### INSTRUMENTATION ####
## OTEL ##
MADARA_ORCHESTRATOR_OTEL_SERVICE_NAME= # OpenTelemetry service name
MADARA_ORCHESTRATOR_OTEL_COLLECTOR_ENDPOINT= # OpenTelemetry collector endpoint

#### SERVER ####
MADARA_ORCHESTRATOR_HOST= # Server host
MADARA_ORCHESTRATOR_PORT= # Server port

#### SERVICE ####
MADARA_ORCHESTRATOR_MAX_BLOCK_NO_TO_PROCESS= # Maximum block number to process (optional)
MADARA_ORCHESTRATOR_MIN_BLOCK_NO_TO_PROCESS= # Minimum block number to process (optional)
MADARA_ORCHESTRATOR_MADARA_RPC_URL= # Madara RPC URL

#### SNOS ####
MADARA_ORCHESTRATOR_RPC_FOR_SNOS= # SNOS RPC URL
117 changes: 117 additions & 0 deletions .env.test
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here

Original file line number Diff line number Diff line change
@@ -0,0 +1,117 @@

#### AWS CONFIG ####

AWS_ACCESS_KEY_ID=AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY=AWS_SECRET_ACCESS_KEY
AWS_REGION=us-east-1
# For Aws sdk
AWS_ENDPOINT_URL=http://localhost.localstack.cloud:4566
# For Omniqueue
AWS_DEFAULT_REGION=localhost

# For EventBridge
MADARA_ORCHESTRATOR_EVENT_BRIDGE_TYPE=rule
MADARA_ORCHESTRATOR_EVENT_BRIDGE_TRIGGER_RULE_NAME=madara-orchestrator-worker-trigger
MADARA_ORCHESTRATOR_EVENT_BRIDGE_TRIGGER_ROLE_NAME=madara-orchestrator-worker-trigger-role
MADARA_ORCHESTRATOR_EVENT_BRIDGE_TRIGGER_POLICY_NAME=madara-orchestrator-worker-trigger-policy



#### ALERTS ####

MADARA_ORCHESTRATOR_AWS_SNS_ARN=arn:aws:sns:us-east-1:000000000000:madara-orchestrator-arn


#### DATA AVAILABILITY ####

## ETHEREUM ##

MADARA_ORCHESTRATOR_ETHEREUM_DA_RPC_URL=https://eth-sepolia.public.blastapi.io


#### DATABASE ####

## MONGODB ##

MADARA_ORCHESTRATOR_MONGODB_CONNECTION_URL=mongodb://localhost:27017
MADARA_ORCHESTRATOR_DATABASE_NAME=orchestrator

#### PROVER ####

## SHARP ##

MADARA_ORCHESTRATOR_SHARP_CUSTOMER_ID=sharp_consumer_id
MADARA_ORCHESTRATOR_SHARP_URL=http://127.0.0.1:6000
MADARA_ORCHESTRATOR_SHARP_USER_CRT=LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUR4ekNDQXErZ0F3SUJBZ0lVTjBSK0xpb1MzL2ZadUZsK291RjZNNFk2RnRZd0RRWUpLb1pJaHZjTkFRRUwKQlFBd2N6RUxNQWtHQTFVRUJoTUNTVTR4RXpBUkJnTlZCQWdNQ2xOdmJXVXRVM1JoZEdVeElUQWZCZ05WQkFvTQpHRWx1ZEdWeWJtVjBJRmRwWkdkcGRITWdVSFI1SUV4MFpERU5NQXNHQTFVRUF3d0VVMVJTU3pFZE1Cc0dDU3FHClNJYjNEUUVKQVJZT1lXSmpRR3RoY201dmRDNTRlWG93SGhjTk1qUXdPREV6TVRNd05UTTBXaGNOTWpVd09ERXoKTVRNd05UTTBXakJ6TVFzd0NRWURWUVFHRXdKSlRqRVRNQkVHQTFVRUNBd0tVMjl0WlMxVGRHRjBaVEVoTUI4RwpBMVVFQ2d3WVNXNTBaWEp1WlhRZ1YybGtaMmwwY3lCUWRIa2dUSFJrTVEwd0N3WURWUVFEREFSVFZGSkxNUjB3Ckd3WUpLb1pJaHZjTkFRa0JGZzVoWW1OQWEyRnlibTkwTG5oNWVqQ0NBU0l3RFFZSktvWklodmNOQVFFQkJRQUQKZ2dFUEFEQ0NBUW9DZ2dFQkFOSEtaUGRqWSs4QWo4ZFV2V0xReEl5NTNrK1BHY001T2FlYnpTV3FER0xGSlBOdgpkVzJvWjFCSnNEb2hobWZFSCt5ZEFoQXEvbzc4NDljblg2VDJTOVhta25wdnNud2dRckU5Z3lqSmV3MUxBRzNHCm10U0lOMWJJSm9peWJ3QUR5NGxPd0xrVzUzdFdueHBSazVVVmZUU1hLYVRRTnlHd2o3Q2xMSGthcnlZYVk3OVkKOXlHMFJ2RkFkb1IzczBveWthNkFLV0d1WjhOdWd4NTY2bysyWllRenJteWVNU1NGYkhNdW1aUkxYb0hpazhBSgpLZXJ0bnNBRC9LMVJRYm80Y21ubHFoTVRhQktiTEFVVjVteFVvMlpveFBJVU9tREE5N3IyMmRTYkRkRlVjeC9kCjhQcDB6VXNycXdQckJlcW5SMXdLOE80MUlHajUzRnUzVmxDeS94MENBd0VBQWFOVE1GRXdIUVlEVlIwT0JCWUUKRkc0T0lvKzcvckJyZlR4S2FFMGx2L1dwRDJ3UE1COEdBMVVkSXdRWU1CYUFGRzRPSW8rNy9yQnJmVHhLYUUwbAp2L1dwRDJ3UE1BOEdBMVVkRXdFQi93UUZNQU1CQWY4d0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFEMURDZkR3CnpoSXRGMWd5YVdhWURZRHErZjJSUHBFRWVaWk1BSDdJV0ZTajRrTzhmVHN1RnN6bFoyNXNlR3ZHYW4xQ3F4alQKYnJ3MXliVlJQeGZMUWgxRlZMMGhFeDZWYXhGditxMmtqUmlCQmZURFBxWGxYcmpaaUYrZTNPS3lKSVhnNkpIUAppbVpBV0dyRFBHNkorQi90bHRaQ3VLZVhLK1FUcnRSOVVCL29hOWVaQWc5RXNkOVJsZDRNeVo5b0NtdUNPU1hmCnk1THFkVlgrNENpTnJXQ3BwM1B2M2MyL28rZ0RMQjUzZ252R056RjR6Q1FIZ0RtN0RNZnpmZlY1TUMwV1MvWXkKVnpyUG11Sys0Y0tSK3dMOFZITVNEeC9ybTFhYnh0dEN2VW92MUw5dVZ1QUNGc29yNmdsR0N1RDNNQ0dIa0pNNgpxaS8rM1haeHhxeGw1Rzg9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
MADARA_ORCHESTRATOR_SHARP_USER_KEY=LS0tLS1CRUdJTiBQUklWQVRFIEtFWS0tLS0tCk1JSUV2UUlCQURBTkJna3Foa2lHOXcwQkFRRUZBQVNDQktjd2dnU2pBZ0VBQW9JQkFRRFJ5bVQzWTJQdkFJL0gKVkwxaTBNU011ZDVQanhuRE9UbW5tODBscWd4aXhTVHpiM1Z0cUdkUVNiQTZJWVpueEIvc25RSVFLdjZPL09QWApKMStrOWt2VjVwSjZiN0o4SUVLeFBZTW95WHNOU3dCdHhwclVpRGRXeUNhSXNtOEFBOHVKVHNDNUZ1ZDdWcDhhClVaT1ZGWDAwbHltazBEY2hzSSt3cFN4NUdxOG1HbU8vV1BjaHRFYnhRSGFFZDdOS01wR3VnQ2xocm1mRGJvTWUKZXVxUHRtV0VNNjVzbmpFa2hXeHpMcG1VUzE2QjRwUEFDU25xN1o3QUEveXRVVUc2T0hKcDVhb1RFMmdTbXl3RgpGZVpzVktObWFNVHlGRHBnd1BlNjl0blVtdzNSVkhNZjNmRDZkTTFMSzZzRDZ3WHFwMGRjQ3ZEdU5TQm8rZHhiCnQxWlFzdjhkQWdNQkFBRUNnZ0VBQU9mcDFiT2xLOVFKeXVlUHhjeDIvTkNVcUMxTEJDL01FdkEyUzVKWGFWbkcKbGhLR0pFb1U0Q0RoVk83dUlLYVZLTFZvMjk4RHFHUnBLM1d0RVE1TE40bytXYTcveTA5c1drMlVzbWxrVWFOZwpSaGtVZEJSK2dLNXVsQ3FKRml2dUJoTEQvRWlnQ1VWUGZKS2JtNG96TnpYcjVSMU5ENlV1aWFtODdtenlFcTBLCmZsVXlhR0RZNGdIdFNBOVBENVBFYlUveFpKeitKaHk5T2l3aVRXV0MrSHoyb2c3UWRDRDE2RlhGcit2VHpQN0MKb2tFb0VDZFNPRWlMalVENjBhS2ZxRmFCVm5MTkVudC9QSytmY1RBM05mNGtSMnFDNk9ZWjVFb09zYm1ka29ZTgpyU3NJZW9XblMxOEhvekZud2w3Z05wTUtjNmRzQzRBTldOVDFsTkhCb1FLQmdRRHlaUDFJSlppZUh6NlExaUVTCm5zd2tnblZCQUQ0SlVLR1ZDMHA3dk4yclNDZXh4c05ZZXFPTEEyZGZCUGpOVjd3blFKcUgxT05XellOMUJVSUUKeThLTCtFZVl6Q3RZa21LL21wSGJIMzNjd2tJODBuMHJROU1BalZMTlJ2YVVEOWp1NFBsRzFqaEFZUVVyTkViZQpKRlVpSk83aDVQa1llZG50SitqSHFpQnRoUUtCZ1FEZGtPbndmL0szYk4xenR0bXZQd0VicjhkVWJjRVh5NDFOCkl5VWwrZW1WSlgzYktKM0duNDZnQ2RsTTdkYmpwS3JVZ3oxL2JsZTgvMkVFckJvSEFRNkMrU2pEaGhvL01CbnIKekZheTBoK3YxbjBnZnNNVzRoOEF4cEFwc25OYnh6K2g1Wm5uSnRTd0srUjB3U0VJVVEzRjAxL2hMWWhLQ2l5OApwbW5HQi9hU3VRS0JnRzdxd1cvVExGd214ZlYyMXBsenFzeUdHZXVObGRXalhOMGIxcEI2b3lDdW11TmhwYUFHCk5uSDFNOGNxT2tPVWd4ZWZHMWRPbGx6eEc5ZGZlWTlDUWhyVW1NYVZucndmK0NuZkxDRU43d1VtcXpLenl1MFMKVXlwc2dOaElRYXNNK1dLTjllTnhRVHBNYXhZVERONjMxM0VSWDNKazJZdFdydDh6cFBSQXFDZ1ZBb0dCQU54egpUa0NMbmJ6aFphbTNlZm9DenlCMEVma3dSdHBkSGxkc3E0NlFqTmRuK1VSd3NpTXBLR2lWeEE3bDZsU1B4NlV3CmU2VHA3Z1JQZUlHRWwxVDJ1VENacGZSODNtcVdlb1FCeVJXZE9nZmplcFkxYWZpL3ZhY3c2Y21ERTRKeXloNVUKYTMveFE5ZVJwSHFDbWxKREMxZ1V5eVlwL3B2a2FjUytNeW5sVEhHSkFvR0FQekdTSzdXOHBUYldSVEFoaTVrSQpwZk5kWk1tcnRodUxNT3F6TGhyRjZublpldk9OdTBoYXVhZktlVElFd2w0clhYZHFKQlJBaWZKMFFsLzZKWFFkCmd1VzFrZWk1Ui8rUFZ5eUhab042c3NXSTNWYklwUUloUmt6UENnTDZhbHEwSzFpT1dlV1lIOHdORGRRdlB1T2UKRkZPOEovSzNxV0NtWjU0ODBBbTNhT0U9Ci0tLS0tRU5EIFBSSVZBVEUgS0VZLS0tLS0K
MADARA_ORCHESTRATOR_SHARP_SERVER_CRT=LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURhekNDQWxPZ0F3SUJBZ0lVRUR0Rjd0YVNTUnVQQTJ6Uk1aNWNzY2JCRm5jd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1JURUxNQWtHQTFVRUJoTUNTVTR4RXpBUkJnTlZCQWdNQ2xOdmJXVXRVM1JoZEdVeElUQWZCZ05WQkFvTQpHRWx1ZEdWeWJtVjBJRmRwWkdkcGRITWdVSFI1SUV4MFpEQWVGdzB5TkRBNE1UTXhNekEzTVROYUZ3MHlOVEE0Ck1UTXhNekEzTVROYU1FVXhDekFKQmdOVkJBWVRBa2xPTVJNd0VRWURWUVFJREFwVGIyMWxMVk4wWVhSbE1TRXcKSHdZRFZRUUtEQmhKYm5SbGNtNWxkQ0JYYVdSbmFYUnpJRkIwZVNCTWRHUXdnZ0VpTUEwR0NTcUdTSWIzRFFFQgpBUVVBQTRJQkR3QXdnZ0VLQW9JQkFRRFRHcEEwNEZ1QlNFaE5PNVYvMGxTaDkvSEgxeVRZT2dRVFdoOG43eDlRCnZGMHpvZFZueVFIdjE5elU5eVdia2xvOEkvOXFBVm9lRzdXTnpUVFg2Q295ZlNjb1YvazN0Q2UwVnVWMlFJTVQKdW82SzJSU05CVHB1TlNqNTlzUiszVTQ2OFRBQnY0YVpsYjU4TU5CRXM3MVRieVpLRHBGRVRkMkg3T0ZKajg4QQpNRi9MaXJkeDZPOFdZL0tDeisxd1ZXL1JRdytYYjRJSWx4bXJFOC9UZ3FNSEo4dFUxYkZiOWJNcTEvOTN5YWtJClU1V2J2NVhXKzFwZFVyTUFNcTFFaC9vZThMN2pFaFdvZXZrNzgyU0kwUk0xeG5MaEtrUUVBYXd6Zkg2ODZiR2YKUHQ3RkFIQ1pGaWJ4KzZzSkg0R1M3S25iK0x5bk9ud3phMWZPUXZEZmcvRm5BZ01CQUFHalV6QlJNQjBHQTFVZApEZ1FXQkJUYlFUdmlUTW1xNXlNK2ZJRVI4VjdTZk1pK3B6QWZCZ05WSFNNRUdEQVdnQlRiUVR2aVRNbXE1eU0rCmZJRVI4VjdTZk1pK3B6QVBCZ05WSFJNQkFmOEVCVEFEQVFIL01BMEdDU3FHU0liM0RRRUJDd1VBQTRJQkFRREYKTllyRnpBa2RIVkhjUkd5SUNsTi9IVGswaldOcTVSdTB1RUpDQ21Dbm9ZY1pRSTlDQlcwTkl3dGpZUkpTKzR1UwordWh4VWpSYTA5YXdOWDhvYmU0dDZjK25HRnhZMGZqamk0cGZnbU1kMWNJeGdsM3E3Nlp0ZkllRGR6alRLRXN1CjRFUTVadnEwMnJvTEZ0ZjEvL3dRVG0xNkNKdFpGWnhNZ1phYnNxc2JRc3M2dWdMUGtTTmdBWjI1L2VhcWhnQ20KTjFUV2FxL0xJMVBLSkxPK085NFlMa2FsNVpyOTJCOXk4Q0VKVUVuSTA1R1N1MmJUOFM2a0ZBMEpadEszTW9SbwpqRWZWV1lQVHR5TFR4amNvRndCcDlHaXZYSDdSdHBxMDlmSmFhU1pNekxmNGlyNHpBdXprbExBNWZvampPNXlKCllnYlVaQUU2aS81N1NFWjR3VmxTCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
# MADARA_ORCHESTRATOR_SHARP_RPC_NODE_URL should be same as MADARA_ORCHESTRATOR_ETHEREUM_SETTLEMENT_RPC_URL
MADARA_ORCHESTRATOR_SHARP_RPC_NODE_URL=http://127.0.0.1:8545
MADARA_ORCHESTRATOR_SHARP_PROOF_LAYOUT=small
MADARA_ORCHESTRATOR_GPS_VERIFIER_CONTRACT_ADDRESS=0x07ec0D28e50322Eb0C159B9090ecF3aeA8346DFe

## ATLANTIC ##

MADARA_ORCHESTRATOR_ATLANTIC_API_KEY="API-KEY"
MADARA_ORCHESTRATOR_ATLANTIC_SERVICE_URL="https://atlantic.api.herodotus.cloud"
MADARA_ORCHESTRATOR_ATLANTIC_MOCK_FACT_HASH="false" # Whether to use mock fact registry
MADARA_ORCHESTRATOR_ATLANTIC_PROVER_TYPE="herodotus" # ("herodotus" | "starkware")
MADARA_ORCHESTRATOR_ATLANTIC_SETTLEMENT_LAYER="ethereum" # ("ethereum" | "starknet")
MADARA_ORCHESTRATOR_ATLANTIC_VERIFIER_CONTRACT_ADDRESS=0x07ec0D28e50322Eb0C159B9090ecF3aeA8346DFe
MADARA_ORCHESTRATOR_ATLANTIC_RPC_NODE_URL=http://127.0.0.1:8545
#### QUEUE ####

## AWS SQS ##

MADARA_ORCHESTRATOR_SQS_PREFIX=madara_orchestrator
MADARA_ORCHESTRATOR_SQS_SUFFIX=queue
MADARA_ORCHESTRATOR_EVENT_BRIDGE_TARGET_QUEUE_NAME=madara_orchestrator_worker_trigger_queue
MADARA_ORCHESTRATOR_SQS_BASE_QUEUE_URL=http://sqs.us-east-1.localhost.localstack.cloud:4566/000000000000

#### SETTLEMENT ####

## ETHEREUM ##

MADARA_ORCHESTRATOR_ETHEREUM_SETTLEMENT_RPC_URL=https://eth-sepolia.public.blastapi.io
MADARA_ORCHESTRATOR_ETHEREUM_PRIVATE_KEY=0xac0974bec39a17e36ba4a6b4d238ff944bacb478cbed5efcae784d7bf4f2ff80
MADARA_ORCHESTRATOR_L1_CORE_CONTRACT_ADDRESS=0xE2Bb56ee936fd6433DC0F6e7e3b8365C906AA057
MADARA_ORCHESTRATOR_STARKNET_OPERATOR_ADDRESS=0x5b98B836969A60FEC50Fa925905Dd1D382a7db43

## STARKNET ##

MADARA_ORCHESTRATOR_STARKNET_SETTLEMENT_RPC_URL=https://starknet-sepolia.public.blastapi.io
MADARA_ORCHESTRATOR_STARKNET_PRIVATE_KEY=0x76f2ccdb23f29bc7b69278e947c01c6160a31cf02c19d06d0f6e5ab1d768b86
MADARA_ORCHESTRATOR_STARKNET_ACCOUNT_ADDRESS=0x3bb306a004034dba19e6cf7b161e7a4fef64bc1078419e8ad1876192f0b8cd1
MADARA_ORCHESTRATOR_STARKNET_CAIRO_CORE_CONTRACT_ADDRESS=
MADARA_ORCHESTRATOR_STARKNET_FINALITY_RETRY_WAIT_IN_SECS=10
# MADARA_ORCHESTRATOR_MADARA_BINARY_PATH=

#### STORAGE ####

## AWS S3 ##

MADARA_ORCHESTRATOR_AWS_S3_BUCKET_NAME=madara-orchestrator-test-bucket


#### INSTRUMENTATION ####

## OTEL ##

MADARA_ORCHESTRATOR_OTEL_SERVICE_NAME=orchestrator

#### SERVER ####

MADARA_ORCHESTRATOR_HOST=127.0.0.1
MADARA_ORCHESTRATOR_PORT=3000


#### SERVICE ####

MADARA_ORCHESTRATOR_MAX_BLOCK_NO_TO_PROCESS=
MADARA_ORCHESTRATOR_MIN_BLOCK_NO_TO_PROCESS=
MADARA_ORCHESTRATOR_MADARA_RPC_URL=http://81.16.176.130:9545


#### SNOS ####

MADARA_ORCHESTRATOR_RPC_FOR_SNOS=http://81.16.176.130:9545
13 changes: 13 additions & 0 deletions .github/workflows/changelog.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
name: "Changelog Workflow"
on:
pull_request:
types: [opened, synchronize, reopened, ready_for_review, labeled, unlabeled]
branches: [main]

jobs:
# Enforces the update of a changelog file on every pull request
# skipLabel default is "Skip-Changelog"
changelog:
runs-on: ubuntu-latest
steps:
- uses: dangoslen/changelog-enforcer@v3
92 changes: 92 additions & 0 deletions .github/workflows/e2e-test.yml
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel like we should tests Madara and SNOS in different CI actions

Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
name: E2E test

on:
workflow_call:
secrets:
RPC_FOR_SNOS:
required: true

jobs:
coverage:
runs-on: karnot-arc-runner-set

services:
localstack:
image: localstack/localstack@sha256:763947722c6c8d33d5fbf7e8d52b4bddec5be35274a0998fdc6176d733375314

env:
DEFAULT_REGION: us-east-1
AWS_ACCESS_KEY_ID: "AWS_ACCESS_KEY_ID"
AWS_SECRET_ACCESS_KEY: "AWS_SECRET_ACCESS_KEY"
ports:
- 4566:4566
mongodb:
image: mongo:latest
ports:
- 27017:27017

steps:
- uses: actions/checkout@v3

- name: Install necessary dependencies
run: |
sudo apt update
sudo apt-get install -y clang llvm libudev-dev protobuf-compiler gcc g++ build-essential libssl-dev pkg-config curl wget git libclang-dev
- name: Install Rust toolchain
uses: actions-rs/toolchain@v1
with:
profile: minimal
toolchain: nightly

- name: Rust Cache
uses: Swatinem/rust-cache@v2

- name: Install Foundry
uses: foundry-rs/foundry-toolchain@v1

- name: Set up Python 3.9
uses: actions/setup-python@v2
with:
python-version: 3.9

- name: Display Python version
run: python --version

- name: Install cairo 0
run: |
# Creating venv
python3.9 -m venv ~/cairo_venv
source ~/cairo_venv/bin/activate
# Installing deps
sudo apt install -y libgmp3-dev
pip3 install ecdsa fastecdsa sympy
# Installing cairo lang
pip3 install cairo-lang
- name: Check Anvil Installation & run Anvil
run: |
if command -v anvil &> /dev/null
then
echo "Anvil is installed. Version information:"
anvil --version
anvil --block-time 1 &
else
echo "Anvil is not installed or not in PATH"
exit 1
fi
- name: Build SNOS files
run: |
make snos
- name: Run e2e test
env:
MADARA_ORCHESTRATOR_ETHEREUM_SETTLEMENT_RPC_URL: ${{ secrets.ETHEREUM_SEPOLIA_BLAST_RPC }}
MADARA_ORCHESTRATOR_RPC_FOR_SNOS: ${{ secrets.RPC_FOR_SNOS }}
# the self hosted runner has a different region so we override it here
AWS_REGION: us-east-1
run: |
RUST_LOG=info cargo test --features testing test_orchestrator_workflow -- --nocapture
45 changes: 45 additions & 0 deletions .github/workflows/rust-build.yml
Copy link
Collaborator

@Trantorian1 Trantorian1 Jan 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We probably want to figure out a way to compile only SNOS or Madara if we change only one of those no? At least I feel like testing Madara against SNOS should be a separate and last step in the CI if all other steps pass. And make it so that if changes are made to SNOS we do not run e2e on Madara.

Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
---
name: Task - Build Rust

on:
workflow_dispatch:
workflow_call:

jobs:
rust_build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3

- run: rustup show

- uses: Swatinem/rust-cache@v2

- name: Set up Python 3.9
uses: actions/setup-python@v2
with:
python-version: 3.9

- name: Display Python version
run: python --version

- name: Install cairo 0
run: |
# Creating venv
python3.9 -m venv ~/cairo_venv
source ~/cairo_venv/bin/activate
# Installing deps
sudo apt install -y libgmp3-dev
pip3 install ecdsa fastecdsa sympy
# Installing cairo lang
pip3 install cairo-lang
- name: Build SNOS files
run: |
make snos
- name: Build the project
run: |
cargo build --release --workspace
15 changes: 15 additions & 0 deletions .gitmodules
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
[submodule "cairo-lang"]
path = cairo-lang
url = https://github.com/starkware-libs/cairo-lang.git
ignore = all
[submodule "madara"]
path = madara
url = https://github.com/madara-alliance/madara.git
ignore = all
[submodule "madara-bootstrapper"]
path = madara-bootstrapper
url = https://github.com/madara-alliance/madara-bootstrapper.git
ignore = all
[submodule "pathfinder"]
path = pathfinder
url = https://github.com/eqlabs/pathfinder.git
Loading