[DON'T MERGE] Feat/oracle v5 devnet

README.md

@@ -3,7 +3,8 @@
 [![License: GPL v3](https://img.shields.io/badge/License-GPLv3-blue.svg)](https://www.gnu.org/licenses/gpl-3.0)
 [![Tests](https://github.com/lidofinance/lido-oracle/workflows/Tests/badge.svg?branch=daemon_v2)](https://github.com/lidofinance/lido-oracle/actions)
 
-Oracle daemon for Lido decentralized staking service: Monitoring the state of the protocol across both layers and submitting regular update reports to the Lido smart contracts.
+Oracle daemon for Lido decentralized staking service: Monitoring the state of the protocol across both layers and submitting regular update
+reports to the Lido smart contracts.
 
 ## How it works
 
@@ -15,7 +16,8 @@ There are 3 modules in the oracle:
 
 ### Accounting module
 
-Accounting module updates the protocol TVL, distributes node-operator rewards, updates information about the number of exited and stuck validators and processes user withdrawal requests.
+Accounting module updates the protocol TVL, distributes node-operator rewards, updates information about the number of exited and stuck
+validators and processes user withdrawal requests.
 Also Accounting module makes decision to turn on/off the bunker.
 
 **Flow**
@@ -37,7 +39,8 @@ The frame includes these stages:
 
 ### Ejector module
 
-Ejector module requests Lido validators to eject via events in Execution Layer when the protocol requires additional funds to process user withdrawals.
+Ejector module requests Lido validators to eject via events in Execution Layer when the protocol requires additional funds to process user
+withdrawals.
 
 **Flow**
 
@@ -58,14 +61,16 @@ Only Oracle:
 - Memory - 8 GB
 
 Oracle + KAPI:
+
 - vCPU - 4
 - Memory - 16 GB
 
 ## Dependencies
 
 ### Execution Client Node
 
-To prepare the report, Oracle fetches up to 10 days old events, makes historical requests for balance data and makes simulated reports on historical blocks. This requires an [archive](https://ethereum.org/en/developers/docs/nodes-and-clients/#archive-node) execution node.
+To prepare the report, Oracle fetches up to 10 days old events, makes historical requests for balance data and makes simulated reports on
+historical blocks. This requires an [archive](https://ethereum.org/en/developers/docs/nodes-and-clients/#archive-node) execution node.
 Oracle needs two weeks of archived data.
 
 | Client                                          | Tested | Notes                                                                                                                                                                           |
@@ -77,19 +82,21 @@ Oracle needs two weeks of archived data.
 
 ### Consensus Client Node
 
-Also, to calculate some metrics for bunker mode Oracle needs [archive](https://ethereum.org/en/developers/docs/nodes-and-clients/#archive-node) consensus node.
+Also, to calculate some metrics for bunker mode Oracle
+needs [archive](https://ethereum.org/en/developers/docs/nodes-and-clients/#archive-node) consensus node.
 
-| Client                                            | Tested | Notes                                                                                                                                           |
-|---------------------------------------------------|:------:|-------------------------------------------------------------------------------------------------------------------------------------------------|
-| [Lighthouse](https://lighthouse.sigmaprime.io/)   |   🟢   | Use `--reconstruct-historic-states` param                                                                                                       |
-| [Lodestar](https://lodestar.chainsafe.io)         |   🔴   | Not tested yet                                                                                                                                  |
-| [Nimbus](https://nimbus.team)                     |   🔴   | Not tested yet                                                                                                                                  |
-| [Prysm](https://github.com/prysmaticlabs/prysm)   |   🟢   | Use <br> `--grpc-max-msg-size=104857600` <br> `--enable-historical-state-representation=true` <br> `--slots-per-archive-point=1024` <br> params |
-| [Teku](https://docs.teku.consensys.net)           |   🟢   | Use <br> `--data-storage-mode=archive` <br>`--data-storage-archive-frequency=1024`<br> `--reconstruct-historic-states=true`<br> params          |
+| Client                                          | Tested | Notes                                                                                                                                           |
+|-------------------------------------------------|:------:|-------------------------------------------------------------------------------------------------------------------------------------------------|
+| [Lighthouse](https://lighthouse.sigmaprime.io/) |   🟢   | Use `--reconstruct-historic-states` param                                                                                                       |
+| [Lodestar](https://lodestar.chainsafe.io)       |   🔴   | Not tested yet                                                                                                                                  |
+| [Nimbus](https://nimbus.team)                   |   🔴   | Not tested yet                                                                                                                                  |
+| [Prysm](https://github.com/prysmaticlabs/prysm) |   🟢   | Use <br> `--grpc-max-msg-size=104857600` <br> `--enable-historical-state-representation=true` <br> `--slots-per-archive-point=1024` <br> params |
+| [Teku](https://docs.teku.consensys.net)         |   🟢   | Use <br> `--data-storage-mode=archive` <br>`--data-storage-archive-frequency=1024`<br> `--reconstruct-historic-states=true`<br> params          |
 
 ### Keys API Service
 
-This is a separate service that uses Consensus and Execution Clients to fetch all lido keys. It stores the latest state of lido keys in database.
+This is a separate service that uses Consensus and Execution Clients to fetch all lido keys. It stores the latest state of lido keys in
+database.
 
 [Lido Keys API repository.](https://github.com/lidofinance/lido-keys-api)
 
@@ -102,9 +109,11 @@ Pull the image using the following command:
 docker pull lidofinance/oracle:{tag}
 ```
 
-Where `{tag}` is a version of the image. You can find the latest version in the [releases](https://github.com/lidofinance/lido-oracle/releases)
+Where `{tag}` is a version of the image. You can find the latest version in
+the [releases](https://github.com/lidofinance/lido-oracle/releases)
 **OR**\
-You can build it locally using the following command (make sure build it from latest [release](https://github.com/lidofinance/lido-oracle/releases)):
+You can build it locally using the following command (make sure build it from
+latest [release](https://github.com/lidofinance/lido-oracle/releases)):
 
 ```bash
 docker build -t lidofinance/oracle .
@@ -124,16 +133,17 @@ Full variables list could be found [here](https://github.com/lidofinance/lido-or
    and your environment is ready to run the oracle.
 
 ## Run the oracle
+
 1. By default, the oracle runs in *dry mode*. It means that it will not send any transactions to the Ethereum network.
-    To run Oracle in *production mode*, set `MEMBER_PRIV_KEY` or `MEMBER_PRIV_KEY_FILE` environment variable:
+   To run Oracle in *production mode*, set `MEMBER_PRIV_KEY` or `MEMBER_PRIV_KEY_FILE` environment variable:
     ```
     MEMBER_PRIV_KEY={value}
     ```
-    Where `{value}` is a private key of the Oracle member account or:
+   Where `{value}` is a private key of the Oracle member account or:
     ```
     MEMBER_PRIV_KEY_FILE={path}
     ```
-    Where `{path}` is a path to the private key of the Oracle member account.
+   Where `{path}` is a path to the private key of the Oracle member account.
 2. Run the container using the following command:
 
    ```bash
@@ -203,6 +213,7 @@ In manual mode all sleeps are disabled and `ALLOW_REPORTING_IN_BUNKER_MODE` is T
 | `CACHE_PATH`                                           | Directory to store cache for CSM module                                                                                                                                  | False    | `.`                                     |
 
 ### Mainnet variables
+
 > LIDO_LOCATOR_ADDRESS=0xC1d0b3DE6792Bf6b4b37EccdcC24e45978Cfd2Eb
 > ALLOW_REPORTING_IN_BUNKER_MODE=False
 
@@ -281,9 +292,12 @@ Special metrics for ejector oracle:
 
 Special metrics for CSM oracle:
 
-| Metric name                       | Description                                 | Labels |
-|-----------------------------------|---------------------------------------------|--------|
-| TBD                               | TBD                                         |        |
+| Metric name                     | Description                            | Labels |
+|---------------------------------|----------------------------------------|--------|
+| csm_current_frame_range_l_epoch | Left epoch of the current frame range  |        |
+| csm_current_frame_range_r_epoch | Right epoch of the current frame range |        |
+| csm_unprocessed_epochs_count    | Unprocessed epochs count               |        |
+| csm_min_unprocessed_epoch       | Minimum unprocessed epoch              |        |
 
 # Development
 

docker-compose.devnet.yml

@@ -0,0 +1,169 @@
+version: '3.9'
+
+networks:
+  devnet:
+    name: ${DOCKER_NETWORK_NAME}
+    external: true
+
+services:
+
+  lido-oracle-accounting:
+    image: lidofinance/oracle:4.1.1
+    restart: unless-stopped
+    depends_on:
+      - keys_api
+    networks:
+      - devnet
+    deploy:
+      resources:
+        limits:
+          cpus: "1"
+          memory: 4G
+    labels:
+      - "prometheus-job=lido-oracle-accounting"
+      - "prometheus-endpoint=/"
+      - "prometheus-port=9000"
+    environment:
+      - "PROMETHEUS_PORT=9000"
+      - "EXECUTION_CLIENT_URI=${EXECUTION_CLIENT_URI}"
+      - "CONSENSUS_CLIENT_URI=${CONSENSUS_CLIENT_URI}"
+      - "KEYS_API_URI=http://keys_api:9030"
+      - "MEMBER_PRIV_KEY=${MEMBER_PRIV_KEY_1}"
+      - "LIDO_LOCATOR_ADDRESS=${LIDO_LOCATOR_ADDRESS}"
+      - "ALLOW_REPORTING_IN_BUNKER_MODE=false"
+    command: accounting
+
+  lido-oracle-ejector:
+    image: lidofinance/oracle:4.1.1
+    restart: unless-stopped
+    depends_on:
+      - keys_api
+    networks:
+      - devnet
+    deploy:
+      resources:
+        limits:
+          cpus: "1"
+          memory: 4G
+    labels:
+      - "prometheus-job=lido-oracle-ejector"
+      - "prometheus-endpoint=/"
+      - "prometheus-port=9000"
+    environment:
+      - "PROMETHEUS_PORT=9000"
+      - "EXECUTION_CLIENT_URI=${EXECUTION_CLIENT_URI}"
+      - "CONSENSUS_CLIENT_URI=${CONSENSUS_CLIENT_URI}"
+      - "KEYS_API_URI=http://keys_api:9030"
+      - "MEMBER_PRIV_KEY=${MEMBER_PRIV_KEY_1}"
+      - "LIDO_LOCATOR_ADDRESS=${LIDO_LOCATOR_ADDRESS}"
+    command: ejector
+
+  lido-oracle-csm:
+    image: lidofinance/oracle:4.1.1
+    restart: unless-stopped
+    depends_on:
+      - keys_api
+    networks:
+      - devnet
+    deploy:
+      resources:
+        limits:
+          cpus: "2"
+          memory: 4G
+    labels:
+      - "prometheus-job=lido-oracle-ejector"
+      - "prometheus-endpoint=/"
+      - "prometheus-port=9000"
+    environment:
+      - "PROMETHEUS_PORT=9000"
+      - "EXECUTION_CLIENT_URI=${EXECUTION_CLIENT_URI}"
+      - "CONSENSUS_CLIENT_URI=${CONSENSUS_CLIENT_URI}"
+      - "KEYS_API_URI=http://keys_api:9030"
+      - "MEMBER_PRIV_KEY=${MEMBER_PRIV_KEY_1}"
+      - "LIDO_LOCATOR_ADDRESS=${LIDO_LOCATOR_ADDRESS}"
+      - "CSM_MODULE_ADDRESS=${CSM_MODULE_ADDRESS}"
+      - "PINATA_JWT=${PINATA_JWT}"
+      - "GW3_ACCESS_KEY=${GW3_ACCESS_KEY}"
+      - "GW3_SECRET_KEY=${GW3_SECRET_KEY}"
+    command: csm
+
+
+  lido-oracle-accounting-v5:
+    build: ./
+    restart: unless-stopped
+    depends_on:
+      - keys_api
+    networks:
+      - devnet
+    deploy:
+      resources:
+        limits:
+          cpus: "1"
+          memory: 4G
+    labels:
+      - "prometheus-job=lido-oracle-accounting"
+      - "prometheus-endpoint=/"
+      - "prometheus-port=9000"
+    environment:
+      - "PROMETHEUS_PORT=9000"
+      - "EXECUTION_CLIENT_URI=${EXECUTION_CLIENT_URI}"
+      - "CONSENSUS_CLIENT_URI=${CONSENSUS_CLIENT_URI}"
+      - "KEYS_API_URI=http://keys_api:9030"
+      - "MEMBER_PRIV_KEY=${MEMBER_PRIV_KEY_2}"
+      - "LIDO_LOCATOR_ADDRESS=${LIDO_LOCATOR_ADDRESS}"
+      - "ALLOW_REPORTING_IN_BUNKER_MODE=false"
+    command: accounting
+
+  lido-oracle-ejector-v5:
+    build: ./
+    restart: unless-stopped
+    depends_on:
+      - keys_api
+    networks:
+      - devnet
+    deploy:
+      resources:
+        limits:
+          cpus: "1"
+          memory: 4G
+    labels:
+      - "prometheus-job=lido-oracle-ejector"
+      - "prometheus-endpoint=/"
+      - "prometheus-port=9000"
+    environment:
+      - "PROMETHEUS_PORT=9000"
+      - "EXECUTION_CLIENT_URI=${EXECUTION_CLIENT_URI}"
+      - "CONSENSUS_CLIENT_URI=${CONSENSUS_CLIENT_URI}"
+      - "KEYS_API_URI=http://keys_api:9030"
+      - "MEMBER_PRIV_KEY=${MEMBER_PRIV_KEY_2}"
+      - "LIDO_LOCATOR_ADDRESS=${LIDO_LOCATOR_ADDRESS}"
+    command: ejector
+
+  lido-oracle-csm-v5:
+    build: ./
+    restart: unless-stopped
+    depends_on:
+      - keys_api
+    networks:
+      - devnet
+    deploy:
+      resources:
+        limits:
+          cpus: "2"
+          memory: 4G
+    labels:
+      - "prometheus-job=lido-oracle-ejector"
+      - "prometheus-endpoint=/"
+      - "prometheus-port=9000"
+    environment:
+      - "PROMETHEUS_PORT=9000"
+      - "EXECUTION_CLIENT_URI=${EXECUTION_CLIENT_URI}"
+      - "CONSENSUS_CLIENT_URI=${CONSENSUS_CLIENT_URI}"
+      - "KEYS_API_URI=http://keys_api:9030"
+      - "MEMBER_PRIV_KEY=${MEMBER_PRIV_KEY_2}"
+      - "LIDO_LOCATOR_ADDRESS=${LIDO_LOCATOR_ADDRESS}"
+      - "CSM_MODULE_ADDRESS=${CSM_MODULE_ADDRESS}"
+      - "PINATA_JWT=${PINATA_JWT}"
+      - "GW3_ACCESS_KEY=${GW3_ACCESS_KEY}"
+      - "GW3_SECRET_KEY=${GW3_SECRET_KEY}"
+    command: csm

fixtures/tests/modules/accounting/test_withdrawal_integration.py/test_happy_path.json

@@ -6,7 +6,7 @@
         "to": "0x8D49f1b4AF30598679D4D37Be4B094da1b459b82",
         "data": "0xa3a3fd5d"
       },
-      "0x8ee61584b9d3e010c55f1fa77a803051f5f783385ec75b4e3fc71e199a86184d"
+      "latest"
     ],
     "response": {
       "jsonrpc": "2.0",
@@ -195,7 +195,7 @@
         "to": "0x4c1F6cA213abdbc19b27f2562d7b1A645A019bD9",
         "data": "0x29fd065d"
       },
-      "0x8ee61584b9d3e010c55f1fa77a803051f5f783385ec75b4e3fc71e199a86184d"
+      "latest"
     ],
     "response": {
       "jsonrpc": "2.0",
@@ -218,4 +218,4 @@
       "result": "0x0000000000000000000000000000000000000000000000037d3047cdfd698705000000000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000030af00000000000000000000000000000000000000000000000000000000000030c000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000002"
     }
   }
-]
+]

src/constants.py

@@ -1,5 +1,7 @@
+from src.types import Gwei
+
 # https://github.com/ethereum/consensus-specs/blob/dev/specs/phase0/beacon-chain.md#misc
-FAR_FUTURE_EPOCH = 2 ** 64 - 1
+FAR_FUTURE_EPOCH = 2**64 - 1
 # https://github.com/ethereum/consensus-specs/blob/dev/specs/phase0/beacon-chain.md#time-parameters-1
 MIN_VALIDATOR_WITHDRAWABILITY_DELAY = 2**8
 SHARD_COMMITTEE_PERIOD = 256
@@ -10,22 +12,31 @@
 PROPORTIONAL_SLASHING_MULTIPLIER_BELLATRIX = 3
 # https://github.com/ethereum/consensus-specs/blob/dev/specs/phase0/beacon-chain.md#gwei-values
 EFFECTIVE_BALANCE_INCREMENT = 2 ** 0 * 10 ** 9
-MAX_EFFECTIVE_BALANCE = 32 * 10 ** 9
+MAX_EFFECTIVE_BALANCE = Gwei(32 * 10 ** 9)
 # https://github.com/ethereum/consensus-specs/blob/dev/specs/capella/beacon-chain.md#execution
-MAX_WITHDRAWALS_PER_PAYLOAD = 2 ** 4
+MAX_WITHDRAWALS_PER_PAYLOAD = 2**4
 # https://github.com/ethereum/consensus-specs/blob/dev/specs/phase0/beacon-chain.md#withdrawal-prefixes
 ETH1_ADDRESS_WITHDRAWAL_PREFIX = '0x01'
+# https://github.com/ethereum/consensus-specs/blob/dev/specs/phase0/beacon-chain.md#withdrawal-prefixes
+COMPOUNDING_WITHDRAWAL_PREFIX = '0x02'
 # https://github.com/ethereum/consensus-specs/blob/dev/specs/phase0/beacon-chain.md#validator-cycle
-MIN_PER_EPOCH_CHURN_LIMIT = 2 ** 2
-CHURN_LIMIT_QUOTIENT = 2 ** 16
+MIN_PER_EPOCH_CHURN_LIMIT = 2**2
+CHURN_LIMIT_QUOTIENT = 2**16
+# https://github.com/ethereum/consensus-specs/blob/dev/specs/electra/beacon-chain.md#validator-cycle
+MIN_PER_EPOCH_CHURN_LIMIT_ELECTRA = Gwei(2**7 * 10**9)
+MAX_PER_EPOCH_ACTIVATION_EXIT_CHURN_LIMIT = Gwei(2**8 * 10**9)
 # https://github.com/ethereum/consensus-specs/blob/dev/specs/phase0/beacon-chain.md#time-parameters
 SLOTS_PER_HISTORICAL_ROOT = 8192
 
+# https://github.com/ethereum/consensus-specs/blob/dev/specs/electra/beacon-chain.md#gwei-values
+MIN_ACTIVATION_BALANCE = Gwei(2**5 * 10**9)
+MAX_EFFECTIVE_BALANCE_ELECTRA = Gwei(2**11 * 10**9)
+
 # Local constants
-GWEI_TO_WEI = 10 ** 9
-SHARE_RATE_PRECISION_E27 = 10 ** 27
+GWEI_TO_WEI = 10**9
+SHARE_RATE_PRECISION_E27 = 10**27
 TOTAL_BASIS_POINTS = 10000
 
 MAX_BLOCK_GAS_LIMIT = 30_000_000
 
-UINT64_MAX = 2 ** 64 - 1
+UINT64_MAX = 2**64 - 1