Merge branch 'next' into feat/etherscan-failover

.vscode/launch.json

@@ -193,38 +193,6 @@
         "DIPDUP_DEBUG": "1"
       }
     },
-    {
-      "name": "demo_tezos_big_maps: run",
-      "type": "debugpy",
-      "request": "launch",
-      "module": "dipdup",
-      "args": [
-        "-e",
-        ".env",
-        "run"
-      ],
-      "console": "integratedTerminal",
-      "cwd": "${workspaceFolder}/src/demo_tezos_big_maps",
-      "env": {
-        "DIPDUP_DEBUG": "1"
-      }
-    },
-    {
-      "name": "demo_tezos_big_maps: init",
-      "type": "debugpy",
-      "request": "launch",
-      "module": "dipdup",
-      "args": [
-        "-e",
-        ".env",
-        "init"
-      ],
-      "console": "integratedTerminal",
-      "cwd": "${workspaceFolder}/src/demo_tezos_big_maps",
-      "env": {
-        "DIPDUP_DEBUG": "1"
-      }
-    },
     {
       "name": "demo_tezos_dao: run",
       "type": "debugpy",

CHANGELOG.md

@@ -11,6 +11,9 @@ Releases prior to 7.0 has been removed from this file to declutter search result
 ### Added
 
 - abi.etherscan: Try to extract ABI from webpage when API call fails.
+- database: support database migrations using [`aerich`](https://github.com/tortoise/aerich)
+- cli: Added `schema` subcommands to manage database migrations: `migrate`, `upgrade`, `downgrade`, `heads` and `history`
+- hasura: Added `ignore` and `ignore_internal` config options to hide specific tables/views.
 
 ## [8.0.0] - 2024-09-10
 

docs/1.getting-started/5.database.md

@@ -16,6 +16,7 @@ DipDup officially supports SQLite, PostgreSQL and TimescaleDB as a database engi
 | SQL scripts          | 🟢                | 🟢            | 🟢                      |
 | Immune tables        | ⚠️                | 🟢            | 🟢                      |
 | Hasura integration   | 🔴                | 🟢            | 🟢                      |
+| Migrations           | 🔴                | 🟢            | 🟢                      |
 
 By default, DipDup uses an in-memory SQLite database that is destroyed after the process exits.
 
@@ -124,3 +125,25 @@ For more information visit the official TimescaleDB documentation:
 
 - [Hypertables](https://docs.timescale.com/use-timescale/latest/hypertables/)
 - [Continuous aggregates](https://docs.timescale.com/use-timescale/latest/continuous-aggregates/)
+
+## Migrations
+
+::banner{type="note"}
+The database migrations feature is optional and is disabled by default. To enable it, you need to install `aerich`, which is available in the `[migrations]` optional dependencies group.
+::
+
+DipDup supports database migrations with [aerich](https://github.com/tortoise/aerich), a Tortoise ORM migration tool. The migration files are stored in the `migrations` directory in the project root.
+
+DipDup provides a set of commands to manage databasek migrations:
+
+- `dipdup schema history` to show the migration history
+- `dipdup schema heads` to show the show the current available heads
+- `dipdup schema migrate` to create a new migration changes file
+- `dipdup schema upgrade` to upgrade the database to the latest or a specified version
+- `dipdup schema downgrade` to downgrade the database to a previous version
+
+The migrations directory should be initialized before using these commands, you can run `dipdup schema init` or simply run the indexer with `dipdup run`.
+
+Theses commands invoke `aerich`'s commands under the hood. You can find more information about `aerich` in the [github repo](https://github.com/tortoise/aerich).
+
+Note that you won't be able to directly use `aerich` commands in your project, as DipDup uses its own way of configuring it.

docs/16.thanks.md

@@ -24,6 +24,7 @@ We are grateful to all the people who helped us with the project.
 - [852Kerfunkle](https://github.com/852Kerfunkle)
 - [aguillon](https://github.com/aguillon)
 - [Anshit Bhardwaj](https://github.com/Anshit01)
+- [Anton Rudenko](https://github.com/MrRoudyk)
 - [arrijabba](https://github.com/arrijabba)
 - [Do Quang Huy](https://github.com/huydo2105)
 - [Fitblip](https://github.com/Fitblip)
@@ -42,6 +43,8 @@ We are grateful to all the people who helped us with the project.
 - [Scott Simpson](https://github.com/scottincrypto)
 - [shuoer86](https://github.com/shuoer86)
 - [Simon Bihel](https://github.com/sbihel)
+- [Soham Das](https://github.com/tosoham)
 - [tomsib2001](https://github.com/tomsib2001)
+- [bigherc18](https://github.com/bigherc18)
 
 If we forgot to mention you, or you want to update your record, please, open an issue or pull request.

docs/7.references/1.cli.md

@@ -365,6 +365,44 @@ Discord:               <a class="reference external" href="https://discord.gg/aG
 </section>
 
 
+### downgrade
+
+<p>Downgrade to specified version.</p>
+<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span>dipdup<span class="w"> </span>schema<span class="w"> </span>downgrade<span class="w"> </span><span class="o">[</span>OPTIONS<span class="o">]</span>
+</pre></div>
+</div>
+<p class="rubric">Options</p>
+<dl class="std option">
+<dt class="sig sig-object std" id="cmdoption-dipdup-schema-downgrade-v">
+<span id="cmdoption-dipdup-schema-downgrade-version"></span><span class="sig-name descname"><span class="pre">-v</span></span><span class="sig-prename descclassname"></span><span class="sig-prename descclassname"><span class="pre">,</span> </span><span class="sig-name descname"><span class="pre">--version</span></span><span class="sig-prename descclassname"> <span class="pre">&lt;version&gt;</span></span></dt>
+<dd><p>Specified version, default to last.</p>
+<dl class="field-list simple">
+<dt class="field-odd" style="color: var(--txt-primary);">Default<span class="colon">:</span></dt>
+<dd class="field-odd"><p><code class="docutils literal notranslate"><span class="pre">-1</span></code></p>
+</dd>
+</dl>
+</dd></dl>
+
+<dl class="std option">
+<dt class="sig sig-object std" id="cmdoption-dipdup-schema-downgrade-d">
+<span id="cmdoption-dipdup-schema-downgrade-delete"></span><span class="sig-name descname"><span class="pre">-d</span></span><span class="sig-prename descclassname"></span><span class="sig-prename descclassname"><span class="pre">,</span> </span><span class="sig-name descname"><span class="pre">--delete</span></span><span class="sig-prename descclassname"></span></dt>
+<dd><p>Delete version files at the same time.</p>
+<dl class="field-list simple">
+<dt class="field-odd" style="color: var(--txt-primary);">Default<span class="colon">:</span></dt>
+<dd class="field-odd"><p><code class="docutils literal notranslate"><span class="pre">False</span></code></p>
+</dd>
+</dl>
+</dd></dl>
+
+<dl class="std option">
+<dt class="sig sig-object std" id="cmdoption-dipdup-schema-downgrade-yes">
+<span class="sig-name descname"><span class="pre">--yes</span></span><span class="sig-prename descclassname"></span></dt>
+<dd><p>Confirm the action without prompting.</p>
+</dd></dl>
+
+</section>
+
+
 ### export
 
 <p>Print SQL schema including scripts from <cite>sql/on_reindex</cite>.</p>
@@ -375,6 +413,24 @@ Discord:               <a class="reference external" href="https://discord.gg/aG
 </section>
 
 
+### heads
+
+<p>Show current available heads in migrate location.</p>
+<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span>dipdup<span class="w"> </span>schema<span class="w"> </span>heads<span class="w"> </span><span class="o">[</span>OPTIONS<span class="o">]</span>
+</pre></div>
+</div>
+</section>
+
+
+### history
+
+<p>List all migrate items.</p>
+<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span>dipdup<span class="w"> </span>schema<span class="w"> </span><span class="nb">history</span><span class="w"> </span><span class="o">[</span>OPTIONS<span class="o">]</span>
+</pre></div>
+</div>
+</section>
+
+
 ### init
 
 <p>Prepare database schema for running DipDup.</p>
@@ -385,6 +441,43 @@ Discord:               <a class="reference external" href="https://discord.gg/aG
 </section>
 
 
+### migrate
+
+<p>Generate migrate changes file.</p>
+<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span>dipdup<span class="w"> </span>schema<span class="w"> </span>migrate<span class="w"> </span><span class="o">[</span>OPTIONS<span class="o">]</span>
+</pre></div>
+</div>
+<p class="rubric">Options</p>
+<dl class="std option">
+<dt class="sig sig-object std" id="cmdoption-dipdup-schema-migrate-name">
+<span class="sig-name descname"><span class="pre">--name</span></span><span class="sig-prename descclassname"> <span class="pre">&lt;name&gt;</span></span></dt>
+<dd><p>Migrate name.</p>
+<dl class="field-list simple">
+<dt class="field-odd" style="color: var(--txt-primary);">Default<span class="colon">:</span></dt>
+<dd class="field-odd"><p><code class="docutils literal notranslate"><span class="pre">'update'</span></code></p>
+</dd>
+</dl>
+</dd></dl>
+
+</section>
+
+
+### upgrade
+
+<p>Upgrade to specified version.</p>
+<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span>dipdup<span class="w"> </span>schema<span class="w"> </span>upgrade<span class="w"> </span><span class="o">[</span>OPTIONS<span class="o">]</span>
+</pre></div>
+</div>
+<p class="rubric">Options</p>
+<dl class="std option">
+<dt class="sig sig-object std" id="cmdoption-dipdup-schema-upgrade-i">
+<span id="cmdoption-dipdup-schema-upgrade-in-transaction"></span><span class="sig-name descname"><span class="pre">-i</span></span><span class="sig-prename descclassname"></span><span class="sig-prename descclassname"><span class="pre">,</span> </span><span class="sig-name descname"><span class="pre">--in-transaction</span></span><span class="sig-prename descclassname"> <span class="pre">&lt;in_transaction&gt;</span></span></dt>
+<dd><p>Make migrations in transaction or not. Can be helpful for large migrations or creating concurrent indexes.</p>
+</dd></dl>
+
+</section>
+
+
 ### wipe
 
 <p>Drop all database tables, functions and views.</p>

docs/7.references/2.config.md

@@ -333,7 +333,7 @@ description: "Config file reference"
 
 ## dipdup.config.HasuraConfig
 
-<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">dipdup.config.</span></span><span class="sig-name descname"><span class="pre">HasuraConfig</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">url</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">admin_secret</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">create_source</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">False</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">source</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">'default'</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">select_limit</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">1000</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">allow_aggregations</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">True</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">allow_inconsistent_metadata</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">False</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">camel_case</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">False</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">rest</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">True</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">http</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em><span class="sig-paren">)</span></dt>
+<em class="property"><span class="pre">class</span><span class="w"> </span></em><span class="sig-prename descclassname"><span class="pre">dipdup.config.</span></span><span class="sig-name descname"><span class="pre">HasuraConfig</span></span><span class="sig-paren">(</span><em class="sig-param"><span class="n"><span class="pre">url</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">admin_secret</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">create_source</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">False</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">source</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">'default'</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">select_limit</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">1000</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">allow_aggregations</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">True</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">allow_inconsistent_metadata</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">False</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">camel_case</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">False</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">rest</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">True</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">http</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">ignore_internal</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">False</span></span></em>, <em class="sig-param"><span class="n"><span class="pre">ignore</span></span><span class="o"><span class="pre">=</span></span><span class="default_value"><span class="pre">None</span></span></em><span class="sig-paren">)</span></dt>
 <dd><p>Config for the Hasura integration.</p>
 <dl class="field-list simple">
 <dt class="field-odd" style="color: var(--txt-primary);">Parameters<span class="colon">:</span></dt>
@@ -348,6 +348,8 @@ description: "Config file reference"
 <li><p><strong>camel_case</strong> (<em>bool</em>) – Whether to use camelCase instead of default pascal_case for the field names.</p></li>
 <li><p><strong>rest</strong> (<em>bool</em>) – Enable REST API both for autogenerated and custom queries.</p></li>
 <li><p><strong>http</strong> (<a class="reference internal" href="#dipdupconfighttpconfig" title="dipdup.config.HttpConfig" target="_self"><em>HttpConfig</em></a><em> | </em><em>None</em>) – HTTP connection tunables</p></li>
+<li><p><strong>ignore_internal</strong> (<em>bool</em>) – Whether to ignore internal tables (prefixed with “dipdup”) when generating the GraphQL API.</p></li>
+<li><p><strong>ignore</strong> (<em>list</em><em>[</em><em>str</em><em>]</em>) – List of additional table names to ignore when generating the GraphQL API.</p></li>
 
 </ul>
 </dd>

docs/8.examples/_demos_table.md

@@ -7,7 +7,6 @@
 | demo_evm_uniswap | EVM | Uniswap V3 pools, positions, etc. (advanced, uses TimescaleDB) | [link](https://github.com/dipdup-io/dipdup/tree/8.0.0/src/demo_evm_uniswap) |
 | demo_starknet_events | Starknet | ERC-20 token transfers (from events) | [link](https://github.com/dipdup-io/dipdup/tree/8.0.0/src/demo_starknet_events) |
 | demo_tezos_auction | Tezos | NFT marketplace (TzColors) | [link](https://github.com/dipdup-io/dipdup/tree/8.0.0/src/demo_tezos_auction) |
-| demo_tezos_big_maps | Tezos | Indexing specific big maps | [link](https://github.com/dipdup-io/dipdup/tree/8.0.0/src/demo_tezos_big_maps) |
 | demo_tezos_dao | Tezos | DAO registry (Homebase DAO) | [link](https://github.com/dipdup-io/dipdup/tree/8.0.0/src/demo_tezos_dao) |
 | demo_tezos_dex | Tezos | DEX balances and liquidity (Quipuswap) | [link](https://github.com/dipdup-io/dipdup/tree/8.0.0/src/demo_tezos_dex) |
 | demo_tezos_domains | Tezos | Domain name service (Tezos Domains) | [link](https://github.com/dipdup-io/dipdup/tree/8.0.0/src/demo_tezos_domains) |

docs/9.release-notes/_8.0_changelog.md

@@ -8,13 +8,16 @@
 - cli: Added `--raw` option to `config export` command to dump config preserving the original structure.
 - cli: Added `-C` option, a shorthand for `-c . -c configs/dipdup.<name>.yaml`.
 - cli: Added `package verify` command to check the package consistency.
+- cli: Added `schema` subcommands to manage database migrations: `migrate`, `upgrade`, `downgrade`, `heads` and `history`
 - cli: Added full project migration support for 3.0 spec.
 - cli: Added spec_version 3.0 support to `migrate` command.
 - config: Publish JSON schemas for config validation and autocompletion.
 - database: Added `dipdup_status` view to the schema.
+- database: support database migrations using [`aerich`](https://github.com/tortoise/aerich)
 - env: Added `DIPDUP_JSON_LOG` environment variable to enable JSON logging.
 - env: Added `DIPDUP_LOW_MEMORY` variable to reduce the size of caches and buffers.
 - env: Added `DIPDUP_PACKAGE_PATH` environment variable to override discovered package path.
+- hasura: Added `ignore` and `ignore_internal` config options to hide specific tables/views.
 - package: Added built-in `batch` handler to modify higher-level indexing logic.
 - starknet.events: Added `starknet.events` index kind to process Starknet events.
 - starknet.node: Added Starknet node datasource for last mile indexing.