Merge branch 'next' into feat/substrate

CHANGELOG.md

@@ -10,6 +10,9 @@ Releases prior to 7.0 has been removed from this file to declutter search result
 
 ### Added
 
+- abi.etherscan: Try to extract ABI from webpage when API call fails.
+- database: support database migrations using [`aerich`](https://github.com/tortoise/aerich)
+- cli: Added `schema` subcommands to manage database migrations: `migrate`, `upgrade`, `downgrade`, `heads` and `history`
 - hasura: Added `ignore` and `ignore_internal` config options to hide specific tables/views.
 
 ## [8.0.0] - 2024-09-10

docs/1.getting-started/5.database.md

@@ -16,6 +16,7 @@ DipDup officially supports SQLite, PostgreSQL and TimescaleDB as a database engi
 | SQL scripts          | 🟢                | 🟢            | 🟢                      |
 | Immune tables        | ⚠️                | 🟢            | 🟢                      |
 | Hasura integration   | 🔴                | 🟢            | 🟢                      |
+| Migrations           | 🔴                | 🟢            | 🟢                      |
 
 By default, DipDup uses an in-memory SQLite database that is destroyed after the process exits.
 
@@ -124,3 +125,25 @@ For more information visit the official TimescaleDB documentation:
 
 - [Hypertables](https://docs.timescale.com/use-timescale/latest/hypertables/)
 - [Continuous aggregates](https://docs.timescale.com/use-timescale/latest/continuous-aggregates/)
+
+## Migrations
+
+::banner{type="note"}
+The database migrations feature is optional and is disabled by default. To enable it, you need to install `aerich`, which is available in the `[migrations]` optional dependencies group.
+::
+
+DipDup supports database migrations with [aerich](https://github.com/tortoise/aerich), a Tortoise ORM migration tool. The migration files are stored in the `migrations` directory in the project root.
+
+DipDup provides a set of commands to manage databasek migrations:
+
+- `dipdup schema history` to show the migration history
+- `dipdup schema heads` to show the show the current available heads
+- `dipdup schema migrate` to create a new migration changes file
+- `dipdup schema upgrade` to upgrade the database to the latest or a specified version
+- `dipdup schema downgrade` to downgrade the database to a previous version
+
+The migrations directory should be initialized before using these commands, you can run `dipdup schema init` or simply run the indexer with `dipdup run`.
+
+Theses commands invoke `aerich`'s commands under the hood. You can find more information about `aerich` in the [github repo](https://github.com/tortoise/aerich).
+
+Note that you won't be able to directly use `aerich` commands in your project, as DipDup uses its own way of configuring it.

docs/16.thanks.md

@@ -45,5 +45,6 @@ We are grateful to all the people who helped us with the project.
 - [Simon Bihel](https://github.com/sbihel)
 - [Soham Das](https://github.com/tosoham)
 - [tomsib2001](https://github.com/tomsib2001)
+- [bigherc18](https://github.com/bigherc18)
 
 If we forgot to mention you, or you want to update your record, please, open an issue or pull request.

docs/7.references/1.cli.md

@@ -365,6 +365,44 @@ Discord:               <a class="reference external" href="https://discord.gg/aG
 </section>
 
 
+### downgrade
+
+<p>Downgrade to specified version.</p>
+<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span>dipdup<span class="w"> </span>schema<span class="w"> </span>downgrade<span class="w"> </span><span class="o">[</span>OPTIONS<span class="o">]</span>
+</pre></div>
+</div>
+<p class="rubric">Options</p>
+<dl class="std option">
+<dt class="sig sig-object std" id="cmdoption-dipdup-schema-downgrade-v">
+<span id="cmdoption-dipdup-schema-downgrade-version"></span><span class="sig-name descname"><span class="pre">-v</span></span><span class="sig-prename descclassname"></span><span class="sig-prename descclassname"><span class="pre">,</span> </span><span class="sig-name descname"><span class="pre">--version</span></span><span class="sig-prename descclassname"> <span class="pre">&lt;version&gt;</span></span></dt>
+<dd><p>Specified version, default to last.</p>
+<dl class="field-list simple">
+<dt class="field-odd" style="color: var(--txt-primary);">Default<span class="colon">:</span></dt>
+<dd class="field-odd"><p><code class="docutils literal notranslate"><span class="pre">-1</span></code></p>
+</dd>
+</dl>
+</dd></dl>
+
+<dl class="std option">
+<dt class="sig sig-object std" id="cmdoption-dipdup-schema-downgrade-d">
+<span id="cmdoption-dipdup-schema-downgrade-delete"></span><span class="sig-name descname"><span class="pre">-d</span></span><span class="sig-prename descclassname"></span><span class="sig-prename descclassname"><span class="pre">,</span> </span><span class="sig-name descname"><span class="pre">--delete</span></span><span class="sig-prename descclassname"></span></dt>
+<dd><p>Delete version files at the same time.</p>
+<dl class="field-list simple">
+<dt class="field-odd" style="color: var(--txt-primary);">Default<span class="colon">:</span></dt>
+<dd class="field-odd"><p><code class="docutils literal notranslate"><span class="pre">False</span></code></p>
+</dd>
+</dl>
+</dd></dl>
+
+<dl class="std option">
+<dt class="sig sig-object std" id="cmdoption-dipdup-schema-downgrade-yes">
+<span class="sig-name descname"><span class="pre">--yes</span></span><span class="sig-prename descclassname"></span></dt>
+<dd><p>Confirm the action without prompting.</p>
+</dd></dl>
+
+</section>
+
+
 ### export
 
 <p>Print SQL schema including scripts from <cite>sql/on_reindex</cite>.</p>
@@ -375,6 +413,24 @@ Discord:               <a class="reference external" href="https://discord.gg/aG
 </section>
 
 
+### heads
+
+<p>Show current available heads in migrate location.</p>
+<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span>dipdup<span class="w"> </span>schema<span class="w"> </span>heads<span class="w"> </span><span class="o">[</span>OPTIONS<span class="o">]</span>
+</pre></div>
+</div>
+</section>
+
+
+### history
+
+<p>List all migrate items.</p>
+<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span>dipdup<span class="w"> </span>schema<span class="w"> </span><span class="nb">history</span><span class="w"> </span><span class="o">[</span>OPTIONS<span class="o">]</span>
+</pre></div>
+</div>
+</section>
+
+
 ### init
 
 <p>Prepare database schema for running DipDup.</p>
@@ -385,6 +441,43 @@ Discord:               <a class="reference external" href="https://discord.gg/aG
 </section>
 
 
+### migrate
+
+<p>Generate migrate changes file.</p>
+<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span>dipdup<span class="w"> </span>schema<span class="w"> </span>migrate<span class="w"> </span><span class="o">[</span>OPTIONS<span class="o">]</span>
+</pre></div>
+</div>
+<p class="rubric">Options</p>
+<dl class="std option">
+<dt class="sig sig-object std" id="cmdoption-dipdup-schema-migrate-name">
+<span class="sig-name descname"><span class="pre">--name</span></span><span class="sig-prename descclassname"> <span class="pre">&lt;name&gt;</span></span></dt>
+<dd><p>Migrate name.</p>
+<dl class="field-list simple">
+<dt class="field-odd" style="color: var(--txt-primary);">Default<span class="colon">:</span></dt>
+<dd class="field-odd"><p><code class="docutils literal notranslate"><span class="pre">'update'</span></code></p>
+</dd>
+</dl>
+</dd></dl>
+
+</section>
+
+
+### upgrade
+
+<p>Upgrade to specified version.</p>
+<div class="highlight-shell notranslate"><div class="highlight"><pre><span></span>dipdup<span class="w"> </span>schema<span class="w"> </span>upgrade<span class="w"> </span><span class="o">[</span>OPTIONS<span class="o">]</span>
+</pre></div>
+</div>
+<p class="rubric">Options</p>
+<dl class="std option">
+<dt class="sig sig-object std" id="cmdoption-dipdup-schema-upgrade-i">
+<span id="cmdoption-dipdup-schema-upgrade-in-transaction"></span><span class="sig-name descname"><span class="pre">-i</span></span><span class="sig-prename descclassname"></span><span class="sig-prename descclassname"><span class="pre">,</span> </span><span class="sig-name descname"><span class="pre">--in-transaction</span></span><span class="sig-prename descclassname"> <span class="pre">&lt;in_transaction&gt;</span></span></dt>
+<dd><p>Make migrations in transaction or not. Can be helpful for large migrations or creating concurrent indexes.</p>
+</dd></dl>
+
+</section>
+
+
 ### wipe
 
 <p>Drop all database tables, functions and views.</p>

docs/9.release-notes/_8.0_changelog.md

@@ -3,14 +3,17 @@
 
 ### Added
 
+- abi.etherscan: Try to extract ABI from webpage when API call fails.
 - cli: Added `--pre` flag to `self` group commands to install pre-release versions.
 - cli: Added `--raw` option to `config export` command to dump config preserving the original structure.
 - cli: Added `-C` option, a shorthand for `-c . -c configs/dipdup.<name>.yaml`.
 - cli: Added `package verify` command to check the package consistency.
+- cli: Added `schema` subcommands to manage database migrations: `migrate`, `upgrade`, `downgrade`, `heads` and `history`
 - cli: Added full project migration support for 3.0 spec.
 - cli: Added spec_version 3.0 support to `migrate` command.
 - config: Publish JSON schemas for config validation and autocompletion.
 - database: Added `dipdup_status` view to the schema.
+- database: support database migrations using [`aerich`](https://github.com/tortoise/aerich)
 - env: Added `DIPDUP_JSON_LOG` environment variable to enable JSON logging.
 - env: Added `DIPDUP_LOW_MEMORY` variable to reduce the size of caches and buffers.
 - env: Added `DIPDUP_PACKAGE_PATH` environment variable to override discovered package path.