From 605b01e6642b12cce6eb1bf9225942c1d0dd19d5 Mon Sep 17 00:00:00 2001 From: Lev Gorodetskiy Date: Mon, 29 Jul 2024 21:13:14 -0300 Subject: [PATCH 1/6] Update F.A.Q page --- docs/1.getting-started/4.package.md | 4 ---- docs/12.faq.md | 12 ++++++++++++ 2 files changed, 12 insertions(+), 4 deletions(-) diff --git a/docs/1.getting-started/4.package.md b/docs/1.getting-started/4.package.md index 904cf6cd7..4c9923859 100644 --- a/docs/1.getting-started/4.package.md +++ b/docs/1.getting-started/4.package.md @@ -41,10 +41,6 @@ DipDup uses contract type information to generate [Pydantic](https://docs.pydant This approach allows working with complex contract types with nested structures and polymorphic variants. -::banner{type="note"} -Currently, we use Pydantic v1, but plan to migrate to v2 one day. -:: - ## Nested packages Callbacks can be grouped into packages to organize the project structure. Add one or multiple dots to the callback name to define nested packages: diff --git a/docs/12.faq.md b/docs/12.faq.md index d3d8c38ec..2c8ca1373 100644 --- a/docs/12.faq.md +++ b/docs/12.faq.md @@ -133,3 +133,15 @@ Don't forget to reindex after this change. When decimal context precision is adj ```text WARNING dipdup.database Decimal context precision has been updated: 28 -> 128 ``` + +## Package + +### What is the symlink in the project root for? + +DipDup project must be a valid discoverable Python package. Python searches for packages in `site-packages` and the current working directory. This symlink is a hack that allows project root to serve as a working directory without requiring a subdirectory. It's created only when `dipdup init` is executed from the project root. While this symlink trick _should_ be harmless, if it interferes with any of your tools, use `DIPDUP_NO_SYMLINK=1` environment variable to disable this behavior. + +## Miscellaneous + +### Why it's called DipDup? + +DipDup (/dɪp dʌp/) was initially developed as a grant project for the [Tezos Foundation](https://tezos.foundation/). Tezos is one of the few blockchains with its own contract language, completely different from Solidity. This low-level, stack-based language is called [Michelson](https://tezos.gitlab.io/active/michelson.html). The name "DipDup" comes from `DIP {DUP}`, a small Michelson program that duplicates the second element on the stack. While today we support dozens of blockchains, Tezos holds a special place in our project's history. From 8271ec213c8daca0f30c3ac473c8b024ca7e16f5 Mon Sep 17 00:00:00 2001 From: Lev Gorodetskiy Date: Mon, 29 Jul 2024 22:38:27 -0300 Subject: [PATCH 2/6] another question --- docs/12.faq.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/docs/12.faq.md b/docs/12.faq.md index 2c8ca1373..9a17569f4 100644 --- a/docs/12.faq.md +++ b/docs/12.faq.md @@ -140,6 +140,20 @@ WARNING dipdup.database Decimal context precision has been updated: 28 -> DipDup project must be a valid discoverable Python package. Python searches for packages in `site-packages` and the current working directory. This symlink is a hack that allows project root to serve as a working directory without requiring a subdirectory. It's created only when `dipdup init` is executed from the project root. While this symlink trick _should_ be harmless, if it interferes with any of your tools, use `DIPDUP_NO_SYMLINK=1` environment variable to disable this behavior. +## Maintenance + +### pipx, Poetry, PDM... What's the difference? + +For historical reasons, Python package management is a mess. There are multiple tools and approaches to manage Python dependencies. Here's a brief comparison: + +- **pip** is a general-purpose package manager. It's simple and robust, but you need to manage venvs and lock files manually. +- **pipx** is meant for applications. It installs packages into separate environments in `~/.local/pipx/venvs` and makes their CLI commands available from any path. pipx is a recommended way to install DipDup CLI. +- **Poetry** and **PDM** are full-fledged project management tools. They handle venvs, lock files, dependency resolution, publishing, etc. + +Using PDM/Poetry is not required to run DipDup, but strongly recommended. Choosing one over the other is a matter of personal preference. _As of writing_, Poetry is [faster](https://lincolnloop.github.io/python-package-manager-shootout/), more popular, and has nicer CLI, while PDM is more PEP-compatible and allows dependency overrides. + +You can choose the preferred tool (or none) when initializing a project with `dipdup new` command. If you change your mind later, modify the `replay.yaml` file and run `dipdup init --base --force`. + ## Miscellaneous ### Why it's called DipDup? From 624ae5729757b27485621c2b2afe1fb2527bd349 Mon Sep 17 00:00:00 2001 From: Lev Gorodetskiy Date: Mon, 29 Jul 2024 23:10:08 -0300 Subject: [PATCH 3/6] hardware question --- docs/12.faq.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/docs/12.faq.md b/docs/12.faq.md index 9a17569f4..ba42ca500 100644 --- a/docs/12.faq.md +++ b/docs/12.faq.md @@ -10,6 +10,14 @@ nested: Resources This page contains answers to the most frequently asked questions about DipDup. +## General + +### What hardware do I need to run DipDup? + +DipDup can run on any amd64/arm64 machine starting from 1 CPU core and 256M of RAM. Aim for a good single-threaded and disk I/O performance. + +Actual RAM requirements depend on multiple factors: a number and complexity of indexes, size of internal queues and caches, usage of `CachedModel`. For the average project 1GB is usually enough. If you're running DipDup on some ultra-low-end instance and getting OOMs, try `DIPDUP_LOW_MEMORY=1` environment variable. + ## Indexing ### How to index similar but not identical contracts as a single entity? From c2c8f13c1fddeafefd973c332b5fa1ac9b2c71a9 Mon Sep 17 00:00:00 2001 From: Lev Gorodetskiy Date: Tue, 30 Jul 2024 00:15:58 -0300 Subject: [PATCH 4/6] migrations --- docs/12.faq.md | 33 ++++++++++++++++++++------------- 1 file changed, 20 insertions(+), 13 deletions(-) diff --git a/docs/12.faq.md b/docs/12.faq.md index ba42ca500..4e24d72ac 100644 --- a/docs/12.faq.md +++ b/docs/12.faq.md @@ -56,7 +56,9 @@ Instead, save raw data in handlers and process it later with hooks when all cond ### How to perform database migrations? -DipDup does not provide any tooling for database migrations. The reason is that in context of indexing, schema changes are rare and usually require reindexing. However, you can perform migrations yourself using any tool you like. First, disable schema hash check in config: +At the moment DipDup does not have a built-in migration system. Framework architecture implies that schema changes are rare and usually require reindexing. However, you can perform migrations yourself using third party tools or write your own scripts and keep them in `sql` project directory. + +You may want to disable schema hash check in config. Alternatively, call the `schema approve` command after every schema change. ```yaml [dipdup.yaml] advanced: @@ -64,30 +66,35 @@ advanced: schema_modified: ignore ``` -You can also use the `schema approve` command for a single schema change. - -To determine changes you need to apply after modifying classes in the `models` module, you can compare raw SQL schema before and after the change. +Now, let's prepare a migration script. To determine changes you need to make, you can compare the SQL schema dump before and after modifying the models. Say you need to add a new field to one of models. ```diff -- timestamp = fields.DatetimeField() -+ timestamp = fields.DatetimeField(auto_now=True) +class Event(Model): + ... ++ timestamp = fields.DatetimeField() ``` + ```shell -dipdup schema export > old-schema.sql -# ...after modifying `models` module... +$ dipdup schema export > old-schema.sql +# [make some changes] dipdup schema export > new-schema.sql diff old-schema.sql new-schema.sql ``` +And here's SQL for the new column: + ```diff -76c76 -< "timestamp" TIMESTAMP NOT NULL, ---- -> "timestamp" TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, ++ "timestamp" TIMESTAMP NOT NULL, ``` -Now you can prepare and execute an `ALTER TABLE` query manually or using SQL hooks. +Now prepare the idempodent migration script and put to the `sql/on_restart` directory. + +```sql [sql/on_restart/00-add-timestamp.sql] +ALTER TABLE "event" ADD COLUMN IF NOT EXISTS "timestamp" TIMESTAMP NOT NULL; + +SELECT dipdup_approve('public'); +``` ### I get `schema_modified` error, but I didn't change anything From 2232c7c93ec8c1f571b1b59e906502fce7643c2d Mon Sep 17 00:00:00 2001 From: Lev Gorodetskiy Date: Tue, 30 Jul 2024 00:54:02 -0300 Subject: [PATCH 5/6] typo --- docs/12.faq.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/docs/12.faq.md b/docs/12.faq.md index 4e24d72ac..d9c1ddf4b 100644 --- a/docs/12.faq.md +++ b/docs/12.faq.md @@ -74,9 +74,8 @@ class Event(Model): + timestamp = fields.DatetimeField() ``` - ```shell -$ dipdup schema export > old-schema.sql +dipdup schema export > old-schema.sql # [make some changes] dipdup schema export > new-schema.sql diff old-schema.sql new-schema.sql From ca4a1952ce2fd05d6a3ff288e13bd41ac2482f50 Mon Sep 17 00:00:00 2001 From: Lev Gorodetskiy Date: Tue, 30 Jul 2024 01:08:46 -0300 Subject: [PATCH 6/6] typos and articles --- docs/12.faq.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/12.faq.md b/docs/12.faq.md index d9c1ddf4b..0f2799ffe 100644 --- a/docs/12.faq.md +++ b/docs/12.faq.md @@ -16,7 +16,7 @@ This page contains answers to the most frequently asked questions about DipDup. DipDup can run on any amd64/arm64 machine starting from 1 CPU core and 256M of RAM. Aim for a good single-threaded and disk I/O performance. -Actual RAM requirements depend on multiple factors: a number and complexity of indexes, size of internal queues and caches, usage of `CachedModel`. For the average project 1GB is usually enough. If you're running DipDup on some ultra-low-end instance and getting OOMs, try `DIPDUP_LOW_MEMORY=1` environment variable. +Actual RAM requirements depend on multiple factors: the number and complexity of indexes, the size of internal queues and caches, and the usage of `CachedModel`. For the average project, 1GB is usually enough. If you're running DipDup on some ultra-low-end instance and getting OOMs, try the `DIPDUP_LOW_MEMORY=1` environment variable. ## Indexing @@ -56,9 +56,9 @@ Instead, save raw data in handlers and process it later with hooks when all cond ### How to perform database migrations? -At the moment DipDup does not have a built-in migration system. Framework architecture implies that schema changes are rare and usually require reindexing. However, you can perform migrations yourself using third party tools or write your own scripts and keep them in `sql` project directory. +At the moment DipDup does not have a built-in migration system. Framework architecture implies that schema changes are rare and usually require reindexing. However, you can perform migrations yourself using third-party tools or write your own scripts and keep them in `sql` project directory. -You may want to disable schema hash check in config. Alternatively, call the `schema approve` command after every schema change. +You may want to disable the schema hash check in config. Alternatively, call the `schema approve` command after every schema change. ```yaml [dipdup.yaml] advanced: @@ -66,7 +66,7 @@ advanced: schema_modified: ignore ``` -Now, let's prepare a migration script. To determine changes you need to make, you can compare the SQL schema dump before and after modifying the models. Say you need to add a new field to one of models. +Now, let's prepare a migration script. To determine the changes you need to make, you can compare the SQL schema dump before and after modifying the models. Say you need to add a new field to one of the models. ```diff class Event(Model): @@ -87,7 +87,7 @@ And here's SQL for the new column: + "timestamp" TIMESTAMP NOT NULL, ``` -Now prepare the idempodent migration script and put to the `sql/on_restart` directory. +Now prepare the idempotent migration script and put it in the `sql/on_restart` directory. ```sql [sql/on_restart/00-add-timestamp.sql] ALTER TABLE "event" ADD COLUMN IF NOT EXISTS "timestamp" TIMESTAMP NOT NULL; @@ -164,7 +164,7 @@ For historical reasons, Python package management is a mess. There are multiple - **pipx** is meant for applications. It installs packages into separate environments in `~/.local/pipx/venvs` and makes their CLI commands available from any path. pipx is a recommended way to install DipDup CLI. - **Poetry** and **PDM** are full-fledged project management tools. They handle venvs, lock files, dependency resolution, publishing, etc. -Using PDM/Poetry is not required to run DipDup, but strongly recommended. Choosing one over the other is a matter of personal preference. _As of writing_, Poetry is [faster](https://lincolnloop.github.io/python-package-manager-shootout/), more popular, and has nicer CLI, while PDM is more PEP-compatible and allows dependency overrides. +Using PDM/Poetry is not required to run DipDup, but strongly recommended. Choosing one over the other is a matter of personal preference. _As of writing_, Poetry is [faster](https://lincolnloop.github.io/python-package-manager-shootout/), more popular, and has a nicer CLI, while PDM is more PEP-compatible and allows dependency overrides. You can choose the preferred tool (or none) when initializing a project with `dipdup new` command. If you change your mind later, modify the `replay.yaml` file and run `dipdup init --base --force`.