Merge pull request #372 from subsquid/misc-fixes-by-abernatskiy-mar07

Misc fixes by abernatskiy mar07

docs/cloud/overview.md

@@ -12,6 +12,10 @@ The deployment is managed by the file `squid.yaml` in the root folder of the squ
 - [which services should be deployed](/cloud/reference/manifest/#deploy) for the squid (e.g. postgres, processor, api, RPC proxy)
 - [what resources should be allocated](/cloud/reference/scale) by Cloud for each squid service (only configurable if you deploy to a [Professional organization](/cloud/resources/organizations/#professional-organizations))
 
+:::warning
+Squids are deployed with a collocated profile by default. This is a highly constrained configuration that's only useful when performance is not important (e.g. in development). Use the [dedicated profile](/cloud/reference/scale/#dedicated) for squids running in production.
+:::
+
 :::tip
 Make sure to check our [best practices guide](/cloud/resources/best-practices) before deploying to production!
 :::

docs/cloud/pricing.md

@@ -6,6 +6,10 @@ description: Subscription types and pay-as-you-go prices
 
 # Subsquid Cloud Pricing
 
+:::tip
+Use our [cost calculator](https://subsquid.io/subsquid-cloud#calculator) if you need a quick estimate
+:::
+
 ## Free squids
 
 Each account gets a playground [organization](/cloud/resources/organizations) where a single free squid can be deployed. Certain limitations apply; check the [playground organization](/cloud/resources/organizations/#playgrounds) section for details.

docs/cloud/reference/scale.md

@@ -14,7 +14,8 @@ The manifest supports the following scaling options:
 
 Default: `dedicated: false`. 
 
-By default, the service deployments are _collocated_ - that is, share resources with other deployments - and so the maximal allowed resource allocation is not guaranteed. With the `dedicated: true` option, the resources are reserved in advance. See the profile specifications below for details.
+By default, the service deployments are _collocated_ - that is, share resources with other deployments - and so the allocation of full resources is not guaranteed. With `dedicated: true`, the resources are reserved in advance.
+
 We recommend setting `dedicated: true` for all squids running in production.
 
 Squid deployment must be dedicated to be a subject of [Subsquid Cloud SLA](/cloud/pricing).

docs/external-tools.md

@@ -9,7 +9,15 @@ sidebar_position: 90
 
 ## `@belopash/typeorm-store`
 
-[`@belopash/typeorm-store`](https://github.com/belopash/squid-typeorm-store) is a [fork](/sdk/resources/persisting-data/overview/#custom-database) of [`@subsquid/typeorm-store`](/sdk/reference/store/typeorm) that automates collecting read and write database requests into [batches](/sdk/resources/basics/batch-processing) and caches the available entity records in RAM. See [this repository](https://github.com/subsquid-labs/belopash-typeorm-store-example) for a minimal example.
+[`@belopash/typeorm-store`](https://github.com/belopash/squid-typeorm-store) is a [fork](/sdk/resources/persisting-data/overview/#custom-database) of [`@subsquid/typeorm-store`](/sdk/reference/store/typeorm) that automates collecting read and write database requests into [batches](/sdk/resources/basics/batch-processing) and caches the available entity records in RAM. Unlike the [standard `typeorm-store`](/sdk/resources/persisting-data/typeorm), @belopash's store is intended to be used with declarative code: it makes it easy to write mapping functions (e.g. event handlers) that explicitly define
+
+ - what data you're going to need from the database
+ - what code has to be executed once the data is available
+ - how to save the results
+
+Data dependencies due to [entity relations](/sdk/reference/schema-file/entity-relations) are handled automatically, along with the caching of intermediate resultsg and in-memory batching of database requests.
+
+See [this repository](https://github.com/subsquid-labs/belopash-typeorm-store-example) for a minimal example.
 
 ## DipDup
 

docs/sdk/how-to-start/squid-from-scratch.mdx

@@ -30,17 +30,12 @@ To make the indexer, follow these steps:
 
 1. Create a new folder and install the packages:
    ```bash
-   npm i @subsquid/evm-processor @subsquid/typeorm-store @subsquid/typeorm-migration @subsquid/graphql-server @subsquid/archive-registry
+   npm i dotenv @subsquid/evm-processor @subsquid/typeorm-store @subsquid/typeorm-migration @subsquid/graphql-server @subsquid/archive-registry
    ```
    ```bash
    npm i typescript @subsquid/typeorm-codegen @subsquid/evm-typegen --save-dev
    ```
 
-   * install dotenv package
-   ```bash
-   npm i dotenv --save
-   ```
-
 2. Add a minimal `tsconfig.json`:
    ```json title="tsconfig.json"
    {

docs/sdk/reference/store/typeorm.md

@@ -26,9 +26,13 @@ The argument of the `TypeormDatabase` class constructor may have the following f
 Upsert a single or multiple entities to the database. **Does not cascade the upsert to the relations.**
 
 ```ts
-await ctx.store.save([new User({id: 'Bob'}), new User({id: 'Alice'}))])
+await ctx.store.upsert([new User({id: 'Bob'}), new User({id: 'Alice'}))])
 ```
 
+#### **`save(e: E | E[])`** {#save}
+
+Deprecated alias for [`upsert()`](#upsert).
+
 #### **`insert(e: E | E[])`** {#insert}
 
 Inserts a given entity or entities into the database. Does not check if the entity(s) exist in the database and will fail if a duplicate is inserted. Executes a primitive INSERT operation **without cascading to the relations**.

docs/sdk/resources/basics/multichain.md

@@ -73,22 +73,27 @@ Also ensure that
 
 ### Handling concurrency
 
-  - The [default isolation level](/sdk/reference/store/typeorm/#typeormdatabase-constructor-arguments) used by `TypeormDatabase` is `SERIALIZABLE`, the most secure and the most restrictive one. Another isolation level commonly used in multichain squids is `READ COMMITTED`, which guarantees that the execution is deterministic for as long as the sets of records that different processors read/write do not overlap.
-  - To avoid overlaps, use per-chain records for volatile data. E.g. if you track account balances across multiple chains you can avoid overlaps by storing the balance for each chain in a different table row.
-  - When you need to combine the records (e.g. get a total of all balaces across chains) use [custom resolvers](/sdk/resources/graphql-server/custom-resolvers) to do it on the GraphQL server side.
-  - It is OK to use cross-chain [entities](/sdk/reference/schema-file/entities) to simplify aggregation. Just don't store any data in them:
-    ```graphql
-    type Account @entity {
-      id: ID! # evm address
-      balances: [Balance!]! @derivedFrom("field": "account")
-    }
-
-    type Balance @entity {
-      id: ID! # chainId + evm address
-      account: Account!
-      value: BigInt!
-    }
-    ```
+- Cross-chain data dependencies are to be avoided. With the [default isolation level](/sdk/reference/store/typeorm/#typeormdatabase-constructor-arguments) used by `TypeormDatabase`, `SERIALIZABLE`, one of the processors will crash with an error whenever two cross-dependent transactions are submitted to the database simultaneously. It will write the correct data when restarted, but such restarts can impact performance, especially in squids that use many (>5) processors.
+
+- The alternative isolation level is `READ COMMITTED`. At this level data dependencies will not crush the processors, but the execution is not guaranteed to be deterministic unless the sets of records that different processors read/write do not overlap.
+
+- To avoid cross-chain data dependencies, use per-chain records for volatile data. E.g. if you track account balances across multiple chains you can avoid overlaps by storing the balance for each chain in a different table row.
+
+  When you need to combine the records (e.g. get a total of all balaces across chains) use a [custom resolver](/sdk/resources/graphql-server/custom-resolvers) to do it on the GraphQL server side.
+
+- It is OK to use cross-chain [entities](/sdk/reference/schema-file/entities) to simplify aggregation. Just don't store any data in them:
+  ```graphql
+  type Account @entity {
+    id: ID! # evm address
+    balances: [Balance!]! @derivedFrom("field": "account")
+  }
+
+  type Balance @entity {
+    id: ID! # chainId + evm address
+    account: Account!
+    value: BigInt!
+  }
+  ```
 
 ## On [file-store](/sdk/resources/persisting-data/file)
 

docs/sdk/troubleshooting.mdx

@@ -0,0 +1,81 @@
+---
+sidebar_position: 72
+title: Troubleshooting
+description: Common issues arising when developing squids
+---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+# Troubleshooting
+
+Many issues can be resolved by following the [best practices guide](/cloud/resources/best-practices) of Subsquid Cloud.
+
+## Processor
+
+<details>
+<summary>`Error: data out-of-bounds` `ethers` errors on EVM</summary>
+
+* Usually this means that you're using the decoder on the wrong data. Make sure that the decoder receives _only_ the data you intend it to.
+
+  Example: suppose you want to add the processing of a `Mint` event to a squid that is currently processing only `Transfer` events. You change the processor configuration to get the `Mint` events for you, but you forget to sort the events in the batch handler and a data item with a `Mint` event finds its way into a decoder of `Transfer`s.
+
+* Another common source of this error is faulty RPC endpoints. If your EVM processor crashes during RPC ingestion on a log with `'0x'` in its data field, try switching to another RPC provider or, if you are developing locally, to another Ethereum network emulator.
+
+</details>
+
+<details>
+<summary>`BAD_DATA` when using a Multicall contract</summary>
+
+This error can occur for a variety of reasons, but one common cause is choosing a Multicall deployment that is newer than the oldest blocks that have to be indexed. When [batch state queries](/sdk/resources/tools/typegen/state-queries/#batch-state-queries) are performed on historical chain state older than the Multicall deployment, EVM detects that and refuses to run.
+
+Solutions:
+1. Use an older Multicall deployment.
+2. Delay your chain state queries until a later block.
+
+These issues are explored in [Part 4 of the BAYC tutorial](/sdk/tutorials/bayc/step-four-optimizations).
+
+</details>
+
+## Data sinks
+
+<details>
+<summary>`QueryFailedError: relation "..." does not exist`</summary>
+
+It is likely that the generated migrations in the `db/migrations` folder are outdated and do not match the schema file.
+Recreate the migrations from scratch as detailed on [this page](/sdk/resources/persisting-data/typeorm/#updating-after-schema-changes).
+
+</details>
+
+<details>
+<summary>`Query runner already released. Cannot run queries anymore`, or `too late to perform db updates, make sure you haven't forgot to await on db query`</summary>
+
+If your squid [saves its data to a database](/sdk/resources/persisting-data/typeorm), all operations with `ctx.store` are asynchronous. Make sure you `await` on all `store` operations like `upsert`, `update`, `find`, `save` etc.
+
+You may find the [require-await](https://eslint.org/docs/latest/rules/require-await) eslint rule to be helpful.
+
+</details>
+
+<details>
+<summary>`QueryFailedError: invalid byte sequence for encoding "UTF8": 0x00`</summary>
+
+PostgreSQL doesn't support storing `NULL (\0x00)` characters in text fields. Usually the error occurs when a raw bytes string (like `UIntArray` or `Bytes`) is inserted into a `String` field. If this is the case, use hex encoding, e.g. using [`util-internal-hex`](https://github.com/subsquid/squid/tree/master/util/util-internal-hex) library. For addresses, use the [`ss58` encoding library](https://github.com/subsquid/squid/tree/master/ss58).
+
+</details>
+
+## GraphQL
+
+<details>
+<summary>API queries are too slow</summary>
+
+- Make sure all the necessary fields are [indexed](/sdk/reference/schema-file/indexes-and-constraints/)
+- Annotate the schema and [set reasonable limits](/sdk/resources/graphql-server/dos-protection/) for the incoming queries to protect against DoS attacks
+
+</details>
+
+<details>
+<summary>`response might exceed the size limit`</summary>
+
+Make sure the input query has limits set or the entities are decorated with `@cardinality`. We recommend using `XXXConnection` queries for pagination. For configuring limits and max response sizes, see [DoS protection](/sdk/resources/graphql-server/dos-protection/).
+
+</details>

redirectRules.js

@@ -727,6 +727,18 @@ const urlList = [
         "from": "/troubleshooting",
         "to": "/sdk/troubleshooting"
     },
+    {
+        "from": "/sdk/troubleshooting/processor",
+        "to": "/sdk/troubleshooting"
+    },
+    {
+        "from": "/sdk/troubleshooting/persisting-data",
+        "to": "/sdk/troubleshooting"
+    },
+    {
+        "from": "/sdk/troubleshooting/graphql",
+        "to": "/sdk/troubleshooting"
+    },
     {
         "from": "/basics",
         "to": "/sdk"