Skip to content

Commit

Permalink
Merge pull request #343 from subsquid/resources-reorg
Browse files Browse the repository at this point in the history
Resources reorg
  • Loading branch information
abernatskiy authored Dec 7, 2023
2 parents 67bd730 + 29451b3 commit 1854a4f
Show file tree
Hide file tree
Showing 76 changed files with 262 additions and 174 deletions.
2 changes: 1 addition & 1 deletion docs/cloud/reference/manifest.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ The manifest header defines the squid metadata
## `build:`

Specifies the way the squid is built into a docker image. Only an empty `build:` section is supported, and the squid must adhere
to the default [folder structure](/sdk/reference/layout).
to the default [folder structure](/sdk/how-to-start/layout).

In particular, the following files and folders **must** be present in the root folder of the squid:

Expand Down
6 changes: 3 additions & 3 deletions docs/cloud/resources/best-practices.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,9 @@ Here is a list of items to check out before you deploy your squid for use in pro

* If your squid [saves its data to a database](/sdk/resources/persisting-data/typeorm), make sure your [schema](/sdk/reference/schema-file) has [`@index` decorators](/sdk/reference/schema-file/indexes-and-constraints) for all entities that will be looked up frequently.

* If your squid serves a [GraphQL API](/sdk/reference/graphql-server), consider:
1. configuring the built-in [DoS protection](/sdk/reference/graphql-server/dos-protection) against heavy queries;
2. configuring [caching](/sdk/reference/graphql-server/caching).
* If your squid serves a [GraphQL API](/sdk/resources/graphql-server), consider:
1. configuring the built-in [DoS protection](/sdk/resources/graphql-server/dos-protection) against heavy queries;
2. configuring [caching](/sdk/resources/graphql-server/caching).

* If you deploy your squid to Subsquid Cloud:
1. Deploy your squid to a [Professional organization](/cloud/resources/organizations/#professional-organizations).
Expand Down
2 changes: 1 addition & 1 deletion docs/cloud/resources/migration.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ deploy:

## 3. (Optional) Revise `cmd`

You may want to add additional flags to `api.cmd`, e.g. to enable [caching](/sdk/reference/graphql-server/caching) or [enforce query limits](/sdk/reference/graphql-server/dos-protection).
You may want to add additional flags to `api.cmd`, e.g. to enable [caching](/sdk/resources/graphql-server/caching) or [enforce query limits](/sdk/resources/graphql-server/dos-protection).

## 4. (Optional) Add `scale:` options

Expand Down
4 changes: 2 additions & 2 deletions docs/cloud/troubleshooting.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,8 @@ npm update -g @subsquid/cli
```bash
npm run update
```
- Check that the squid adheres to the expected [structure](/sdk/reference/layout)
- Make sure you can [build and run Docker images locally](/sdk/resources/self-hosting)
- Check that the squid adheres to the expected [structure](/sdk/how-to-start/layout)
- Make sure you can [build and run Docker images locally](/sdk/resources/basics/self-hosting)

### `Validation error` when releasing a squid

Expand Down
4 changes: 2 additions & 2 deletions docs/glossary.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ An SDK (software development kit) and a smart-contract language for developing W

### OpenReader

An open-source GraphQL server that automatically generates an expressive API from an input schema file. See the [repo](https://github.com/subsquid/squid/tree/master/openreader) and [details](/sdk/reference/graphql-server).
An open-source GraphQL server that automatically generates an expressive API from an input schema file. See the [repo](https://github.com/subsquid/squid/tree/master/openreader) and [details](/sdk/resources/graphql-server).

### Palette

Expand Down Expand Up @@ -86,7 +86,7 @@ The ETL part of the squid. Extracts on-chain data from an Archive, transforms, o

### Squid API

The data presentation part of the squid. Typically, it's a GraphQL API auto-generated from the schema file. See details [here](/sdk/reference/graphql-server).
The data presentation part of the squid. Typically, it's a GraphQL API auto-generated from the schema file. See details [here](/sdk/resources/graphql-server).

### Substrate

Expand Down
10 changes: 5 additions & 5 deletions docs/overview.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -49,10 +49,10 @@ For real-time use cases such as app-specific APIs use [Squid SDK](/sdk): it'll u
[Squid SDK](/sdk) is a powerful Typescript toolkit for building indexers on top of Subsquid Network, featuring

- [High-level libraries](/sdk/reference/processors) for extracting and filtering the Subsquid Network data in what can be though of as Extract-Transform-Load (ETL) pipelines
- [Ergonomic tools](/sdk/reference/typegen) for decoding and normalizing raw data and efficiently accessing [network state](/sdk/reference/typegen/state-queries)
- [Ergonomic tools](/sdk/resources/tools/typegen) for decoding and normalizing raw data and efficiently accessing [network state](/sdk/resources/tools/typegen/state-queries)
- Pluggable [data sinks](/sdk/reference/store) to save data into Postgres, files (local or s3) or BigQuery
- Expressive [GraphQL server](/sdk/reference/graphql-server) with a schema-based [config](/sdk/reference/schema-file)
- Seamless handling of [unfinalized blocks and chain reorganizations](/sdk/resources/unfinalized-blocks) for real-time data ingestion
- Expressive [GraphQL server](/sdk/resources/graphql-server) with a schema-based [config](/sdk/reference/schema-file)
- Seamless handling of [unfinalized blocks and chain reorganizations](/sdk/resources/basics/unfinalized-blocks) for real-time data ingestion
- rapid data extraction and decoding [for local analytics](/sdk/tutorials/file-csv)

The SDK is a go-to choice for production solutions and prototypes of
Expand All @@ -78,10 +78,10 @@ A Platform-as-a-Service for deploying Squid SDK indexers, featuring
<TabItem value="sdk" label="SDK">
```

- Learn about [squid components](/sdk/overview), [combining them](/sdk/squid-from-scratch) or follow the [end-to-end development guide](/sdk/squid-development)
- Learn about [squid components](/sdk/overview), [combining them](/sdk/how-to-start/squid-from-scratch) or follow the [end-to-end development guide](/sdk/how-to-start/squid-development)
- Explore [tutorials](/sdk/tutorials) or [examples](/sdk/examples)
- Learn how to [migrate from The Graph](/sdk/resources/migrate/migrate-subgraph)
- Explore the [GraphQL server options](/sdk/reference/graphql-server) including custom extensions, caching and DoS protection in production
- Explore the [GraphQL server options](/sdk/resources/graphql-server) including custom extensions, caching and DoS protection in production

```mdx-code-block
</TabItem>
Expand Down
6 changes: 3 additions & 3 deletions docs/sdk/examples.mdx
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
sidebar_position: 31
sidebar_position: 80
title: Examples
description: Example squids
---
Expand All @@ -17,7 +17,7 @@ Browse a dedicated repository with [EVM squid examples](https://github.com/subsq

- A [squid indexing ERC20 transfers](https://github.com/subsquid-labs/evm-logs-example) by extracting `Transfer(address,address,uint256)` logs emitted by the USDC contract.
- A [squid indexing DEX trades](https://github.com/subsquid-labs/evm-factory-example) while dynamically tracking Uniswap v3 pool contracts as they are deployed. Shows [factory contract](/sdk/resources/evm/factory-contracts) indexing in action.
- A [NFT indexing squid](https://github.com/subsquid-labs/evm-multicall-example) that additionally queries the NFT metadata from the contract state. Illustrates batching RPC calls using the [Multicall contract](/sdk/reference/typegen/generation/#batching-contract-state-calls-using-the-multicall-contract).
- A [NFT indexing squid](https://github.com/subsquid-labs/evm-multicall-example) that additionally queries the NFT metadata from the contract state. Illustrates batching RPC calls using the [Multicall contract](/sdk/resources/tools/typegen/generation/#batching-contract-state-calls-using-the-multicall-contract).
- A [multichain squid](https://github.com/subsquid-labs/squid-multichain-template) tracking USDC `Transfer` events on both Ethereum and BSC.

and a few others.
Expand Down Expand Up @@ -62,7 +62,7 @@ processor.run(new TypeormDatabase(), async (ctx) => {
<details>
<summary>Index all USDC Transfer events on Ethereum with real time updates</summary>

Real time data is fetched from a chain node RPC; a Database object with hot blocks support is required to store it (see [this page](/sdk/resources/unfinalized-blocks) for more details). [Full squid here](https://github.com/subsquid-labs/showcase01-all-usdc-transfers).
Real time data is fetched from a chain node RPC; a Database object with hot blocks support is required to store it (see [this page](/sdk/resources/basics/unfinalized-blocks) for more details). [Full squid here](https://github.com/subsquid-labs/showcase01-all-usdc-transfers).

```ts
export const USDC_CONTRACT_ADDRESS = '0xa0b86991c6218b36c1d19d4a2e9eb0ce3606eb48'
Expand Down
8 changes: 4 additions & 4 deletions docs/sdk/faq.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
sidebar_position: 35
sidebar_position: 80
title: FAQ
description: Frequently asked questions
---
Expand All @@ -16,19 +16,19 @@ Here is an incomplete list:

### How does Squid SDK handle unfinalized blocks?

The Subsquid Network only serves finalized blocks and is typically ~1000 blocks behind the tip. The most recent blocks, as well as the unfinalized blocks are seemlesly handled by the SDK from a complementary RPC data source, set by the `chain` config. Potential chain reorgs are automatically handled under the hood. See [Indexing unfinalized blocks](/sdk/resources/unfinalized-blocks) for details.
The Subsquid Network only serves finalized blocks and is typically ~1000 blocks behind the tip. The most recent blocks, as well as the unfinalized blocks are seemlesly handled by the SDK from a complementary RPC data source, set by the `chain` config. Potential chain reorgs are automatically handled under the hood. See [Indexing unfinalized blocks](/sdk/resources/basics/unfinalized-blocks) for details.

### What is the latency for the data served by the squid?

Since the ArrowSquid release, the Squid SDK has the option to ingest unfinalized blocks directly from an RPC endpoint, making the indexing real-time.

### How do I enable GraphQL subscriptions for local runs?

Add `--subscription` flag to the `serve` command defined in `commands.json`. See [Subscriptions](/sdk/reference/graphql-server/subscriptions) for details.
Add `--subscription` flag to the `serve` command defined in `commands.json`. See [Subscriptions](/sdk/resources/graphql-server/subscriptions) for details.

### How do squids keep track of their sync progress?

Depends on the data sink used. Squid processors that use [`TypeormDatabase`](/sdk/resources/persisting-data/typeorm) keep their state in a [schema](https://www.postgresql.org/docs/current/sql-createschema.html), not in a table. By default the schema is called `squid_processor` (name must be overridden in [multiprocessor squids](/sdk/resources/multichain)). You can view it with
Depends on the data sink used. Squid processors that use [`TypeormDatabase`](/sdk/resources/persisting-data/typeorm) keep their state in a [schema](https://www.postgresql.org/docs/current/sql-createschema.html), not in a table. By default the schema is called `squid_processor` (name must be overridden in [multiprocessor squids](/sdk/resources/basics/multichain)). You can view it with
```sql
select * from squid_processor.status;
```
Expand Down
12 changes: 12 additions & 0 deletions docs/sdk/how-to-start/_category_.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
{
"position": 20,
"label": "Getting started",
"collapsible": true,
"collapsed": true,
"className": "red",
"link": {
"type": "generated-index",
"slug": "/sdk/how-to-start",
"title": "Getting started with Squid SDK"
}
}
7 changes: 7 additions & 0 deletions docs/sdk/how-to-start/cli-cheatsheet.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
---
sidebar_position: 50
title: CLI cheatsheet
description: Commonly used CLI commands
---

# Squid CLI cheatsheet
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
sidebar_position: 20
sidebar_position: 10
description: >-
Prepare for working with Squid SDK
---
Expand Down
11 changes: 4 additions & 7 deletions docs/sdk/reference/layout.md → docs/sdk/how-to-start/layout.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
sidebar_position: 40
title: Layout convention
description: The folder layout of a squid
title: Project structure
description: The folder layout of a squid project
---

# Squid structure
Expand All @@ -15,7 +15,7 @@ A squid is expected to follow the folder structure conventions listed below.
+ `/src/processor.ts` -- Processor object ([EVM](/sdk) or [Substrate](/sdk)) definition and configuration.
+ `/src/model/generated` -- The folder for the TypeORM entities generated from `schema.graphql`.
+ `/src/model` -- The module exporting the entity classes.
+ `/src/server-extension/resolvers` -- An (optional) folder for [user-defined GraphQL resolvers](/sdk/reference/graphql-server/custom-resolvers).
+ `/src/server-extension/resolvers` -- An (optional) folder for [user-defined GraphQL resolvers](/sdk/resources/graphql-server/custom-resolvers).
+ `/src/types` -- An (optional) output folder for typescript definitions of the Substrate data generated with [`squid-substrate-typegen`[(/sdk/tutorials/batch-processor-in-action).
+ `/src/abi` -- An (optional) output folder for the [EVM typegen[(/sdk/reference/typegen/state-queries) and [WASM typegen](https://github.com/subsquid/squid-sdk/tree/master/substrate/ink-typegen) tools that generate type definitions and the decoding boilerplate.
- `/assets` -- (optional) A designated folder for custom user-provided files (e.g. static data files to seed the squid processor with).
Expand All @@ -25,10 +25,7 @@ A squid is expected to follow the folder structure conventions listed below.
- `docker-compose.yml` -- A Docker compose file for local runs. Has a Postgres service definition by default. Ignored by Subsquid Cloud.
- `.env` -- Defines environment variables used by `docker-compose.yml` and when the squid is run locally. Ignored by Subsquid Cloud.
- `typegen.json` -- (optional) The config file for `squid-substrate-typegen`. Ignored by Subsquid Cloud.
- `Makefile` -- (optional) Script definitions for `Make`. Ignored by Subsquid Cloud.
- `commands.json` -- User-defined scripts picked up by [Squid CLI](/squid-cli).
- `tsconfig.json`, `package-lock.json`, `package.json` -- **(Required)** The `npm` and `tsc` configs.
- `/archive` -- An (optional) folder for running a local Archive. Ignored by Subsquid Cloud.

The templates also define the following auxiliary scripts (optional):
- `build` to build the squid.
- `update` to update the squid packages to the latest version.
Loading

0 comments on commit 1854a4f

Please sign in to comment.