Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add drizzle persistence plugin #122

Merged
merged 2 commits into from
Nov 30, 2024

Conversation

jaipaljadeja
Copy link
Member

No description provided.

@jaipaljadeja jaipaljadeja requested a review from fracek November 30, 2024 16:21
Copy link

coderabbitai bot commented Nov 30, 2024

📝 Walkthrough

Walkthrough

The changes in this pull request introduce a persistence layer for the @apibara/indexer using PostgreSQL and Drizzle ORM. Key modifications include the addition of new imports, the establishment of a PostgreSQL client, and the creation of database tables for storing checkpoints and filters. A new plugin, drizzlePersistence, is integrated into the indexer, along with hooks for managing database interactions. Additionally, updates to the package.json files reflect new dependencies and development requirements, enhancing the project's capabilities.

Changes

File Path Change Summary
examples/cli/indexers/2-starknet.indexer.ts Added imports for drizzlePersistence, sql, drizzle, and Client. Initialized PostgreSQL client and persistence database. Introduced plugins array with drizzlePersistence and added run:before hook for table management.
examples/cli/package.json Added @types/pg as a dev dependency. Added @electric-sql/pglite, drizzle-orm, and pg as dependencies.
packages/indexer/build.config.ts Added "./src/plugins/drizzle-persistence.ts" to the entries array.
packages/indexer/package.json Added export entry for ./plugins/drizzle-persistence with paths for types and modules. Updated devDependencies and peerDependencies to include @electric-sql/pglite.
change/@apibara-indexer-3ab2f0f6-4d43-4499-a0a3-55188fe8eb53.json Introduced a new JSON file for prerelease metadata, including comments about the drizzle persistence plugin.
packages/indexer/src/plugins/drizzle-persistence.ts Created drizzle-persistence.ts defining checkpoints and filters tables. Implemented drizzlePersistence function and DrizzlePersistence class with methods for managing data persistence.

Possibly related PRs

Suggested reviewers

  • fracek

Poem

🐇 In the fields where data flows,
A persistence layer now grows.
With PostgreSQL, we store with grace,
Checkpoints and filters find their place.
Drizzle ORM, a friend so true,
In our indexer, we bid adieu! 🌼


Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

❤️ Share
🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Generate unit testing code for this file.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai generate unit testing code for this file.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read src/utils.ts and generate unit testing code.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
    • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Inline review comments failed to post. This is likely due to GitHub's limits when posting large numbers of comments.

Actionable comments posted: 8

🧹 Outside diff range and nitpick comments (2)
examples/cli/indexers/2-starknet.indexer.ts (2)

16-22: Ensure Proper Disposal of Database Connections

While setting up the SQLite database, ensure that database connections are properly closed when no longer needed to prevent potential resource leaks.

Consider adding code to close the database connection when the indexer stops or encounters an error.


Line range hint 43-46: Clean Up Commented-Out Code

The writer.insert code block is commented out. If it's not needed, consider removing it to keep the codebase clean. If it's intended for future use, add a comment explaining its purpose.

🛑 Comments failed to post (8)
packages/indexer/build.config.ts (1)

15-15: ⚠️ Potential issue

Possible Typo in Filename 'dirzzle-persistence.ts'

It appears that the filename dirzzle-persistence.ts may be misspelled. Considering that you're integrating with Drizzle ORM, the correct filename should likely be drizzle-persistence.ts.

Apply this diff to correct the filename:

     "./src/index.ts",
     "./src/sinks/sqlite.ts",
     "./src/sinks/csv.ts",
     "./src/sinks/drizzle/index.ts",
     "./src/testing/index.ts",
     "./src/vcr/index.ts",
     "./src/plugins/index.ts",
     "./src/plugins/kv.ts",
     "./src/plugins/logger.ts",
     "./src/plugins/persistence.ts",
-    "./src/plugins/dirzzle-persistence.ts",
+    "./src/plugins/drizzle-persistence.ts",
   ],

Committable suggestion skipped: line range outside the PR's diff.

examples/cli/indexers/2-starknet.indexer.ts (3)

63-87: 🛠️ Refactor suggestion

Consider Using Migrations for Database Schema Management

Directly executing SQL statements for schema creation within the code is not recommended. Using a migration system ensures consistency and easier maintenance of the database schema.

Consider implementing migrations using Drizzle ORM's migration capabilities or another migration tool to handle table creation and schema updates. This approach allows for versioning of schema changes and better collaboration among team members.


2-2: ⚠️ Potential issue

Possible Typo in Import Path 'dirzzle-persistence'

The import path @apibara/indexer/plugins/dirzzle-persistence may contain a typo. The correct path should likely be @apibara/indexer/plugins/drizzle-persistence, reflecting the Drizzle ORM integration.

Apply this diff to correct the import statement:

 import { defineIndexer, useSink } from "@apibara/indexer";
-import { drizzlePersistence } from "@apibara/indexer/plugins/dirzzle-persistence";
+import { drizzlePersistence } from "@apibara/indexer/plugins/drizzle-persistence";
 import { useLogger } from "@apibara/indexer/plugins/logger";
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

import { drizzlePersistence } from "@apibara/indexer/plugins/drizzle-persistence";

24-26: ⚠️ Potential issue

Avoid Hardcoding Database Credentials

The database connection string contains hardcoded credentials, which is a security risk. It's recommended to use environment variables to manage sensitive information.

Apply this diff to use environment variables for the connection string:

 // Persistence Database
 const client = new Client({
-  connectionString: "postgres://postgres:postgres@localhost:5432/postgres",
+  connectionString: process.env.DATABASE_URL,
 });

Ensure that the DATABASE_URL environment variable is properly set in your environment.

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

  const client = new Client({
    connectionString: process.env.DATABASE_URL,
  });
packages/indexer/package.json (1)

72-77: ⚠️ Potential issue

Fix Typo in Export Path for drizzle-persistence Plugin

There's a typo in the export path "./plugins/dirzzle-persistence"; it should be "./plugins/drizzle-persistence" to correctly reference the plugin and maintain consistency.

Apply this diff to fix the typo:

     },
-    "./plugins/dirzzle-persistence": {
+    "./plugins/drizzle-persistence": {
       "types": "./dist/plugins/drizzle-persistence.d.ts",
       "import": "./dist/plugins/drizzle-persistence.mjs",
       "require": "./dist/plugins/drizzle-persistence.cjs",
       "default": "./dist/plugins/drizzle-persistence.mjs"
     }
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

    },
    "./plugins/drizzle-persistence": {
      "types": "./dist/plugins/drizzle-persistence.d.ts",
      "import": "./dist/plugins/drizzle-persistence.mjs",
      "require": "./dist/plugins/drizzle-persistence.cjs",
      "default": "./dist/plugins/drizzle-persistence.mjs"
packages/indexer/src/plugins/dirzzle-persistence.ts (3)

1-207: ⚠️ Potential issue

Correct Typo in Filename and Ensure Consistency

The file is named dirzzle-persistence.ts, but the exported function is drizzlePersistence (line 42). This inconsistency can lead to confusion and import errors. Please rename the file to drizzle-persistence.ts to match the function name and update any related import statements and export paths accordingly.


22-22: ⚠️ Potential issue

Use bigint Data Type for orderKey to Prevent Precision Loss

  • Table Definition (Line 22): orderKey is defined as integer("order_key").notNull(). Since orderKey represents a BigInt, using integer may cause overflow issues for large values.
  • Data Insertion (Lines 147-148, 153-154): Converting cursor.orderKey (a BigInt) to Number may result in precision loss.

Recommend using bigint in your table definition and storing orderKey as string or BigInt.

Apply these diffs:

Update Table Definition:

 export const checkpoints = pgTable("checkpoints", {
   id: text("id").notNull().primaryKey(),
-  orderKey: integer("order_key").notNull(),
+  orderKey: bigint("order_key").notNull(),
   uniqueKey: text("unique_key")
     .$type<`0x${string}` | undefined>()
     .notNull()
     .default(undefined),
 });

Update _putCheckpoint Method:

       .values({
         id: this._indexerName,
-        orderKey: Number(cursor.orderKey),
+        orderKey: cursor.orderKey,
         uniqueKey: cursor.uniqueKey,
       })
       .onConflictDoUpdate({
         target: checkpoints.id,
         set: {
-          orderKey: Number(cursor.orderKey),
+          orderKey: cursor.orderKey,
           uniqueKey: cursor.uniqueKey,
         },
       });

Update _getCheckpoint Method (Lines 137-139):

No change needed if orderKey is retrieved directly as BigInt.

Also applies to: 147-148, 153-154


184-184: ⚠️ Potential issue

Use Consistent Data Types for fromBlock and toBlock in filters Table

  • Table Definition (Lines 34-35): fromBlock and toBlock are declared as integer. Since block numbers can be large, consider using bigint to accommodate larger values.
  • Data Handling (Lines 184, 193, 199): Converting endCursor.orderKey to Number may lead to precision loss.

Apply these diffs:

Update Table Definition:

       filter: text("filter").notNull(),
-      fromBlock: integer("from_block").notNull(),
-      toBlock: integer("to_block"),
+      fromBlock: bigint("from_block").notNull(),
+      toBlock: bigint("to_block"),
     },

Update _putFilter Method:

     // Update existing filter's to_block
     await this._db
       .update(filters)
-      .set({ toBlock: Number(endCursor.orderKey) })
+      .set({ toBlock: endCursor.orderKey })
       .where(and(eq(filters.id, this._indexerName), isNull(filters.toBlock)));

     // Insert new filter
     await this._db
       .insert(filters)
       .values({
         id: this._indexerName,
         filter: serialize(filter as Record<string, unknown>),
-        fromBlock: Number(endCursor.orderKey),
+        fromBlock: endCursor.orderKey,
       })
       .onConflictDoUpdate({
         target: [filters.id, filters.fromBlock],
         set: {
           filter: serialize(filter as Record<string, unknown>),
-          fromBlock: Number(endCursor.orderKey),
+          fromBlock: endCursor.orderKey,
         },
       });

Also applies to: 193-193, 199-199

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🧹 Outside diff range and nitpick comments (2)
examples/cli/indexers/2-starknet.indexer.ts (2)

35-40: Consider adding type safety to plugin configuration

The plugin configuration is functional but could benefit from explicit type definitions.

Consider adding type safety:

plugins: [
  drizzlePersistence({
    database: persistDatabase,
-   indexerName: "2-starknet",
+   indexerName: "2-starknet" as const,
  }),
],

66-71: TODO Comments Indicate Incomplete Implementation

The comments suggest that table definitions should be imported from the plugin package instead of being created directly with SQL.

Would you like me to help implement proper schema imports and migrations setup?

📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL

📥 Commits

Reviewing files that changed from the base of the PR and between 2cb90f8 and fc8449e.

📒 Files selected for processing (3)
  • change/@apibara-indexer-3ab2f0f6-4d43-4499-a0a3-55188fe8eb53.json (1 hunks)
  • examples/cli/indexers/2-starknet.indexer.ts (2 hunks)
  • examples/cli/package.json (2 hunks)
✅ Files skipped from review due to trivial changes (1)
  • change/@apibara-indexer-3ab2f0f6-4d43-4499-a0a3-55188fe8eb53.json
🚧 Files skipped from review as they are similar to previous changes (1)
  • examples/cli/package.json
🔇 Additional comments (2)
examples/cli/indexers/2-starknet.indexer.ts (2)

2-2: LGTM: Import statements are properly structured

The new imports for Drizzle ORM and PostgreSQL client are correctly added and align with the PR's objective of adding persistence functionality.

Also applies to: 8-10


Line range hint 1-91: Verify Plugin Integration

Let's verify that this implementation matches the expected usage of the drizzle persistence plugin.

✅ Verification successful

Based on the plugin implementation from the source code, I can now generate the final response:

The drizzle persistence plugin implementation is correct

The implementation matches the expected usage pattern of the drizzle persistence plugin:

  1. Proper plugin configuration with required parameters:

    • database: Correctly passed as a drizzle PgDatabase instance
    • indexerName: Optional parameter with default value "default"
  2. Correct table structure creation in the run:before hook:

    • checkpoints table with required fields (id, order_key, unique_key)
    • filters table with required fields (id, filter, from_block, to_block)
  3. Proper connection handling:

    • Database client connection established in run:before hook
    • Tables created with correct schema matching the plugin's exported table definitions

The implementation follows all the plugin's requirements and best practices for persistence management.

🏁 Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Search for other implementations or tests of the drizzle persistence plugin
# to ensure consistent usage patterns

# Look for other files using the drizzle persistence plugin
rg -l "drizzlePersistence" --type ts

# Look for test files related to the plugin
fd -e test.ts -e spec.ts | xargs rg "drizzlePersistence"

# Look for documentation or example usage
fd README.md | xargs rg -l "drizzlePersistence"

Length of output: 237


Script:

#!/bin/bash
# Let's examine the plugin implementation to understand the expected usage
cat packages/indexer/src/plugins/dirzzle-persistence.ts

# Also look for any related configuration types or interfaces
ast-grep --pattern 'interface $_PersistenceConfig {
  $$$
}'

# Search for any exports from the plugin file
ast-grep --pattern 'export const $_ = $_'

Length of output: 100794

Comment on lines +23 to +27
// Persistence Database
const client = new Client({
connectionString: "postgres://postgres:postgres@localhost:5432/postgres",
});
const persistDatabase = drizzle(client);
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Security & Configuration Issues in Database Setup

Several critical issues need to be addressed:

  1. Hardcoded database credentials in the connection string pose a security risk
  2. No error handling for database connection failures
  3. Missing configuration options for different environments (dev/staging/prod)

Consider implementing the following changes:

- const client = new Client({
-   connectionString: "postgres://postgres:postgres@localhost:5432/postgres",
- });
+ const client = new Client({
+   connectionString: runtimeConfig.persistenceConnectionString ?? 
+     process.env.PERSISTENCE_DB_URL ??
+     throw new Error("Database connection string not configured"),
+ });

Also, consider adding error handling:

try {
  await client.connect();
} catch (error) {
  logger.error("Failed to connect to persistence database:", error);
  throw error;
}

Comment on lines +72 to +86
await persistDatabase.execute(sql`
CREATE TABLE IF NOT EXISTS checkpoints (
id TEXT NOT NULL PRIMARY KEY,
order_key INTEGER NOT NULL,
unique_key TEXT
);

CREATE TABLE IF NOT EXISTS filters (
id TEXT NOT NULL,
filter TEXT NOT NULL,
from_block INTEGER NOT NULL,
to_block INTEGER,
PRIMARY KEY (id, from_block)
);
`);
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Improve Database Schema Management

Several improvements are recommended for the schema management:

  1. Use Drizzle's schema definitions instead of raw SQL
  2. Implement proper migrations
  3. Add indexes for better query performance

Consider refactoring to use Drizzle's schema definitions:

import { pgTable, text, integer } from 'drizzle-orm/pg-core';

export const checkpoints = pgTable('checkpoints', {
  id: text('id').primaryKey(),
  orderKey: integer('order_key').notNull(),
  uniqueKey: text('unique_key')
});

export const filters = pgTable('filters', {
  id: text('id').notNull(),
  filter: text('filter').notNull(),
  fromBlock: integer('from_block').notNull(),
  toBlock: integer('to_block'),
  // Add composite primary key
}, (table) => ({
  pk: primaryKey(table.id, table.fromBlock)
}));

Comment on lines 62 to 88
hooks: {
async "run:before"() {
await client.connect();

// Normally user will do migrations of both tables, which are defined in
// ```
// import { checkpoints, filters } from "@apibara/indexer/plugins/dirzzle-persistence"
// ```,
// but just for quick testing and example we create them here directly

await persistDatabase.execute(sql`
CREATE TABLE IF NOT EXISTS checkpoints (
id TEXT NOT NULL PRIMARY KEY,
order_key INTEGER NOT NULL,
unique_key TEXT
);

CREATE TABLE IF NOT EXISTS filters (
id TEXT NOT NULL,
filter TEXT NOT NULL,
from_block INTEGER NOT NULL,
to_block INTEGER,
PRIMARY KEY (id, from_block)
);
`);
},
},
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Add Cleanup Hook

Consider adding a cleanup hook to properly close the database connection.

Add a run:after hook:

 hooks: {
   async "run:before"() {
     await client.connect();
     // ... table creation
   },
+  async "run:after"() {
+    try {
+      await client.end();
+    } catch (error) {
+      logger.error("Error closing database connection:", error);
+    }
+  },
 },
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
hooks: {
async "run:before"() {
await client.connect();
// Normally user will do migrations of both tables, which are defined in
// ```
// import { checkpoints, filters } from "@apibara/indexer/plugins/dirzzle-persistence"
// ```,
// but just for quick testing and example we create them here directly
await persistDatabase.execute(sql`
CREATE TABLE IF NOT EXISTS checkpoints (
id TEXT NOT NULL PRIMARY KEY,
order_key INTEGER NOT NULL,
unique_key TEXT
);
CREATE TABLE IF NOT EXISTS filters (
id TEXT NOT NULL,
filter TEXT NOT NULL,
from_block INTEGER NOT NULL,
to_block INTEGER,
PRIMARY KEY (id, from_block)
);
`);
},
},
hooks: {
async "run:before"() {
await client.connect();
// Normally user will do migrations of both tables, which are defined in

Comment on lines 120 to 123
public async del() {
await this._delCheckpoint();
await this._delFilter();
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not used so I think it can go away.

Comment on lines 159 to 163
private async _delCheckpoint() {
await this._db
.delete(checkpoints)
.where(eq(checkpoints.id, this._indexerName));
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not needed.

Comment on lines 204 to 206
private async _delFilter() {
await this._db.delete(filters).where(eq(filters.id, this._indexerName));
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not needed.

@@ -69,6 +69,12 @@
"import": "./dist/plugins/persistence.mjs",
"require": "./dist/plugins/persistence.cjs",
"default": "./dist/plugins/persistence.mjs"
},
"./plugins/dirzzle-persistence": {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
"./plugins/dirzzle-persistence": {
"./plugins/drizzle-persistence": {

Comment on lines 74 to 77
"types": "./dist/plugins/dirzzle-persistence.d.ts",
"import": "./dist/plugins/dirzzle-persistence.mjs",
"require": "./dist/plugins/dirzzle-persistence.cjs",
"default": "./dist/plugins/dirzzle-persistence.mjs"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
"types": "./dist/plugins/dirzzle-persistence.d.ts",
"import": "./dist/plugins/dirzzle-persistence.mjs",
"require": "./dist/plugins/dirzzle-persistence.cjs",
"default": "./dist/plugins/dirzzle-persistence.mjs"
"types": "./dist/plugins/drizzle-persistence.d.ts",
"import": "./dist/plugins/drizzle-persistence.mjs",
"require": "./dist/plugins/drizzle-persistence.cjs",
"default": "./dist/plugins/drizzle-persistence.mjs"

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be drizzle-persistence.ts

@@ -1,26 +1,43 @@
import { defineIndexer, useSink } from "@apibara/indexer";
import { drizzlePersistence } from "@apibara/indexer/plugins/dirzzle-persistence";
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
import { drizzlePersistence } from "@apibara/indexer/plugins/dirzzle-persistence";
import { drizzlePersistence } from "@apibara/indexer/plugins/drizzle-persistence";


// Normally user will do migrations of both tables, which are defined in
// ```
// import { checkpoints, filters } from "@apibara/indexer/plugins/dirzzle-persistence"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// import { checkpoints, filters } from "@apibara/indexer/plugins/dirzzle-persistence"
// import { checkpoints, filters } from "@apibara/indexer/plugins/drizzle-persistence"

@@ -12,6 +12,7 @@ export default defineBuildConfig({
"./src/plugins/kv.ts",
"./src/plugins/logger.ts",
"./src/plugins/persistence.ts",
"./src/plugins/dirzzle-persistence.ts",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
"./src/plugins/dirzzle-persistence.ts",
"./src/plugins/drizzle-persistence.ts",

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Outside diff range and nitpick comments (1)
examples/cli/indexers/2-starknet.indexer.ts (1)

16-17: Consider Consistency in Database Usage

The indexer is using both SQLite (for the sink database) and PostgreSQL (for persistence). Evaluate whether it's necessary to use two different database systems. Consolidating to a single database could simplify the architecture and reduce overhead.

📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL

📥 Commits

Reviewing files that changed from the base of the PR and between fc8449e and 76f1e94.

⛔ Files ignored due to path filters (1)
  • pnpm-lock.yaml is excluded by !**/pnpm-lock.yaml
📒 Files selected for processing (6)
  • change/@apibara-indexer-3ab2f0f6-4d43-4499-a0a3-55188fe8eb53.json (1 hunks)
  • examples/cli/indexers/2-starknet.indexer.ts (2 hunks)
  • examples/cli/package.json (2 hunks)
  • packages/indexer/build.config.ts (1 hunks)
  • packages/indexer/package.json (3 hunks)
  • packages/indexer/src/plugins/drizzle-persistence.ts (1 hunks)
🚧 Files skipped from review as they are similar to previous changes (4)
  • packages/indexer/build.config.ts
  • packages/indexer/package.json
  • examples/cli/package.json
  • change/@apibara-indexer-3ab2f0f6-4d43-4499-a0a3-55188fe8eb53.json
🔇 Additional comments (7)
examples/cli/indexers/2-starknet.indexer.ts (6)

2-2: Import Statement Added Correctly

The drizzlePersistence plugin is correctly imported from @apibara/indexer/plugins/drizzle-persistence.


88-88: Suggestion: Add a Cleanup Hook to Close the Database Connection

Consider adding a run:after hook to properly close the database connection when the indexer stops running. This ensures that resources are released properly and prevents potential connection leaks.

Apply this diff to add the cleanup hook:

 hooks: {
   async "run:before"() {
     // ...existing code...
   },
+  async "run:after"() {
+    try {
+      await client.end();
+    } catch (error) {
+      const logger = useLogger();
+      logger.error("Error closing database connection:", error);
+    }
+  },
 },

35-40: Integration of drizzlePersistence Plugin

The drizzlePersistence plugin is correctly integrated into the indexer configuration. This enables efficient persistence of checkpoint and filter data using Drizzle ORM and PostgreSQL.


24-26: ⚠️ Potential issue

Security Risk: Hardcoded Database Credentials

The hardcoded database connection string in the Client constructor poses a security risk and lacks flexibility across different environments (development, staging, production). It's recommended to use environment variables or configuration files to manage database credentials securely.

Apply this diff to address the issue:

- const client = new Client({
-   connectionString: "postgres://postgres:postgres@localhost:5432/postgres",
- });
+ const client = new Client({
+   connectionString: runtimeConfig.persistenceConnectionString ?? 
+     process.env.PERSISTENCE_DB_URL ??
+     throw new Error("Database connection string not configured"),
+ });

64-64: ⚠️ Potential issue

Lack of Error Handling for Database Connection

The call to await client.connect(); lacks error handling. If the database connection fails, it could lead to unhandled exceptions and application crashes. Wrap the connection logic in a try-catch block to handle potential errors gracefully.

Apply this diff to add error handling:

 async "run:before"() {
+  try {
     await client.connect();
+  } catch (error) {
+    const logger = useLogger();
+    logger.error("Failed to connect to persistence database:", error);
+    throw error;
+  }

     // Normally user will do migrations of both tables...

72-86: 🛠️ Refactor suggestion

Refactor Suggestion: Use Drizzle's Schema Definitions Instead of Raw SQL

Using raw SQL statements to create tables is less maintainable and bypasses the benefits of using an ORM. Consider using Drizzle ORM's schema definitions and migration system to manage your database schema. This approach enhances maintainability and leverages Drizzle's features.

Here's how you can define your tables using Drizzle:

import { checkpoints, filters } from "@apibara/indexer/plugins/drizzle-persistence";

// Use Drizzle's migration system to create tables
await persistDatabase.runMigrations();

Ensure you have proper migration files set up according to Drizzle's documentation.

packages/indexer/src/plugins/drizzle-persistence.ts (1)

1-192: LGTM: Implementation of drizzlePersistence Plugin

The drizzlePersistence plugin is well-implemented, providing a robust persistence layer using Drizzle ORM. The definitions of the checkpoints and filters tables are appropriate, and the plugin correctly integrates with the indexer's lifecycle hooks. The methods for managing checkpoints and filters are clearly defined and utilize Drizzle's capabilities effectively.

Comment on lines +24 to +26
.$type<`0x${string}` | undefined>()
.notNull()
.default(undefined),
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Inconsistency in Column Definition: uniqueKey

The uniqueKey column in the checkpoints table is defined with .notNull().default(undefined). Setting a default value of undefined for a NOT NULL column may not behave as intended. In SQL, undefined is not a valid default value.

Consider adjusting the column definition to allow uniqueKey to be nullable if appropriate:

-    .$type<`0x${string}` | undefined>()
-    .notNull()
-    .default(undefined),
+    .$type<`0x${string}` | null>()
+    .default(null),

Alternatively, if uniqueKey should always have a value, provide a valid default or ensure it's populated correctly when inserting records.

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
.$type<`0x${string}` | undefined>()
.notNull()
.default(undefined),
.$type<`0x${string}` | null>()
.default(null),

@jaipaljadeja jaipaljadeja requested a review from fracek November 30, 2024 17:08
@fracek
Copy link
Contributor

fracek commented Nov 30, 2024

Looks good now!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants