Initial commit

.gitignore

@@ -0,0 +1,69 @@
+# Node.js
+node_modules/
+npm-debug.log*
+yarn-debug.log*
+yarn-error.log*
+package-lock.json
+yarn.lock
+.pnpm-store
+
+# Editor directories and files
+.idea
+.vscode
+*.suo
+*.ntvs*
+*.njsproj
+*.sln
+*.sw?
+
+# Operating System Files
+.DS_Store
+Thumbs.db
+
+# Build output
+/dist
+/build
+/out
+
+# Logs
+logs
+*.log
+npm-debug.log*
+yarn-debug.log*
+yarn-error.log*
+
+# Dependency directories
+node_modules/
+jspm_packages/
+
+# Optional npm cache directory
+.npm
+
+# Optional eslint cache
+.eslintcache
+
+# Optional REPL history
+.node_repl_history
+
+# Output of 'npm pack'
+*.tgz
+
+# Yarn Integrity file
+.yarn-integrity
+
+# dotenv environment variables file
+.env
+.env.test
+
+# parcel-bundler cache (https://parceljs.org/)
+.cache
+.parcel-cache
+
+# Next.js build output
+.next
+.next/cache
+
+
+.vercel
+
+service-account.json

README.md

@@ -0,0 +1,82 @@
+<img width="1200" alt="github photo (2)" src="https://github.com/user-attachments/assets/2ac55f44-e343-4ce0-9e04-98965f7ddbe5">
+
+<br>
+<br>
+
+## What is 🍙 Dubbie?
+
+**[Dubbie](https://dubbie.com) is an open-source AI dubbing studio that costs $0.1/min**, which is about ~20x less than alternatives like ElevenLabs, RaskAI, or Speechify. While still in early development and not at feature parity with these alternatives, Dubbie offers enough features to create dubs for basic videos.
+
+Here we focus on the technical aspects of Dubbie. For more on motivations and mission, see [why we made Dubbie](http://dubbie.com/blog/why).
+
+For questions/bugs/contributions, join our [Discord server](https://discord.gg/qJNV93PY2e).
+
+<br>
+
+## What is Dubbie built with?
+
+- **NextJS 14**: Client app (app.dubbie.com)
+- **Tailwind**: Styling
+- **ShadcnUI**: Components
+- **Prisma**: Database interface (Postgres)
+- **Clerk**: User authentication
+- **Stripe**: Payments
+- **Openrouter**: LLM selection for best-fit tasks
+- **Azure/OpenAI**: Voice generation
+- **Firebase**: Storage
+- **NodeJS**: Longer running functions (initialization/exporting)
+
+Note: These choices are more due to personal preference/experience rather than the "best".
+
+<br>
+
+## How are the folders structured?
+This project is a monorepo with 4 packages
+1. `/next`
+2. `/node`
+3. `/shared`
+4. `/db`
+
+
+`next` and `node` are applications that are deployed to vercel/railway.
+`db` contains our prisma schema + client. 
+`shared` contains individual functions that are used for inside of both `next` and `node`.
+
+You be wondering, next is frontend(web runtime) and node is backend(node runtime), how can they use the same functions? To be honest, the code is not perfectly organized, and not all code in `shared` can be used in both web and node runtime. However, since NextJS is _actually_ just a node server that's serving web pages, most of the code in shared can be used in the "Server Actions" side of NextJS. This may be a little confusing if you've not familiar with Next14, RSC and Server Actions. But it really does make things a lot easier!
+
+<br>
+
+## How does the dubbing initialization process work?
+
+1. The user uploads the video and click “create project”
+2. Upload the video to firebase storage
+3. Extract the audio and upload it to firebase storage as well
+4. Transcribe the audio via Whisper
+    - Which will give us the entire transcription in a big paragraph, as well as time stamps for each word.
+5. Use an LLM to break down the entire paragraph into individual sentences.
+6. Match the individual sentences with the word level timestamps to figure out when each sentence begins and ends.
+    - Since the LLM output may not be “perfect” match, we will then use an approximation algorithm.
+7. Use an LLM to translate each sentence it into the language the user selected.
+    - We do this translation chunk by chunk, and use certain techniques to ensure the output matches the input.
+8. Use a text to speech API(currently just Azure and OpenAI) to generate audio!
+9. Upload those audio files to firebase storage, and save the URLs to our database via Prisma.
+10. The frontend client updates and renders all of that so users can preview realtime and edit
+
+<br>
+
+## How does the frontend editor work?
+On a high level: there are 3 elements that we need to sync
+1. Video element
+2. Timeline scrubber
+3. Invisible audio player
+
+Tone.js connects individual audio URLs and serves as the main timer. See `useAudioTrack.ts` for implementation details.
+
+<br>
+
+## Known limits
+
+- client bugs
+  - Regenerating segment and moving segments sometimes crashes the Next.js app. Unclear why. But there's some sorts of race condition happening.
+- backend scale
+  - currently there's only one instance of the backend. There are no limits/queues either. So it's almost guarantee'd that if many people uses it at once, it'll crash. I don't have too much experience with this, so if you wanna help that'd be very much appreciated :)

biome.json

@@ -0,0 +1,52 @@
+{
+  "$schema": "https://biomejs.dev/schemas/1.8.1/schema.json",
+  "organizeImports": { "enabled": true },
+  "linter": {
+    "enabled": true,
+    "rules": {
+      "recommended": true,
+      "a11y": {
+        "useKeyWithClickEvents": {
+          "level": "off"
+        }
+      },
+      "style": {
+        "useImportType": {
+          "level": "off"
+        }
+      },
+      "suspicious": {
+        "noExplicitAny": {
+          "level": "off"
+        },
+        "noImplicitAnyLet": "off",
+        "noArrayIndexKey": "off"
+      }
+    },
+    "ignore": ["node_modules/", ".next"]
+  },
+  "formatter": {
+    "enabled": true,
+    "formatWithErrors": false,
+    "attributePosition": "auto",
+    "indentStyle": "space",
+    "indentWidth": 2,
+    "lineEnding": "lf",
+    "lineWidth": 80
+  },
+  "javascript": {
+    "formatter": {
+      "arrowParentheses": "always",
+      "bracketSameLine": false,
+      "bracketSpacing": true,
+      "jsxQuoteStyle": "double",
+      "quoteProperties": "asNeeded",
+      "semicolons": "always"
+    }
+  },
+  "json": {
+    "formatter": {
+      "trailingCommas": "none"
+    }
+  }
+}

nixpacks.toml

@@ -0,0 +1,2 @@
+[phases.setup]
+nixPkgs = ["...", "ffmpeg"]

package.json

@@ -0,0 +1,23 @@
+{
+  "name": "dubbie-mono",
+  "private": true,
+  "workspaces": [
+    "packages/*"
+  ],
+  "packageManager": "[email protected]",
+  "scripts": {
+    "next:build": "pnpm --filter next build",
+    "next:start": "pnpm --filter next start",
+    "node:start": "pnpm --filter node start",
+    "postinstall": "pnpm g",
+    "d": "pnpm --filter next dev",
+    "dn": "pnpm --filter node dev",
+    "t": "pnpm --filter node test",
+    "g": "pnpm --filter @dubbie/db generate",
+    "dp": "pnpm --filter @dubbie/db db-push",
+    "sd": "pnpm --filter @dubbie/db studio"
+  },
+  "devDependencies": {
+    "@biomejs/biome": "1.8.1"
+  }
+}

packages/db/index.ts

@@ -0,0 +1,15 @@
+import { PrismaClient } from "@prisma/client";
+
+const prismaClientSingleton = () => {
+  return new PrismaClient();
+};
+
+declare global {
+  var prismaGlobal: undefined | ReturnType<typeof prismaClientSingleton>;
+}
+
+const prisma = globalThis.prismaGlobal ?? prismaClientSingleton();
+if (process.env.NODE_ENV !== "production") globalThis.prismaGlobal = prisma;
+
+export { prisma };
+export * from "@prisma/client"; // This line re-exports all exports from Prisma client

packages/db/package.json

@@ -0,0 +1,19 @@
+{
+  "name": "@dubbie/db",
+  "version": "1.0.0",
+  "description": "",
+  "scripts": {
+    "generate": "pnpm exec prisma generate",
+    "db-push": "pnpm exec prisma db push",
+    "studio": "pnpm exec prisma studio"
+  },
+  "keywords": [],
+  "author": "",
+  "license": "ISC",
+  "dependencies": {
+    "@prisma/client": "5.12.1"
+  },
+  "devDependencies": {
+    "prisma": "5.12.1"
+  }
+}

packages/db/prisma/migrations/20240516123611_rename_track_column/migration.sql

@@ -0,0 +1,65 @@
+-- CreateEnum
+CREATE TYPE "AcceptedLanguage" AS ENUM ('original', 'english', 'mandarin', 'spanish', 'hindi', 'arabic', 'portuguese', 'bengali', 'russian', 'japanese', 'french');
+
+-- CreateEnum
+CREATE TYPE "ProjectStatus" AS ENUM ('NOT_STARTED', 'TRANSCRIBING', 'FORMATTING', 'TRANSLATING', 'AUDIO_PROCESSING', 'COMPLETED');
+
+-- CreateEnum
+CREATE TYPE "ExportStatus" AS ENUM ('NOT_STARTED', 'EXPORTING', 'EXPORTED', 'ERROR');
+
+-- CreateEnum
+CREATE TYPE "MediaType" AS ENUM ('AUDIO', 'VIDEO');
+
+-- CreateTable
+CREATE TABLE "Project" (
+    "userId" TEXT NOT NULL,
+    "status" "ProjectStatus" NOT NULL DEFAULT 'NOT_STARTED',
+    "name" TEXT NOT NULL,
+    "originalMediaType" "MediaType" NOT NULL,
+    "originalUrl" TEXT NOT NULL,
+    "originalLanguage" "AcceptedLanguage",
+    "targetLanguage" "AcceptedLanguage" NOT NULL,
+    "thumbnailUrl" TEXT NOT NULL,
+    "transcription" TEXT,
+    "exportedUrl" TEXT,
+    "exportType" "MediaType",
+    "exportStatus" "ExportStatus" NOT NULL DEFAULT 'NOT_STARTED',
+    "id" TEXT NOT NULL,
+    "createdAt" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP,
+    "updatedAt" TIMESTAMP(3) NOT NULL,
+
+    CONSTRAINT "Project_pkey" PRIMARY KEY ("id")
+);
+
+-- CreateTable
+CREATE TABLE "Track" (
+    "projectId" TEXT NOT NULL,
+    "language" "AcceptedLanguage" NOT NULL,
+    "id" TEXT NOT NULL,
+
+    CONSTRAINT "Track_pkey" PRIMARY KEY ("id")
+);
+
+-- CreateTable
+CREATE TABLE "Segment" (
+    "id" TEXT NOT NULL,
+    "index" INTEGER NOT NULL,
+    "startTime" DOUBLE PRECISION NOT NULL,
+    "endTime" DOUBLE PRECISION NOT NULL,
+    "text" TEXT NOT NULL,
+    "audioUrl" TEXT,
+    "createdAt" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP,
+    "updatedAt" TIMESTAMP(3) NOT NULL,
+    "trackId" TEXT NOT NULL,
+
+    CONSTRAINT "Segment_pkey" PRIMARY KEY ("id")
+);
+
+-- CreateIndex
+CREATE UNIQUE INDEX "Track_projectId_language_key" ON "Track"("projectId", "language");
+
+-- AddForeignKey
+ALTER TABLE "Track" ADD CONSTRAINT "Track_projectId_fkey" FOREIGN KEY ("projectId") REFERENCES "Project"("id") ON DELETE RESTRICT ON UPDATE CASCADE;
+
+-- AddForeignKey
+ALTER TABLE "Segment" ADD CONSTRAINT "Segment_trackId_fkey" FOREIGN KEY ("trackId") REFERENCES "Track"("id") ON DELETE RESTRICT ON UPDATE CASCADE;

packages/db/prisma/migrations/migration_lock.toml

@@ -0,0 +1,3 @@
+# Please do not edit this file manually
+# It should be added in your version-control system (i.e. Git)
+provider = "postgresql"