Trying to do transpiled and bundled standalone server #1165

thomasjm · 2023-10-06T11:02:24Z

Description

I would really like to build a server entrypoint that is not only transpiled (as in #562) but also bundled, so that the resulting dist folder is completely standalone. I don't want to worry about providing a separate node_modules folder in production with dependencies like express.

I filed an issue in vavite about this, but I think it may properly be solved at the Vike level. None of the solutions I could find online have worked, but I've come up with a solution which I think is really close. The demo repo is here:

https://github.com/thomasjm/vike-standalone

The idea is simple: I add an extra entry to build.rollupOptions.input pointing at the server/index_prod.js file. (For simplicity, I created a separate index_prod.js file, which is like index.js but with development stuff removed.) Then I set ssr.noExternal to prevent externalizing just about everything (except Node builtins).

At this point, I can build a dist folder which contains dist/server/index.mjs, and just about everything is bundled. The server actually starts up when you run it!

The only issue I'm tripping on is related to @brillout/vite-plugin-import-build. This is the point where I could really use some help, as it seems the tricks of this plugin aren't working in this scenario. The output is below. Could I trouble you @brillout ?

npm run try-standalone

...
Copying dist dir to /tmp/tmp.DO0XbUx8u7
Switching to dir and starting standalone server...
Server running at http://localhost:3000
...
Error: [@brillout/[email protected]][Bug] You stumbled upon a bug in the source code of @brillout/vite-plugin-import-build. Reach out at https://github.com/brillout/vite-plugin-import-build/issues/new and include this error stack (the error stack is usually enough to fix the problem). A maintainer will fix the bug (usually under 24 hours). Don't hesitate to reach out as it makes @brillout/vite-plugin-import-build more robust. Debug info (this is for the @brillout/vite-plugin-import-build maintainers; you can ignore this): `{"distImporterDir":"/tmp/tmp.DO0XbUx8u7/dist/server","distImporterPathUnresolved":"/tmp/tmp.DO0XbUx8u7/dist/server/importBuild.cjs"}`
    at createErrorWithCleanStackTrace (file:///tmp/tmp.DO0XbUx8u7/dist/server/chunks/chunk-d66c6d0c.js:1907:11)
    at assert (file:///tmp/tmp.DO0XbUx8u7/dist/server/chunks/chunk-d66c6d0c.js:1968:93)
    at loadWithNodejs (file:///tmp/tmp.DO0XbUx8u7/dist/server/chunks/chunk-d66c6d0c.js:2235:24)
    at async Object.loadServerBuild$1 [as loadServerBuild] (file:///tmp/tmp.DO0XbUx8u7/dist/server/chunks/chunk-d66c6d0c.js:2181:15)
    at async loadImportBuild (file:///tmp/tmp.DO0XbUx8u7/dist/server/chunks/chunk-d66c6d0c.js:2254:5)
    at async initGlobalContext (file:///tmp/tmp.DO0XbUx8u7/dist/server/chunks/chunk-d66c6d0c.js:2853:26)
    at async renderPageAndPrepare (file:///tmp/tmp.DO0XbUx8u7/dist/server/chunks/chunk-d66c6d0c.js:6380:5)
    at async renderPage_wrapper (file:///tmp/tmp.DO0XbUx8u7/dist/server/chunks/chunk-d66c6d0c.js:6355:22)
    at async renderPage (file:///tmp/tmp.DO0XbUx8u7/dist/server/chunks/chunk-d66c6d0c.js:6371:48)
    at async file:///tmp/tmp.DO0XbUx8u7/dist/server/index.mjs:121876:25

The text was updated successfully, but these errors were encountered:

brillout · 2023-10-06T19:48:48Z

thomasjm/vike-standalone#1

The example is still not working but it seems to be because of an error about how you resolve the static dir; it should be fixable on the user land.

Let me know if you have further questions, and please update the example to make it work so that I can share with others. Keep me updated how it goes.

Thanks for opening this ticket. Closing in favor of #562:

Edit:

Ideally with a bundled/standalone option, see Trying to do transpiled and bundled standalone server #1165 (which includes an example of how Vite can be configured to bundle everything).

thomasjm · 2023-10-06T22:08:29Z

Many thanks for the fast response @brillout ! I tried the PR and it works (if you've already built the project once). I added a question there about how this solution could be made better.

thomasjm · 2023-10-07T00:31:10Z

Maybe I'll continue the discussion here if you don't mind. I just did some more work trying to resolve the importBuild.cjs issue and came up with this: thomasjm/vike-standalone@3eb35dd

The tricky thing is that we want to bundle the reference to vike/dist/esm/node/runtime/globalContext/loadImportBuild.js, but we want to leave the references to ./pageFiles.mjs etc. to be resolved at runtime.

It doesn't quite work yet though and I'd appreciate any help -- it currently fails with this:

Error: [vike][Wrong Usage] Re-build your app (you're using [email protected] but your app was built with vike@undefined)
    at assertPluginManifest (file:///tmp/tmp.B834vb8whI/dist/server/chunks/chunk-4db35243.js:2819:3)
    at initGlobalContext (file:///tmp/tmp.B834vb8whI/dist/server/chunks/chunk-4db35243.js:2875:5)
    at async renderPageAndPrepare (file:///tmp/tmp.B834vb8whI/dist/server/chunks/chunk-4db35243.js:6397:5)
    at async renderPage_wrapper (file:///tmp/tmp.B834vb8whI/dist/server/chunks/chunk-4db35243.js:6372:22)
    at async renderPage (file:///tmp/tmp.B834vb8whI/dist/server/chunks/chunk-4db35243.js:6388:48)
    at async file:///tmp/tmp.B834vb8whI/dist/server/index.mjs:121884:25

thomasjm · 2023-10-07T00:50:14Z

Actually never mind, I got it to work as of thomasjm/vike-standalone@d9bbe82 !

It's super fragile and weird (uses sed to patch up the index.mjs file), but should serve as a proof of concept.

TBH, unless I'm missing something, it's beyond me why #562 and vitejs/vite#12165 are proposing such complicated solutions to this problem. It seems to me there's no need to bring in vavite or vite-node, when the approach I used here could just be bundled up in a Vike plugin.

brillout · 2023-10-07T16:44:20Z

Simpler solution: https://github.com/brillout/vike-standalone/tree/simpler.

it's beyond me why #562 and vitejs/vite#12165 are proposing such complicated solutions to this problem.

It's more complicated in a dev context.

thomasjm · 2023-10-07T23:35:37Z

Simpler solution: https://github.com/brillout/vike-standalone/tree/simpler.

Huh? That's a solution for the static assets problem, which I already fixed: thomasjm/vike-standalone@b521db6

The work I was doing above is to resolve the other problem, the one I've been trying to communicate on thomasjm/vike-standalone#1.

It's more complicated in a dev context.

Is it though?

If you want bundling but no transpilation: my example main branch already works fine for dev and prod
If you want bundling AND transpilation: simply use ts-node in dev, demonstrated in this branch: https://github.com/thomasjm/vike-standalone/tree/transpile-dev. Works in dev and prod.

brillout · 2023-10-08T09:11:36Z

For the other problem, I'm busy with a high prio ticket; I'll then have a look at it. I will circle back on this.

(With dev context I mean without ts-node. E.g. it isn't clear how HMR / auto-reload can be implemented.)

thomasjm · 2023-10-08T09:25:11Z

Sounds good, thanks! No rush, I'm happy since I have a workaround now.

With dev context I mean without ts-node. E.g. it isn't clear how HMR / auto-reload can be implemented.

If you ask me, HMR / auto-reload is a nice-to-have. It's easy to survive without it; the current disclaimer here clearly states what the user has to do.

On the other hand, the ability to create a standalone build at all is a must-have, or else it severely limits how you can deploy a Vike app.

brillout · 2023-10-08T09:26:27Z

I agree.

…

On Sun 8. Oct 2023 at 11:25, Tom McLaughlin ***@***.***> wrote: Sounds good, thanks! No rush, I'm happy since I have a workaround now. With dev context I mean without ts-node. E.g. it isn't clear how HMR / auto-reload can be implemented. If you ask me, HMR / auto-reload is a nice-to-have. It's easy to survive without it; the current disclaimer here <https://github.com/vikejs/vike/blob/ac0bb99f8828d98d88a2525564b90f7c0b9eeabe/examples/graphql-apollo-react/server/index.js#L1-L5> clearly states what the user has to do. On the other hand, the ability to create a standalone build *at all* is a must-have, or else it severely limits how you can deploy a Vike app. — Reply to this email directly, view it on GitHub <#1165 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAHVQRXSZUCH4P6HPRIWJOLX6JWQFAVCNFSM6AAAAAA5VUSGJOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONJRHE3TINZQHA> . You are receiving this because you modified the open/close state.Message ID: ***@***.***>

brillout · 2023-10-09T08:13:16Z

No rush, I'm happy since I have a workaround now.

Actually, I'd like to take some time before tackling this ticket because 1. I want to do some research before starting the implementation and 2. I want to release the V1 design (and soft-deprecate the current design) sooner rather than later as it's blocking the official release of Bati and vike-{react,vue,solid}.

So it's going to take a little while, but I'd be up for working on this right after the V1 design is released (not to be confused with the V1 release). Rough ETA: this month.

nitedani · 2023-10-31T01:18:32Z

https://github.com/vercel/ncc seems to work https://github.com/nitedani/vite-plugin-standalone/tree/main/examples/vike

brillout · 2023-10-31T08:56:34Z

@nitedani 👍

Esbuild usually also works (and it's fast). Any reason you chose ncc over esbuild?

nitedani · 2023-10-31T11:19:38Z

I would like to use esbuild, but esbuild in itself doesn't handle bundling/copying binaries/assets. (prisma, sharp, argon2, etc...) Initially, I tried not to involve https://github.com/vercel/ncc (uses webpack to bundle) and use https://github.com/vercel/nft (only discovery, no bundle) instead to discover the binaries/assets and copy them to the dist folder, and let vite/rollup handle the rest, setting the ssr.externals and ssr.noExternals options. I had issues with https://github.com/vercel/nft, not working correctly in my pnpm monorepo, so I commented the code. I will try it again later.

brillout · 2023-10-31T11:53:13Z

I see. There is also ssr.noExternal: true.

dkourilov · 2023-11-30T18:25:01Z

I recently built a website with Vike and managed to do standalone client, server and SSR HTML builds with react and styled-components in a slightly different way than @thomasjm did.

The ssr.noExternal: true didn't really work for me as it wants to bundle built-ins for server build:

vite v4.5.0 building SSR bundle for production...                                                                                                              
✓ 31 modules transformed.                                                                                                                                      
✓ built in 486ms                                                                                                                                               
[commonjs--resolver] Cannot bundle Node.js built-in "stream" imported from "node_modules/.pnpm/[email protected][email protected]/node_modules/react-dom/cjs/react-d
om-server-legacy.node.production.min.js". Consider disabling ssr.noExternal or remove the built-in dependency.

But a match-all regexp works perfectly, e.g. ssr: {noExternal: /^.*$/}.

Alternatively, if you want to bundle all but package1 and package2, it'll be /^((?!(^package1$|^package2$)).)*$/.

brillout · 2023-11-30T19:56:08Z

@dkourilov Good to know, thanks for sharing. It's a bummer Vite doesn't exclude built-ins automatically.

dkourilov · 2023-11-30T20:17:41Z

@brillout Surprisingly, it doesn't work in vike-react-styled-components-grommet repo of yours 🤦

12:09:18 AM [vike][Warning] Cannot pre-render page /pages/star-wars/@id because it has a non-static route, while no onBeforePrerenderStart() hook returned any URL matching the page's route. ...

With vite.config.ts:

import ssr from "vike/plugin";
import react from "@vitejs/plugin-react";
import { defineConfig } from "vite";

export default defineConfig({
  ssr: { noExternal: /^.*$/ },
  plugins: [
    react({
      jsxRuntime: "classic",
    }),
    ssr({
      prerender: true
    })
  ],
});

But it looks like Vike issue now, with new layout and router. I was using old vite-plugin-ssr convention for my website, thanks for maintaining the backward compat and good luck fixing this.

brillout · 2023-11-30T20:26:31Z

@dkourilov PR welcome to update the example.

brillout · 2023-12-09T13:21:48Z

Lean, robust, and I guess (very) fast: https://github.com/nitedani/vite-plugin-standalone. (See source code to understand what it does). It's quite a neat idea and we'll likely make it a built-in Vike feature.

simplecommerce · 2024-01-30T16:41:47Z

Member

I've tested this approach on my test project and it seems to work very nicely.
I encountered an issue where I didn't know what to do, so I reverted some code I had changed in the past to fix an issue.

If we fork processes using fork("process.js"), the process.js file doesn't get bundled up and compiled.
I had to manually add it to the roll up options but I then encountered the issue where it was trying to import modules that weren't bundled either.

Do you guys have ideas or suggestions on how to approach this in these specific cases?

I don't need it anymore, but in case I ever do, I figured i'd ask.

nitedani · 2024-01-31T04:31:43Z

@simplecommerce
I tried to add support for multiple server-only entries.
Check out https://github.com/nitedani/vite-plugin-standalone/tree/main/examples/argon2
Now, the worker.js is also bundled, but needs to be set in vite.config.ts like so:

  standalone({ workers: ["./src/worker.js"] }),

simplecommerce · 2024-01-31T09:06:00Z

@simplecommerce I tried to add support for multiple server-only entries. Check out https://github.com/nitedani/vite-plugin-standalone/tree/main/examples/argon2 Now, the worker.js is also bundled, but needs to be set in vite.config.ts like so:
  standalone({ workers: ["./src/worker.js"] }),

I will try that, thanks a lot!

UPDATE

@nitedani

It seems to compile the code and add it to the bundle but it doesn't seem to bundle the imports that the worker code has inside.

For example, in my code I have:

import WebSocket from "ws";
import { createClient } from "graphql-ws";
import { isEmpty } from "lodash-es";

And in the output code I see

import o from"ws";import{createClient as r}from"graphql-ws";import{isEmpty as n}from"lodash-es";

Am I missing something?

nitedani · 2024-02-01T03:22:17Z

@simplecommerce
I opened an issue here: nitedani/vite-plugin-standalone#1

thomasjm · 2024-05-02T22:03:44Z

Hmm, I don't think I'm a fan of the vite-plugin-standalone approach because it uses a whole separate set of tools (vercel/nft and esbuild). I use Vike because I like Vite, and I'd like my server build to be compiled using the same stuff as the client build. (Also, it's not clear to me that it has all the features of a proper bundler, like tree-shaking?)

I was happy with my workaround which I posted above, which I put in this commit: thomasjm/vike-standalone@3eb35dd. This worked great for building everything with Vike/Vite.

However, I recently wanted to upgrade to the Vike v1 API + versions of Vike later than 0.4.149, but the workaround I had before breaks. It seems like there have been some changes with how server auto-importing works, perhaps involving vite-plugin-server-entry?

I saw that there was this commit f555646 which claims to make standalone builds easier, but I'm having trouble understanding how :).

Any chance we could get an example in the repo examples directory of doing a standalone build?

thomasjm · 2024-05-02T22:33:16Z

Update: I've updated my example and gotten it working with a slightly different weird hack:

https://github.com/thomasjm/vike-standalone/

I'd love some help understanding why this hack is needed. Once it's fixed I'd be happy to help make this into an official example.

nitedani · 2024-05-03T01:24:11Z

Hmm, I don't think I'm a fan of the vite-plugin-standalone approach because it uses a whole separate set of tools (vercel/nft and esbuild).

Esbuild and Rollup are shipped with Vite, and Esbuild is much faster than Rollup for bundling. In the future I imagine we can use Rolldown instead and remove the Esbuild step, making it more similar to your example. In standalone mode, Rollup is still used to bundle your server code(not library imports), and only the imported libraries are bundled with Esbuild in the final step. This is similar to how Vite works in dev for server code (Esbuild is used for dep-optimization in dev).
vercel/nft is only used for discovering binaries, those can't be bundled with javascript. The code needs to be analyzed for imported binaries, and the imported binaries/assets need to be copied to create a standalone output. This is needed for, for example node-rs, sharp, prisma, ... libraries

brillout · 2024-05-03T08:50:35Z

The plan is to have it built-in; nitedani already implemented a PR but there are a couple of blockers to merge it (most notably the Vike CLI which is a breaking change). Hopefully we can land the PRs sooner rather than later (I'm currently busy with other high priority tickets). In the meantime, if your company is up for it, you can consider sponsoring (also for getting a priority bump).

Update: I've updated my example and gotten it working

👍 If you run into another issue, update the example to be a reproduction and we'll have a look at it.

thomasjm · 2024-10-01T03:45:20Z

Hello, it's been a few months so I thought I'd ask if there's been any progress on making standalone builds a built-in part of Vike. My weird hacks still work, but it's a bit of a pain to keep tweaking them when I update to new versions of Vike or Vite.

I'd love to know if there's an ETA for this, or some way to follow along with the progress. Is #1434 the PR to watch? Thanks!

brillout · 2024-10-01T06:21:42Z

@thomasjm In case you didn't see it already, have a look at vike-node. It's ready (or correct me if I'm wrong @nitedani) although we want to restructure it a bit. In a nutshell: you can use it (and we recommend you to), but you should expect a breaking change soon.

Note

We want to wrap vike-node for each server framework (vike-express, vike-hono, vike-fastify, vike-cloudflare, ...) to include universal-middleware so that the universal server middleware of tools (Vike, Telefunc, Auth.js, etc.) are automatically integrated.

@nitedani I'm thinking, one thing that could be nice is to offer different "levels of support":

Using vike-node only => no universal-middleware.
Using vike-{express,hono,...} => vike-node + universal-middleware

The user would have the choice, although we'd only advertise vike-{express,hono,...}.

thomasjm added the bug 💥 label Oct 6, 2023

brillout mentioned this issue Aug 30, 2023

Transpile server code #562

Open

brillout closed this as completed Oct 6, 2023

brillout added enhancement ✨ New feature or request and removed bug 💥 labels Oct 8, 2023

brillout reopened this Oct 8, 2023

brillout added the v1 release 🌟 label Oct 24, 2023

brillout mentioned this issue Oct 24, 2023

Dist file generated in workspace, makes dist unusable #1204

Closed

brillout added contribution-welcome 💕 highest-prio 🔥 labels Nov 22, 2023

brillout added the server 🌐 label Dec 9, 2023

brillout added a commit that referenced this issue Dec 11, 2023

fix: make standalone builds easier (#1165, #1342)

f555646

brillout added a commit to brillout/telefunc that referenced this issue Dec 11, 2023

fix: make standalone builds easier (vikejs/vike#1165, vikejs/vike#1342)

3765c56

nitedani pushed a commit that referenced this issue Dec 11, 2023

fix: make standalone builds easier (#1165, #1342)

eef0855

smart94423 added a commit to smart94423/telefunc that referenced this issue Jan 30, 2024

fix: make standalone builds easier (vikejs/vike#1165, vikejs/vike#1342)

81fb19d

brillout removed the server 🌐 label Mar 14, 2024

brillout removed the contribution-welcome 💕 label Nov 1, 2024

thomasjm mentioned this issue Jan 20, 2025

Bug when trying to use vike-node prod build #2094

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Trying to do transpiled and bundled standalone server #1165

Trying to do transpiled and bundled standalone server #1165

thomasjm commented Oct 6, 2023

brillout commented Oct 6, 2023

thomasjm commented Oct 6, 2023

thomasjm commented Oct 7, 2023

thomasjm commented Oct 7, 2023

brillout commented Oct 7, 2023

thomasjm commented Oct 7, 2023

brillout commented Oct 8, 2023 •

edited

Loading

thomasjm commented Oct 8, 2023

brillout commented Oct 8, 2023 via email

brillout commented Oct 9, 2023

nitedani commented Oct 31, 2023

brillout commented Oct 31, 2023

nitedani commented Oct 31, 2023

brillout commented Oct 31, 2023

dkourilov commented Nov 30, 2023 •

edited

Loading

brillout commented Nov 30, 2023

dkourilov commented Nov 30, 2023

brillout commented Nov 30, 2023

brillout commented Dec 9, 2023

simplecommerce commented Jan 30, 2024

nitedani commented Jan 31, 2024 •

edited

Loading

simplecommerce commented Jan 31, 2024 •

edited

Loading

nitedani commented Feb 1, 2024

thomasjm commented May 2, 2024

thomasjm commented May 2, 2024

nitedani commented May 3, 2024

brillout commented May 3, 2024

thomasjm commented Oct 1, 2024

brillout commented Oct 1, 2024

Trying to do transpiled and bundled standalone server #1165

Trying to do transpiled and bundled standalone server #1165

Comments

thomasjm commented Oct 6, 2023

Description

brillout commented Oct 6, 2023

thomasjm commented Oct 6, 2023

thomasjm commented Oct 7, 2023

thomasjm commented Oct 7, 2023

brillout commented Oct 7, 2023

thomasjm commented Oct 7, 2023

brillout commented Oct 8, 2023 • edited Loading

thomasjm commented Oct 8, 2023

brillout commented Oct 8, 2023 via email

brillout commented Oct 9, 2023

nitedani commented Oct 31, 2023

brillout commented Oct 31, 2023

nitedani commented Oct 31, 2023

brillout commented Oct 31, 2023

dkourilov commented Nov 30, 2023 • edited Loading

brillout commented Nov 30, 2023

dkourilov commented Nov 30, 2023

brillout commented Nov 30, 2023

brillout commented Dec 9, 2023

simplecommerce commented Jan 30, 2024

nitedani commented Jan 31, 2024 • edited Loading

simplecommerce commented Jan 31, 2024 • edited Loading

nitedani commented Feb 1, 2024

thomasjm commented May 2, 2024

thomasjm commented May 2, 2024

nitedani commented May 3, 2024

brillout commented May 3, 2024

thomasjm commented Oct 1, 2024

brillout commented Oct 1, 2024

brillout commented Oct 8, 2023 •

edited

Loading

dkourilov commented Nov 30, 2023 •

edited

Loading

nitedani commented Jan 31, 2024 •

edited

Loading

simplecommerce commented Jan 31, 2024 •

edited

Loading