diff --git a/docs/ESM-dependency-upgrade/How_we_upgraded_the_ESM_dependency_using_Chalk_as.md b/docs/ESM-dependency-upgrade/How_we_upgraded_the_ESM_dependency_using_Chalk_as.md new file mode 100644 index 00000000000..9b5764ebdff --- /dev/null +++ b/docs/ESM-dependency-upgrade/How_we_upgraded_the_ESM_dependency_using_Chalk_as.md @@ -0,0 +1,202 @@ +# How we upgraded the ESM dependency using Chalk as the first + +## Introduction + +ESM and CommonJS are two of Javascript module systems. Modules are pieces of code that have can be shared and reused in a module system. CJS was the default module system for Node.js. Each file is a separate module which can be imported and exported in your code. However, browsers did not have a module system, and it has taken a long time for one to be developed as it first needs to be standardised by ECMAScript and then be able to be used in browsers. This has now been completed in ES6 and now ESM are the standard for using modules in the browser. It is now supported natively in most web browsers, and is fast and comes with new features. You may recognise some of the syntax already as it has been used in React before it became standard, and has had to be transpiled by the tool Babel we use. + +So there are now two module systems, ESM for browsers and CJS for Node. Unfortunately, these two systems use different syntax and work a bit differently. To use both of them, we need to change some things. CJS modules can be imported in ESM, but ESMs need workarounds to be used in a codebase of mostly CommonJS. + +Here is a brief summary of the differences between these two ways of managing the code, from https://adamcoster.com/blog/commonjs-and-esm-importexport-compatibility-examples: + +- CommonJS uses the `require('./file.js')` syntax for importing other modules and the `module.exports =` syntax for exporting stuff from modules +- ESM uses the `import {stuff} from './file.js'` syntax for importing and the `export stuff` syntax for exports +- CommonJS files can use the `.cjs` extension to tell Node that they are in CommonJS +- ESM files can use the `.mjs` extension to tell Node that they are in ESM +- CommonJS imports are _synchronous_ +- ESM imports are _asynchronous_ (which also allows for [top-level](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/await#top_level_await) `[await](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/await#top_level_await)`) +- CommonJS works in Node but does **not** work in browsers +- ESM is supported by all modern browsers and the latest versions of Node, but does **not** work at all in Node versions below 12 +- Tons of the core JavaScript ecosystem tooling was developed in Node, and Node only recently started to support ESM, so a huge fraction of existing Node projects are in CommonJS + +## Starting the upgrade with Chalk as an example + +We started by upgrading chalk in the package.json of the entire application. + +![chalk version update](./images/chalk-version-update.png) + + + +After running `yarn`, you will also see changes in the yarn.lock file + +![](./images/chalk%20yarn%20lock.png) + + + +After this, we allowed the github actions to run on the branch to see where we would get errors that informed us to where we needed to change syntax. The rest of the document will show what changes we had to make to use the ESM chalk. When you are upgrading other ESM dependencies, the places you need to make changes will differ, however some of the changes will be similar e.g adding the “type”: “module” declaration, changing the way you import and export files, using a different mocking method in tests and using asynchronous imports. + +## Mark code as ES Module + +One important code change that needs to happen to be able to use an ESM in CJS is declaring “type”: “module” in a package.json that is within the folder of your code using the ESM. This is telling node that ESM syntax will be used in this folder’s files. + +![](./images/chalk%20package%20json.png) + + +In this screenshot, we have a package.json declaring this within the bundleSize folder. It is here because this is the folder in which Chalk, the ESM we are first using to test upgrading ESM dependencies, is being used in the code. + +There is another way to mark code as ESM, which is used if you don’t want to it for a whole folder, but just for one file. You can change the file extension to .mjs to show that it is a script with which you want to use ESM. Similarly, if you have declared your code as ESM package-wide, you can declare particular files within that as CommonJS with the file extension .cjs. I have not tried doing this yet so haven’t tested that it works, but the documentation says it is possible. + +## Changing the build and test scripts + +We also need to execute node with ` --``experimental-modules `. Yes it is still experimental. Yes it will be ‘fun’ for a while. + +We added this command to the scripts for `build` and `test:unit`. This allows ESM to work for command line script rather than CommonJS. + +![](./images/build%20script.png) + +![](./images/tes%20unit%20script.png) + + +And to run this locally when developing, I would use ` --``experimental-modules ` in my terminal command. + +In the `build` command you can see we also call the seperate file runBundleSize.js that pulls in our new modularised code and runs the script. + +## Jest version + +To be able to use the most up to date methods of mocking in ESM, we needed to upgrade to the latest version of Jest. This will not be covered here as it is not related to ESM dependency upgrades apart from you may need to keep it up to date to use all the features. It may need to continue to be updated to get the newest features of ESM that may be necessary. + +Development of ESM tracking ticket https://github.com/jestjs/jest/issues/9430 + +## Imports + +The way things are imported must be changed. Screenshot from https://dev.to/abbeyperini/tldr-commonjs-vs-esm-47dk. + +![](./images/imports%20external%20screenshot.png) + + +The `import` keyword must be used. Here are some examples of the import changes we made in our code: + +![](./images/getBundleData%20imports.png) + + +![](./images/bundleSize%20index%20imports.png) + + +As Andrew says here, JSON can no longer be directly imported and instead we need to use `readFile` to read the file and `JSON.parse` to parse the JSON. + +![](./images/readFile%20JSON%20parse.png) + + +Jest also needs to be imported into each test file. + +![](./images/import%20jest.png) + + +## Exports + +The way things are exported must be changed. Screenshots from https://dev.to/abbeyperini/tldr-commonjs-vs-esm-47dk. + +![](./images/exports%20commonjs%20external.png) + +![](./images/exports%20esm%20external.png) + + +The `export` keyword must be used. +And with examples in our code: + +![](./images/export%20pages.png) + +![](./images/export%20bundlesize%20config.png) + +![](./images/service%20list.png) + +![](images/export%20getBundleData.png) + +![](./images/export%20averagebundlesize.png) + + +And an example with an `import` and an `export`: + +![](./images/import%20and%20export.png) + + +VScode recognises the need to change the type of export, and has a quick fix for it: +Hover over the highlighted problem (3 dots may also appear underneath), and when the pop up appears with the Quick Fix, you can select the ‘Convert to ES module” and it will do some of this for you. + +![](./images/convert%20to%20es%20module%20vscode.png) + + +## Mocking + +One of the new features the latest version of Jest allows us to use is `jest.unstable_mockModule`. + +Jest doesn’t yet have full support for ES modules, but it does have experimental support. Normally we would use `jest.mock()`, but as this can’t be used with ESM yet (Jest maintainers plan to support it in the future but do not yet have an official API ready), we instead can use the in-development experimental API. + +There are differences in how we use this, compared to `jest.mock()`. For `jest.unstable_mockModule`, the previously optional factory function is now mandatory. A factory function is a function that creates an object and returns it, and can be considered like a constructor function. ` j``est.mock() ` has a useful automatic mock that mocks classes or constructor functions for you. When using `jest.mock()` you might still want to mock manually for more control with fake data, but the great thing about it is it allows for concise code and speed - one line of code for your mocking and not needing to write all the rest in yourself. With `jest.unstable_mockModule` we do not have the auto-mocking ability, and we must write the inside of the mock out by hand - ‘manual’ mocking. You can see this in the following screenshots where we have had to specify the named export mock functions in `fs` that we want to use. We would not be able to access and use `readdirSync`, for example, automatically. + +![](./images/unstable%20mockModule%20servicelist.png) + + +![](./images/unstable%20mockModule%20fs%20promises.png) + + +![](./images/unstable%20mockModule%20chalk.png) + + +![](./images/unstable%20mockModule%20ora.png) + + +If you have defined mocks in a `__mocks__` folder , these now also need to be manually loaded. +Example within a jest.unstable_mockModule: + +![](./images/import%20mocks%20external%20screenshot.png) + + +Example within our code of it in an import: + +![](./images/import%20mocks%20internal%20screenshot.png) + + +## Asynchronous dynamic importing + +`jest.unstable_mockModule` must be called at the top level, before the import for the module you are mocking. Then we need to use a dynamic import (instead of `require`). If we were using CommonJS, jest would automatically hoist the mock calls so that the mocking would happen before the imports. This does not work in ESM, so we must _explicitly_ use `jest.unstable_mockModule` before the mock modules that use the module resources are imported. See the following screenshots for examples of importing dynamically and awaiting those imports: + +![](./images/async%20import%20ora%20chalk.png) + + +![](./images/async%20import%20pageTypeBundleExtractor.png) + + +![](./images/async%20import%20fs.png) + + +![](./images/async%20import%20bundlesize%20index.png) + + +![](./images/async%20import%20getPageBundleData.png) + + +![](./images/async%20import%20bundlesize%20index%202.png) + + +![](./images/async%20import%20bundlesize%20index%203.png) + + +![](./images/async%20import%20bundlesize%20index%204.png) + + +See how in all these examples we have the word ‘await’ and also the word ‘async’ as the function is now asyncronous. + +AN IMPORTANT NOTE: + +During our work on updating this dependency and using `jest.unstable_mockModule` for the first time, we found that because the API is not yet mature and is experimental, if you break ONE mock, it breaks ALL the mocks. Something to do with the injection across modules affecting them all. This is a problem with the API and will likely get better with more development time on it. In the meantime, make sure that your errors aren’t being caused by one broken mock before you start making lots of changes across the code trying to work out how to fix the error. We spent a lot of time on this before Andrew worked out what was happening. It was not mentioned in any docs online and led us on a wild goose chase. + +## Typescript + +We had some problems running the bundleSize script with typescript files from outside the directory. We tried to use `loadableConfig.ts`, which had a list of all the services for use in the tests, but it is outside the bundle size tests folder, and we found that we had difficulty running the bundleSize script from the command line, as typescript compiles the script into JS to run on the command line. We couldn’t get this compilation to happen, and decided it was not worth the time trying to do it this way, and instead to make a new file with the list of service that isn’t typescript and was in the same directory. + +![](./images/service%20list.png) + + +For further examples of code changes we made to upgrade chalk, please refer to the [Upgrade Chalk Dependency (ESM) PR](https://github.com/bbc/simorgh/pull/10937/files) + + diff --git a/docs/ESM-dependency-upgrade/images/async import bundlesize index 2.png b/docs/ESM-dependency-upgrade/images/async import bundlesize index 2.png new file mode 100644 index 00000000000..16748e62da8 Binary files /dev/null and b/docs/ESM-dependency-upgrade/images/async import bundlesize index 2.png differ diff --git a/docs/ESM-dependency-upgrade/images/async import bundlesize index 3.png b/docs/ESM-dependency-upgrade/images/async import bundlesize index 3.png new file mode 100644 index 00000000000..0b36876ee34 Binary files /dev/null and b/docs/ESM-dependency-upgrade/images/async import bundlesize index 3.png differ diff --git a/docs/ESM-dependency-upgrade/images/async import bundlesize index 4.png b/docs/ESM-dependency-upgrade/images/async import bundlesize index 4.png new file mode 100644 index 00000000000..1e68b439237 Binary files /dev/null and b/docs/ESM-dependency-upgrade/images/async import bundlesize index 4.png differ diff --git a/docs/ESM-dependency-upgrade/images/async import bundlesize index.png b/docs/ESM-dependency-upgrade/images/async import bundlesize index.png new file mode 100644 index 00000000000..eebd9e6f87b Binary files /dev/null and b/docs/ESM-dependency-upgrade/images/async import bundlesize index.png differ diff --git a/docs/ESM-dependency-upgrade/images/async import fs.png b/docs/ESM-dependency-upgrade/images/async import fs.png new file mode 100644 index 00000000000..452982adc68 Binary files /dev/null and b/docs/ESM-dependency-upgrade/images/async import fs.png differ diff --git a/docs/ESM-dependency-upgrade/images/async import getPageBundleData.png b/docs/ESM-dependency-upgrade/images/async import getPageBundleData.png new file mode 100644 index 00000000000..0889eccb03f Binary files /dev/null and b/docs/ESM-dependency-upgrade/images/async import getPageBundleData.png differ diff --git a/docs/ESM-dependency-upgrade/images/async import ora chalk.png b/docs/ESM-dependency-upgrade/images/async import ora chalk.png new file mode 100644 index 00000000000..854b97df331 Binary files /dev/null and b/docs/ESM-dependency-upgrade/images/async import ora chalk.png differ diff --git a/docs/ESM-dependency-upgrade/images/async import pageTypeBundleExtractor.png b/docs/ESM-dependency-upgrade/images/async import pageTypeBundleExtractor.png new file mode 100644 index 00000000000..6d8896715ff Binary files /dev/null and b/docs/ESM-dependency-upgrade/images/async import pageTypeBundleExtractor.png differ diff --git a/docs/ESM-dependency-upgrade/images/build script.png b/docs/ESM-dependency-upgrade/images/build script.png new file mode 100644 index 00000000000..e1b2ee78549 Binary files /dev/null and b/docs/ESM-dependency-upgrade/images/build script.png differ diff --git a/docs/ESM-dependency-upgrade/images/bundleSize index imports.png b/docs/ESM-dependency-upgrade/images/bundleSize index imports.png new file mode 100644 index 00000000000..511d4645071 Binary files /dev/null and b/docs/ESM-dependency-upgrade/images/bundleSize index imports.png differ diff --git a/docs/ESM-dependency-upgrade/images/chalk package json.png b/docs/ESM-dependency-upgrade/images/chalk package json.png new file mode 100644 index 00000000000..c427fc288ad Binary files /dev/null and b/docs/ESM-dependency-upgrade/images/chalk package json.png differ diff --git a/docs/ESM-dependency-upgrade/images/chalk yarn lock.png b/docs/ESM-dependency-upgrade/images/chalk yarn lock.png new file mode 100644 index 00000000000..bd99277a1c4 Binary files /dev/null and b/docs/ESM-dependency-upgrade/images/chalk yarn lock.png differ diff --git a/docs/ESM-dependency-upgrade/images/chalk-version-update.png b/docs/ESM-dependency-upgrade/images/chalk-version-update.png new file mode 100644 index 00000000000..3816168e2ee Binary files /dev/null and b/docs/ESM-dependency-upgrade/images/chalk-version-update.png differ diff --git a/docs/ESM-dependency-upgrade/images/convert to es module vscode.png b/docs/ESM-dependency-upgrade/images/convert to es module vscode.png new file mode 100644 index 00000000000..18b4df55fc0 Binary files /dev/null and b/docs/ESM-dependency-upgrade/images/convert to es module vscode.png differ diff --git a/docs/ESM-dependency-upgrade/images/export averagebundlesize.png b/docs/ESM-dependency-upgrade/images/export averagebundlesize.png new file mode 100644 index 00000000000..7bf7c3f3d13 Binary files /dev/null and b/docs/ESM-dependency-upgrade/images/export averagebundlesize.png differ diff --git a/docs/ESM-dependency-upgrade/images/export bundlesize config.png b/docs/ESM-dependency-upgrade/images/export bundlesize config.png new file mode 100644 index 00000000000..6f93882cdbe Binary files /dev/null and b/docs/ESM-dependency-upgrade/images/export bundlesize config.png differ diff --git a/docs/ESM-dependency-upgrade/images/export getBundleData.png b/docs/ESM-dependency-upgrade/images/export getBundleData.png new file mode 100644 index 00000000000..aad3fc08d15 Binary files /dev/null and b/docs/ESM-dependency-upgrade/images/export getBundleData.png differ diff --git a/docs/ESM-dependency-upgrade/images/export pages.png b/docs/ESM-dependency-upgrade/images/export pages.png new file mode 100644 index 00000000000..873e3dab721 Binary files /dev/null and b/docs/ESM-dependency-upgrade/images/export pages.png differ diff --git a/docs/ESM-dependency-upgrade/images/exports commonjs external.png b/docs/ESM-dependency-upgrade/images/exports commonjs external.png new file mode 100644 index 00000000000..4a556739fb5 Binary files /dev/null and b/docs/ESM-dependency-upgrade/images/exports commonjs external.png differ diff --git a/docs/ESM-dependency-upgrade/images/exports esm external.png b/docs/ESM-dependency-upgrade/images/exports esm external.png new file mode 100644 index 00000000000..722e5f78e41 Binary files /dev/null and b/docs/ESM-dependency-upgrade/images/exports esm external.png differ diff --git a/docs/ESM-dependency-upgrade/images/getBundleData imports.png b/docs/ESM-dependency-upgrade/images/getBundleData imports.png new file mode 100644 index 00000000000..79885b46f73 Binary files /dev/null and b/docs/ESM-dependency-upgrade/images/getBundleData imports.png differ diff --git a/docs/ESM-dependency-upgrade/images/import and export.png b/docs/ESM-dependency-upgrade/images/import and export.png new file mode 100644 index 00000000000..84931fa97c8 Binary files /dev/null and b/docs/ESM-dependency-upgrade/images/import and export.png differ diff --git a/docs/ESM-dependency-upgrade/images/import jest.png b/docs/ESM-dependency-upgrade/images/import jest.png new file mode 100644 index 00000000000..d8794180ff2 Binary files /dev/null and b/docs/ESM-dependency-upgrade/images/import jest.png differ diff --git a/docs/ESM-dependency-upgrade/images/import mocks external screenshot.png b/docs/ESM-dependency-upgrade/images/import mocks external screenshot.png new file mode 100644 index 00000000000..6dc52951eb0 Binary files /dev/null and b/docs/ESM-dependency-upgrade/images/import mocks external screenshot.png differ diff --git a/docs/ESM-dependency-upgrade/images/import mocks internal screenshot.png b/docs/ESM-dependency-upgrade/images/import mocks internal screenshot.png new file mode 100644 index 00000000000..2df2a3fd5ec Binary files /dev/null and b/docs/ESM-dependency-upgrade/images/import mocks internal screenshot.png differ diff --git a/docs/ESM-dependency-upgrade/images/imports external screenshot.png b/docs/ESM-dependency-upgrade/images/imports external screenshot.png new file mode 100644 index 00000000000..e6e426d9aaf Binary files /dev/null and b/docs/ESM-dependency-upgrade/images/imports external screenshot.png differ diff --git a/docs/ESM-dependency-upgrade/images/readFile JSON parse.png b/docs/ESM-dependency-upgrade/images/readFile JSON parse.png new file mode 100644 index 00000000000..da6dd9b8bc7 Binary files /dev/null and b/docs/ESM-dependency-upgrade/images/readFile JSON parse.png differ diff --git a/docs/ESM-dependency-upgrade/images/service list.png b/docs/ESM-dependency-upgrade/images/service list.png new file mode 100644 index 00000000000..01e4640a0dc Binary files /dev/null and b/docs/ESM-dependency-upgrade/images/service list.png differ diff --git a/docs/ESM-dependency-upgrade/images/tes unit script.png b/docs/ESM-dependency-upgrade/images/tes unit script.png new file mode 100644 index 00000000000..d14390ba628 Binary files /dev/null and b/docs/ESM-dependency-upgrade/images/tes unit script.png differ diff --git a/docs/ESM-dependency-upgrade/images/unstable mockModule chalk.png b/docs/ESM-dependency-upgrade/images/unstable mockModule chalk.png new file mode 100644 index 00000000000..4f1c1327974 Binary files /dev/null and b/docs/ESM-dependency-upgrade/images/unstable mockModule chalk.png differ diff --git a/docs/ESM-dependency-upgrade/images/unstable mockModule fs promises.png b/docs/ESM-dependency-upgrade/images/unstable mockModule fs promises.png new file mode 100644 index 00000000000..492f255306d Binary files /dev/null and b/docs/ESM-dependency-upgrade/images/unstable mockModule fs promises.png differ diff --git a/docs/ESM-dependency-upgrade/images/unstable mockModule ora.png b/docs/ESM-dependency-upgrade/images/unstable mockModule ora.png new file mode 100644 index 00000000000..a8269e99c08 Binary files /dev/null and b/docs/ESM-dependency-upgrade/images/unstable mockModule ora.png differ diff --git a/docs/ESM-dependency-upgrade/images/unstable mockModule servicelist.png b/docs/ESM-dependency-upgrade/images/unstable mockModule servicelist.png new file mode 100644 index 00000000000..badd7c64227 Binary files /dev/null and b/docs/ESM-dependency-upgrade/images/unstable mockModule servicelist.png differ