forked from w3c/webref
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Update tools and jobs to publish curated data (w3c#468)
The goal of this update is to create a curated data view along with the raw data view and the npm packages views (see w3c#277). Curation means applying patches to the raw data and re-generating the `idlparsed`, `idlnames` and `idlnamesparsed` folders. The latter two will only contain IDL names targeted at browsers, although note that actual spec filtering remains a TODO at this stage (see corresponding TODO comments in `prepare-curated.js` and `prepare-packages.js`). To create the curated data view, this update introduces new tools: - a `prepare-curated.js` tool that copies the raw data to the given folder, applies patches (CSS, elements, IDL) when needed, re-generates the `idlparsed` folder, re-generates the `idlnames` and `idlnamesparsed` folders and adjusts the `index.json` and `idlnames.json` files accordingly. - a `prepare-packages.js` tool (replaces the now gone `packages/prepare.js`) that copies relevant curated data from the curated folder to the packages folder. - a `commit-curated.js` tool that updates the `curated` branch with the contents of the given curated folder. Goal is to have the `curated` branch be the one published as GitHub Pages. The test logic was partially re-written to run the tests against the curated data, and against both the curated data and the NPM packages data when tests may yield different results. A new `curate.yml` job publishes the curated data whenever the crawl data is updated. The job also takes care or preparing package release PRs as needed, replacing the previous prepare-xxx-release jobs. The release workflow becomes: 1. Crawled data is updated (`update-ed.yml`) 2. Curated data and package data get generated (`curate.yml`) 3. Curated data and package data get tested (`curate.yml`) 4. The `curated` branch gets updated with the curated data (`curate.yml`) 5. Npm package pre-release PR gets created (`curate.yml`) 6. Someone reviews and merges the PR 7. New version of npm packages are released (`release-package.yml`) 8. A `Raw data for @webref/[email protected]` tag gets added to the relevant commit on the `main` branch. 9. A `@webref/[email protected]` tag gets added to the relevant commit on the `curated` branch. 10. The `@webref/ttt@latest` tag gets updated to point to the relevant commit on the `curated` branch. Note that, in order for a release to be created, curated data needs to have changed.
- Loading branch information
Showing
27 changed files
with
2,412 additions
and
564 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,66 @@ | ||
| name: Curate data & Prepare package PRs | ||
|
|
||
| on: | ||
| # Runs whenever the `ed` data gets updated on the default branch. | ||
| # Crawl pushes to default branch but that push won't trigger the workflow | ||
| # (a workflow cannot be triggered by a push made by another workflow). Hence | ||
| # the need to react on "workflow_run". | ||
| workflow_run: | ||
| workflows: | ||
| - "Update ED report" | ||
| types: | ||
| - completed | ||
| push: | ||
| branches: | ||
| - main | ||
| paths: | ||
| - 'ed/**' | ||
| - 'packages/**' | ||
| - 'tools/**' | ||
| workflow_dispatch: | ||
|
|
||
| jobs: | ||
| prepare: | ||
| runs-on: ubuntu-latest | ||
| steps: | ||
| - name: Setup node.js | ||
| uses: actions/setup-node@v2 | ||
| with: | ||
| node-version: '16' | ||
|
|
||
| - name: Checkout webref | ||
| uses: actions/checkout@v2 | ||
|
|
||
| - name: Prepare curated and packages data | ||
| # Note that "ci" runs the "prepare" script | ||
| run: npm ci | ||
|
|
||
| - name: Test curated and packages data | ||
| run: npm run test | ||
|
|
||
| - name: Commit curated data | ||
| run: node tools/commit-curated.js stay-on-curated-branch | ||
|
|
||
| - name: Push changes to curated branch | ||
| uses: ad-m/[email protected] | ||
| with: | ||
| github_token: ${{ secrets.GITHUB_TOKEN }} | ||
| branch: curated | ||
|
|
||
| - name: Get back to main branch | ||
| run: git checkout main | ||
|
|
||
| - name: Prepare a pre-release PR for the @webref/css package if needed | ||
| run: node tools/prepare-release.js css | ||
| env: | ||
| GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} | ||
|
|
||
| - name: Prepare a pre-release PR for the @webref/elements package if needed | ||
| run: node tools/prepare-release.js elements | ||
| env: | ||
| GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} | ||
|
|
||
| - name: Prepare a pre-release PR for the @webref/idl package if needed | ||
| run: node tools/prepare-release.js idl | ||
| env: | ||
| GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} |
This file was deleted.
Oops, something went wrong.
This file was deleted.
Oops, something went wrong.
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,9 +1,6 @@ | ||
| name: Test | ||
| on: | ||
| pull_request: {} | ||
| push: | ||
| branches: | ||
| - main | ||
| jobs: | ||
| test: | ||
| name: Test | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -6,3 +6,4 @@ packages/elements/*.json | |
| !packages/elements/package.json | ||
| packages/idl/*.idl | ||
| wpt/ | ||
| curated | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.