Validate theme.json files against the JSON Schema #221

ajlende · 2024-06-25T06:16:42Z

Adds a command to validate the theme.json schemas.

$ npm run schema:validate ./**/theme.json

> [email protected] schema:validate
> node ./theme-utils.mjs validate-schema ./archivist/theme.json ./atlas/theme.json ./blue-note/theme.json ./poetry/theme.json ./purr/theme.json ./stacks/theme.json ./term/theme.json ./tt1-blocks/theme.json

■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ 8/8
[
  {
    "file": "./atlas/theme.json",
    "schema": "https://schemas.wp.org/wp/6.3/theme.json",
    "error": [
      {
        "instancePath": "/settings/color/duotone/0/slug",
        "schemaPath": "#/definitions/settingsPropertiesColor/properties/color/properties/duotone/items/properties/slug/type",
        "keyword": "type",
        "params": {
          "type": "string"
        },
        "message": "must be string"
      }
    ]
  },
  {
    "file": "./blue-note/theme.json",
    "schema": "https://schemas.wp.org/wp/6.2/theme.json",
    "error": [
      {
        "instancePath": "/styles/elements",
        "schemaPath": "#/additionalProperties",
        "keyword": "additionalProperties",
        "params": {
          "additionalProperty": "textarea"
        },
        "message": "must NOT have additional properties"
      },
      {
        "instancePath": "/styles/elements",
        "schemaPath": "#/additionalProperties",
        "keyword": "additionalProperties",
        "params": {
          "additionalProperty": "input"
        },
        "message": "must NOT have additional properties"
      }
    ]
  },
  {
    "file": "./stacks/theme.json",
    "schema": "https://schemas.wp.org/trunk/theme.json",
    "error": [
      {
        "instancePath": "/version",
        "schemaPath": "#/properties/version/enum",
        "keyword": "enum",
        "params": {
          "allowedValues": [
            3
          ]
        },
        "message": "must be equal to one of the allowed values"
      },
      {
        "instancePath": "/styles/elements/h1/typography/lineHeight",
        "schemaPath": "#/properties/typography/properties/lineHeight/oneOf/0/type",
        "keyword": "type",
        "params": {
          "type": "string"
        },
        "message": "must be string"
      },
      {
        "instancePath": "/styles/elements/h1/typography/lineHeight",
        "schemaPath": "#/definitions/refComplete/type",
        "keyword": "type",
        "params": {
          "type": "object"
        },
        "message": "must be object"
      },
      {
        "instancePath": "/styles/elements/h1/typography/lineHeight",
        "schemaPath": "#/properties/typography/properties/lineHeight/oneOf",
        "keyword": "oneOf",
        "params": {
          "passingSchemas": null
        },
        "message": "must match exactly one schema in oneOf"
      },
      {
        "instancePath": "/styles/elements/h2/typography/lineHeight",
        "schemaPath": "#/properties/typography/properties/lineHeight/oneOf/0/type",
        "keyword": "type",
        "params": {
          "type": "string"
        },
        "message": "must be string"
      },
      {
        "instancePath": "/styles/elements/h2/typography/lineHeight",
        "schemaPath": "#/definitions/refComplete/type",
        "keyword": "type",
        "params": {
          "type": "object"
        },
        "message": "must be object"
      },
      {
        "instancePath": "/styles/elements/h2/typography/lineHeight",
        "schemaPath": "#/properties/typography/properties/lineHeight/oneOf",
        "keyword": "oneOf",
        "params": {
          "passingSchemas": null
        },
        "message": "must match exactly one schema in oneOf"
      }
    ]
  },
  {
    "file": "./tt1-blocks/theme.json",
    "error": [
      {
        "instancePath": "",
        "schemaPath": "#/required",
        "keyword": "required",
        "params": {
          "missingProperty": "$schema"
        },
        "message": "must have required property '$schema'"
      }
    ]
  }
]

The command is set up to be run with lint-staged which runs on the precommit hook thanks to husky. It can be run manually too.

$ npx lint-staged

theme-utils.mjs

MaggieCabrera · 2024-06-25T08:39:19Z

Thank you for this @ajlende ! It seems like the action is running on the pre commit hook but when I made a change to one of the theme.json files it doesn't fail as expected. On bluenote I changed this line:

		"layout": {
			"contentSize": "874px",
			"wideSize": "1300px"
		},

to this

		"layout": {
			"contentSize": 874,
			"wideSize": "1300px"
		},

the precommit validation seemed to pass.

Running the command manually gives me a log like so:


➜  community git:(add/schema-validation) ✗ npm run schema:validate ./**/theme.json


> [email protected] schema:validate
> node ./theme-utils.mjs validate-schema ./archivist/theme.json ./atlas/theme.json ./blue-note/theme.json ./poetry/theme.json ./purr/theme.json ./stacks/theme.json ./term/theme.json ./tt1-blocks/theme.json

■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ 8/8
[
  {
    "file": "./atlas/theme.json",
    "error": [
      {
        "instancePath": "/styles/blocks/core~1navigation/spacing/blockGap",
        "schemaPath": "#/properties/spacing/properties/blockGap/oneOf/0/type",
        "keyword": "type",
        "params": {
          "type": "string"
        },
        "message": "must be string"
      },
      {
        "instancePath": "/styles/blocks/core~1navigation/spacing/blockGap",
        "schemaPath": "#/definitions/refComplete/type",
        "keyword": "type",
        "params": {
          "type": "object"
        },
        "message": "must be object"
      },
      {
        "instancePath": "/styles/blocks/core~1navigation/spacing/blockGap",
        "schemaPath": "#/properties/spacing/properties/blockGap/oneOf",
        "keyword": "oneOf",
        "params": {
          "passingSchemas": null
        },
        "message": "must match exactly one schema in oneOf"
      },
      {
        "instancePath": "/styles/blocks/core~1navigation/spacing/blockGap",
        "schemaPath": "#/properties/spacing/properties/blockGap/oneOf/0/type",
        "keyword": "type",
        "params": {
          "type": "string"
        },
        "message": "must be string"
      },
      {
        "instancePath": "/styles/blocks/core~1navigation/spacing/blockGap",
        "schemaPath": "#/definitions/refComplete/type",
        "keyword": "type",
        "params": {
          "type": "object"
        },
        "message": "must be object"
      },
      {
        "instancePath": "/styles/blocks/core~1navigation/spacing/blockGap",
        "schemaPath": "#/properties/spacing/properties/blockGap/oneOf",
        "keyword": "oneOf",
        "params": {
          "passingSchemas": null
        },
        "message": "must match exactly one schema in oneOf"
      }
    ]
  },
  {
    "file": "./blue-note/theme.json",
    "error": [
      {
        "instancePath": "/settings/layout/contentSize",
        "schemaPath": "#/definitions/settingsPropertiesLayout/properties/layout/properties/contentSize/type",
        "keyword": "type",
        "params": {
          "type": "string"
        },
        "message": "must be string"
      },
      {
        "instancePath": "/styles/elements",
        "schemaPath": "#/additionalProperties",
        "keyword": "additionalProperties",
        "params": {
          "additionalProperty": "textarea"
        },
        "message": "must NOT have additional properties"
      },
      {
        "instancePath": "/styles/elements",
        "schemaPath": "#/additionalProperties",
        "keyword": "additionalProperties",
        "params": {
          "additionalProperty": "input"
        },
        "message": "must NOT have additional properties"
      }
    ]
  },
  {
    "file": "./stacks/theme.json",
    "error": [
      {
        "instancePath": "/version",
        "schemaPath": "#/properties/version/enum",
        "keyword": "enum",
        "params": {
          "allowedValues": [
            3
          ]
        },
        "message": "must be equal to one of the allowed values"
      },
      {
        "instancePath": "/styles/elements/h1/typography/lineHeight",
        "schemaPath": "#/properties/typography/properties/lineHeight/oneOf/0/type",
        "keyword": "type",
        "params": {
          "type": "string"
        },
        "message": "must be string"
      },
      {
        "instancePath": "/styles/elements/h1/typography/lineHeight",
        "schemaPath": "#/definitions/refComplete/type",
        "keyword": "type",
        "params": {
          "type": "object"
        },
        "message": "must be object"
      },
      {
        "instancePath": "/styles/elements/h1/typography/lineHeight",
        "schemaPath": "#/properties/typography/properties/lineHeight/oneOf",
        "keyword": "oneOf",
        "params": {
          "passingSchemas": null
        },
        "message": "must match exactly one schema in oneOf"
      },
      {
        "instancePath": "/styles/elements/h2/typography/lineHeight",
        "schemaPath": "#/properties/typography/properties/lineHeight/oneOf/0/type",
        "keyword": "type",
        "params": {
          "type": "string"
        },
        "message": "must be string"
      },
      {
        "instancePath": "/styles/elements/h2/typography/lineHeight",
        "schemaPath": "#/definitions/refComplete/type",
        "keyword": "type",
        "params": {
          "type": "object"
        },
        "message": "must be object"
      },
      {
        "instancePath": "/styles/elements/h2/typography/lineHeight",
        "schemaPath": "#/properties/typography/properties/lineHeight/oneOf",
        "keyword": "oneOf",
        "params": {
          "passingSchemas": null
        },
        "message": "must match exactly one schema in oneOf"
      }
    ]
  },
  {
    "file": "./tt1-blocks/theme.json",
    "error": [
      {
        "instancePath": "",
        "schemaPath": "#/required",
        "keyword": "required",
        "params": {
          "missingProperty": "$schema"
        },
        "message": "must have required property '$schema'"
      }
    ]
  }
]

MaggieCabrera · 2024-06-25T08:41:46Z

Looking at the log, which maybe could be simplified for readability, but we can look into that on a separate PR, it looks like it's treating the schema version being 2 instead of 3 as an error

ajlende · 2024-06-25T14:13:12Z

it looks like it's treating the schema version being 2 instead of 3 as an error

./stacks/theme.json looks like it's using the trunk schema causing version 2 error. I'm going to include which schema is being tested against in the error output so that is more clear.

MaggieCabrera · 2024-06-25T14:21:58Z

./stacks/theme.json looks like it's using the trunk schema causing version 2 error. I'm going to include which schema is being tested against in the error output so that is more clear.

Ugh, you are right, I made a PR to update the ones that had the wrong version but apparently I missed stacks...

ajlende · 2024-06-25T14:52:57Z

Ugh, you are right, I made a PR to update the ones that had the wrong version but apparently I missed stacks...

Imagine if you had a nice script like this when you were updating them to confirm which ones were using the wrong schema :)

ajlende · 2024-06-25T14:54:03Z

It might also be helpful to set up a GitHub action to run the verification on the files touched in the PR. I can try to do that in a follow-up. I didn't want to do that here for two reasons.

We have other errors in the existing schemas which we might want to update alongside the update so folks don't have to fix things when changing something unrelated.
The additionalProperties errors in Blue Note seemed suspicious, like they might be an error with the schema.

ajlende · 2024-06-25T14:57:48Z

Looking at the log, which maybe could be simplified for readability, but we can look into that on a separate PR

I updated it to use console.dir which formats JSON closer to the AJV CLI and uses colors.

They format the errors like this:

$ npx ajv-cli -s ../gutenberg/schemas/json/theme.json -d ./blue-note/theme.json --allowMatchingProperties --allErrors
./blue-note/theme.json invalid
[
  {
    instancePath: '/styles/elements',
    schemaPath: '#/additionalProperties',
    keyword: 'additionalProperties',
    params: { additionalProperty: 'textarea' },
    message: 'must NOT have additional properties'
  },
  {
    instancePath: '/styles/elements',
    schemaPath: '#/additionalProperties',
    keyword: 'additionalProperties',
    params: { additionalProperty: 'input' },
    message: 'must NOT have additional properties'
  }
]

I didn't use that CLI here because it requires the schemas to be saved to disk. It was more complicated to download all the schemas to a temp directory and run the CLI rather than just use the AJV library directly.

ajlende · 2024-06-25T15:02:11Z

It seems like the action is running on the pre commit hook but when I made a change to one of the theme.json files it doesn't fail as expected.

Fixed in 8fb8156.

It wasn't failing because I missed renaming the command in .lintstagedrc.json and the script doesn't exit with an error when you type an incorrect command.

MaggieCabrera · 2024-06-26T07:53:13Z

./stacks/theme.json looks like it's using the trunk schema causing version 2 error. I'm going to include which schema is being tested against in the error output so that is more clear.

Ugh, you are right, I made a PR to update the ones that had the wrong version but apparently I missed stacks...

MaggieCabrera · 2024-06-26T08:07:31Z

Now it works as intended when I run the command manually, but the pre commit hook says

→ No staged files match any configured task.

When I try to commit a faulty theme.json. I tried to debug why but your code seems fine and it should be working :S

github-actions · 2024-06-26T17:34:41Z

Preview changes

I've detected changes to the following themes in this PR: Purr.

You can preview these changes by following the links below:

Preview changes for Purr

I will update this comment with the latest preview links as you push more changes to this PR.

⚠️ Note: The preview sites are created using WordPress Playground. You can add content, edit settings, and test the themes as you would on a real site, but please note that changes are not saved between sessions.

MaggieCabrera · 2024-06-27T07:40:09Z

.lintstagedrc.json

@@ -0,0 +1,3 @@
+{
+	"{**/theme.json,**/styles/*.json,**/assets/fonts/*.json}": "node ./theme-utils.mjs validate-schema"


🤦‍♀️

MaggieCabrera

Thank you for the follow up! It works nicely, let's bring it in

ajlende added the enhancement New feature or request label Jun 25, 2024

ajlende requested review from scruffian and MaggieCabrera June 25, 2024 06:16

MaggieCabrera reviewed Jun 25, 2024

View reviewed changes

theme-utils.mjs Show resolved Hide resolved

Base automatically changed from update/node-20 to trunk June 25, 2024 11:32

ajlende added 7 commits June 25, 2024 09:30

Install AJV validator

87bfca1

Add validate script

f380375

Update validateSchema

8417ca3

Add lint-staged for theme.json schemas

e2b92d3

Update filename to file

3ba70b5

Add pre-commit hook

9b54d8c

Fix lint-staged

8fb8156

ajlende force-pushed the add/schema-validation branch from 6abb898 to 8fb8156 Compare June 25, 2024 14:31

Add schema url to the error output

e76b51e

ajlende added 7 commits June 25, 2024 10:16

Leave error schema as undefined when missing

4e95fa1

Add error exit code for invalid commands

3ba9b5a

Rewrite for latest ajv

b8caa5d

Revert to ajv@6

585d2b5

Support both draft-04 and draft-07

42cad57

Add colored JSON output

432d2aa

Add variations and font collection validation

d5749db

Fix glob for running validation in lint-staged

97e93d4

ajlende requested a review from MaggieCabrera June 26, 2024 17:58

MaggieCabrera reviewed Jun 27, 2024

View reviewed changes

MaggieCabrera approved these changes Jun 27, 2024

View reviewed changes

MaggieCabrera merged commit 299bb8c into trunk Jun 27, 2024
1 check passed

scruffian deleted the add/schema-validation branch July 2, 2024 12:31

carolinan mentioned this pull request Jul 3, 2024

Should this repository use the same utilities as the community themes? WordPress/twentytwentyfive#4

Closed

ajlende mentioned this pull request Jul 19, 2024

How JSON schema versioning works and how it's different from the theme.json version WordPress/developer-blog-content#260

Closed

ajlende mentioned this pull request Aug 27, 2024

Add automated JSON schema validation Automattic/themes#8039

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Validate theme.json files against the JSON Schema #221

Validate theme.json files against the JSON Schema #221

ajlende commented Jun 25, 2024 •

edited

Loading

MaggieCabrera commented Jun 25, 2024

MaggieCabrera commented Jun 25, 2024

ajlende commented Jun 25, 2024 •

edited

Loading

MaggieCabrera commented Jun 25, 2024

ajlende commented Jun 25, 2024

ajlende commented Jun 25, 2024

ajlende commented Jun 25, 2024 •

edited

Loading

ajlende commented Jun 25, 2024

MaggieCabrera commented Jun 26, 2024

MaggieCabrera commented Jun 26, 2024

github-actions bot commented Jun 26, 2024

MaggieCabrera Jun 27, 2024

MaggieCabrera left a comment

		@@ -0,0 +1,3 @@
		{
		"{/theme.json,/styles/.json,/assets/fonts/.json}": "node ./theme-utils.mjs validate-schema"

Validate theme.json files against the JSON Schema #221

Validate theme.json files against the JSON Schema #221

Conversation

ajlende commented Jun 25, 2024 • edited Loading

MaggieCabrera commented Jun 25, 2024

MaggieCabrera commented Jun 25, 2024

ajlende commented Jun 25, 2024 • edited Loading

MaggieCabrera commented Jun 25, 2024

ajlende commented Jun 25, 2024

ajlende commented Jun 25, 2024

ajlende commented Jun 25, 2024 • edited Loading

ajlende commented Jun 25, 2024

MaggieCabrera commented Jun 26, 2024

MaggieCabrera commented Jun 26, 2024

github-actions bot commented Jun 26, 2024

Preview changes

MaggieCabrera Jun 27, 2024

Choose a reason for hiding this comment

MaggieCabrera left a comment

Choose a reason for hiding this comment

ajlende commented Jun 25, 2024 •

edited

Loading

ajlende commented Jun 25, 2024 •

edited

Loading

ajlende commented Jun 25, 2024 •

edited

Loading