Theme check fails even when it passes

[wrainbird]

Hello!

I've set this up and all appeared to be going well, I've been testing it with deliberate theme check errors on a PR.

However, now I have fixed the errors, the theme check still fails. I thought this may be similar to #28 - but I do have changed liquid files in the PR.

Comparing the results to a successful action in Dawn, I can't see what is causing this. However we do have our sections inside a _sections folder (we run our own build tool for shared schemas which gets compiled into a sections folder before we run shopify theme package), which theme check does pick up, but maybe that could be causing the error code?

My workflow:

name: Push
on: [push]

jobs:
  theme-check:
    name: Theme Check
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v4
        
       # Add configuration to .theme-check.yml
      - name: Update .theme-check.yml
        run: |
          echo "" >> .theme-check.yml
          echo "RequiredDirectories:" >> .theme-check.yml
          echo "  enabled: false" >> .theme-check.yml

      # Theme Check
      - name: Theme Check
        uses: shopify/theme-check-action@v1
        with:
          token: ${{ github.token }}
          base: development

Screenshots:

Similar successful action on a Dawn commit:

[jamesmengo]

Hey @wrainbird!

Trying to understand if this is on our side or an issue with Github Actions 🤔

Could you try running this same workflow on a different branch and on a theme without the _sections folder and letting me know if that works?

Alternatively, please try with themecheck@v2 and let us know if that works!

Any additional logs will also help!

[wrainbird]

Hi @jamesmengo

Thanks for getting back to me.

I've tried updating shopify/theme-check-action@v2 - it's still failing but no more annotations/errors.

I wonder if this is something to do with npm install (related to #34)

We use yarn, so there could be a conflict?

Here are logs from updated to V2:

##[debug]Evaluating condition for step: 'Theme Check'
##[debug]Evaluating: success()
##[debug]Evaluating success:
##[debug]=> true
##[debug]Result: true
##[debug]Starting: Theme Check
##[debug]Loading inputs
##[debug]Evaluating: github.token
##[debug]Evaluating Index:
##[debug]..Evaluating github:
##[debug]..=> Object
##[debug]..Evaluating String:
##[debug]..=> 'token'
##[debug]=> '***'
##[debug]Result: '***'
##[debug]Loading env
Run shopify/theme-check-action@v[2](https://github.com/switchthemes/exhibit-actions-test/actions/runs/9215541399/job/25354222420#step:5:2)
/usr/local/bin/npm install --no-package-lock --no-save @shopify/cli
npm WARN deprecated [email protected]: This module is not supported, and leaks memory. Do not use it. Check out lru-cache if you want a good and tested way to coalesce async requests by a key value, which is much more comprehensive and powerful.
npm WARN deprecated [email protected]: 🙌  Thanks for using Babel: we recommend using babel-preset-env now: please read https://babeljs.io/env to update!
npm WARN deprecated @babel/[email protected]: This proposal has been merged to the ECMAScript standard and thus this plugin is no longer maintained. Please use @babel/plugin-transform-class-properties instead.
npm WARN deprecated [email protected]: core-js@<[3](https://github.com/switchthemes/exhibit-actions-test/actions/runs/9215541399/job/25354222420#step:5:3).23.3 is no longer maintained and not recommended for usage due to the number of issues. Because of the V8 engine whims, feature detection in old core-js versions could cause a slowdown up to 100x even if nothing is polyfilled. Some versions have web compatibility issues. Please, upgrade your dependencies to the actual version of core-js.
added 702 packages, and audited 910 packages in 59s
13[4](https://github.com/switchthemes/exhibit-actions-test/actions/runs/9215541399/job/25354222420#step:5:4) packages are looking for funding
  run `npm fund` for details
40 vulnerabilities (3 moderate, 2 high, 3[5](https://github.com/switchthemes/exhibit-actions-test/actions/runs/9215541399/job/25354222420#step:5:5) critical)
To address issues that do not require attention, run:
  npm audit fix
Some issues need review, and may require choosing
a different dependency.
Run `npm audit` for details.
/usr/bin/git fetch origin development
From https://github.com/switchthemes/exhibit-actions-test
 * branch            development -> FETCH_HEAD
Creating GitHub check...
Updating GitHub Checks...
##[debug]Node Action run completed with exit code 1
##[debug]Finishing: Theme Check

[jamesmengo]

I looked into this and here's my 2 cents - what I think is happening here:

Theme Check is likely still failing in your github action, but the Errors / Warnings are not showing up in the error report

Why?
I believe this comes from the base configuration provided in your action

This configuration means that we will only surface errors/annotations for files that are different from the base branch for any given execution run

If there are errors or warnings related to files already on the base branch, the action should fail, but you won't see any of those relevant errors or annotations

Fix
Try removing the base option from your config and re-running the action

Here's what I used:

name: Theme Check
on: [push]
jobs:
  theme-check:
    name: Theme Check
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Theme Check
        uses: shopify/theme-check-action@v2
        with:
          token: ${{ github.token }}
          base: main

Can you please try that out and let me know if it works?
I'm by no means a subject matter expert in this domain, so feel free to correct me if I'm off base here

[lukeh-shopify]

👋🏻 Hey @wrainbird, because there hasn’t been activity in the last couple of weeks we’ll be closing this issue. However if you’re still experiencing the same problem, please reopen the issue and share the outstanding context we’ve requested. Thanks!

[wrainbird]

@jamesmengo @lukeh-shopify

My apologies, I forgot to thank James for his suggestion as that worked - I was working off a main branch that had theme errors so once I cleaned it up it fixed it!