Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

github-bot broken? #3720

Closed
richardlau opened this issue May 14, 2024 · 17 comments · Fixed by nodejs/github-bot#511 or nodejs/github-bot#512
Closed

github-bot broken? #3720

richardlau opened this issue May 14, 2024 · 17 comments · Fixed by nodejs/github-bot#511 or nodejs/github-bot#512
Labels

Comments

@richardlau
Copy link
Member

I noticed that the bot reported starting a CI run in nodejs/node#52973 (comment) but no following comment linking to the Jenkins run nor statuses posted back to the PR. It looks like a Jenkins run was started: https://ci.nodejs.org/job/node-test-pull-request/59205/

@richardlau
Copy link
Member Author

post-build-status-update at first glance looks green, but clicking on the jobs and looking at the "Pipeline steps", e.g. https://ci.nodejs.org/job/post-build-status-update/2261443/flowGraphTable/, shows that the "httpRequest" step failed.
image

The console output can't find the build log, e.g.

java.io.FileNotFoundException: /var/lib/jenkins/jobs/post-build-status-update/builds/2261443/log (No such file or directory)
	at java.base/java.io.RandomAccessFile.open0(Native Method)
...

I'll see if I can find it on the Jenkins server (I think they get compressed there).

@richardlau
Copy link
Member Author

On ci:

root@infra-digitalocean-ubuntu14-x64-1:~# ls -al /var/lib/jenkins/jobs/post-build-status-update/builds/2261443
total 976
drwxr-xr-x   3 jenkins jenkins   4096 May 14 07:56 .
drwxr-xr-x 102 jenkins jenkins 933888 May 14 08:37 ..
-rw-r--r--   1 jenkins jenkins  35675 May 14 07:56 build.xml
-rw-r--r--   1 jenkins jenkins      0 May 14 07:56 changelog1986113076434187246.xml
-rw-r--r--   1 jenkins jenkins   6021 May 14 07:56 log.gz
-rw-r--r--   1 jenkins jenkins    102 May 14 07:56 log-index
drwxr-xr-x   2 jenkins jenkins   4096 May 14 07:56 workflow
root@infra-digitalocean-ubuntu14-x64-1:~#

From the uncompressed log:

[2024-05-14T11:56:22.706Z] Fetching upstream changes from https://github.com/nodejs/build.git
[2024-05-14T11:56:22.706Z]  > git --version # timeout=10
[2024-05-14T11:56:22.711Z]  > git --version # 'git version 2.34.1'
[2024-05-14T11:56:22.711Z]  > git fetch --tags --force --progress -- https://github.com/nodejs/build.git +refs/heads/*:refs/remotes/origin/* # timeout=10
[2024-05-14T11:56:23.047Z]  > git rev-parse origin/main^{commit} # timeout=10
[2024-05-14T11:56:23.258Z]  > git config core.sparsecheckout # timeout=10
[2024-05-14T11:56:23.264Z]  > git read-tree -mu HEAD # timeout=10
[2024-05-14T11:56:23.272Z]  > git checkout -f e2397a4d6638c7c644464be8ae219f8580ee6100 # timeout=10
[2024-05-14T11:56:23.963Z] HttpMethod: POST
[2024-05-14T11:56:23.963Z] URL: http://github-bot.nodejs.org:3333/node/jenkins/end
[2024-05-14T11:56:23.964Z] Content-Type: application/json; charset=UTF-8
[2024-05-14T11:56:23.995Z] Sending request to url: http://github-bot.nodejs.org:3333/node/jenkins/end
[2024-05-14T11:56:24.193Z] Treating class org.apache.http.conn.HttpHostConnectException(Connect to github-bot.nodejs.org:3333 [github-bot.nodejs.org/23.253.100.79] failed: Connection refused) as 408 Request Timeout

@richardlau
Copy link
Member Author

The bot isn't running -- on gh-bot:

root@infra-rackspace-debian8-x64-1:~# systemctl status github-bot
● github-bot.service - github-bot
   Loaded: loaded (/lib/systemd/system/github-bot.service; enabled)
   Active: activating (auto-restart) (Result: exit-code) since Tue 2024-05-14 13:00:13 UTC; 8s ago
  Process: 29335 ExecStart=/usr/bin/node server.js (code=exited, status=1/FAILURE)
 Main PID: 29335 (code=exited, status=1/FAILURE)

May 14 13:00:13 infra-rackspace-debian8-x64-1 node[29335]: code: 'MODULE_NOT_FOUND',
May 14 13:00:13 infra-rackspace-debian8-x64-1 node[29335]: requireStack: [
May 14 13:00:13 infra-rackspace-debian8-x64-1 node[29335]: '/home/iojs/github-bot/node_modules/path-scurry/dist/commonjs/index.js',
May 14 13:00:13 infra-rackspace-debian8-x64-1 node[29335]: '/home/iojs/github-bot/node_modules/glob/dist/commonjs/glob.js',
May 14 13:00:13 infra-rackspace-debian8-x64-1 node[29335]: '/home/iojs/github-bot/node_modules/glob/dist/commonjs/index.js',
May 14 13:00:13 infra-rackspace-debian8-x64-1 node[29335]: '/home/iojs/github-bot/server.js'
May 14 13:00:13 infra-rackspace-debian8-x64-1 node[29335]: ]
May 14 13:00:13 infra-rackspace-debian8-x64-1 node[29335]: }
May 14 13:00:13 infra-rackspace-debian8-x64-1 systemd[1]: github-bot.service: main process exited, code=exited, status=1/FAILURE
May 14 13:00:13 infra-rackspace-debian8-x64-1 systemd[1]: Unit github-bot.service entered failed state.
root@infra-rackspace-debian8-x64-1:~#

@richardlau
Copy link
Member Author

From journalctl

-- Logs begin at Tue 2024-05-14 10:38:04 UTC, end at Tue 2024-05-14 13:02:12 UTC. --
May 14 10:38:04 infra-rackspace-debian8-x64-1 systemd[1]: Stopping github-bot...
May 14 10:38:04 infra-rackspace-debian8-x64-1 systemd[1]: Starting github-bot...
May 14 10:38:04 infra-rackspace-debian8-x64-1 systemd[1]: Started github-bot.
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: internal/modules/cjs/loader.js:883
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: throw err;
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: ^
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: Error: Cannot find module 'node:path'
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: Require stack:
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: - /home/iojs/github-bot/node_modules/path-scurry/dist/commonjs/index.js
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: - /home/iojs/github-bot/node_modules/glob/dist/commonjs/glob.js
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: - /home/iojs/github-bot/node_modules/glob/dist/commonjs/index.js
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: - /home/iojs/github-bot/server.js
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: at Function.Module._resolveFilename (internal/modules/cjs/loader.js:880:15)
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: at Function.Module._load (internal/modules/cjs/loader.js:725:27)
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: at Module.require (internal/modules/cjs/loader.js:952:19)
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: at require (internal/modules/cjs/helpers.js:88:18)
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: at Object.<anonymous> (/home/iojs/github-bot/node_modules/path-scurry/dist/commonjs/index.js:28:21)
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: at Module._compile (internal/modules/cjs/loader.js:1063:30)
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: at Object.Module._extensions..js (internal/modules/cjs/loader.js:1092:10)
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: at Module.load (internal/modules/cjs/loader.js:928:32)
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: at Function.Module._load (internal/modules/cjs/loader.js:769:14)
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: at Module.require (internal/modules/cjs/loader.js:952:19) {
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: code: 'MODULE_NOT_FOUND',
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: requireStack: [
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: '/home/iojs/github-bot/node_modules/path-scurry/dist/commonjs/index.js',
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: '/home/iojs/github-bot/node_modules/glob/dist/commonjs/glob.js',
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: '/home/iojs/github-bot/node_modules/glob/dist/commonjs/index.js',
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: '/home/iojs/github-bot/server.js'
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: ]
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: }
May 14 10:38:05 infra-rackspace-debian8-x64-1 systemd[1]: github-bot.service: main process exited, code=exited, status=1/FAILURE
May 14 10:38:05 infra-rackspace-debian8-x64-1 systemd[1]: Unit github-bot.service entered failed state.

which is #3214 and the inability to update Node.js there until we update the OS all over again.

@targos
Copy link
Member

targos commented May 14, 2024

nodejs/github-bot@c5ba723

Landed 18 hours ago.

@targos
Copy link
Member

targos commented May 14, 2024

Can't we update Node.js to 14.21.3 (the last 14.x version), or at least 14.18? isaacs/node-glob@57f5551

@Trott
Copy link
Member

Trott commented May 14, 2024

Oof, sorry I did this with that glob update. That sure seemed safe.... 🥴

@Trott
Copy link
Member

Trott commented May 14, 2024

If upgrading to Node.js 14.18 or later is trouble, and even if it's not, a quick fix would be to revert the glob update.

Trott added a commit to nodejs/github-bot that referenced this issue May 14, 2024
@Trott
Copy link
Member

Trott commented May 14, 2024

If upgrading to Node.js 14.18 or later is trouble, and even if it's not, a quick fix would be to revert the glob update.

Oops, probably not because the bot will still install the latest glob and this was introduced in a patch release. I'll open a PR to pin glob instead, which isn't great, but will fix things if the Node.js upgrade isn't something that can happen quickly.

Trott added a commit to nodejs/github-bot that referenced this issue May 14, 2024
We can unpin it once the bot is running on Node.js 14.18 or newer.

Fixes: nodejs/build#3720
@Trott
Copy link
Member

Trott commented May 14, 2024

nodejs/github-bot#511 for a temporary workaround

@richardlau
Copy link
Member Author

FWIW apt update offers no update or newer verson of Node.js 14.

Trott added a commit to nodejs/github-bot that referenced this issue May 14, 2024
We can unpin it once the bot is running on Node.js 14.18 or newer.

Fixes: nodejs/build#3720
@Trott
Copy link
Member

Trott commented May 15, 2024

Hmm, pinning glob doesn't seem to have fixed it. I'm not sure how to look at the log file to confirm that glob is still the problem, but I'll reopen this.

@Trott Trott reopened this May 15, 2024
@targos
Copy link
Member

targos commented May 15, 2024

Is the machine still in sync with its playbook? If it's not, I suggest we remove the Node.js installed via apt and instead just download it from nodejs.org and extract it to /usr/local

Trott added a commit to nodejs/github-bot that referenced this issue May 15, 2024
Trott added a commit to nodejs/github-bot that referenced this issue May 15, 2024
Trott added a commit to nodejs/github-bot that referenced this issue May 15, 2024
@richardlau
Copy link
Member Author

hmm. bot is still failing to start.

It seems like the deployment still has path-scurry 1.11.1 installed.

# npm ls
[email protected] /home/iojs/github-bot
├─┬ @octokit/[email protected]
│ ├─┬ @octokit/[email protected]
│ │ ├─┬ @octokit/[email protected]
│ │ │ └── @octokit/[email protected] deduped
│ │ ├─┬ @octokit/[email protected]
│ │ │ ├── @octokit/[email protected] deduped
│ │ │ ├── @octokit/[email protected] deduped
│ │ │ └── [email protected] deduped
│ │ ├─┬ @octokit/[email protected]
│ │ │ ├─┬ @octokit/[email protected]
│ │ │ │ ├── @octokit/[email protected] deduped
│ │ │ │ ├── [email protected] deduped
│ │ │ │ └── [email protected] deduped
│ │ │ ├── @octokit/[email protected] deduped
│ │ │ ├── @octokit/[email protected] deduped
│ │ │ ├── [email protected]
│ │ │ ├─┬ [email protected]
│ │ │ │ └─┬ [email protected]
│ │ │ │   ├── [email protected]
│ │ │ │   └── [email protected]
│ │ │ └── [email protected] deduped
│ │ ├─┬ @octokit/[email protected]
│ │ │ ├── @octokit/[email protected] deduped
│ │ │ ├── [email protected] deduped
│ │ │ └─┬ [email protected]
│ │ │   └── [email protected]
│ │ ├─┬ @octokit/[email protected]
│ │ │ └── @octokit/[email protected]
│ │ ├── [email protected]
│ │ └── [email protected]
│ ├─┬ @octokit/[email protected]
│ │ ├── @octokit/[email protected]
│ │ └── @octokit/[email protected] deduped
│ ├── @octokit/[email protected]
│ └─┬ @octokit/[email protected]
│   ├── @octokit/[email protected] deduped
│   └── [email protected]
├─┬ [email protected]
│ └── [email protected]
├─┬ [email protected]
│ └── [email protected]
├─┬ [email protected]
│ ├── [email protected]
│ ├── [email protected]
│ ├─┬ [email protected]
│ │ └── [email protected]
│ ├── [email protected]
│ ├── [email protected]
│ ├─┬ [email protected]
│ │ ├── [email protected] deduped
│ │ ├── [email protected]
│ │ ├── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ └── [email protected]
│ ├─┬ [email protected]
│ │ └── [email protected]
│ ├─┬ [email protected]
│ │ └── [email protected]
│ ├─┬ [email protected]
│ │ └─┬ [email protected]
│ │   ├─┬ [email protected]
│ │   │ ├── [email protected]
│ │   │ └── [email protected] deduped
│ │   ├─┬ [email protected]
│ │   │ ├── [email protected] deduped
│ │   │ ├─┬ [email protected]
│ │   │ │ └── [email protected] deduped
│ │   │ ├── [email protected]
│ │   │ └── [email protected]
│ │   └── [email protected]
│ ├─┬ [email protected]
│ │ ├── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ └── [email protected] deduped
│ ├─┬ [email protected]
│ │ ├── [email protected]
│ │ └─┬ [email protected]
│ │   └── [email protected]
│ └── [email protected]
├─┬ [email protected]
│ ├─┬ [email protected]
│ │ └── [email protected]
│ ├── [email protected]
│ ├─┬ [email protected]
│ │ ├─┬ [email protected]
│ │ │ └── [email protected]
│ │ ├── [email protected]
│ │ └─┬ [email protected]
│ │   └─┬ [email protected]
│ │     ├─┬ [email protected]
│ │     │ ├── [email protected] deduped
│ │     │ └── [email protected] deduped
│ │     ├── [email protected] deduped
│ │     ├─┬ [email protected]
│ │     │ └─┬ [email protected]
│ │     │   ├── [email protected] deduped
│ │     │   └── [email protected]
│ │     ├── [email protected] deduped
│ │     └── [email protected]
│ └── [email protected]
├─┬ [email protected]
│ ├─┬ @types/[email protected]
│ │ └── @types/[email protected]
│ ├─┬ @types/[email protected]
│ │ ├─┬ @types/[email protected]
│ │ │ ├─┬ @types/[email protected]
│ │ │ │ └── @types/[email protected] deduped
│ │ │ └── @types/[email protected] deduped
│ │ ├─┬ @types/[email protected]
│ │ │ ├── @types/[email protected] deduped
│ │ │ ├── @types/[email protected] deduped
│ │ │ └── @types/[email protected]
│ │ ├── @types/[email protected]
│ │ └─┬ @types/[email protected]
│ │   ├── @types/[email protected]
│ │   └── @types/[email protected] deduped
│ └── [email protected]
├─┬ [email protected]
│ ├─┬ [email protected]
│ │ ├── [email protected]
│ │ ├─┬ [email protected]
│ │ │ └── [email protected]
│ │ └─┬ [email protected]
│ │   └── [email protected]
│ ├─┬ [email protected]
│ │ ├── [email protected] deduped
│ │ └── [email protected]
│ ├── [email protected]
│ └─┬ [email protected]
│   └─┬ [email protected]
│     └─┬ [email protected]
│       └── [email protected]
├─┬ [email protected]
│ └── [email protected]
├── [email protected]
├── [email protected]
├─┬ [email protected]
│ ├─┬ [email protected]
│ │ ├── [email protected] deduped
│ │ └── [email protected]
│ ├── [email protected]
│ ├── [email protected] deduped
│ ├─┬ [email protected]
│ │ └── [email protected] deduped
│ ├── [email protected] deduped
│ ├── [email protected]
│ ├── [email protected]
│ ├─┬ [email protected]
│ │ └── [email protected]
│ ├── [email protected] deduped
│ ├── [email protected]
│ ├── [email protected]
│ ├── [email protected]
│ ├─┬ [email protected]
│ │ ├─┬ [email protected]
│ │ │ └── [email protected]
│ │ ├── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ └── [email protected] deduped
│ ├── [email protected]
│ ├── [email protected] deduped
│ ├── [email protected]
│ ├── [email protected]
│ ├── [email protected] deduped
│ ├── [email protected]
│ ├── [email protected]
│ ├─┬ [email protected]
│ │ ├── [email protected]
│ │ └── [email protected]
│ ├── [email protected] deduped
│ ├── [email protected]
│ ├── [email protected]
│ ├─┬ [email protected]
│ │ ├─┬ [email protected]
│ │ │ └── [email protected]
│ │ ├── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ ├── [email protected]
│ │ ├── [email protected]
│ │ ├── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ └── [email protected] deduped
│ ├─┬ [email protected]
│ │ ├── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ ├── [email protected] deduped
│ │ └── [email protected] deduped
│ ├── [email protected]
│ ├── [email protected]
│ ├── [email protected] deduped
│ ├── [email protected]
│ └── [email protected]
└─┬ [email protected]
  ├─┬ [email protected]
  │ ├── [email protected] deduped
  │ └── [email protected]
  ├─┬ [email protected]
  │ ├─┬ @isaacs/[email protected]
  │ │ ├─┬ [email protected]
  │ │ │ ├── [email protected]
  │ │ │ ├── [email protected]
  │ │ │ └── [email protected] deduped
  │ │ ├─┬ string-width-cjs@npm:[email protected]
  │ │ │ ├── [email protected]
  │ │ │ ├── [email protected]
  │ │ │ └─┬ [email protected]
  │ │ │   └── [email protected]
  │ │ ├─┬ [email protected]
  │ │ │ └── [email protected]
  │ │ ├─┬ strip-ansi-cjs@npm:[email protected]
  │ │ │ └── [email protected]
  │ │ ├─┬ [email protected]
  │ │ │ ├── [email protected]
  │ │ │ ├── [email protected] deduped
  │ │ │ └── [email protected] deduped
  │ │ └─┬ wrap-ansi-cjs@npm:[email protected]
  │ │   ├─┬ [email protected]
  │ │   │ └─┬ [email protected]
  │ │   │   └── [email protected]
  │ │   ├─┬ [email protected]
  │ │   │ ├── [email protected]
  │ │   │ ├── [email protected] deduped
  │ │   │ └── [email protected] deduped
  │ │   └─┬ [email protected]
  │ │     └── [email protected]
  │ └── @pkgjs/[email protected]
  ├─┬ [email protected]
  │ └─┬ [email protected]
  │   └── [email protected]
  ├── [email protected]
  └─┬ [email protected]
    ├── [email protected]
    └── [email protected]

Maybe trying to get a manual Node.js 14 installed as a temporary solution would be worth pursuing.

@richardlau
Copy link
Member Author

Is the machine still in sync with its playbook? If it's not, I suggest we remove the Node.js installed via apt and instead just download it from nodejs.org and extract it to /usr/local

I've downloaded Node.js 14.21.3 onto the machine and extracted it to /usr/local. Had to edit /lib/systemd/system/github-bot.service so that it used /usr/local/bin/node instead of /usr/bin/node. It looks like the bot has now been able to start:

root@infra-rackspace-debian8-x64-1:/home/iojs# systemctl status github-bot
● github-bot.service - github-bot
   Loaded: loaded (/lib/systemd/system/github-bot.service; enabled)
   Active: active (running) since Wed 2024-05-15 17:52:29 UTC; 1s ago
 Main PID: 19176 (node)
   CGroup: /system.slice/github-bot.service
           └─19176 /usr/local/bin/node server.js

May 15 17:52:29 infra-rackspace-debian8-x64-1 systemd[1]: Started github-bot.
May 15 17:52:29 infra-rackspace-debian8-x64-1 node[19176]: {"name":"bot","hostname":"infra-rackspace-debian8-x64-1","pid":19176,"level":30,"msg":"Loading: ./scripts/ping.js","time":"2024-05-15T1...591Z","v":0}
May 15 17:52:29 infra-rackspace-debian8-x64-1 node[19176]: {"name":"bot","hostname":"infra-rackspace-debian8-x64-1","pid":19176,"level":30,"msg":"Loading: ./scripts/node-ping-owners.js","time":"...595Z","v":0}
May 15 17:52:29 infra-rackspace-debian8-x64-1 node[19176]: {"name":"bot","hostname":"infra-rackspace-debian8-x64-1","pid":19176,"level":30,"msg":"Loading: ./scripts/jenkins-status.js","time":"20...687Z","v":0}
May 15 17:52:29 infra-rackspace-debian8-x64-1 node[19176]: {"name":"bot","hostname":"infra-rackspace-debian8-x64-1","pid":19176,"level":30,"msg":"Loading: ./scripts/event-relay.js","time":"2024-...688Z","v":0}
May 15 17:52:29 infra-rackspace-debian8-x64-1 node[19176]: {"name":"bot","hostname":"infra-rackspace-debian8-x64-1","pid":19176,"level":30,"msg":"Listening on port 3333","time":"2024-05-15T17:52...693Z","v":0}
Hint: Some lines were ellipsized, use -l to show in full.
root@infra-rackspace-debian8-x64-1:/home/iojs#

@richardlau
Copy link
Member Author

richardlau commented May 15, 2024

httpRequest step in the pipeline (see #3720 (comment)) is passing now and I can see status checks being posted back to PRs.

@Trott
Copy link
Member

Trott commented May 20, 2024

@richardlau Should we land nodejs/github-bot#514 and keep an eye on things?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants