tool-cache: Use --force-local for tar on Windows #165

Ecco · 2019-09-30T07:53:24Z

The extractTar nearly works on Windows, except that it cannot deal with absolute path.
Indeed, C:\foo\bar is interpreted as a remote path by tar by default.

…th paths

bryanmacfarlane · 2019-10-02T22:16:50Z

Thanks! Can you include a workflow with a repro? I want to validate with and without your changes.

Ecco · 2019-10-03T07:49:55Z

Hi! I'm not sure what you're asking for exactly.

My understanding is that a workflow.yml is not going to use this directly. But another action may use tool-cache as a dependency.

As far as a test case could go, I think this should do the trick:

const myArchive = await tc.downloadTool("http://host.com/path/to/tarball.tgz");
const archiveExtractedFolder = await tc.extractTar(node12Path, ".");

Ecco · 2019-10-03T07:55:06Z

Note that when testing this patch you might also get hit by #180 which is yet another bug on Windows 😄

mroch · 2019-10-19T14:00:37Z

thanks for this! it (and #180) were necessary for me too.

ericsciple · 2019-12-07T03:18:52Z

packages/tool-cache/src/tool-cache.ts

+    // On Windows, absolute path use a ':' (e.g. C:\foo), which is incorrectly recognized as a remote path by tar unless this flag is passed
+    args.push('--force-local')
+  }
+  args.push('-C', dest, '-f', file)


~~According to C:\windows\system32\tar.exe --help the option -C is only for create, it is not a common option.~~

Also @joshmgross informed me bsd tar does not support --force-local. And C:\windows\system32\tar.exe is bsd tar.

Instead of -C I think dest should be the working directory (exec option).

Thoughts?

ExecOptions.cwd

@ericsciple, while using option cwd works, I am not sure about --force-local. It seems to be required. See:

https://github.com/eine/setup-msys2/commit/4278cd9b1b01d0d7fc0a0cc42a0b2769d44fb5f3/checks?check_suite_id=346924107

https://github.com/eine/setup-msys2/commit/cd6ad53aa10f2461e6affd7cdc81715eca390f4d/checks?check_suite_id=346927420

msys2/setup-msys2@cwd...eine:cwd-noforce

Oh I see. Because of the other file path 🤦

The problem is the hosted machines prepend the gnu tools to the PATH, rather than append to the PATH.

Self hosted runners would more commonly have %windir%\System32\tar.exe (bsd tar) in the PATH.

And bsd tar works correctly. It also doesn't require forward slashes.

I think the correct solution is: on Windows don't which tar.

Instead do:

`${process.env['windir']}\\System32\\tar.exe`

If tar doesn't exist at that location, fail.

Thoughts?

@ericsciple, thanks for the details. If that is the case, I think that more effort is required than just fixing if (IS_WINDOWS) in this repo:

The toolkit should provide a tar command that would hide the inconsistencies in the implementation of the runners from the user. Not using which has the side effect that the arguments might need to be explicitly different for windows-latest, ubuntu-latest and macos-latest.

Your comment about gnu tools being prepended to the default (bsd flavour), should be added to the docs as an important warning.

Then, it will be up to action developers to use the command or to handle it with custom logic.

I dont like the solution i proposed anymore. @thboop good point that it doesnt cover older windows which the runner supports.

ericsciple

see comment

dhouseatx · 2019-12-07T03:31:52Z

Ppl please stop emailing me...have no idea what this about

…

On Fri, Dec 6, 2019, 9:28 PM eric sciple ***@***.***> wrote: ***@***.**** requested changes on this pull request. see comment — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#165?email_source=notifications&email_token=AH6VJYAZ4SC7WT3ZK7DYD53QXMJ6BA5CNFSM4I3XMMD2YY3PNVWWK3TUL52HS4DFWFIHK3DMKJSXC5LFON2FEZLWNFSXPKTDN5WW2ZLOORPWSZGOCOKRA6A#pullrequestreview-328536184>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AH6VJYHPQM4EGI2YMDLI363QXMJ6BANCNFSM4I3XMMDQ> .

ericsciple · 2019-12-16T19:58:09Z

@Ecco @mroch @eine also curious about the scenario. Many tools offer multiple archive formats for users on different platforms (zip or 7z for Windows)

And the tar command doesn't exist on older versions of Windows (which are supported by self hosted runners)

eine · 2019-12-16T21:17:37Z

@ericsciple, the scenario is numworks/setup-msys2:

https://github.com/numworks/setup-msys2/blob/0cf6aff498679e8ce523121a8e8839ab8d26277f/index.js#L24-L31

See also numworks/setup-msys2#23.

The big picture is that the bash shell provided in windows-latest seems to be some limited variant of MINGW installation, such as Git for Windows. Therefore, it lacks pacman and the ability to add other MSYS2/MINGW32/MINGW64 packages. Being able to use pacman (install additional dependencies) is a requirement to build PKGBUILD files on windows-latest GHA jobs. There are currently at least ~1500 tools available as PKGBUILD files for MSYS2: https://github.com/msys2/MINGW-packages.

As commented in actions/starter-workflows#95, this is, overall, the best but limited workaround we can use to overcome the fact that GHA are not MSYS2-friendly at all. Should GitHub provide built-in msys2, mingw64, mingw32 and/or wsl shells (instead of just bash), many related issues would be fixed: #232, numworks/setup-msys2#21, numworks/setup-msys2#22, etc.

Many tools offer multiple archive formats for users on different platforms (zip or 7z for Windows)

Unfortunately, pre-built versions of MSYS2 are distributed as tar.xz only (http://repo.msys2.org/distrib/x86_64/). Is it possible to extract it with any other tool available in windows-latest which is not tar?

As commented in https://github.community/t5/GitHub-Actions/Windows-MSYS2-Ruby/m-p/33946/highlight/true#M1727, before using setup-msys2, I was using chocolatey to install MSYS2. However, it takes 5min to do so. setup-msys2 needs around 30s.

And the tar command doesn't exist on older versions of Windows (which are supported by self hosted runners)

I don't think this is an issue, since maintainers of self-hosted runners can (should) install MSYS2 as part of the base image. Hence, step setup-msys2 might just be skipped.

ericsciple · 2019-12-17T02:31:05Z

@eine thanks, that helps. I checked - { shell: cmd, run: where bash } and it looks like only Git Bash is installed. WSL Bash not installed at all, I was a little surprised but maybe related to HDD size constraints 🤷‍♂. Note the repo for the images is here: https://github.com/actions/virtual-environments

eine · 2019-12-17T14:32:41Z

@ericsciple, precisely, see actions/runner-images#30 and actions/runner-images#50.

joshmgross · 2019-12-26T16:33:05Z

Fixed by #264

tool-cache: Use --force-local for tar on Windows to correctly deal wi…

d473d33

…th paths

joshmgross approved these changes Oct 24, 2019

View reviewed changes

eine approved these changes Nov 19, 2019

View reviewed changes

ericsciple reviewed Dec 7, 2019

View reviewed changes

ericsciple mentioned this pull request Dec 7, 2019

tool-cache: Work around a weird GNU Tar behavior on Windows #180

Closed

ericsciple requested changes Dec 7, 2019

View reviewed changes

eine mentioned this pull request Dec 7, 2019

Use ExecOptions.cwd numworks/setup-msys2#23

Closed

ericsciple mentioned this pull request Dec 8, 2019

Extract tar.gz on windows-latest failed. #194

Closed

This was referenced Dec 16, 2019

Add MSYS2 to Windows actions/runner-images#30

Closed

Add Windows Subsystem for Linux to Windows actions/runner-images#50

Closed

joshmgross closed this Dec 26, 2019

Ecco deleted the fix-tool-cache-tar-on-windows branch December 27, 2019 02:42

Ecco mentioned this pull request Jan 17, 2020

tar.exe: Option --force-local is not supported numworks/setup-msys2#26

Closed

eine mentioned this pull request Feb 1, 2020

Install WSL and MSYS2 on Windows Images actions/runner-images#342

Closed

eine mentioned this pull request Feb 17, 2020

Jobs stuck when trying to untar an archive actions/runner-images#282

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tool-cache: Use --force-local for tar on Windows #165

tool-cache: Use --force-local for tar on Windows #165

Ecco commented Sep 30, 2019

bryanmacfarlane commented Oct 2, 2019

Ecco commented Oct 3, 2019 •

edited

Loading

Ecco commented Oct 3, 2019

mroch commented Oct 19, 2019

ericsciple Dec 7, 2019 •

edited

Loading

ericsciple Dec 7, 2019

eine Dec 7, 2019

ericsciple Dec 7, 2019

eine Dec 9, 2019 •

edited

Loading

ericsciple Dec 9, 2019

ericsciple left a comment

dhouseatx commented Dec 7, 2019 via email

ericsciple commented Dec 16, 2019

eine commented Dec 16, 2019 •

edited

Loading

ericsciple commented Dec 17, 2019

eine commented Dec 17, 2019

joshmgross commented Dec 26, 2019

tool-cache: Use --force-local for tar on Windows #165

tool-cache: Use --force-local for tar on Windows #165

Conversation

Ecco commented Sep 30, 2019

bryanmacfarlane commented Oct 2, 2019

Ecco commented Oct 3, 2019 • edited Loading

Ecco commented Oct 3, 2019

mroch commented Oct 19, 2019

ericsciple Dec 7, 2019 • edited Loading

Choose a reason for hiding this comment

ericsciple Dec 7, 2019

Choose a reason for hiding this comment

eine Dec 7, 2019

Choose a reason for hiding this comment

ericsciple Dec 7, 2019

Choose a reason for hiding this comment

eine Dec 9, 2019 • edited Loading

Choose a reason for hiding this comment

ericsciple Dec 9, 2019

Choose a reason for hiding this comment

ericsciple left a comment

Choose a reason for hiding this comment

dhouseatx commented Dec 7, 2019 via email

ericsciple commented Dec 16, 2019

eine commented Dec 16, 2019 • edited Loading

ericsciple commented Dec 17, 2019

eine commented Dec 17, 2019

joshmgross commented Dec 26, 2019

Ecco commented Oct 3, 2019 •

edited

Loading

ericsciple Dec 7, 2019 •

edited

Loading

eine Dec 9, 2019 •

edited

Loading

eine commented Dec 16, 2019 •

edited

Loading