Add MSVC support for Response Files

.github/workflows/integration-tests.yml

@@ -340,3 +340,61 @@ jobs:
           ${SCCACHE_PATH} --show-stats
 
           ${SCCACHE_PATH} --show-stats | grep -e "Cache hits\s*[1-9]"
+
+  test-mock-msvc:
+    runs-on: windows-2019
+    env:
+      TARGET: x86_64-pc-windows-msvc
+      SCCACHE_EXE: ${{ github.workspace }}\\target\\x86_64-pc-windows-msvc\\debug\\sccache.exe
+      SCCACHE_LOG: "trace"
+      SCCACHE_ERROR_LOG: "${{ github.workspace }}\\server_log.txt"
+
+    steps:
+      - uses: ilammy/msvc-dev-cmd@v1
+
+      - name: Clone repository
+        uses: actions/checkout@v3
+
+      - name: Install rust
+        uses: ./.github/actions/rust-toolchain
+        with:
+          toolchain: "stable"
+          target: $TARGET
+
+      - name: Build
+        run: cargo build --bin sccache --target $env:TARGET --features=openssl/vendored
+
+      - name: Compile MSVC (no cache)
+        shell: bash
+        working-directory: ./tests/msvc
+        run: cl "@args.rsp"
+
+      - name: Start Server
+        shell: bash
+        run: $SCCACHE_EXE --start-server
+
+      - name: Compile - Cache Miss
+        shell: bash
+        working-directory: ./tests/msvc
+        run: |
+          $SCCACHE_EXE "$(where cl.exe)" -c "@args.rsp"
+          $SCCACHE_EXE --show-stats
+          $SCCACHE_EXE --show-stats | grep -e "Cache misses\s*[1-9]"
+
+      - name: Compile - Cache Hit
+        shell: bash
+        working-directory: ./tests/msvc
+        run: |
+          $SCCACHE_EXE "$(where cl.exe)" -c "@args.rsp"
+          $SCCACHE_EXE --show-stats
+          $SCCACHE_EXE --show-stats | grep -e "Cache hits\s*[1-9]"
+
+      - name: Stop Server
+        if: success() || failure()
+        shell: bash
+        run: $SCCACHE_EXE --stop-server
+
+      - name: Show Server Log
+        if: success() || failure()
+        shell: bash
+        run: cat "$SCCACHE_ERROR_LOG"

Cargo.toml

@@ -36,6 +36,7 @@ opendal = { version= "0.27.1", optional=true }
 reqsign = {version="0.8.3", optional=true}
 clap = { version = "4.0.32", features = ["derive", "env", "wrap_help"] }
 directories = "4.0.1"
+encoding = "0.2"
 env_logger = "0.10"
 filetime = "0.2"
 flate2 = { version = "1.0", optional = true, default-features = false, features = ["rust_backend"] }

README.md

@@ -99,7 +99,7 @@ export RUSTC_WRAPPER=/path/to/sccache
 cargo build
 ```
 
-sccache supports gcc, clang, MSVC, rustc, NVCC, and [Wind River's diab compiler](https://www.windriver.com/products/development-tools/#diab_compiler).
+sccache supports gcc, clang, MSVC, rustc, NVCC, and [Wind River's diab compiler](https://www.windriver.com/products/development-tools/#diab_compiler). Both gcc and msvc support Response Files, read more about their implementation [here](docs/ResponseFiles.md).
 
 If you don't [specify otherwise](#storage-options), sccache will use a local disk cache.
 

docs/ResponseFiles.md

@@ -0,0 +1,36 @@
+# Response Files
+
+Response files are a way for compilers to accept arguments that would otherwise overflow the character limit in the command line. [On Windows in particular](https://learn.microsoft.com/en-us/troubleshoot/windows-client/shell-experience/command-line-string-limitation), the character limit per command is 8191 characters. These files can contain additional options that the compiler will read and process as if they were provided in the original command. Each compiler that supports response files has different formats/expectations and implementations. Support for response files are also re-implemented per compiler by sccache so it can cache compilations accurately. There is currently support for response files on the gcc and msvc implementations in sccache.
+
+## GCC
+
+As defined by the [gcc docs](https://gcc.gnu.org/onlinedocs/gcc-4.6.3/gcc/Overall-Options.html#Overall-Options):
+
+1. Options in a response file are inserted in-place in the original command line. If the file does not exist or cannot be read, the option will be treated literally, and not removed.
+2. Options in a response file are separated by whitespace.
+3. Single or double quotes can be used to include whitespace in an option.
+4. Any character (including a backslash) may be included by prefixing the character to be included with a backslash (e.g. `\\`, `\?`, `\@`, etc).
+5. The response file may itself contain additional @file options; any such options will be processed recursively.
+
+Implementation details:
+- The gcc implementation in sccache supports all of these **except** #3. If a response file contains **any** quotations (`"` or `'`), the @file arg is treated literally and not removed (and its content not processed).
+- Additionally, sccache will not expand concatenated arguments such as `-include@foo` (see [#150](https://github.com/mozilla/sccache/issues/150#issuecomment-318586953) for more on this).
+- Recursive files are processed depth-first; when an @file option is encountered, its contents are read and each option is evaluated in-place before continuing to options following the @file.
+
+## MSVC
+
+Per the [MSVC docs](https://learn.microsoft.com/en-us/cpp/build/reference/cl-command-files?view=msvc-170):
+
+1. The contents of a response file are inserted in-place in the original command.
+2. Response files can contain multiple lines of options, but each option must begin and end on the same line.
+3. Backslashes (`\`) cannot be used to combine options across multiple lines.
+4. The `/link` directive has special treatment:
+    1. Entering an @file: if the `/link` option is provided prior to an `@file` in the command line, the `/link` directive does not affect any options within the `@file`.
+    2. Newlines: A `/link` directive provided in an `@file` on one line does not affect the next line.
+    3. Exitting an @file: A `/link` directive on the final line of a response file does not affect options following the `@file` option in the command line.
+5. A response file cannot contain additional `@file` options, they are not recursive. (found in a [separate doc](https://learn.microsoft.com/en-us/cpp/build/reference/at-specify-a-compiler-response-file?view=msvc-170))
+6. (implied) options can be wrapped in double-quotes (`"`), which allows whitespace to be preserved within the option
+
+The msvc implementaion in sccache supports all of these **except** #4, because sccache doesn't accept the `/link` directive. 
+
+Additionally, because `msbuild` generates response files using an encoding other than `utf-8`, all text files under the [WHATWG encoding standard](https://encoding.spec.whatwg.org/) are supported. This includes both `utf-8` and `utf-16`.