Merge pull request #5238 from neduard/jit-update-readme

development.markdown

@@ -42,7 +42,7 @@ Some tests are executables instead:
 
 * `stack exec transcripts` runs the transcripts-related integration tests, found in `unison-src/transcripts`. You can add more tests to this directory.
 * `stack exec transcripts -- prefix-of-filename` runs only transcript tests with a matching filename prefix.
-* `stack exec integration-tests` runs the additional integration tests for cli. These tests are not triggered by `tests` or `transcripts`.
+* `stack exec cli-integration-tests` runs the additional integration tests for cli. These tests are not triggered by `tests` or `transcripts`.
 * `stack exec unison -- transcript unison-src/transcripts-round-trip/main.md` runs the pretty-printing round trip tests
 * `stack exec unison -- transcript unison-src/transcripts-manual/benchmarks.md` runs the benchmark suite. Output goes in unison-src/transcripts-manual/benchmarks/output.txt.
 
@@ -220,3 +220,7 @@ nix develop '.#cabal-unison-parser-typechecker'
 cd unison-cli
 cabal run --enable-profiling unison-cli-main:exe:unison -- +RTS -p
 ```
+
+## Native compilation
+
+See the [readme](scheme-libs/racket/unison/Readme.md).

scheme-libs/racket/unison/.gitignore

@@ -0,0 +1,6 @@
+compiled/
+boot-generated.ss
+builtin-generated.ss
+compound-wrappers.ss
+data-info.ss
+simple-wrappers.ss

scheme-libs/racket/unison/Readme.md

@@ -1,5 +1,18 @@
 This directory contains libraries necessary for building and running
-unison programs via Racket Scheme.
+unison programs via Racket Scheme.  The rough steps are as follows:
+
+* Build Racket libraries from the current Unison version.
+* Build the `unison-runtime` binary.
+* Pass the path to `unison-runtime` to `ucm`.
+
+Native compilation is done via the `compile.native` `ucm` command.
+Under-the-hood, Unison does the following:
+
+* Convert the function to bytecode (similar to how `compile` command works).
+* Call `unison-runtime` which will convert the bytecode to a temporary Racket
+  file.  The Racket file is usually placed in your `.cache/unisonlanguage`.
+* folder. Call `raco exe file.rkt -o executable` which will create a native
+  executable from the Racket source code.
 
 ## Prerequisites
 
@@ -9,20 +22,56 @@ You'll need to have a couple things installed on your system:
 * [Racket](https://racket-lang.org/), with the executable `racket` on your path somewhere
 * [BLAKE2](https://github.com/BLAKE2/libb2) (you may need to install this manually)
 
-
-In particular, our crypto functions require on both `libcrypto` (from openssl) and `libb2`. You may have to tell racket where to find `libb2`, by adding an entry to the hash table in your [`config.rktd` file](https://docs.racket-lang.org/raco/config-file.html). This is what I had, for an M1 mac w/ libb2 installed via Homebrew:
+In particular, our crypto functions require both `libcrypto` (from openssl or
+eg. libressl) and `libb2`. You may have to tell racket where to find `libb2`, by
+adding an entry to the hash table in your
+[`config.rktd` file](https://docs.racket-lang.org/raco/config-file.html).
+This is what I had, for an M1 mac with `libb2` installed via Homebrew:
 ```
-(lib-search-dirs . (#f "/opt/homebrew/Cellar/libb2/0.98.1/lib/"))
+$ cat scheme-libs/racket/config/config.rktd
+#hash(
+  (lib-search-dirs . (#f "/opt/homebrew/Cellar/libb2/0.98.1/lib/"))
+)
 ```
 You'll also need to install `x509-lib` with `raco pkg install x509-lib`
 
+Finally, some distributions only package `racket-minimal`.  You'll need to
+install the full compiler suite using `raco pkg install compiler-lib`
+([source](https://www.dbrunner.de/blog/2016/01/12/using-racket-minimal-and-raco/))
+
+## Building
+
+First, make sure unison is built (see [development](../../../development.markdown))
+
+Next, use unison to generate the racket libraries.  These are dependencies for
+building `unison-runtime`.
+* Read [gen-racket-libs.md](../../../unison-src/transcripts-manual/gen-racket-libs.md).
+  It will contain two things:
+  * `ucm` and `unison` transcripts that generate the libraries
+  * Instructions on how to build `unison-runtime` using `raco`
+
+If everything went well you should now have a new executable in `scheme-libs/racket/unison-runtime`.
+For example:
+```
+$ file scheme-libs/racket/unison-runtime
+scheme-libs/racket/unison-runtime: Mach-O 64-bit executable arm64
+```
 
 ## Running the unison test suite
 
-To run the test suite, first `stack build` (or `stack build --fast`), then:
+Note that if you set up `config.rktd` above, you'll need to pass the path to its
+folder in `PLTCONFIGDIR` before invoking unison or the test scripts:
+
+```
+export PLTCONFIGDIR=$(pwd)/scheme-libs/racket/config
+```
+
+If you don't, some of the tests will fail with eg `ffi-lib: could not load foreign library`.
+
+To run the test suite you can do:
 
 ```
-./unison-src/builtin-tests/jit-tests.sh $(stack exec which unison) --runtime-path <path>
+./unison-src/builtin-tests/jit-tests.sh $(stack exec which unison) --runtime-path scheme-libs/racket/unison-runtime
 ```
 
 OR if you want to run the same tests in interpreted mode:
@@ -31,7 +80,9 @@ OR if you want to run the same tests in interpreted mode:
 ./unison-src/builtin-tests/interpreter-tests.sh
 ```
 
-The above scripts fetch and cache a copy of base and the scheme-generating libraries, and copy this directory to `$XDG_DATA_DIRECTORY/unisonlanguage/scheme-libs`.
+The above scripts fetch and cache a copy of base and the scheme-generating
+libraries, and copy this directory to `$XDG_DATA_DIRECTORY/unisonlanguage/scheme-libs`.
+Both scripts _should_ pass.
 
 ## Iterating more quickly
 

unison-src/transcripts-manual/gen-racket-libs.md

@@ -20,12 +20,11 @@ complement of unison libraries for a given combination of ucm version
 and @unison/internal version.
 
 To set up racket to use these files, we need to create a package with
-them. This is accomplished by running.
+them. This is accomplished by running:
 
-    raco pkg install -t dir unison
+    raco pkg install -t dir scheme-libs/racket/unison
 
-in the directory where the `unison` directory is located. Then the
-runtime executable can be built with
+After, the runtime executable can be built with
 
     raco exe scheme-libs/racket/unison-runtime.rkt
 

unison-src/transcripts-manual/gen-racket-libs.output.md

@@ -1,22 +1,21 @@
-
 When we start out, `./scheme-libs/racket` contains a bunch of library files that we'll need. They define the Unison builtins for Racket.
 
 Next, we'll download the jit project and generate a few Racket files from it.
 
-```ucm
+``` ucm
 jit-setup/main> lib.install @unison/internal/releases/0.0.18
 
-  Downloaded 14917 entities.
+  Downloaded 14949 entities.
 
   I installed @unison/internal/releases/0.0.18 as
   unison_internal_0_0_18.
 
 ```
-```unison
+``` unison
 go = generateSchemeBoot "scheme-libs/racket"
 ```
 
-```ucm
+``` ucm
 
   Loading changes detected in scratch.u.
 
@@ -29,7 +28,7 @@ go = generateSchemeBoot "scheme-libs/racket"
       go : '{IO, Exception} ()
 
 ```
-```ucm
+``` ucm
 jit-setup/main> run go
 
   ()
@@ -42,16 +41,23 @@ and @unison/internal version.
 To set up racket to use these files, we need to create a package with
 them. This is accomplished by running.
 
-    raco pkg install -t dir unison
+``` 
+raco pkg install -t dir unison
+```
 
-in the directory where the `unison directory is located. Then the
+in the directory where the `unison` directory is located. Then the
 runtime executable can be built with
 
-    raco exe scheme-libs/racket/unison-runtime.rkt
+``` 
+raco exe scheme-libs/racket/unison-runtime.rkt
+```
 
 and a distributable directory can be produced with:
 
-    raco distribute <output-dir> scheme-libs/racket/unison-runtime
+``` 
+raco distribute <output-dir> scheme-libs/racket/unison-runtime
+```
 
 At that point, <output-dir> should contain the executable and all
 dependencies necessary to run it.
+