Support Cosmopolitan libc (APE)

[PallHaraldsson]

Hi,
See https://justine.lol/cosmopolitan/ and APE

It may not be a priority for you to be "actually portable", but curious, why is Windows not supported? Is it simply because you've not gotten around to it or don't have access to it?

It's no huge concern of mine (nor macOS), I run Linux, I just thought with Cosmopolitan/APE would be helpful to you (and you could boot to Julia code?!).

I've opened similar issues elsewhere, e.g. at julialang, but they've been closed. It's supposedly a one line to compile with GCC, but I've not gotten around to it. I also foresee at least one problem, since it doesn't support threads, but neither do you, so might be ok for you.

See also:
https://discourse.julialang.org/t/now-that-julia-w-o-llvm-is-a-thing-how-long-before-i-can-run-julia-on-rpi-pico/87676/6

[tshort]

It blows my mind that Cosmopolitan can be that portable. Wow.

As far as windows support, it's just buggy, and tests fail. It's likely problems with tooling (but I'm not sure). There's more info in #52. No one has spent much time getting it to work on windows. You can use WSL under windows.

[brenhinkeller]

That looks very cool! Can one compile an executable using that on linux and then have it run on Windows? And/or can it be applied to LLVM IR or bitcode? If so then we might be able to get that to work.

[PallHaraldsson]

Yes, that's the point, and not just Windows. It doesn't work with LLVM IR/bitcode, just x86-64 instructions. Those work "every Linux distro in addition to Mac OS X, Windows NT, FreeBSD, OpenBSD, and NetBSD too". Apple's future is ARM, but it works, as I understand (still) emulated, because that's what you already have on the Mac (until apple drops it, possibly, I guess, far into the future).

[Working an all Linux distros is actually problematic already, until Cosmopolitan, some do not have GNU libc (and it has sometimes broken because of updates, e.g. where I work, with our in-house language, it supported older Ubuntu, then not automatically later, until fixed); some might have e.g. musl, libc on Alpine. There's also apt vs rpm, and snap vs flatpak.]

I think doing this in this project might be easy and a good proof-of-concept for Julia. Later you could do it for Julia runtime itself, then it will be portable (maybe just excluding threads, should be fixable), and Julia would take care of LLVM so source code, or LLVM IR in some form, should work too.

You do not want to ship an emulator with your binary, then it's going to be that much larger, so I'm unclear on how web browsers are supported too.

I'm not on Windows, so I don't know how easy WSL2 is, do you have to enable it somehow, I think it's mostly for developers who know Linux, and you have to go into that shell (and then use the Linux kernel API, no longer Windows), but Cosmopolitan will use the Windows API. It will not hide all the differences between operating systems, only for text work (and e.g. web [server]), i.e. what a libc is supposed to do (so GUI stuff still about as problematic as without this). libuv hides some, but I believe you don't use that.

[PallHaraldsson]

I'm trying to use both your packages, with docs as is, and I can't (docs here work, not at other package, but error here), get pipeline_error for this line:

        run(`$cc $wrapper_path $cflags $obj_path -o $exec_path`)

Does something need to be preinstalled (I added clang, maybe more recent than clang-13 needed, then which and how?)?

julia> filepath = compile_executable(print_args, (Int64, Ptr{Ptr{UInt8}}), "./")
./wrapper.c:3:5: warning: implicit declaration of function 'julia_print_args' is invalid in C99 [-Wimplicit-function-declaration]
    julia_print_args(argc, argv);
    ^
1 warning generated.
/usr/bin/x86_64-linux-gnu-ld: ./print_args.o: in function `julia_print_args':
text:(.text+0x11): undefined reference to `__stack_chk_guard'
clang-13: error: linker command failed with exit code 1 (use -v to see invocation)
ERROR: failed process: Process(setenv(`/home/pharaldsson/.julia/artifacts/75ed94858c831a8af1fcfa5151509c116570ef05/tools/clang ./wrapper.c ./print_args.o -o ./print_args`,["PATH=/home/pharaldsson/.julia/artifacts/75ed94858c831a8af1fcfa5151509c116570ef05/tools:/home/pharaldsson/.julia/juliaup/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin", "QT_ACCESSIBILITY=1", "LD_LIBRARY_PATH=/home/pharaldsson/.julia/juliaup/julia-1.8.1+0.x64/bin/../lib/julia:/home/pharaldsson/.julia/juliaup/julia-1.8.1+0.x64/bin/../lib/julia:/home/pharaldsson/.julia/juliaup/julia-1.8.1+0.x64/bin/../lib", "DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus", "XDG_SEAT_PATH=/org/freedesktop/DisplayManager/Seat0", "XDG_SESSION_DESKTOP=cinnamon", "GTK_OVERLAY_SCROLLING=1", "XDG_SESSION_TYPE=x11", "SYSTEMD_EXEC_PID=1829", "USER=pharaldsson"  …  "GPG_AGENT_INFO=/run/user/1000/gnupg/S.gpg-agent:0:1", "GNOME_TERMINAL_SCREEN=/org/gnome/Terminal/screen/83c32b9d_85e4_40fa_9dbf_fac5f1c3e3ad", "GNOME_TERMINAL_SERVICE=:1.71", "PWD=/home/pharaldsson/NewSysImageEnv", "XDG_SESSION_CLASS=user", "DISPLAY=:0", "LANGUAGE=is", "XDG_GREETER_DATA_DIR=/var/lib/lightdm-data/pharaldsson", "XDG_SESSION_PATH=/org/freedesktop/DisplayManager/Session0", "LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:"]), ProcessExited(1)) [1]

Stacktrace:
 [1] pipeline_error
   @ ./process.jl:565 [inlined]
 [2] run(::Cmd; wait::Bool)
   @ Base ./process.jl:480
 [3] run
   @ ./process.jl:477 [inlined]
 [4] generate_executable(f::Function, tt::Type, path::String, name::String, filename::String; cflags::Cmd, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
   @ StaticCompiler ~/.julia/packages/StaticCompiler/Avq3a/src/StaticCompiler.jl:386
 [5] compile_executable(f::Function, types::Tuple{DataType, DataType}, path::String, name::String; filename::String, cflags::Cmd, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
   @ StaticCompiler ~/.julia/packages/StaticCompiler/Avq3a/src/StaticCompiler.jl:244
 [6] compile_executable (repeats 2 times)
   @ ~/.julia/packages/StaticCompiler/Avq3a/src/StaticCompiler.jl:229 [inlined]
 [7] top-level scope
   @ REPL[49]:1

I like the example to work before a go further, but then any pointers where I would patch this code (I think I know though...)?

[brenhinkeller]

The key thing here is probably the line ``` text:(.text+0x11): undefined reference to `__stack_chk_guard’ ``` In general, anything beginning with "undefined reference to” means that a function name has made it into your llvm ir that is not actually defined in the same ir module. This generally happens when you’re trying to use a Julia feature or function that requires the Julia stdlib and thus isn’t supported here.

[PallHaraldsson]

Thanks, but I was just using the example from the docs, so maybe it could be changed, only have working example? Also I understand ccall can't be used, and it would be good to document that, even better point to an llvm workaround, or exactly how. Or is ccall likely to be supported soon?

[brenhinkeller]

Which example from the docs?

[PallHaraldsson]

Sory if it wasn't clear, I meant the example there ending with:
https://github.com/brenhinkeller/StaticTools.jl

using StaticCompiler
filepath = compile_executable(print_args, (Int64, Ptr{Ptr{UInt8}}), "./")

[brenhinkeller]

Ah, well that one we certainly do want to work! What's your OS and Julia version?

[brenhinkeller]

I get

julia> using StaticTools

julia> function print_args(argc::Int, argv::Ptr{Ptr{UInt8}})
           # c"..." lets you construct statically-sized, stack allocated `StaticString`s
           # We also have m"..." and MallocString if you want the same thing but on the heap
           printf(c"Argument count is %d:\n", argc)
           for i=1:argc
               # iᵗʰ input argument string
               pᵢ = unsafe_load(argv, i) # Get pointer
               strᵢ = MallocString(pᵢ) # Can wrap to get high-level interface
               println(strᵢ)
               # No need to `free` since we didn't allocate this memory
           end
           println(c"That was fun, see you next time!")
           return 0
       end
print_args (generic function with 1 method)

julia> # Compile executable
       using StaticCompiler

julia> filepath = compile_executable(print_args, (Int64, Ptr{Ptr{UInt8}}), "./")
"/Users/cbkeller/print_args"

shell> ./print_args 1 2 foo
Argument count is 4:
./print_args
1
2
foo
That was fun, see you next time!

and it's in the integration test suite for both StaticTools and StaticCompiler, but maybe we need to add more platforms

[PallHaraldsson]

I'm on Linux Mint 21 ("based on Ubuntu 22.04 LTS", both based on Debian) in case it matters. I don't think it should. Only major difference I've seen with Mint (besides windows manager), is snap isn't installed by default and prevented for use (but I got around that with instructions), and that's very unlikely to be related to this.

[brenhinkeller]

Hmm, odd. Are you on the official Julia binary? And x86 or other architecture?

[PallHaraldsson]

No, 64-bit. I do also have with Juliaup "1.8.1+0.x86" but I'm pretty sure I didn't use it, so I tried again just in case, and even downloaded 1.8.2 (both with juliaup and even tried bypassing it ~/.julia/juliaup/julia-1.8.2+0.x64/bin/julia and (seemingly) always get same error):

julia> versioninfo()
Julia Version 1.8.2
Commit 36034abf260 (2022-09-29 15:21 UTC)
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 16 × Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-13.0.1 (ORCJIT, broadwell)
  Threads: 1 on 16 virtual cores

[brenhinkeller]

Just an update:

• I've found the issue behind the __stack_chk_guard error and how to work around it; it's because of an apparently on-by-default feature in some newer compilers to enable stack smashing protection (which is great, but requires us to set a value of __stack_chk_guard as a canary, e.g. https://antoinealb.net/programming/2016/06/01/stack-smashing-protector-on-microcontrollers.html)

• I've gotten cosmopolitan executables to build on Linux, will add new options for doing this by default after figuring out some open PRs

[PallHaraldsson]

The smallest executable you can make right now is 17 (or was it 16?) KB for hello world. I'm curious what the current lower limit is, even for (a dummy no-op), and what you think might be the actual lower limit (with or without Cosmopolitan, or even if any libc can be skipped). Or possibly one just returning a non-default (or maybe rnd() ?) error code. What's currently the largest single component of the executable?

FYI: https://github.com/niklas-heer/speed-comparison Julia is awesome there, it would be nice if tiny bit faster (and/or smaller, not sure that's what needed), to make top spot. Note also, intriguing to me, accuracy higher for Julia than C/C++, while lower tan for Fortran, and it's unclear what it means to me, or why.

[brenhinkeller]

Currently executable size depends on platform... the smallest hello world I can get without Cosmopolitan on most x86 systems is ~8.4 kb, but on my M1 mac it's 33 kb; there are probably other platforms that are in between. Currently for me the same hello world with cosmopolitan is about 70kb, but there's also "cosmopolitan-tiny", which I haven't tried yet.