OpenFL + Gramine

[lamiaa404]

I am trying to replicate the example of using OpenFL with Gramine: . When using gramine-direct the federated learning works without any problems. However when using gramine-sgx the process just hangs. Any tips how to debug this issue? Attached are two screenshots one with gramine-direct when the federation ends and the other one with gramine-sgx when the process just hangs.

[kailun-qin]

@lamiaa404: Thanks for the question!

May I know:

1. were you following the OpenFL+Gramine Manual and using the associated artifects?
2. which Gramine version were you running w/?

However when using gramine-sgx the process just hangs

Note that it's mentioned in the OpenFL+Gramine known issues: "Kvasir experiment: aggregation takes really long, debug log-level does not show the reason". So are we sure that it hung rather than running too slowly?

Any tips how to debug this issue?

Well, we could normally check Gramine logs and/or GDB into it. But let's figure out the above questions first.

[lamiaa404]

Hey @kailun-qin! Thanks for the quick reply!

I am currently following the mentioned Manual but for the "Torch_cnn_gramine_ready" experiment. After reading your reply I let it run for much longer time and indeed it just taking realllly long time. So it seems it's an issue not only relating to the Kvasir experiment as written in the known issues but is the case for other workspaces too.

Thanks again for the help.

[lamiaa404]

@kailun-qin Hey again, I have another question :) Whenever I run the federation I get this error: Error: Emulating a raw syscall instruction. This degrades performance, consider patching your application to use Gramine syscall API. Any idea what is causing this error and is there any workaround? I read that this error appears for GO applications. Here in the tutorial PyTorch is used. Thanks!

[kailun-qin]

Whenever I run the federation I get this error: Error: Emulating a raw syscall instruction. This degrades performance, consider patching your application to use Gramine syscall API.

I read that this error appears for GO applications.

Any idea what is causing this error

It's not an error but rather a warning -- statically-built go apps would use raw SYSCALL instructions which'll lead gramine-sgx to a "trap-then-emulate" slow path (which would thus impact performance). Pls see more details here:

gramine/Documentation/devel/features.md

Lines 3232 to 3244 in e3d3aa3

	1. Fast path, through patched C standard library (e.g. Glibc or musl): Gramine ships patched Glibc
	and musl where raw `SYSCALL` instructions are replaced with function calls into Gramine's syscall
	entry point.
	2. Slow path, through an exception-handling mechanism:
	- In case of Linux backend, Gramine sets up a seccomp policy that redirects all syscall requests
	from the Linux kernel back into the Gramine process.
	- In case of SGX backend, Intel SGX hardware itself forbids the `SYSCALL` instruction and instead
	generates a `#UD` (illegal instruction) exception, which is delivered into the Gramine process.
	The fast path is recommended for all applications. However, some applications bypass Glibc/musl and
	issue raw `SYSCALL` instructions (e.g., Golang statically compiled binaries); in this case the slow
	path is activated.

is there any workaround?

If the performance caused by ^ is a concern, you can either try https://go.dev/doc/install/gccgo to produce a dynamically-linked go app, or directly patch the go toolchain to create binaries with the Gramine-specific syscall ABI. (I'd suggest the former as a first shot.)