Running a graminized signed GSC image on AKS reports error related to FSGSBASE CPU instructions missing

[tiagorvmartins]

Been trying to run a specific image that was built and signed using GSC on a AKS environment with sgx plugin installed but facing issues.

Kernel is 5.15 so should have support for FSGSBASE.
Also ran this on one of the nodes:

$ LD_SHOW_AUXV=1 /bin/true | grep AT_HWCAP2
AT_HWCAP2:            0x2

A normal and simple gsc-python image runs with success.

apiVersion: batch/v1
kind: Job
metadata:
  namespace: blockchain
  name: gsc-aks-python
  labels:
      app: gsc-aks-python
spec:
  template:
      metadata:
        labels:
            app: gsc-aks-python
      spec:
        containers:
        - name: gsc-aks-python
          image:  username/gsc-python
          imagePullPolicy: Always
          args: ["-c", "print('HelloWorld!')"] 
          resources:
              limits:
                sgx.intel.com/epc: "1Mi"
        restartPolicy: Never
  backoffLimit: 0

However an image based on golang:1.20-bullseye with the following entry point:
ENTRYPOINT ["/bin/sh"]

and built and signed with gsc using the following manifest raises issues:

sys.enable_sigterm_injection = true

sys.stack.size = "16M"
sys.enable_extra_runtime_domain_names_conf = true
sys.experimental__enable_flock = true
sgx.nonpie_binary = true
sgx.enclave_size = "2G"
sgx.edmm_enable = false
sgx.max_threads = 32

fs.mounts = [
  { path = "/root/.folder", uri = "file:/root/.folder" },
  { path = "/etc/app", uri = "file:/etc/app" }  
]

sgx.trusted_files = [
  "file:/usr/local/bin/app",
  "file:/usr/local/bin/app2",
  "file:/root/.folder/"
]

sgx.allowed_files = [
  "file:/usr/local/bin/app",
  "file:/root/.folder/",
  "file:/root/.folder2/",
  "file:/etc/app/"  
]

The logs containing the error when running the pod:

/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
Gramine is starting. Parsing TOML manifest file, this may take some time...
-----------------------------------------------------------------------------------------------------------------------
Gramine detected the following insecure configurations:

  - loader.insecure__use_cmdline_argv = true   (forwarding command-line args from untrusted host to the app)
  - sys.experimental__enable_flock = true      (flock syscall is enabled; still under development and may contain bugs)
  - sgx.allowed_files = [ ... ]                (some files are passed through from untrusted host without verification)

Gramine will continue application execution, but this configuration must not be used in production!
-----------------------------------------------------------------------------------------------------------------------

/gramine/app_files/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
Emulating a raw syscall instruction. This degrades performance, consider patching your application to use Gramine syscall API.
error: Gramine with Linux-SGX backend requires support for FSGSBASE CPU instructions in the host kernel. Please update your system.

What am I missing or doing wrong?
Maybe because of the experimental flock flag being true?
Thank you!

[dimakuv]

Maybe because of the experimental flock flag being true?

No, this is definitely not the reason.

This is a surprising issue you have. Can you enable Gramine logs (in GSC, you need to build with ./gsc build --debug) and show them to us?

Two suspicions:

1. Some AKS nodes have older Linux versions, or just versions that hide the FSGSBASE feature? (This feature is detected by Gramine via the auxiliary vector AT_HWCAP2, see our code)
2. Your Go application spawns a child process, which means that a new (child) Gramine process is spawned on the host, and this child process somehow loses access to the auxiliary vector AT_HWCAP2 (or to all auxiliary vectors). I don't know why this would happen, we never saw this before.

Is there anything specific about these AKS nodes? What is the OS distro used? Maybe some additional security policies are applied to these AKS nodes?

[dimakuv]

Also, did you try to run this GSC image on your local machine (with e.g. minikube or just docker run)?

[tiagorvmartins]

Hello @dimakuv thank you for the prompt replies,
Sorry, it's all working now.

• I did remove all dangling images and also rebuilt everything from scratch with --no-cache
• Since I have /bin/sh as docker entrypoint, I also did change the flag sgx.nonpie_binary from true to false (anyway not sure if it was the right thing to do)

Not sure what exactly fixed it, but it was on my side apparently.