OpenWRT VM images

[tregubovav-dev]

Linux containers Image server distributes OpenWRT containers-only images, which is suitable for multiple use cases. However, containers are unsuitable for some cases, especially when specific devices and kernel modules are required. The VM is a good solution as OpenWRT resources requirements are low. For example, a customer needs internet access via a mobile network solution running in the Incus environment in ARM-bases SBC. So OpenWRT container could not be used for him as container can't handle kernel modules and devices for mobile modem.
I was able to build an OpenWRT VM (aarch64) image for him using the LXC/Incus distrobuilder with some caveats:

• Unable to prepare EFI bootloader with the distrobuilder (Issue 880). The distrobuilder uses grub-setup packages to build the EFI bootloader in other Linux VMS. However, OpenWRT does not provide such a package, but the EFI partition content could be taken directly from the combined image.
• Unable to include the incus-agent to the build, making it impossible to access the VM console from the Incus client.

What needs to be done to build OpenWRT VM image:

• Distrobuilder needs to be able to build an EFI partition and upload the following files from the EFI partition of generic-ext4-combined.img :
  • /efi/boot/bootaa64.efi
  • /efi/openwrt/bootaa64.efi
  • /efi/openwrt/grub.cfg
  • /efi/openwrt/vmlinux
• Distrobuilder must be able to substitute PARTUUIDs in the /efi/openwrt/grub.cfg file
• Distrobuilder must install incus-agent into VM. OpenWRT uses a "procd" based init system.

One more caveats is OpenWRT VM unable to give access to boot console. The command "incus start owrt-vm --console" displays the error "Error: Failed to swap console ring buffer with socket: Chardev user does not support chardev hotswap" instead of console access.

[tregubovav-dev]

I was tried to lunch an incus-agent in the OpenWRT VM and it fails:

incus-agent --verbose --debug
INFO   [2024-12-31T03:07:19Z] Starting
INFO   [2024-12-31T03:07:19Z] Loading vsock module
INFO   [2024-12-31T03:07:19Z] Stopped
Error: Unable to load the vsock kernel module: Failed to run: modprobe -b vsock: exit status 255 (modprobe: unrecognized option: b)

This issue displays at least 2 problems:

1. Most of OpenWRT commands have reduced list of options.
2. Most of modules must be installed via package manager
3. OpenWRT armsr/armv8 (aarch64) kernel is not built with CONFIG_VHOST_VSOCK flag and there is no package which includes vsock module.
   I looked to the incus-agent and find that it can load several kernel modules. How can I know which modules are required and which are optional? I'm going to check whether these module available and ask to add support for missing ones.

[stgraber]

The only hard dependency for the agent is vsock.
Our LoadModule calls will usually detect an already loaded module and just move on, so if you set things up to load the vsock module prior to starting the agent, things should just work.

[tregubovav-dev]

Thank you Stephane for the confirmation!

I looked through the code and see how does it check already loaded modules.
The biggest problem now is that the none of vhost_vsock or vsock module exists in OpwnWRT (nor with bundled kernel, now as a package). I need to see whether it possible build this kernel module and then make request for include it in the snapshot.

Happy New Year!

[tregubovav-dev]

Looking through the qemu-agent I see it can use three communication methods between guest and host. The default method is a serial, which does not require loading any additional modules. Do you think the incus-agent functionality can be extended with this method ?

[stgraber]

It's not something we're interested in attending, no.

The incus-agent is designed for high throughput parallel network transfers, it exposes a full REST API over HTTPS to the host system. Trying to carry all of that onto a single serial connection wouldn't be pleasant.

[tregubovav-dev]

Understood.

[tregubovav-dev]

One more question, Stephane.

Is it possible to use custom local rootfs tar file or directory structure to build image using the distrobuilder? I tried to see the available options, but did not find the solution. The distrobuilder downloads rootfs image every time

[tregubovav-dev]

One more question, Stephane.

Is it possible to use custom local rootfs tar file or directory structure to build image using the distrobuilder? I tried to see the available options, but did not find the solution. The distrobuilder downloads rootfs image every time

pack-incus command resolved the question.