talos apply-config: Specified install disk does not exist

[squishykid]

Bug Report

When configuring talos, it is sometimes not possible to install to eMMC, even though the node booted from that storage.

Description

When performing talos apply-config to a RockPi 4A node, I receive the following message for some nodes:

$ talosctl apply-config --insecure -f worker.yaml --nodes 192.168.1.173
error applying new configuration: rpc error: code = Unknown desc = configuration validation failed: 1 error occurred:
        * specified install disk does not exist: "/dev/mmcblk1"

This is odd, since the talos node has just booted from the eMMC module plugged into the board. Other nodes with identical hardware configuration have no problems discovering the eMMC module and installing to it. I have even tried to take a known good eMMC module from another node and it did not appear in talosctl disks (and also failed during install)

When I run talosctl disks on the affected node, I only see the attached NVMe disk:

$ talosctl -n 192.168.1.173 disks --insecure
DEV            MODEL                    SERIAL             TYPE   UUID   WWID   MODALIAS   NAME   SIZE     BUS_PATH
/dev/nvme0n1   KINGSTON SA2000M81000G   50026B728275C1E9   NVME   -      -      -          -      1.0 TB   /platform/f8000000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/nvme/nvme0/nvme0n1

Logs

Environment

• Talos version: v1.1.0-alpha.1
• Kubernetes version: N/A
• Platform:
  RockPi 4A with eMMC, SPI flash bootloader has never been written.

To Reproduce

• dd talos image from v1.1.0-alpha.1 to eMMC
• insert eMMC module and power-up rock pi board
• run talosctl -n <ip_of_node> disks --insecure

[frezbo]

linking siderolabs/sidero#838, seems something changed, I'll spend some time next week to investigate

[squishykid]

I did just notice that rebooting and trying again can very occasionally make the disk show up.

[frezbo]

would you be able to provide the whole dmesg when the disk shows up and when it doesn't?

[squishykid]

I’m not sure how to get the dmesg before the node is configured. My understanding is that I would have to run the talosctl command with -insecure.

I get a certificate error when running talosctl dmesg against the unconfigured node without -insecure

is there another way?

[squishykid]

I guess the talosctl read command has an insecure option. I’ll have to google how to read dmesg directly- or even if it’s possible.

[squishykid]

I flashed this release and now the installer on the affected node appears to be able to find the emmc disk: https://github.com/siderolabs/talos/releases/tag/v1.1.0-alpha.2

I will try the rest later today

[squishykid]

Seems to be fixed since 1.1