Talos install not persisting after initial boot / config from ISO

[MonMothma1ABY]

Hi - ive got a fun issue where, i can boot to the ISO, apply a config, bootstrap and see all running nodes however when I reboot and remove the USB stick the node fails with errors about etcd.

Error message after removing boot medium.

I can't reboot into the Talos OS, as it just reverts back to my initial Ubuntu OS.

Static IP / network config is applied correctly as well so the config is getting loaded nicely. Maybe just ignoring some of the install parameters? I've disabled Both Legacy Support and SecureBoot in BIOS. I just can't get it to persist.

This is the output from the disk selector, they seem to tie.

NODE            DEV            MODEL                SERIAL            TYPE   UUID   WWID                                                                                     MODALIAS      NAME   SIZE     BUS_PATH                                                               SUBSYSTEM          READ_ONLY   SYSTEM_DISK
192.168.68.71   /dev/nvme0n1   HP SSD EX900 500GB   HASE32421402668   NVME   -      nvme.126f-484153453332343231343032363638-485020535344204558393030203530304742-00000001   -             -      500 GB   /pci0000:00/0000:00:1d.0/0000:01:00.0/nvme/nvme0/nvme0n1               /sys/class/block               
192.168.68.71   /dev/sda       Flash Drive          -                 HDD    -      -                                                                                        scsi:t-0x00   -      128 GB   /pci0000:00/0000:00:14.0/usb2/2-3/2-3:1.0/host0/target0:0:0/0:0:0:0/   /sys/class/block               *

Relevant config below - have tried to include the metal platform args, being verbose about installing a bootloader and then as a hail maray put legacyBIOS support to true which did not help.

    install:
        disk: /dev/nvme0n1
        image: ghcr.io/siderolabs/installer:v1.7.6
        wipe: true
        extraKernelArgs:
            - talos.platform=metal
        bootloader: true

I've seen #8806, and #9060 but I may not be understanding the issues here.

[rothgar]

Try booting the machine from usb and before you apply the config remove the USB drive. This will eliminate the /dev/sda device being available at all to make sure the install happens on the nvme disk.

[MonMothma1ABY]

The machine instantly reboots landing me on my secondary boot option. Had tried that a few times before my initial post however, putting the logs here may put me down a path. Ended up being able to grab a snap (below) of the error... typed it out for brevity.

[sda] Synchronize Cache(10) failed: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK

It does however remove the drive from the talosctl drives list... curiously on push of the config it goes straight to the sda drive. Do I need to create partitions in the nvme? I had thought the OS was loaded into memory removing dependencies on any physical medium.

Additionally used the device selector on and off to no avail. I've been investigating PXE boot, which has given me a bit of a headache however something I need to figure out eventually but... more of a me problem. Would really like to be able to get this cluster up and running with the .iso and play with the cluster before I jump head first into the lovely word of IaC - (Digital Rebar, Matchbox, Terraform, Ansible)

[rothgar]

When you pull the USB the device reboots immediately? That shouldn't happen because once the USB boots the OS should be running in RAM.

[MonMothma1ABY]

Not instant sorry... was late.

Pull the drive, the device removal is recognized. All good so far, as soon as I push the config without the drive it I get the error above and the device throws some additional logs about shutting down and a 10 second timer starts. The machine then reboots. I can pull those logs in a few if they are relevant, seemed innocuous enough after the error above.

[smira]

Can you do talosctl get discoveredvolumes while Talos is (presumably) installed to rule out any inconsistencies.

[MonMothma1ABY]

Strange update... with a fresh WSL install (new talosctl etc..) + new burn of the .iso (same file) seems to be working - sorta. The "install" cycle after pushing the config this go around took WAY longer which makes me think the last one was not actually installing to disk.

My repo was cloned into the new WSL install that contained all of my configs so, that didn't change. The environment was pretty clean the other day... only thing I can think of is the boot .iso got corrupted in some way when ripping it which is kind of a stretch?

I did notice a different grub menu on boot previously it was something like A*Talos... or, reset back to maintenance mode. Now I just see "Talos ISO" or "Reset Talos Installation" - not sure if there is any difference in what the grub menu shows that might shed some light as to where / how I ended up with the setup that I had.

discoveredvolumes shows the 6 partitions written to the nvme which is good. So thank you.

That being said, after the initial config push... the machine crashes pretty hard. My remote desktop via MeshCommander goes down and is unable to connect - at all, the only way to bring the machine back up is to power cycle via the wall wart. I can replicate this behavior by issuing a talosctl reboot command which via the logs it seems to do post the config completion.

I suppose this issue here is largely closed, will add additional documentation related to the crashing if deemed related. Appreciate everyone's time. Will leave this open for a few more days in case any discussion on the above issue is warranted.

Documentation

Environments Used
LocalDev: WSL2 on Windows11. AMD 7900X, 64GB DDR5
TargetMachine: HP EliteDesk 800 G3 DM 65W, i5-6500 CPU, 8GB DDR4

Remediation Steps

1. in LocalDev cmdline wsl --unregister Ubuntu and remove from "Add/Remove Programs"
2. in LocalDev cmdline wsl --install
3. In WSL cmdline apt-get update && apt-get dist-upgrade
4. In WSL cmdline /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)" -get brew
5. In WSL cmdline brew install llvm
6. In WSL cmdline brew install siderolabs/tap/talosctl
7. Format and burn metal-amd64 to USB Stick from https://factory.talos.dev/image/376567988ad370138ad8b2698212367b8edcb69b5fd68c80be1f2ec7d603b4ba/v1.7.6/metal-amd64.iso
8. Boot machine to .iso - pushed config via talosctl (references above)
9. Bootstrapped etcd

Summary: Issue with Talos not persisting to nvme drive after config/bootstrap push. No root cause found however, refresh of both local development environment and re burn of boot .iso seemed to have remediated the issue.

[rothgar]

If the grub menu says "Talos ISO" then it's still booting off an attached USB/CD. You should remove the drive and reboot to get it to boot from the local disk

[MonMothma1ABY]

Yes, correct. Only when I select the usb from the BIOS does that come up. I only mentioned it because when I was facing the issue previously... the GRUB menu was slightly different.

I can use the installed Talos OS without the thumb drive now.

Thanks for the tips.