Dropping squashfs

[Itxaka]

As nothing seems to be using squashfs and there doesnt seem to be any features from it that we can use I propose dropping support for squashfs altogether.

PROS:

• Real RO filesystem
• I dunno lol

CONS:

• Needs 2 extra modules on grub (squash4 and xzio)
• Signed grub doesnt usually provide those modules (no secureboot for you)
• Extra configuration for CI
• Extra configuration for install/upgrade/reset
• Extra flags to support squash and its compression options
• Extra steps in testing
• Cant add tools to recovery (similar to using rd.cos.debugrw but for recovery)
• Extra grub complexity
• Failures on grub when not using it (could no load squash, could not load modules if they are not in there)

I propose we fully drop support from it + tests, then we drop support for its path and flags from elemental-cli

[Itxaka]

cc @mudler @davidcassany @kkaempf @bk201 (Not sure if harvester uses squash for the recovery)

[bk201]

Just try the recovery from GRUB menu in Harvester, but it doesn't seem to work:

error: ../../grub-core/commands/search.c.:296:no such device:
/cOS/recovery.squashfs.

(/cOS/recovery.img does exists in the COS_RECOVERY partition)

I guess recovery is a rarely used feature. But does dropping the squashfs only impact recovery?
There are some places we use it:

• PXE boot: the squashfs image is loaded to RAM.
• Installation: basically this is just handle by elemental CLI.
• Upgrade: we download the squashfs image to local and upgrade from it: https://github.com/harvester/harvester/blob/bd1d49a9d88832c925b8f2775e3cc4510f5bc2a8/package/upgrade/upgrade_node.sh#L423-L426

cc @tjjh89017 @guangbochen

[Itxaka]

Upgrade: we download the squashfs image to local and upgrade from it: harvester/harvester@bd1d49a/package/upgrade/upgrade_node.sh#L423-L426

Interesting, any reason you are not using the upgrade directly pointing to the image? Where is the squashfs image from that part come from?? UPGRADE_REPO_SQUASHFS_IMAGE

[mudler]

mmm missing a bit of context here, dropping squashfs as in use of the recovery and just use img all the way?

[Itxaka]

yep! If you want to support secureboot for example, all modules need to be bundled as part of the grub.efi images, as module loading is forbidden under secureboot.

This means that the upstream signed grub.efi image needs to include those modules. To insert squashfs is to add 2 modules as part of the grub builds. Its doable but difficult.

Not only that but as far as I can see the squashfs doesn't really offer anything on top of our implementation of the inmutable rootfs based on loop devices and img files other than the true read-only nature of it. But it means that we also have to consider different paths for reset,install,upgrade which makes everything more complex.

Basically what Im proposing is to drop it and only support nonsquashfs (img files) to make everything less complex and be able to drop code and support secureboot in the future.

[mudler]

all in for it for what's worth my cents. the only requirement tied to it was (at the time of writeup) to be able to build raw disk inside Kube in an unprivileged way (squahfs don't require any mount) - I think that's why we ended up at the end supporting squashfs at recovery. To use it to bootstrap the full system after first-boot - just depend if we want to still support that use case there (but I don't think so?)

[davidcassany]

I am not against dropping squashfs, but I consider most of the issues listed here are not a problem with squashfs. Also note that by using squashfs for all images (recovery, active and passive) we could consider building full disks in non privileged containers (using GPT and EFI), hence inside k8s clusters. This is doable, there is just quite some tech debt to get rid of, including our current booting procedure. Rather than coding issue is an issue about how to organize and verify changes.

In short my take is that squashfs should be a transparent option. IMHO the fact that squashfs it tricky to handle in our elemental-cli code and in elemental-tookit mostly reveals arguable design choices:

1. Use dmsquashfs-live dracut module for squashfs images. This is wrong as this creates a new and different booting path hence extra CI paths. Immutable rootfs module should have been used.
2. Name image files with extension recovery.squashfs vs recovery.img. It should have been only recovery.img, relaying on a name file to assume or not squashfs was a smelly choice.
3. upgrade and reset should not perform runtime checks on the host (to detect running setup) and relay only on the state yaml file to merge required configurations (no bios/efi checks, no squashfs checks, no runtime only heuristics). Just load config and apply it.
4. grub config (kernel params and each menu options details) should not be static from install, upgrade and reset perspective. I'd say each menu entry (active, fallback and recovery) should source its own specific file sourced by grub.cfg (so intead of bootargs.cfg, having active.cfg, passive.cfg and recovery.cfg). So in each upgrade the full kernel call could be updated if needed. upgrade, install and reset should be capable to tune the in image provided grub config (e.g. kernel parameters or grub modules) as the logic of setting some of these belongs there, when config choices are actually taken.
5. immutable rootfs dracut module should not relay on image labels. note this is redundant data, we include the image label and the image file in kernel parameters. Systemd is perfectly capable to mount image files, there is no need to discover labels.
6. we should use the squashfs creation root-tree procedure for all filesystems. The more I think about it the more convinced I am. Basically create the tree in a temporary directory and then sync it to the image, this allows us to create images that fit perfectly the root-tree size and simplifies quite a lot our elemental-cli logic. The only counter part is some extra IO operations (the root-tree is copied twice) during installation and upgrade.

Note most of the above is not strictly related to the squashfs support and would make squashfs mostly transparent (as it was ext2, ext3, xfs or something else). Also mksquashfs awareness and its flags management in elemental-cli cannot be dropped, this is required for the ISO build.

So my take on that is that there is little gain on dropping squashfs support, feels more like a shortcut and a workaround than anything else.

[mudler]

I am not against dropping squashfs, but I consider most of the issues listed here are not a problem with squashfs. Also note that by using squashfs for all images (recovery, active and passive) we could consider building full disks in non privileged containers (using GPT and EFI), hence inside k8s clusters. This is doable, there is just quite some tech debt to get rid of, including our current booting procedure. Rather than coding issue is an issue about how to organize and verify changes.

In short my take is that squashfs should be a transparent option. IMHO the fact that squashfs it tricky to handle in our elemental-cli code and in elemental-tookit mostly reveals arguable design choices:

1. Use dmsquashfs-live dracut module for squashfs images. This is wrong as this creates a new and different booting path hence extra CI paths. Immutable rootfs module should have been used.

Agreed, but AFAICR we didn't had a specific dracut module back then - but I doubt this have a real impact on the feature at hand? how a recovery boots is defined in the bootargs

2. Name image files with extension `recovery.squashfs` vs `recovery.img`. It should have been only `recovery.img`, relaying on a name file to assume or not squashfs was a smelly choice.

3. `upgrade` and `reset` should not perform runtime checks on the host (to detect running setup) and relay only on the state yaml file to merge required configurations (no bios/efi checks, no squashfs checks, no runtime only heuristics). Just load config and apply it.

4. grub config (kernel params and each menu options details) should not be static from `install`, `upgrade` and `reset` perspective. I'd say each menu entry (active, fallback and recovery) should source its own specific file sourced by grub.cfg (so intead of `bootargs.cfg`, having `active.cfg`, `passive.cfg` and `recovery.cfg`). So in each upgrade the full kernel call could be updated if needed. `upgrade`, `install` and `reset` should be capable to tune the in image provided grub config (e.g. kernel parameters or grub modules) as the logic of setting some of these belongs there, when config choices are actually taken.

Isn't this just comsetic changes? the bootargs.cfg already have logic to separate when we are booting into active, passive or recovery.

If I got the discussion right, this is to remove golang code that handles the squashfs to avoid maintenance in handling the squashfs type at all. I think those changes ^ will require more or less the same efforts as still you would need to support both (generation, detecting - that is runtime or static, still need to have that business logic in)

5. immutable rootfs dracut module should not relay on image labels. note this is redundant data, we include the image label and the image file in kernel parameters. Systemd is perfectly capable to mount image files, there is no need to discover labels.

6. we should use the squashfs creation root-tree procedure for all filesystems. The more I think about it the more convinced I am. Basically create the tree in a temporary directory and then sync it to the image, this allows us to create images that fit perfectly the root-tree size and simplifies quite a lot our elemental-cli logic. The only counter part is some extra IO operations (the root-tree is copied twice) during installation and upgrade.

Do you mean to switch to all squashfs instead of raw img? that is the opposite of this discussion but I'm either fine - it will make things easier to maintain in the long run. The real issue I see here is trying to support both - as in any case would require doubling efforts.

[kkaempf]

2. Name image files with extension `recovery.squashfs` vs `recovery.img`. It should have been only `recovery.img`, relaying on a name file to assume or not squashfs was a smelly choice.

The current ISO build also generates rootfs.squashfs 🤔

[davidcassany]

Agreed, but AFAICR we didn't had a specific dracut module back then - but I doubt this have a real impact on the feature at hand? how a recovery boots is defined in the bootargs

Well the recovery boot is causing and ugly configuration in grub.cfg with some if/else. If it is on squashfs uses dmsquashfs to boot, if it is not it uses the immutable-rootfs. This is different boot stack and different boot code. Since quite a while I regret of this setup, immutable-rootfs should be simplified to support all cases, this is simple, but requires a grub.cfg change that is likely not to be backward compatible. I blame myself for the current setup this regard.

Isn't this just comsetic changes? the bootargs.cfg already have logic to separate when we are booting into active, passive or recovery.

This is relevant to facilitate upgrades. Current bootargs.cfg is provided by the image. In each grub menu entry we source bootargs.cfg as:

menuentry "${display_name}" --id cos {
  search --no-floppy --label --set=root @STATE_LABEL@
  set img=/cOS/active.img
  set label=@ACTIVE_LABEL@
  loopback loop0 /$img
  set root=($root)
  source (loop0)/etc/cos/bootargs.cfg
  linux (loop0)$kernel $kernelcmd ${extra_cmdline} ${extra_active_cmdline}
  initrd (loop0)$initramfs
}

So now if we want to upgrade from image A to image B it is assumed that image B contains a bootargs.cfg that compatible with config provided by A. I'd like to reduce this dependency by doing something like:

menuentry "${display_name}" --id cos {
  search --no-floppy --label --set=root @STATE_LABEL@
  source ($root)/cOS/active.cfg
}

active.cfg could be still provided from the OS image (imagine a /etc/cos/grub_menuentry.cfg), just extracted at install/upgrade time. Then on upgrades you could simply copy /cOS/active.img to /cOS/passive.img and /cOS/active.cfg to /cOS/passive.cfg. This gives you freedom to vastly modify grub2 configuration on upgrades at the time we keep backward compatibility.

Also this way would be easier from elemental-cli PoV to run modifications to the menu entry config at install/upgrade time (e.g. extra kernel arguments such as selinux or similar stuff).

Note that now one of the blockers to simplify and make our boot procedure simpler is keeping grub setup backward compatibility.

Do you mean to switch to all squashfs instead of raw img? that is the opposite of this discussion but I'm either fine - it will make things easier to maintain in the long run. The real issue I see here is trying to support both - as in any case would require doubling efforts.

No, this is not what I mean, sorry I wrote my thoughts in rush and probably I did not make my point clear enough. Currently for ext2 images we create the image file, create loop device for it, format it, mount it and then dump the OS in there. Obviously this is not applicable for squashfs, hence for squashfs we have a parallel path that creates a tmp dir with the root tree and then creates the squashfs img out of it. What I mean is that we could refactor that to a single path for all cases. Consider this logic:

1. Create tmp dir
2. Apply OS image to tmp dir (from channel, docker or dir)
3. apply after-*-chroot hooks, selinux, install/upgrade grub conf, etc.
4. Create image file (sync root-tree to a loop device or call mkquashfs) with the benefit of being capable to create images of the root-tree size.

In short, my whole idea is making our code more generic so squashfs is not a weird bird causing if/else paths in code. So in config we could set the filesystem to ext2 or squashfs or whatever and the execution path is amost the same everywhere, with only differing in calling the mkfs wrapper or mksquashfs wrapper.

No secure-boot while using squashfs? Fine, this is a limitation of the underlying distro, not Elemental.

[Itxaka]

active.cfg could be still provided from the OS image (imagine a /etc/cos/grub_menuentry.cfg), just extracted at install/upgrade time. Then on upgrades you could simply copy /cOS/active.img to /cOS/passive.img and /cOS/active.cfg to /cOS/passive.cfg. This gives you freedom to vastly modify grub2 configuration on upgrades at the time we keep backward compatibility.

Ugh this seems like a pita. We already have like 5 different grub config files that we load for several things where you can overload stuff (boot assesment, next entry, grubenv, grubconf, grub_oem). Adding extra config files is sure to make this more unmaintainable in the future.

The idea here was to simplify. Less grub code, less grub modules, less paths to test, less paths to maintain in the install/reset/upgrade, less stuff to maintain forever..

Especially with the current bandwidth we have for the toolkit, less is better. Otherwise I spent half a week each week fixing CI tests, adjusting minor things here and there and fighting with the whole process 😢

[davidcassany]

active.cfg could be still provided from the OS image (imagine a /etc/cos/grub_menuentry.cfg), just extracted at install/upgrade time. Then on upgrades you could simply copy /cOS/active.img to /cOS/passive.img and /cOS/active.cfg to /cOS/passive.cfg. This gives you freedom to vastly modify grub2 configuration on upgrades at the time we keep backward compatibility.

Ugh this seems like a pita. We already have like 5 different grub config files that we load for several things where you can overload stuff (boot assesment, next entry, grubenv, grubconf, grub_oem). Adding extra config files is sure to make this more unmaintainable in the future.

As far as I am aware we only have three grub config files, the ones from system/grub2-config. Anything else are runtime grubenv variables and runtime created files, none of this is mandatory to boot and they are meant to be customized from cloud-init. I am not suggesting to add extra files, I just suggest to repurpose the bootargs.cfg to a wider scope so it is easier to evolve the overall grub.cfg on simple OS image upgrades.

The idea here was to simplify. Less grub code, less grub modules, less paths to test, less paths to maintain in the install/reset/upgrade, less stuff to maintain forever..

Yes, this is the idea, less execution paths, less tests in CI, less logic in elemental-toolkit and a robust generic API to maintain in elemental-cli.

[davidcassany]

So far here we have few diverging opinions, but still, I believe there is a first step we probably can agree on:

Do you agree on dropping any squashfs recovery related test in CI? mostly thinking on squashfs install from ISOs. As far as I understood Harvester is only using the squashfs from the ISO to extract it to disk or for PXE booting (no squashfs based recovery). That would already reduce the test matrix and alleviate the CI.

For that, and also thinking on secure boot, I would make a small change in grub configuration, I would remove all insmod squash4 (probably we can remove some others too) and check this could be added on demand by /grubcustom if needed (probably it needs to be sourced before menus in grub.cfg). This way we also have a way to load extra custom modules if needed.

@Itxaka @mudler @bk201 @rancher/elemental