hyper2kvm

Enhanced Chroot for Bootloader Regeneration

Status: ✅ Production Ready (v0.1.0+) Platforms: openSUSE, SUSE, Fedora, RHEL, Ubuntu, Debian Component: VMCraft Offline Fixer

Overview

The Enhanced Chroot feature provides reliable bootloader regeneration during offline VM migrations by automatically bind-mounting pseudo-filesystems (/proc, /dev, /sys) before executing bootloader commands.

This solves a critical issue where bootloader tools like grub2-mkconfig would fail with errors such as:

error: cannot find a device for / (is /dev mounted?).
awk: fatal: cannot open file `/proc/self/mountinfo'

Problem Statement

Background

During offline VM migrations, hyper2kvm uses a chroot environment to execute commands within the guest filesystem without booting the VM. This is essential for:

Regenerating GRUB configuration (grub2-mkconfig)
Installing bootloader to disk (grub2-install)
Updating initramfs with virtio drivers (dracut, update-initramfs)

The Challenge

Modern bootloader tools require access to kernel pseudo-filesystems:

Filesystem	Purpose	Example Usage
`/proc`	Process and kernel information	`grub2-mkconfig` reads `/proc/self/mountinfo`
`/dev`	Device nodes	`grub2-install` needs block device access
`/sys`	Kernel subsystem information	Hardware detection, driver information

A simple chroot (just changing root directory) doesn’t provide these filesystems, causing bootloader commands to fail.

Impact Before Fix

Without enhanced chroot, migrations would fail on distributions that rely on these tools:

❌ openSUSE Leap 15.4+: grub2-mkconfig failed
❌ SUSE Enterprise Linux: Same GRUB2 infrastructure
❌ Fedora: Modern GRUB2 versions require pseudo-filesystems
❌ RHEL 9/10: Bootloader regeneration unreliable
❌ Ubuntu/Debian: update-grub failures

Result: VMs would not boot after migration, or would boot with incorrect bootloader configuration.

Solution: Enhanced Chroot

How It Works

The enhanced chroot implementation (command_with_mounts() method) follows this sequence:

Check if pseudo-filesystems are already mounted (avoid double-mount)
Bind-mount /proc, /dev, /sys into the chroot environment
Execute the bootloader command
Unmount in reverse order (LIFO: sys, dev, proc)
Guarantee cleanup even if command fails (finally block)

Implementation Details

Location: hyper2kvm/core/vmcraft/main.py:3514-3576

Key Features:

✅ Automatic bind mounting: Creates /proc, /dev, /sys bind mounts
✅ Double-mount prevention: Checks with mountpoint -q before mounting
✅ Guaranteed cleanup: Uses finally block to ensure unmounting
✅ Reverse-order unmount: LIFO order prevents busy mount errors
✅ Debug logging: Detailed logs for troubleshooting
✅ Quiet mode: Suppresses expected failures for better logs

Performance Overhead: ~510ms per command (1.2% of total migration time)

Auto-Detection in GRUB Fixer

The GRUB fixer automatically detects bootloader commands and uses enhanced chroot:

Location: hyper2kvm/fixers/bootloader/grub.py:132-169

Detected Commands (8 total):

grub2-mkconfig, grub-mkconfig
update-grub, update-grub2
grub2-install, grub-install
grub2-probe, grub-probe

When any of these commands are detected, the fixer automatically uses command_with_mounts() instead of the simple command().

Backward Compatibility

The implementation is fully backward compatible:

Feature Detection: Uses hasattr(g, 'command_with_mounts') to check availability
Graceful Fallback: Falls back to command_quiet() if enhanced chroot not available
libguestfs Compatible: Works with both VMCraft and libguestfs backends

Usage

Automatic (Recommended)

No user action required. The enhanced chroot is used automatically during offline migrations when:

Using --regen-initramfs or --update-grub flags
Running OfflineFSFix with bootloader regeneration enabled
Any bootloader command is executed via VMCraft

Example:

# Enhanced chroot is used automatically
hyper2kvm migrate vm.vmdk \
  --output vm.qcow2 \
  --regen-initramfs \
  --update-grub

Programmatic (API)

For custom scripts using VMCraft directly:

from hyper2kvm.core.vmcraft.main import VMCraft

with VMCraft() as g:
    g.add_drive_opts("/path/to/disk.qcow2", readonly=False)
    g.launch()

    # Mount root filesystem
    g.mount("/dev/sda2", "/")

    # Use enhanced chroot for bootloader commands
    # This provides /proc, /dev, /sys automatically
    output = g.command_with_mounts(["grub2-mkconfig", "-o", "/boot/grub2/grub.cfg"])

    # Regular commands use standard chroot
    output = g.command(["cat", "/etc/fstab"])

Validation Results

Test Platform: openSUSE Leap 15.4

Test Type: End-to-end offline migration with GRUB regeneration Duration: 40.8 seconds Result: ✅ SUCCESS

Before Enhanced Chroot

{
  "bootloader": {
    "attempts": [
      {
        "cmd": ["grub2-mkconfig", "-o", "/boot/grub2/grub.cfg"],
        "ok": false,
        "out": "error: cannot find a device for / (is /dev mounted?)."
      }
    ],
    "success": false
  }
}

After Enhanced Chroot

{
  "bootloader": {
    "attempts": [
      {
        "cmd": ["grub2-mkconfig", "-o", "/boot/grub2/grub.cfg"],
        "ok": true,
        "out": ""
      }
    ],
    "success": true
  }
}

Key Metrics:

grub2-mkconfig: SUCCESS ✅ (was failing)
Enhanced chroot overhead: ~510ms (1.2% of total)
No /proc, /dev, or /sys errors
All bind mounts cleaned up successfully

Supported Distributions

Distribution	Status	GRUB Command	Validation
openSUSE Leap 15.4+	✅ Validated	`grub2-mkconfig`	End-to-end test passed
SUSE Enterprise Linux	✅ Expected	`grub2-mkconfig`	Same GRUB2 infrastructure
Fedora 38+	✅ Expected	`grub2-mkconfig`	Same bootloader tools
RHEL 9/10	✅ Expected	`grub2-mkconfig`	Same GRUB2 version
Ubuntu 22.04+	✅ Expected	`update-grub`	Detected and handled
Debian 12+	✅ Expected	`update-grub`	Detected and handled

All distributions using GRUB2 benefit from this fix automatically.

Technical Architecture

Bind Mount Lifecycle

┌─────────────────────────────────────────────────────────┐
│ 1. Pre-Mount Checks                                     │
│    - mountpoint -q /mnt/guest/proc → Not mounted (1)    │
│    - mountpoint -q /mnt/guest/dev  → Not mounted (1)    │
│    - mountpoint -q /mnt/guest/sys  → Not mounted (1)    │
│    Return code 1 = not mounted (proceed with mount)    │
└─────────────────────────────────────────────────────────┘
                          │
                          ▼
┌─────────────────────────────────────────────────────────┐
│ 2. Bind Mount Creation                                  │
│    mount --bind /proc /mnt/guest/proc                  │
│    mount --bind /dev  /mnt/guest/dev                   │
│    mount --bind /sys  /mnt/guest/sys                   │
│    Track mounts: [proc, dev, sys]                      │
└─────────────────────────────────────────────────────────┘
                          │
                          ▼
┌─────────────────────────────────────────────────────────┐
│ 3. Execute Command in Chroot                            │
│    chroot /mnt/guest grub2-mkconfig -o /boot/grub2/... │
│    Command has access to:                               │
│    - /proc/self/mountinfo (via bind mount)             │
│    - /dev/* device nodes (via bind mount)              │
│    - /sys kernel information (via bind mount)          │
└─────────────────────────────────────────────────────────┘
                          │
                          ▼
┌─────────────────────────────────────────────────────────┐
│ 4. Cleanup (Reverse Order - LIFO)                       │
│    umount /mnt/guest/sys    (last mounted, first unmount)│
│    umount /mnt/guest/dev                                │
│    umount /mnt/guest/proc   (first mounted, last unmount)│
│    Guaranteed by finally block (even on errors)         │
└─────────────────────────────────────────────────────────┘

Error Handling Flow

try:
    # Mount phase
    for mount_point in ["proc", "dev", "sys"]:
        if not already_mounted(mount_point):
            bind_mount(mount_point)
            track_for_cleanup(mount_point)

    # Command execution phase
    result = chroot_command(cmd)
    return result

finally:
    # Cleanup phase (ALWAYS runs, even on errors)
    for mount in reversed(cleanup_list):
        try:
            unmount(mount)
        except:
            log_warning(f"Failed to unmount {mount}")
            # Continue with other unmounts

Key Design Decisions:

LIFO unmount order: Prevents “busy” errors from nested mounts
finally block: Guarantees cleanup even if command fails or throws exception
Best-effort unmount: Continue unmounting other mounts even if one fails
Debug-level logging: Unmount failures are logged but don’t block

Troubleshooting

Symptom: GRUB regeneration fails with “/dev not found”

Cause: Enhanced chroot not being used (possible VMCraft version issue)

Solution:

# Verify VMCraft version
python3 -c "from hyper2kvm.core.vmcraft.main import VMCraft; print(hasattr(VMCraft, 'command_with_mounts'))"
# Should print: True

# If False, update hyper2kvm:
pip install --upgrade hyper2kvm

Symptom: “Device or resource busy” during unmount

Cause: Processes still accessing bind-mounted filesystems

Solution: This is handled automatically by reverse-order unmount. If issues persist:

# Check for processes in the mount
sudo lsof | grep /tmp/hyper2kvm-guestfs-

Symptom: Performance degradation during migrations

Cause: Multiple bind mounts for each bootloader command

Impact: Enhanced chroot adds ~510ms overhead per command (negligible)

Verification:

# Check migration report for timing breakdown
cat /tmp/migration-report.json | jq '.analysis.timings.regen_initramfs_and_bootloader'

Developer Information

Testing

Unit Tests: tests/unit/test_vmcraft_enhanced_chroot.py (13 tests)

Run tests:

pytest tests/unit/test_vmcraft_enhanced_chroot.py -v

Test Coverage:

Bind mount creation and cleanup
Double-mount prevention
Reverse-order unmounting
Error handling and recovery
GRUB fixer integration
Backward compatibility

API Reference

`VMCraft.command_with_mounts(cmd: list[str], quiet: bool = False) -> str`

Execute command in guest filesystem with pseudo-filesystems bind-mounted.

Parameters:

cmd: Command and arguments to execute (e.g., ["grub2-mkconfig", "-o", "/boot/grub2/grub.cfg"])
quiet: If True, suppress output for expected failures (default: False)

Returns: Command stdout as string

Raises:

RuntimeError: If not launched or command fails
CommandError: If command execution fails