hyper2kvm

hyper2kvm: vSphere Control-Plane + Data-Plane Design

VMware API Integration: hyper2kvm leverages hypersdk (VMware’s modern Python SDK, formerly known as pyvmomi/pyVmomi) for enterprise-grade, type-safe vCenter/vSphere API access.

Table of Contents

Prerequisites

Before following this guide, you should have:

Overview

hyper2kvm is a specialized tool for integrating with VMware vSphere, treating it in a realistic manner: inventory and orchestration exist in one domain, while disk byte movement operates in another. This intentional separation ensures the vSphere integration remains fast, predictable, and highly debuggable. By avoiding the mixing of control-plane and data-plane operations, the tool prevents common pitfalls like performance bottlenecks in large inventories or hidden failures during exports.

The philosophy is embodied in two key files:

This structure allows for modular reuse: the engine can be imported independently for scripting, while the CLI provides a user-friendly interface.

Design Principles

The core principles guide the tool’s behavior to address real-world vSphere challenges:

Control-Plane ≠ Data-Plane (Don’t Mix Them)

Don’t Scan the Universe Unless Asked

vCenter inventories can be massive, and naive “list everything” approaches lead to sluggish tools. To counter this:

Correct Compute Paths for Libvirt ESX (Host-System Path)

hyper2kvm resolves a common failure where libvirt rejects cluster-only paths:

Bytes Should Be Explicit (Download ≠ Convert)

Operators need control over operations to avoid surprises. hyper2kvm exposes distinct data-plane modes:

Async Where It Matters, Sync Where It’s Safe

Never Hide the Real Failure

vSphere failures often involve cryptic issues (e.g., TLS mismatches, invalid thumbprints, path errors, or verbose stderr). vmware_client.py addresses this with:

Architecture Diagram

Philosophy & Design Principles

The vSphere integration follows these core principles:

Main Architecture Components

graph TB
    subgraph HV["hyper2kvm vSphere Integration"]
        CLI[VsphereMode.py<br/>CLI Entrypoint<br/>Sync w/ Threads]
        Engine[VMwareClient.py<br/>Reusable Engine<br/>Async-First]
        DataPlane[Data-Plane<br/>Bytes Movement]

        CLI <--> Engine
        Engine <--> DataPlane

        subgraph CP["Control-Plane"]
            CP1[pyvmomi/pyVim]
            CP2[Connect/Session]
            CP3[DC/Host Cache]
            CP4[VM Lookup]
            CP5[Disk Enum]
            CP6[Snapshot/CBT]
            CP7[DS Browsing]
        end

        subgraph AF["Actions/Flags"]
            AF1[Wires to Options]
            AF2[Calls Engine]
        end

        subgraph DP["Data-Plane Modes"]
            DP1[Direct Export]
            DP2[HTTP Download]
            DP3[VDDK Disk Pull]
        end

        subgraph CBT["CBT Sync Workflow"]
            CBT1[1. Enable CBT]
            CBT2[2. Quiesced Snap]
            CBT3[3. Query Changes]
            CBT4[4. Range HTTP Pull]
            CBT1 --> CBT2 --> CBT3 --> CBT4
        end

        Engine --> CP1
        Engine --> CP2
        Engine --> CP3
        Engine --> CP4
        Engine --> CP5
        Engine --> CP6
        Engine --> CP7

        CLI --> AF1
        CLI --> AF2

        DataPlane --> DP1
        DataPlane --> DP2
        DataPlane --> DP3
    end

    style CLI fill:#FF9800,stroke:#E65100,color:#fff
    style Engine fill:#2196F3,stroke:#1565C0,color:#fff
    style DataPlane fill:#4CAF50,stroke:#2E7D32,color:#fff
    style CP fill:#9C27B0,stroke:#6A1B9A,color:#fff
    style DP fill:#00BCD4,stroke:#006064,color:#fff
    style CBT fill:#F44336,stroke:#C62828,color:#fff

Export Modes

graph LR
    subgraph Modes["Export Mode Options via ExportOptions.export_mode"]
        Export["export Default<br/>━━━━━━━<br/>✓ Converted Output<br/>✓ qcow2/raw Local<br/>✓ Direct Export<br/>✓ VDDK/SSH Transport"]

        Download["download_only<br/>━━━━━━━<br/>✓ Exact VM Folder<br/>✓ Byte-for-Byte<br/>✓ HTTPS /folder<br/>✓ Globs/Concurrency"]

        VDDK["vddk_download<br/>━━━━━━━<br/>✓ Single Disk Raw<br/>✓ Fast VDDK Pull<br/>✓ No Conversion<br/>✓ Sector Reads"]
    end

    style Export fill:#4CAF50,stroke:#2E7D32,color:#fff
    style Download fill:#2196F3,stroke:#1565C0,color:#fff
    style VDDK fill:#FF9800,stroke:#E65100,color:#fff

Export Flow

flowchart TD
    User[User/CLI] --> VsphereMode[VsphereMode]
    VsphereMode --> AsyncExport[VMwareClient.async_export_vm]
    AsyncExport --> ModeCheck{Export Mode?}

    ModeCheck -->|export| ExportMode[Direct Export<br/>Convert]
    ModeCheck -->|download_only| DownloadOnly[Download Only<br/>Exact Copy]
    ModeCheck -->|vddk_download| VDDKDownload[VDDK Disk<br/>Download]

    ExportMode --> Output1[Local qcow2/raw]
    DownloadOnly --> Output2[VM Folder Files]
    VDDKDownload --> Output3[Raw Disk Image]

    style User fill:#9C27B0,stroke:#6A1B9A,color:#fff
    style VsphereMode fill:#FF9800,stroke:#E65100,color:#fff
    style AsyncExport fill:#2196F3,stroke:#1565C0,color:#fff
    style ModeCheck fill:#FFC107,stroke:#F57C00,color:#000
    style ExportMode fill:#4CAF50,stroke:#2E7D32,color:#fff
    style DownloadOnly fill:#00BCD4,stroke:#006064,color:#fff
    style VDDKDownload fill:#F44336,stroke:#C62828,color:#fff
    style Output1 fill:#8BC34A,stroke:#558B2F,color:#fff
    style Output2 fill:#8BC34A,stroke:#558B2F,color:#fff
    style Output3 fill:#8BC34A,stroke:#558B2F,color:#fff

Detailed Architecture Breakdown

Where pyvmomi Ends and Data-Plane Begins

Control-Plane: pyvmomi / pyVim in hyper2kvm

The control-plane leverages pyvmomi for non-I/O tasks:

Key Patterns:

Data-Plane Options in hyper2kvm

  1. Direct Export Mode (export_mode="export"):
    • Implemented in VMwareClient.
    • Builds correct paths for disk access.
    • Validates/resolves vddk-libdir for VDDK transport.
    • Streams subprocess output safely with chunking.
    • Emits local output to output_dir.
    • This is the primary “conversion/export” path, supporting transports like vddk or ssh.
  2. HTTP /folder Download-Only Mode (export_mode="download_only"):
    • Used in both engine and CLI.
    • Mechanics: vCenter exposes https://<vc>/folder/<ds_path>?dcPath=<dc>&dsName=<ds>.
    • Authentication via session cookie from pyvmomi (si._stub.cookie).
    • ds_path is URL-encoded with slashes preserved (quote(..., safe="/")).
    • In VMwareClient: Async downloads with aiohttp + aiofiles (if available), controlled concurrency via download_only_concurrency, and globs/max files for safety.
    • In VsphereMode: File listing via datastore browser, parallel downloads with ThreadPoolExecutor, mirrors layout under --output-dir.
    • This provides a “byte-for-byte VM directory pull” without guest inspection.
  3. VDDK Single-Disk Pull (export_mode="vddk_download"):
    • Implemented in VMwareClient.
    • Control-plane resolves runtime ESXi host and disk backing filename ([ds] folder/disk.vmdk).
    • Data-plane uses VDDKESXClient for sector downloads to local files.
    • Handles: vddk-libdir validation (must contain libvixDiskLib.so), thumbprint normalization/auto-computation (unless no_verify), rate-limited progress logging.
    • This is the “get one disk fast, don’t convert” path.

Why There Are Two Download-Only Implementations (Engine + CLI)

Currently, hyper2kvm features dual implementations for download-only:

CBT Sync in hyper2kvm (Control-Plane + Data-Plane Hybrid)

cbt_sync exemplifies the split’s value for incremental workflows: Control-Plane Steps:

  1. Optionally enable CBT.
  2. Create a quiesced snapshot.
  3. Query changed disk areas via QueryChangedDiskAreas(...).

Data-Plane Step:

  1. For each changed extent, fetch byte ranges using HTTP Range requests (Range: bytes=<start>-<end>) and write to local disk at the offset. This transforms vSphere into an efficient incremental block source, avoiding full disk re-downloads.

Encoding + Typing Choices Used Across hyper2kvm

Mode Selection Cheatsheet (for hyper2kvm)

| Need | Mode | Transport/Details | |——|——|——————-| | Converted qcow2/raw | export_mode="export" | vddk or ssh | | Exact VM folder contents from datastore | export_mode="download_only" | HTTP /folder | | One disk as raw bytes via VDDK | export_mode="vddk_download" | VDDK client | | Incremental updates on local disk | cbt_sync | CBT + ranged HTTP reads |

Usage Examples

Example 1: Basic VM Export

# Export VM
python -m hyper2kvm vsphere \
  --vcenter vcenter.example.com \
  --username admin@vsphere.local \
  --password-file ~/.vcenter_pass \
  --vs-action export-vm \
  --vs-vm-name production-web \
  --output-dir /data/exports

Example 2: Download-Only Mode

# Download exact VM folder contents
python -m hyper2kvm vsphere \
  --vcenter vcenter.example.com \
  --username admin \
  --vs-action download-vm \
  --vs-vm-name backup-server \
  --output-dir /backups

Example 3: VDDK Fast Transfer

# Use VDDK for high-speed transfer
python -m hyper2kvm vsphere \
  --vcenter vcenter.example.com \
  --username admin \
  --vs-action vddk-export \
  --vs-vm-name database-01 \
  --vddk-libdir /usr/lib/vmware-vix-disklib \
  --output-dir /data/vms

Example 4: Programmatic Usage

from hyper2kvm.vmware.clients.client import VMwareClient
from hyper2kvm.vmware.vsphere.mode import ExportOptions

# Initialize client
client = VMwareClient(
    host='vcenter.example.com',
    user='admin@vsphere.local',
    pwd='password'
)

# Configure export
options = ExportOptions(
    vm_name='web-server',
    output_dir='/data/exports',
    export_mode='export',
    transport='vddk',
    vddk_libdir='/usr/lib/vmware-vix-disklib'
)

# Execute export
await client.async_export_vm(options)

Enhancements & Best Practices

For code-level details, see vmware_client.py. If further expansions or examples are needed, provide specifics!

Next Steps

Continue your migration journey:

Getting Help

Found an issue? Report it on GitHub

This site is open source. Improve this page.