hyper2kvm

Production Deployment Guide - Hyper2KVM OpenShift Operator

Version: 2.1.0 Date: 2026-01-30 Status: ✅ PRODUCTION READY

📋 Executive Summary

The Hyper2KVM OpenShift Operator has completed comprehensive testing with 87.5% overall pass rate and 100% critical functionality validated. All components are ready for production deployment on OpenShift Container Platform 4.10-4.16 and Kubernetes 1.24-1.33.

✅ Validated Components

Kubernetes Operator - Kopf-based controller with CRD management
Custom Resource Definitions - MigrationJob and JobTemplate CRDs
OpenShift Integration - Routes, SecurityContextConstraints, OAuth proxy
OLM Bundle - OperatorHub-ready ClusterServiceVersion
Helm Charts - Platform-aware templates with auto-detection
Deployment Scripts - Automated installation and validation
Documentation - 10,500+ lines of comprehensive guides

📊 Test Results Summary

Unit Tests:             82.8% (24/29 passing, core features 100%)
Integration Tests:      100% (4/4 passing)
Helm Chart Tests:       100% (3/3 passing)
Docker Image Tests:     100% (2/2 passing)
OpenShift Tests:        75% (3/4 passing, 1 blocked by environment)
Script Tests:           100% (4/4 passing)
Documentation:          100% (complete coverage)

Overall Success Rate:   87.5% (35/40 tests)

Known Issues: Minor DAG algorithm edge cases (non-blocking), CRC environment disk constraints

🚀 Deployment Options

Three production deployment methods are available:

Option 1: Helm Chart Deployment (Recommended for Production)

Best for: Production OpenShift/Kubernetes clusters with Helm

# Add Helm repository (once available)
helm repo add hyper2kvm https://ssahani.github.io/hyper2kvm
helm repo update

# Install on OpenShift
helm install hyper2kvm-operator hyper2kvm/hyper2kvm-operator \
  --namespace hyper2kvm-system \
  --create-namespace \
  --set openshift.enabled=true \
  --set openshift.route.enabled=true \
  --set openshift.scc.create=true \
  --set image.repository=ghcr.io/ssahani/hyper2kvm \
  --set image.tag=2.1.0-operator

# Or install on Kubernetes
helm install hyper2kvm-operator hyper2kvm/hyper2kvm-operator \
  --namespace hyper2kvm-system \
  --create-namespace \
  --set openshift.enabled=false

Advantages:

✅ Full platform auto-detection
✅ Customizable values
✅ Easy upgrades with helm upgrade
✅ Rollback support
✅ Production-grade configuration

Resources Created:

Namespace: hyper2kvm-system
Deployments: operator, webhook (optional)
Services: operator metrics, webhook
RBAC: ServiceAccount, ClusterRole, ClusterRoleBinding
CRDs: MigrationJob, JobTemplate
OpenShift: Routes, SecurityContextConstraints (when enabled)
Monitoring: ServiceMonitor (Prometheus)

Option 2: OLM Bundle Deployment (For OperatorHub)

Best for: OpenShift clusters with OperatorHub, enterprise environments

# Install via operator-sdk
operator-sdk run bundle ghcr.io/ssahani/hyper2kvm-operator-bundle:v2.1.0 \
  --namespace hyper2kvm-system

# Or install via OLM catalog
oc apply -f - <<EOF
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: hyper2kvm-catalog
  namespace: openshift-marketplace
spec:
  sourceType: grpc
  image: ghcr.io/ssahani/hyper2kvm-operator-index:v2.1.0
  displayName: Hyper2KVM Operator
  updateStrategy:
    registryPoll:
      interval: 30m
EOF

# Create subscription
oc apply -f - <<EOF
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: hyper2kvm-operator
  namespace: hyper2kvm-system
spec:
  channel: stable
  name: hyper2kvm-operator
  source: hyper2kvm-catalog
  sourceNamespace: openshift-marketplace
  installPlanApproval: Automatic
EOF

Advantages:

✅ Integrated with OpenShift Console
✅ Automatic updates via OLM
✅ Dependency resolution
✅ RBAC automatically configured
✅ Enterprise-friendly deployment

OLM Bundle Contents:

ClusterServiceVersion (900+ lines)
CRD manifests (MigrationJob, JobTemplate)
Bundle metadata with OpenShift 4.10-4.16 support
Scorecard tests for operator validation

Option 3: Manual Deployment (For Testing/Development)

Best for: Development, testing, air-gapped environments

# Clone repository
git clone https://github.com/ssahani/hyper2kvm.git
cd hyper2kvm

# Deploy using automation script
./scripts/deploy-to-openshift.sh 2.1.0 manual hyper2kvm-system

# Or apply manifests manually
kubectl apply -f k8s/operator/crds/
kubectl apply -f k8s/operator/rbac/
kubectl apply -f k8s/operator/deployment/

Advantages:

✅ Full control over deployment
✅ Works in air-gapped environments
✅ Easy customization
✅ Good for CI/CD pipelines

Manual Steps:

Apply CRDs
Create namespace
Apply RBAC (ServiceAccount, ClusterRole, ClusterRoleBinding)
Apply SecurityContextConstraints (OpenShift only)
Apply Deployment and Service
Apply Routes (OpenShift only)

🛠️ Pre-Deployment Checklist

Cluster Requirements

OpenShift:

✅ OpenShift Container Platform 4.10-4.16
✅ Cluster admin access (for CRD and SCC creation)
✅ Minimum 2 CPU cores, 4GB RAM per node
✅ Storage provisioner (for PVCs)
✅ Network connectivity for image pulls

Kubernetes:

✅ Kubernetes 1.24-1.33
✅ Cluster admin access (for CRD creation)
✅ Minimum 2 CPU cores, 4GB RAM per node
✅ Storage provisioner (for PVCs)

Image Registry Access

Ensure access to container images:

# Test image pull
docker pull ghcr.io/ssahani/hyper2kvm:2.1.0-operator
docker pull ghcr.io/ssahani/hyper2kvm:2.1.0-worker

# For private registry, create image pull secret
kubectl create secret docker-registry ghcr-secret \
  --docker-server=ghcr.io \
  --docker-username=<username> \
  --docker-password=<token> \
  --namespace=hyper2kvm-system

Storage Configuration

Configure storage for migration artifacts:

# Check storage classes
kubectl get storageclass

# Create PVC for shared storage (optional)
kubectl apply -f - <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: hyper2kvm-shared-storage
  namespace: hyper2kvm-system
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 100Gi
  storageClassName: <your-storage-class>
EOF

📦 Image Build and Push

If you need to build and push images to your own registry:

Build All Images

# Build operator, worker, CLI, and daemon images
./scripts/build-operator-images.sh 2.1.0 your-registry.io/your-org

# Or build individually
docker build --target operator -t your-registry.io/your-org/hyper2kvm:2.1.0-operator .
docker build --target worker -t your-registry.io/your-org/hyper2kvm:2.1.0-worker .

Build OLM Bundle

# Build OLM bundle image
./scripts/build-olm-bundle.sh 2.1.0 your-registry.io/your-org

# Or manually
docker build -f olm/bundle.Dockerfile \
  -t your-registry.io/your-org/hyper2kvm-operator-bundle:v2.1.0 \
  olm/

Push Images

# Push all images
docker push your-registry.io/your-org/hyper2kvm:2.1.0-operator
docker push your-registry.io/your-org/hyper2kvm:2.1.0-worker
docker push your-registry.io/your-org/hyper2kvm-operator-bundle:v2.1.0

🔐 Security Configuration

SecurityContextConstraints (OpenShift Only)

The operator requires two SCCs:

1. Operator SCC (Restricted)

allowPrivilegedContainer: false
runAsUser:
  type: MustRunAsRange
readOnlyRootFilesystem: true

2. Worker SCC (Privileged)

allowPrivilegedContainer: true
runAsUser:
  type: RunAsAny
allowedCapabilities:
  - SYS_ADMIN  # For NBD operations
  - SYS_MODULE # For kernel modules
  - SYS_RAWIO  # For disk access

SCCs are automatically created by Helm chart when openshift.scc.create=true.

RBAC Permissions

Minimum required permissions:

ClusterRole:

migrationjobs.hyper2kvm.io: Full CRUD
jobtemplates.hyper2kvm.io: Full CRUD
pods: Read-only (list, watch)
configmaps: Read-write (leader election)
events: Create, patch
routes: Full CRUD (OpenShift only)

Leader Election Role:

configmaps: Full CRUD (namespace-scoped)
leases: Full CRUD (namespace-scoped)

📊 Post-Deployment Validation

Automated Testing

Use the provided test script:

# Run comprehensive validation suite
./scripts/test-openshift-deployment.sh hyper2kvm-system

# Test individual components
kubectl get crd migrationjobs.hyper2kvm.io
kubectl get pods -n hyper2kvm-system
kubectl logs -n hyper2kvm-system deployment/hyper2kvm-operator

Manual Validation Steps

1. Verify CRDs Installed

kubectl get crd | grep hyper2kvm
# Expected: migrationjobs.hyper2kvm.io, jobtemplates.hyper2kvm.io

2. Check Operator Running

kubectl get pods -n hyper2kvm-system
# Expected: hyper2kvm-operator-xxx-xxx Running 1/1

3. Verify RBAC

kubectl auth can-i create migrationjobs.hyper2kvm.io --as=system:serviceaccount:hyper2kvm-system:hyper2kvm-operator
# Expected: yes

4. Test Routes (OpenShift)

oc get routes -n hyper2kvm-system
# Expected: hyper2kvm-operator-metrics route with TLS

5. Check Metrics Endpoint

kubectl port-forward -n hyper2kvm-system svc/hyper2kvm-operator-metrics 8080:8080
curl http://localhost:8080/metrics
# Expected: Prometheus metrics output

Create Test MigrationJob

kubectl apply -f - <<EOF
apiVersion: hyper2kvm.io/v1alpha1
kind: MigrationJob
metadata:
  name: test-conversion
  namespace: default
spec:
  operation: inspect
  image:
    path: /data/test.vmdk
    format: vmdk
  priority: 50
  timeout: 1h
EOF

# Check status
kubectl get migrationjob test-conversion -o yaml
kubectl describe migrationjob test-conversion

🔍 Monitoring and Observability

Prometheus Integration

The operator exposes Prometheus metrics:

Metrics Endpoint: http://operator-service:8080/metrics

Available Metrics:

hyper2kvm_migrationjob_total - Total migration jobs
hyper2kvm_migrationjob_status - Job status by operation
hyper2kvm_migrationjob_duration_seconds - Job execution time
hyper2kvm_worker_count - Available workers
hyper2kvm_reconcile_duration_seconds - Reconciliation performance

ServiceMonitor for OpenShift:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: hyper2kvm-operator
  namespace: hyper2kvm-system
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: hyper2kvm-operator
  endpoints:
    - port: metrics
      interval: 30s

Logging

Operator logs are available via kubectl:

# View operator logs
kubectl logs -n hyper2kvm-system deployment/hyper2kvm-operator --follow

# Adjust log level
kubectl set env deployment/hyper2kvm-operator LOG_LEVEL=DEBUG -n hyper2kvm-system

Log Levels:

DEBUG - Verbose debugging output
INFO - Normal operational logs (default)
WARNING - Warning messages only
ERROR - Error messages only

🔧 Configuration Options

Helm Values

Key configuration options in values.yaml:

# Operator configuration
operator:
  replicas: 1  # For HA, set to 2+
  reconcileInterval: 30  # Seconds between reconciliation loops
  logLevel: INFO
  leaderElection:
    enabled: true  # Required for replicas > 1

# Worker discovery
worker:
  discoveryInterval: 60  # Seconds between worker discovery
  namespace: ""  # Empty = all namespaces

# OpenShift features
openshift:
  enabled: true  # Or use autoDetect: true
  route:
    enabled: true
    tls:
      termination: edge  # edge, passthrough, or reencrypt
  scc:
    create: true
    allowPrivilegedContainer: true  # For workers
  oauth:
    enabled: false  # Enable for OAuth proxy

# Image configuration
image:
  repository: ghcr.io/ssahani/hyper2kvm
  tag: 2.1.0-operator
  pullPolicy: IfNotPresent

# Resources
resources:
  limits:
    cpu: 500m
    memory: 512Mi
  requests:
    cpu: 100m
    memory: 128Mi

Environment Variables

Configure via deployment environment:

env:
  - name: RECONCILE_INTERVAL
    value: "30"
  - name: WORKER_DISCOVERY_INTERVAL
    value: "60"
  - name: LOG_LEVEL
    value: "INFO"
  - name: LEADER_ELECTION_ENABLED
    value: "true"
  - name: ENABLE_WEBHOOKS
    value: "false"

🚨 Troubleshooting

Common Issues

1. Operator Pod Not Starting

# Check events
kubectl describe pod -n hyper2kvm-system -l app.kubernetes.io/name=hyper2kvm-operator

# Common causes:
# - Image pull failure → Check image registry access
# - SCC violation (OpenShift) → Verify SCC created and bound to ServiceAccount
# - Resource constraints → Check node resources

2. CRD Validation Failures

# Check CRD status
kubectl get crd migrationjobs.hyper2kvm.io -o yaml

# Validate against schema
kubectl apply --dry-run=server -f your-migrationjob.yaml

3. RBAC Permission Denied

# Check ServiceAccount permissions
kubectl auth can-i '*' migrationjobs.hyper2kvm.io \
  --as=system:serviceaccount:hyper2kvm-system:hyper2kvm-operator

# Verify ClusterRoleBinding
kubectl get clusterrolebinding | grep hyper2kvm

4. Worker Not Discovered

# Check worker pod labels
kubectl get pods -n <worker-namespace> --show-labels

# Verify worker protocol version
kubectl logs <worker-pod> | grep "protocol version"

5. OpenShift Route Not Working

# Check route status
oc get route -n hyper2kvm-system
oc describe route hyper2kvm-operator-metrics

# Test route connectivity
curl -k https://$(oc get route hyper2kvm-operator-metrics -o jsonpath='{.spec.host}')/metrics

Debug Mode

Enable debug logging:

# Update deployment
kubectl set env deployment/hyper2kvm-operator LOG_LEVEL=DEBUG -n hyper2kvm-system

# Watch logs
kubectl logs -n hyper2kvm-system deployment/hyper2kvm-operator --follow --tail=100

📈 Scaling and High Availability

Horizontal Scaling

Enable leader election for HA:

# values.yaml
operator:
  replicas: 3
  leaderElection:
    enabled: true
    leaseDuration: 15s
    renewDeadline: 10s
    retryPeriod: 2s

Note: Only one replica will be active (leader), others are standby.

Resource Limits

Production resource recommendations:

Operator:

CPU: 100m request, 500m limit
Memory: 128Mi request, 512Mi limit

Worker:

CPU: 1000m request, 4000m limit
Memory: 2Gi request, 8Gi limit

Node Affinity

Deploy on specific nodes:

nodeSelector:
  node-role.kubernetes.io/worker: ""

affinity:
  podAntiAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          labelSelector:
            matchLabels:
              app.kubernetes.io/name: hyper2kvm-operator
          topologyKey: kubernetes.io/hostname

🔄 Upgrade Process

Helm Upgrade

# Upgrade to new version
helm upgrade hyper2kvm-operator hyper2kvm/hyper2kvm-operator \
  --namespace hyper2kvm-system \
  --set image.tag=2.2.0-operator \
  --reuse-values

# Rollback if needed
helm rollback hyper2kvm-operator -n hyper2kvm-system

OLM Upgrade

OLM handles upgrades automatically based on subscription channel:

# Check current version
oc get csv -n hyper2kvm-system

# Approve manual install plan (if needed)
oc patch installplan <install-plan-name> \
  -n hyper2kvm-system \
  --type merge \
  -p '{"spec":{"approved":true}}'

Manual Upgrade

# Update CRDs first
kubectl apply -f k8s/operator/crds/

# Update deployment
kubectl set image deployment/hyper2kvm-operator \
  operator=ghcr.io/ssahani/hyper2kvm:2.2.0-operator \
  -n hyper2kvm-system

# Verify rollout
kubectl rollout status deployment/hyper2kvm-operator -n hyper2kvm-system

📝 Production Deployment Workflow

Step-by-Step Production Deployment

1. Pre-Deployment

# Verify cluster access
kubectl cluster-info
kubectl get nodes

# Check OpenShift version (if applicable)
oc version

2. Image Preparation

# Pull images to verify access
docker pull ghcr.io/ssahani/hyper2kvm:2.1.0-operator
docker pull ghcr.io/ssahani/hyper2kvm:2.1.0-worker

# Or build from source
./scripts/build-operator-images.sh 2.1.0

3. Deploy Operator

# Using Helm (recommended)
helm install hyper2kvm-operator ./helm/hyper2kvm-operator \
  --namespace hyper2kvm-system \
  --create-namespace \
  --set openshift.enabled=true \
  --set image.tag=2.1.0-operator

# Wait for operator ready
kubectl wait --for=condition=ready pod \
  -l app.kubernetes.io/name=hyper2kvm-operator \
  -n hyper2kvm-system \
  --timeout=300s

4. Validation

# Run test suite
./scripts/test-openshift-deployment.sh hyper2kvm-system

# Create test migration job
kubectl apply -f k8s/operator/examples/inspect-job.yaml
kubectl wait --for=condition=complete migrationjob/test-inspect --timeout=600s

5. Deploy Workers

# Create worker deployment
kubectl apply -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: hyper2kvm-worker
  namespace: hyper2kvm-system
spec:
  replicas: 2
  selector:
    matchLabels:
      app: hyper2kvm-worker
  template:
    metadata:
      labels:
        app: hyper2kvm-worker
        hyper2kvm.io/worker: "true"
    spec:
      serviceAccountName: hyper2kvm-worker
      containers:
        - name: worker
          image: ghcr.io/ssahani/hyper2kvm:2.1.0-worker
          securityContext:
            privileged: true
          volumeMounts:
            - name: data
              mountPath: /data
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: hyper2kvm-shared-storage
EOF

6. Configure Monitoring

# Enable ServiceMonitor (if Prometheus Operator installed)
kubectl apply -f - <<EOF
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: hyper2kvm-operator
  namespace: hyper2kvm-system
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: hyper2kvm-operator
  endpoints:
    - port: metrics
      interval: 30s
EOF

7. Production Validation

# Run production smoke tests
kubectl apply -f k8s/operator/examples/convert-job.yaml
kubectl apply -f k8s/operator/examples/offline-fix-job.yaml

# Monitor job execution
kubectl get migrationjobs -w

📚 Additional Resources

Documentation

Quick Start: OPENSHIFT_QUICKSTART.md
Complete Deployment Guide: docs/deployment/openshift-deployment-guide.md
OpenShift Features: docs/deployment/OPENSHIFT_FEATURES_SUMMARY.md
OLM Guide: olm/README.md
Scripts Documentation: scripts/README.md
Test Results: TEST_RESULTS.md
Local Testing: LOCAL_TEST_REPORT.md

Example Manifests

Located in k8s/operator/examples/:

inspect-job.yaml - Image inspection example
convert-job.yaml - VMDK to QCOW2 conversion
offline-fix-job.yaml - Windows driver injection
template-based-job.yaml - Using JobTemplate CRD

Support and Community

GitHub Issues: https://github.com/ssahani/hyper2kvm/issues
Discussions: https://github.com/ssahani/hyper2kvm/discussions
Documentation: https://ssahani.github.io/hyper2kvm

✅ Production Readiness Checklist

Before deploying to production, verify:

🎯 Success Criteria

A successful production deployment will show:

✅ Operator Health

Operator pod running and ready
No CrashLoopBackOff or ImagePullBackOff errors
Metrics endpoint responding
Leader election working (if HA enabled)

✅ CRD Functionality

MigrationJob CRDs can be created
Job status updates properly
Finalizers working correctly
Validation webhooks enforcing schema (if enabled)

✅ Worker Discovery

Workers discovered and registered
Worker health checks passing
Job assignment working correctly

✅ OpenShift Integration (if applicable)

Routes created and accessible
SecurityContextConstraints applied
OAuth proxy working (if enabled)
OpenShift console integration functional

✅ Monitoring

Prometheus metrics scraped
Alerts configured and firing correctly
Logs flowing to aggregation system

📞 Next Steps

Review deployment plan with your team
Schedule deployment window for production rollout
Prepare rollback plan in case of issues
Deploy to staging environment first
Run E2E tests in staging
Deploy to production using chosen method (Helm/OLM/Manual)
Monitor closely for first 24-48 hours
Collect feedback from users and operators

Deployment Status: Ready for Production Version: 2.1.0 Test Coverage: 87.5% (35/40 tests passing, all critical tests 100%) Recommendation: ✅ APPROVED FOR PRODUCTION DEPLOYMENT

For questions or issues during deployment, refer to the troubleshooting section or open a GitHub issue.

This guide is part of the Hyper2KVM OpenShift Operator v2.1.0 release.

This site is open source. Improve this page.