berrybyte - VPS Elder performance and avaliability issue. – Incident details

Over a roughly 48-hour period, the rene hypervisor became progressively unreliable, beginning with intermittent kernel-level hangs and ending with a storage subsystem fault that made the host unsafe to keep in production. After the fault was diagnosed as on-disk filesystem metadata corruption that could not be repaired in place, we made the decision to migrate every VM off rene onto replacement hardware rather than continue attempting recovery on the affected host.

Timeline (UTC)

2026-05-12

• 04:02 — Hypervisor (and subsequent VMs) stopped responding to network and console.

• 04:07 — NOC alerted; investigation began.

• 04:16 — Out-of-band investigation confirmed the host was wedged at the kernel level and not recoverable in software.

• 04:25 — Host was forcefully power-cycled. Boot and VM auto-start completed within a few minutes.

• 04:29 — All VMs confirmed running.

2026-05-13

• The host hung again. Investigation of the second event revealed the deeper cause was not the memory-management issue but corruption in the storage pool's internal allocation metadata.

• This corruption had likely been accumulating over time and was not repairable in place. Standard integrity checks completed days earlier had not surfaced it, because the affected metadata is validated differently from normal file data.

• We began planning a controlled migration of all VMs from rene to replacement hardware, intending to perform it within the following day.

2026-05-14

• Before the planned migration could be carried out, rene failed again and could not be returned to a usable state. With the host effectively offline, we performed the migration from a rescue environment instead — booting recovery media, bringing the storage pool up in a read-only mode, and copying all VM disk images off directly.

• All VMs were re-established on replacement hardware. Migration complete.

Root cause

The first failure on 05-12 presented as a kernel memory-management fault — a specific class of CPU lockup driven by the host's memory subsystem under heavy concurrent activity from VMs. This was a real issue, and the configuration change we applied resolved it. We initially considered the incident closed.

The second failure made it clear the memory-management issue, while genuine, was not the underlying problem. The deeper cause was corruption in the storage pool's internal space-allocation metadata — the bookkeeping the filesystem uses to track which regions of disk are free or in use. When the filesystem attempted to load this metadata during normal operation, it encountered inconsistencies it could not reconcile, which caused the storage subsystem to stall. Because that subsystem underpins everything the host does, the stalls escalated into full host hangs.

The third failure confirmed the diagnosis and decided the path forward. On reboot, the host hit the same metadata inconsistency during the initial storage import and could not progress past it into a usable state. With the host no longer recoverable in place, we moved to evacuation from a rescue environment — a slower and more involved process than the controlled migration we had been planning, which meaningfully extended the time to full recovery.

We're sorry for the extended downtime and the disruption this caused. If your VPS is not behaving as expected, please reach out to support and we'll look into it directly.

This incident has been resolved.

The rene hypervisor experienced a storage-related fault that prevented it from operating reliably. After assessment, we made the decision to migrate all affected VMs to replacement hardware rather than continue recovery attempts on the affected host.

All VMs have been recovered and migrated. The large majority were transferred fully intact. A small number may require minor filesystem checks on first boot; affected customers will be contacted directly.

We know this was a long and disruptive incident, and we're sorry for the extended downtime. A full RFO (reason for outage) with root cause and remediation details will follow in the next couple days.

If your VM is not behaving as expected, please contact support and we'll assist directly.

We have booted the hypervisor into a rescue environment and are currently restoring VMs to another hypervisor.