Duration: 6 hours and 4 minutes

Affected Components:

Oct 4, 05:22:35 GMT+0
Resolved - This incident has been resolved..

Oct 4, 00:00:00 GMT+0
Monitoring - We have not observed any further packet loss over the past 30 minutes, and affected machines appear to be stable. We are still waiting on confirmation from our data center provider regarding the root cause, so we're continuing to monitor closely..

Oct 3, 23:18:23 GMT+0
Investigating - We are currently investigating this incident..

Duration: 2 hours

Affected Components: , , , , , ,

Sep 9, 16:00:00 GMT+0
Completed - Maintenance has completed successfully.

Sep 9, 14:00:01 GMT+0
Identified - Maintenance is now in progress.

Sep 1, 15:39:12 GMT+0
Identified - We will be performing scheduled maintenance to apply security updates and infrastructure upgrades across our systems. Maintenance window: September 9, 2026, 14:00–16:00 UTC Expected impact: Nodes will be updated on a rolling basis. Game servers hosted on an affected node may experience an outage of up to 15 minutes while that node is being updated. Not all nodes will be updated at the same time. Affected services: Game servers We will work to minimize disruption throughout the maintenance window and will monitor services as each node is brought back online. If you have any questions or concerns, please don't hesitate to reach out to our support team..

Duration: 1 hour and 52 minutes

Affected Components: , , ,

Aug 13, 12:52:21 GMT+0
Completed - Maintenance has completed successfully..

Aug 13, 11:00:01 GMT+0
Identified - Maintenance is now in progress.

Aug 8, 12:58:53 GMT+0
Identified - We will be performing scheduled upgrades to the infrastructure that supports the following services: * Legacy Game Panel * Legacy Billing Panel * Bare Metal Control Panel * VPS Control Panel During this maintenance window, these services may be temporarily unavailable for up to 30 minutes. Your active game/bare metal/VPS services will remain unaffected; only access to the following panels will be impacted..

Duration: 39 minutes

Affected Components:

Jul 31, 11:38:56 GMT+0
Completed - Maintenance has completed successfully..

Jul 31, 11:00:01 GMT+0
Identified - Maintenance is now in progress.

Jul 26, 14:34:07 GMT+0
Identified - We're are moving snow-game-geg to upgraded hardware. As part of the transition, your server will be assigned a new IP address within our own network space. During this time, the server will be temporarily unavailable for around 30 minutes. However we have scheduled the maintenance block for 1 hour as a precaution. After the migration, your new server connection details will be visible in your control panel..

Duration: 51 minutes

Affected Components:

Jul 31, 10:50:59 GMT+0
Completed - Maintenance has completed successfully..

Jul 31, 10:00:01 GMT+0
Identified - Maintenance is now in progress.

Jul 30, 16:55:17 GMT+0
Identified - We’ll be performing a scheduled upgrade to the infrastructure that supports our VPS Control Panel. During this maintenance window, the VPS Control Panel may be temporarily unavailable for up to 30 minutes. Your active VPS services will remain unaffected; only access to the Control Panel will be impacted..

Duration: 49 minutes

Affected Components: , , , , , ,

Jul 9, 15:48:40 GMT+0
Completed - Maintenance has completed successfully. If you happen to run into any issues, please don't hesitate to reach out to support..

Jul 9, 15:00:01 GMT+0
Identified - Maintenance is now in progress.

Jul 9, 08:40:13 GMT+0
Identified - We will be performing scheduled maintenance to apply routine security and infrastructure updates across our systems. Window: 2026-07-09, 15:00–16:30 UTC Expected impact: Nodes hosting game servers will be updated on a rolling basis. Game servers on an affected node may experience an outage of up to 15 minutes while that node is updated. Not all nodes will be updated at the same time. Affected services: Game servers If you have any questions or concerns, please don't hesitate to reach out to our support team..

Duration: 33 minutes

Affected Components:

Jul 8, 09:57:44 GMT+0
Resolved - This incident has been resolved..

Jul 8, 09:30:08 GMT+0
Identified - We are currently in the process of migrating the server to replacement hardware..

Jul 8, 09:24:53 GMT+0
Investigating - We are currently investigating this incident..

Duration: 1 hour and 7 minutes

Affected Components:

Jul 8, 02:38:35 GMT+0
Resolved - This incident has been resolved..

Jul 8, 01:31:58 GMT+0
Investigating - We are currently investigating this incident..

Duration: 36 minutes

Affected Components: ,

Jul 7, 14:06:24 GMT+0
Completed - Maintenance has completed successfully..

Jul 7, 13:30:01 GMT+0
Identified - Maintenance is now in progress.

Jul 7, 07:57:14 GMT+0
Identified - We will be performing emergency maintenance across our entire VPS fleet. All VPS instances will experience a brief interruption as hosts are restarted in sequence. We expect downtime per instance to be no longer than 30 minutes. We apologise for the short notice. Updates will be posted here as the maintenance progresses..

Duration: 17 minutes

Affected Components:

Jun 2, 08:33:54 GMT+0
Resolved - This incident has been resolved..

Jun 2, 08:16:48 GMT+0
Investigating - We are currently investigating this incident..

Duration: 2 hours and 46 minutes

May 21, 15:23:30 GMT+0
Postmortem - Over a roughly 24-hour period, some users may have experienced intermittent errors or unavailability on services hosted on the `aiba` hypervisor. **What happened** The hypervisor experienced a hardware fault (CPU cooler failure), and we temporarily migrated its workload to a backup machine to keep services running. About a day later, the backup machine developed a separate hardware issue (memory instability) that caused the disruption during this window. We have repaired the original hypervisor and migrated the workload back to it to restore service..

May 18, 21:01:14 GMT+0
Resolved - This incident has been resolved. A full RFO (reason for outage) will follow within the next couple days..

May 18, 20:54:46 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

May 18, 18:15:26 GMT+0
Investigating - We are currently investigating this incident..

Duration: 17 minutes

May 18, 10:47:29 GMT+0
Completed - Maintenance has completed successfully..

May 18, 10:30:01 GMT+0
Identified - Maintenance is now in progress.

May 18, 10:30:00 GMT+0
Identified - We are planning for a emergency scheduled maintenance during that time..

Duration: 30 minutes

May 17, 16:00:14 GMT+0
Completed - Maintenance has completed successfully..

May 17, 15:30:01 GMT+0
Identified - Maintenance is now in progress.

May 17, 15:30:00 GMT+0
Identified - We are planning for a scheduled maintenance during that time..

Duration: 5 hours and 33 minutes

May 17, 16:03:16 GMT+0
Resolved - This incident has been resolved..

May 17, 16:01:13 GMT+0
Monitoring - Drives from the Hypervisor have been moved to a replacement hypervisor and services are currently booting..

May 17, 11:03:37 GMT+0
Identified - We have identified the issue to a faulty CPU cooler and have dispatched a technician to the datacenter..

May 17, 10:30:21 GMT+0
Investigating - We are currently investigating this incident..

Duration: 47 minutes

Affected Components:

May 14, 13:39:07 GMT+0
Resolved - This incident has been resolved..

May 14, 12:52:03 GMT+0
Investigating - We are currently investigating this incident..

Duration: 7 hours and 16 minutes

May 15, 09:12:12 GMT+0
Postmortem - Over a roughly 48-hour period, the `rene` hypervisor became progressively unreliable, beginning with intermittent kernel-level hangs and ending with a storage subsystem fault that made the host unsafe to keep in production. After the fault was diagnosed as on-disk filesystem metadata corruption that could not be repaired in place, we made the decision to migrate every VM off `rene` onto replacement hardware rather than continue attempting recovery on the affected host. ### Timeline (UTC) **2026-05-12** * 04:02 — Hypervisor (and subsequent VMs) stopped responding to network and console. * 04:07 — NOC alerted; investigation began. * 04:16 — Out-of-band investigation confirmed the host was wedged at the kernel level and not recoverable in software. * 04:25 — Host was forcefully power-cycled. Boot and VM auto-start completed within a few minutes. * 04:29 — All VMs confirmed running. **2026-05-13** * The host hung again. Investigation of the second event revealed the deeper cause was not the memory-management issue but corruption in the storage pool's internal allocation metadata. * This corruption had likely been accumulating over time and was not repairable in place. Standard integrity checks completed days earlier had not surfaced it, because the affected metadata is validated differently from normal file data. * We began planning a controlled migration of all VMs from `rene` to replacement hardware, intending to perform it within the following day. **2026-05-14** * Before the planned migration could be carried out, `rene` failed again and could not be returned to a usable state. With the host effectively offline, we performed the migration from a rescue environment instead — booting recovery media, bringing the storage pool up in a read-only mode, and copying all VM disk images off directly. * All VMs were re-established on replacement hardware. Migration complete. ### Root cause The first failure on 05-12 presented as a kernel memory-management fault — a specific class of CPU lockup driven by the host's memory subsystem under heavy concurrent activity from VMs. This was a real issue, and the configuration change we applied resolved it. We initially considered the incident closed. The second failure made it clear the memory-management issue, while genuine, was not the underlying problem. The deeper cause was corruption in the storage pool's internal space-allocation metadata — the bookkeeping the filesystem uses to track which regions of disk are free or in use. When the filesystem attempted to load this metadata during normal operation, it encountered inconsistencies it could not reconcile, which caused the storage subsystem to stall. Because that subsystem underpins everything the host does, the stalls escalated into full host hangs. The third failure confirmed the diagnosis and decided the path forward. On reboot, the host hit the same metadata inconsistency during the initial storage import and could not progress past it into a usable state. With the host no longer recoverable in place, we moved to evacuation from a rescue environment — a slower and more involved process than the controlled migration we had been planning, which meaningfully extended the time to full recovery. We're sorry for the extended downtime and the disruption this caused. If your VPS is not behaving as expected, please reach out to support and we'll look into it directly..

May 14, 17:49:37 GMT+0
Resolved - This incident has been resolved. The rene hypervisor experienced a storage-related fault that prevented it from operating reliably. After assessment, we made the decision to migrate all affected VMs to replacement hardware rather than continue recovery attempts on the affected host. All VMs have been recovered and migrated. The large majority were transferred fully intact. A small number may require minor filesystem checks on first boot; affected customers will be contacted directly. We know this was a long and disruptive incident, and we're sorry for the extended downtime. A full RFO (reason for outage) with root cause and remediation details will follow in the next couple days. If your VM is not behaving as expected, please contact support and we'll assist directly..

May 14, 17:09:03 GMT+0
Identified - We have booted the hypervisor into a rescue environment and are currently restoring VMs to another hypervisor..

May 14, 16:41:32 GMT+0
Identified - We are continuing to work on a fix for this incident..

May 14, 11:21:47 GMT+0
Identified - We are continuing to work on a fix for this incident..

May 14, 10:33:36 GMT+0
Investigating - We are currently investigating this incident..

Duration: 47 minutes

Affected Components:

May 13, 19:16:32 GMT+0
Completed - Maintenance has completed successfully..

May 13, 18:30:01 GMT+0
Identified - Maintenance is now in progress.

May 13, 18:30:00 GMT+0
Identified - We’ll be performing a scheduled upgrade to the infrastructure that supports our Billing Panel. During this maintenance window, the Billing Panel may be temporarily unavailable for up to 30 minutes. Please note that this maintenance will not affect any of your active services, including game servers, VPS, or dedicated servers..

Duration: 21 minutes

May 12, 04:54:49 GMT+0
Postmortem - On 2026-05-12 at approximately 04:02 UTC, the `rene: vps-dfw-7950X` hypervisor became unreachable due to a kernel fault on the host. The host was recovered via power cycle and all VMs returned to service automatically. **Timeline (UTC)** * 04:02 - Hypervisor (and subsequent VMs) stopped responding to network and console. * 04:07 - NOC alerted; investigation began. * 04:16 - Out-of-band investigation confirmed the host was wedged at the kernel level and not recoverable in software. * 04:25 - Host was forcefully power-cycled. Boot and VM auto-start completed within a few minutes. * 04:29 - All VMs confirmed running. We've identified the kernel condition that caused the host to hang and applied a configuration change to prevent it from recurring. Additional monitoring has been added to detect early warning signs of this class of issue across our fleet. We're sorry for the disruption. If you're still experiencing issues with your VM, please contact support and we'll take a look..

May 12, 04:41:53 GMT+0
Resolved - This incident has been resolved..

May 12, 04:32:34 GMT+0
Monitoring - We implemented a fix and are currently monitoring the result..

May 12, 04:20:58 GMT+0
Investigating - We are currently investigating this incident..

Duration: 1 hour

May 7, 12:00:00 GMT+0
Completed - Maintenance has completed successfully.

May 7, 11:00:01 GMT+0
Identified - Maintenance is now in progress.

May 7, 11:00:00 GMT+0
Identified - We are scheduling planned maintenance to upgrade the underlying hypervisor infrastructure for VPS Elder Services. As part of this process, virtual servers will be handled individually, with each instance undergoing a brief reboot on the updated hypervisors. During this time, the affected service will be offline for a few minutes. This maintenance window is scheduled for 1 hour. Most users will only experience a short interruption during their specific reboot period. If you encounter any issues following the maintenance, please contact support for assistance..

Duration: 2 hours and 10 minutes

Affected Components:

Apr 29, 18:10:29 GMT+0
Completed - Maintenance has completed successfully..

Apr 29, 16:00:01 GMT+0
Identified - Maintenance is now in progress.

Apr 29, 16:00:00 GMT+0
Identified - We are planning for a scheduled maintenance during that time for eric-game-geg. During this time, the server will be temporarily unavailable for around 30 minutes. However we have scheduled the maintenance block for 1 hour as a precaution..

Duration: 6 hours and 18 minutes

Affected Components:

Apr 25, 12:13:16 GMT+0
Resolved - This incident has been resolved..

Apr 25, 05:55:44 GMT+0
Investigating - We are currently investigating this incident..

Duration: 22 minutes

Apr 21, 09:22:21 GMT+0
Completed - Maintenance has completed successfully..

Apr 21, 09:00:01 GMT+0
Identified - Maintenance is now in progress.

Apr 21, 09:00:00 GMT+0
Identified - We are scheduling planned maintenance to migrate customers from the temporary machine (aiba: vps-dfw-7950X), used during the prior issues on the Carolina machine, back to Carolina as the hardware issues have been resolved. Incident details are available here: <https://berrybytestatus.com/cmn5nm57508itnbqnorqgl5n2> Virtual servers will be migrated one‑by‑one from the temporary machine to Carolina. Each migration will involve a brief reboot on the new hypervisor, which will place the affected server offline for a couple minutes. This maintenance window is scheduled for 1 hour; most customers will only experience a short downtime during their individual reboot. If you encounter any issues after the maintenance, please contact support for assistance..

Duration: 2 hours and 23 minutes

Affected Components:

Apr 18, 16:23:01 GMT+0
Completed - Maintenance has completed successfully..

Apr 18, 14:00:01 GMT+0
Identified - Maintenance is now in progress.

Apr 18, 14:00:00 GMT+0
Identified - We will be performing scheduled maintenance in our US West (Seattle, WA) location tomorrow. Our datacenter provider is rebalancing power across racks, and as part of this, we will be proactively swapping PDUs (one phase at a time) to ensure stability and avoid any unplanned issues. **Impact:** We do not expect any downtime. Work is being done in a phased manner to minimize risk, but brief interruptions are possible. **Time:** 7AM-8AM CST We’ll keep you updated throughout the maintenance window. If you notice anything unusual, please reach out to support..

Duration: 15 minutes

Affected Components:

Apr 10, 12:54:46 GMT+0
Resolved - This incident has been resolved..

Apr 10, 12:42:04 GMT+0
Monitoring - A fix was implemented by Cosmic and we are currently monitoring the result..

Apr 10, 12:39:40 GMT+0
Investigating - We are currently investigating this incident. Seems to only affect connections between both regions..

Duration: 1 hour and 1 minute

Affected Components: , ,

Apr 3, 06:31:39 GMT+0
Resolved - This incident has been resolved..

Apr 3, 05:42:40 GMT+0
Monitoring - Cosmic has implemented a fix and we are currently monitoring the result. Ref: <https://status.as30456.net/cmnigw4ki0ejswkl8ij6nn44f>.

Apr 3, 05:30:53 GMT+0
Investigating - We are currently investigating this incident..

Duration: 1 hour and 23 minutes

Affected Components:

Apr 2, 01:22:52 GMT+0
Completed - Maintenance has completed successfully..

Apr 2, 01:00:00 GMT+0
Identified - Maintenance has been rescheduled to April 2 at 12:00 AM UK time due to datacenter technician availability..

Apr 2, 00:00:01 GMT+0
Identified - Maintenance is now in progress.

Apr 2, 00:00:00 GMT+0
Identified - We’re scheduling a follow-up maintenance window to implement the permanent fix for our recent hardware issue (<https://berrybytestatus.com/cmmm345iz0b4yn9qfrgoioujt>) for heretic: game-lon-7900. We expect the maintenance to take under 1 hour and 30 minutes, but as a precaution, we have scheduled a 3 hour window to ensure sufficient time for safe completion and validation. We appreciate your understanding and patience as we resolve this issue promptly to maintain system reliability and performance..

Duration: 3 hours and 9 minutes

Affected Components:

Mar 25, 09:28:41 GMT+0
Resolved - The issue has been resolved, and all services are now running on temporary replacement hardware. We’ll schedule maintenance in the next couple of weeks to migrate services back to the original hardware after resolving the issues on it..

Mar 25, 08:51:22 GMT+0
Identified - The machine is now back online. However, due to ongoing issues with the host operating system, we will be migrating all customers to another machine temporarily. This will allow us to reinstall and rebuild the original machine on a new host OS. Once the reinstall is complete and stable, we’ll migrate all customers back to it..

Mar 25, 07:31:01 GMT+0
Identified - We are continuing to work on a fix for this incident..

Mar 25, 06:19:58 GMT+0
Investigating - We are currently investigating this incident..

Duration: 6 hours and 50 minutes

Affected Components:

Mar 23, 16:30:09 GMT+0
Resolved - This incident has been resolved..

Mar 23, 15:41:49 GMT+0
Monitoring - CosmicGuard has now fixed the issue, and we are continuing to monitor the situation to ensure everything remains stable..

Mar 23, 14:53:42 GMT+0
Identified - CosmicGuard (our DDoS mitigation partner) is still investigating the issue. You can monitor their progress here: <https://status.as30456.net/cmn31qz0511dmt3c6v3eht9fm>..

Mar 23, 10:19:21 GMT+0
Identified - Our DDoS mitigation partner, CosmicGuard, has identified an issue on their end and is currently investigating. We’ll provide updates as more information becomes available..

Mar 23, 09:40:17 GMT+0
Investigating - We are currently investigating this incident. This issue seems to be primarily affecting UDP connections..

Duration: 2 hours and 14 minutes

Affected Components:

Mar 11, 21:29:51 GMT+0
Completed - Maintenance has completed successfully. We’ve implemented a temporary solution to ensure stability and minimize downtime while we wait for new hardware to arrive. Please note that a follow-up maintenance window will be scheduled soon to complete the permanent fix. We’ll share the details in advance to minimize any disruption..

Mar 11, 20:36:20 GMT+0
Identified - The hardware replacement for the machine has been completed. We are currently investigating an issue with degraded network performance that was observed afterward..

Mar 11, 19:16:02 GMT+0
Identified - Maintenance is now in progress..

Mar 11, 19:15:00 GMT+0
Identified - Due to unexpected hardware issues, we need to perform urgent maintenance on heretic: game-lon-7900\. This work is essential to prevent further degradation of system performance and stability. During this time, the server will be temporarily unavailable while we conduct diagnostics and hardware repairs. We expect the maintenance to take under 1 hour, but as a precaution, we have scheduled a 2 hour and 30 minute window to ensure sufficient time for safe completion and validation. We appreciate your understanding and patience as we resolve this issue promptly to maintain system reliability and performance..

Duration: 7 minutes

Affected Components:

Mar 8, 10:07:17 GMT+0
Completed - Maintenance has completed successfully..

Mar 8, 10:00:01 GMT+0
Identified - Maintenance is now in progress.

Mar 8, 10:00:00 GMT+0
Identified - We’ll be performing a scheduled upgrade to the infrastructure that supports our Billing Panel. During this maintenance window, the Billing Panel may be temporarily unavailable for up to 30 minutes. Please note that this maintenance will not affect any of your active services, including game servers, VPS, or dedicated servers..

Duration: 2 hours and 19 minutes

Affected Components:

Mar 4, 17:26:42 GMT+0
Resolved - This incident has been resolved..

Mar 4, 15:46:34 GMT+0
Identified - We are continuing to work on a fix for this incident..

Mar 4, 15:07:21 GMT+0
Investigating - We are currently investigating this incident..

Duration: 31 minutes

Affected Components:

Mar 2, 14:30:29 GMT+0
Resolved - This incident has been resolved..

Mar 2, 13:59:19 GMT+0
Investigating - We are currently investigating this incident..

Duration: 7 hours and 16 minutes

Affected Components:

Feb 26, 16:43:18 GMT+0
Resolved - This incident has been resolved..

Feb 26, 14:38:11 GMT+0
Identified - We are continuing to work on a fix for this incident..

Feb 26, 11:45:03 GMT+0
Identified - We are continuing to work on a fix for this incident..

Feb 26, 09:27:44 GMT+0
Investigating - We are currently investigating this incident..

Duration: 9 minutes

Affected Components:

Feb 16, 23:09:00 GMT+0
Resolved - This incident has been resolved..

Feb 16, 23:00:00 GMT+0
Investigating - We are currently investigating this incident..

Duration: 41 minutes

Affected Components:

Feb 11, 02:02:09 GMT+0
Resolved - This incident has been resolved..

Feb 11, 01:20:45 GMT+0
Investigating - We are currently investigating this incident..

Duration: 1 hour and 30 minutes

Affected Components:

Jan 29, 10:58:51 GMT+0
Resolved - This incident has been resolved..

Jan 29, 09:32:54 GMT+0
Identified - We have identified the issue to be partial connectivity loss affecting specific regions. This has been escalated to the data center provider..

Jan 29, 09:28:27 GMT+0
Investigating - We are currently investigating this incident..

Duration: 2 hours and 21 minutes

Affected Components:

Jan 19, 16:57:39 GMT+0
Resolved - This incident has been resolved..

Jan 19, 14:36:30 GMT+0
Investigating - We are currently investigating this incident..

Duration: 3 minutes

Affected Components:

Feb 11, 01:23:27 GMT+0
Resolved - This incident has been resolved..

Jan 10, 04:41:21 GMT+0
Resolved - This incident has been resolved..

Jan 10, 04:38:49 GMT+0
Investigating - We are currently investigating this incident..

Duration: 3 hours and 31 minutes

Affected Components:

Jan 6, 05:12:45 GMT+0
Resolved - This incident has been resolved..

Jan 6, 01:42:10 GMT+0
Investigating - We are currently investigating this incident..

Duration: 1 hour and 6 minutes

Affected Components:

Dec 5, 11:01:19 GMT+0
Resolved - This incident has been resolved..

Dec 5, 09:55:03 GMT+0
Investigating - We are currently investigating this incident. This seems to be isolated with 1.1.1.1, and not 1.0.0.1 for nameservers.

Duration: 8 minutes

Affected Components:

Dec 3, 07:06:22 GMT+0
Resolved - This incident has been resolved..

Dec 3, 07:02:41 GMT+0
Monitoring - A fix was implemented and we are currently monitoring the result..

Dec 3, 06:58:19 GMT+0
Investigating - We are currently investigating this incident..