HostVenom & WinterNode - Notice history

Our team has successfully restored all game servers to the latest available backup, which was taken on August 23, 2025, at 2 PM Central Time.

We would like to remind our clients that according to our Server Backups article, MySQL databases are not backed up and therefore, would have been lost in this process. Fortunately, no client databases were created on `gs103.chi`.

Thank you for your patience and understanding throughout this incident. Please reach out to our support team if you have any further concerns.

We experienced difficulties installing the operating system due to package download issues. These have cleared itself up naturally(?), and we have now begun restoring client server data. While we cannot provide an exact timeline for full restoration, we've got promising results so far with the speed it is restoring at.

We have determined gs103.chi has experienced a disk failure. We are currently provisioning a new server and restoring the backup data onto it from August 23, 2025 @ 2PM Central Time.

We have swapped the PSU of this node during its downtime that should resolve the issue. Marking as resolved.

The node is now back online and we are currently monitoring the result and working with the upstream provider regarding the incident.

This incident has been resolved with no further alerts triggered.

We have received the following RFO by our Service Provider:

The intervention has been completed.

This operation was closed at 2025-07-10 16:22:16 AEST (UTC +10:00)

Here are the details of this operation:

Network Device Check

Date 2025-07-10 15:55:49 AEST (UTC +10:00), Network Device Check:

There was preventative electrical maintenance at SYD2 today.

The top of rack switch lost power briefly while it was swapped to the alternate source.

The server hardware did not lose power during the maintenance.

The server's network connection is back up and ping is good.

Server did not lose power during maintenance.

An automated ticket has been opened by our service provider at 10.07.2025 01:54:31 (possibly Eastern Time).

Our automated monitoring system is now reporting SYD2 as accessible again. We will continue to monitor for any further issues. From our observations, we believe this might be a network related issue, as all game servers appear to have remained only during this incident.

No further reports. Marking as resolved.

The node is now back online and we are continuing the monitor the server. We believe this was a network blip on our upstream provider.

This incident has been resolved with no further alerts triggered.

We've moved the drive to another node. The node is now back online and are currently monitoring the result.

We have swapped the PSU on this node as we suspect this was the cause of the outage. We'll continue to monitor this node, but as of now, it is back online.

No further tickets have been raised regarding this issue and no Panel Team intervention has been noticed. Marking as resolved.

No further alerts have been triggered and we have not seen any support requests filed. This incident has been resolved.

Unfortunately, we're back to the "Identified" stage. Our Panel Team is actively looking into this issue.

Note: Game Servers are not affected and remain online if previously turned on prior to this incident.

A fix has been applied by the Panel Team and our automated alerts are now showing the Game Panel as online again. We will be monitoring the result and support queues to ensure there are no lingering issues.

Our automated alerts have been triggered for the WinterNode Game Panel reporting "Error 1016 Origin DNS error". Our Panel Team acknowledges this issue and are currently working on a fix.

Our Panel Team has implemented a temporary workaround and we are currently monitoring the result. We have noticed a reduction in support requests regarding this issue.