HostVenom & WinterNode - Notice history

This incident has been resolved with no further alerts triggered.

We have received the following RFO by our Service Provider:

The intervention has been completed.

This operation was closed at 2025-07-10 16:22:16 AEST (UTC +10:00)

Here are the details of this operation:

Network Device Check

Date 2025-07-10 15:55:49 AEST (UTC +10:00), Network Device Check:

There was preventative electrical maintenance at SYD2 today.

The top of rack switch lost power briefly while it was swapped to the alternate source.

The server hardware did not lose power during the maintenance.

The server's network connection is back up and ping is good.

Server did not lose power during maintenance.

An automated ticket has been opened by our service provider at 10.07.2025 01:54:31 (possibly Eastern Time).

Our automated monitoring system is now reporting SYD2 as accessible again. We will continue to monitor for any further issues. From our observations, we believe this might be a network related issue, as all game servers appear to have remained only during this incident.

No further reports. Marking as resolved.

The node is now back online and we are continuing the monitor the server. We believe this was a network blip on our upstream provider.

This incident has been resolved with no further alerts triggered.

We've moved the drive to another node. The node is now back online and are currently monitoring the result.

We have swapped the PSU on this node as we suspect this was the cause of the outage. We'll continue to monitor this node, but as of now, it is back online.

No further tickets have been raised regarding this issue and no Panel Team intervention has been noticed. Marking as resolved.

No further alerts have been triggered and we have not seen any support requests filed. This incident has been resolved.

Unfortunately, we're back to the "Identified" stage. Our Panel Team is actively looking into this issue.

Note: Game Servers are not affected and remain online if previously turned on prior to this incident.

A fix has been applied by the Panel Team and our automated alerts are now showing the Game Panel as online again. We will be monitoring the result and support queues to ensure there are no lingering issues.

Our automated alerts have been triggered for the WinterNode Game Panel reporting "Error 1016 Origin DNS error". Our Panel Team acknowledges this issue and are currently working on a fix.

Our Panel Team has implemented a temporary workaround and we are currently monitoring the result. We have noticed a reduction in support requests regarding this issue.

The panel is now back operational.

No further issues have been reported and our automated alerts are no longer triggering for downtime.

No further issues have been reported. Marking as Resolved.

Our upstream service provider has identified and resolved the root cause of this disruption. The issue was traced to a failed line card in their core Coventry router, which triggered a reboot of the remaining cards in the system. The faulty hardware has since been removed, and the remaining components have been inspected and restarted.

Our uptime monitoring confirms that services are coming back online, and we are seeing restoration across the network. We will continue to closely monitor the situation to ensure stability.

If you are still experiencing any connectivity issues, please don't hesitate to reach out to our Support Team.

We are actively tracking an open incident with our service provider regarding this outage.

Customers on MC.LON5 will likely be facing issues connecting and managing their game servers.

No further issues have been reported. Marking as Resolved.

The migration has been completed. The node is now back online. We will continue to monitor for any additional alerts.

No further issues have been reported. Marking as Resolved.

If you are experiencing any kind of issue, please reach out to us.

We have received an update from the upstream that the issue should be resolved.
All all carrier connections have come online without issues.

The Game Control Panel is now reporting fully as online and panel functionality is restored. Connections to the servers should have stabilized back to normal now.

Game servers power shouldn't affected by the incident, if you are still experiencing any issues, please reach to us. Chicago-based servers should be back to normal state.
We are continuing to monitor the issue for any further issues for the next few hours.

We are currently observing a major outage impacting Zayo, which is affecting both halves of our fiber ring into 603 Discovery as of approximately 15 minutes ago. This issue appears to stem from a dark fiber and ISP-level disruption impacting the backbone infrastructure used by multiple carriers traversing that path.

While our network is designed with redundancy in place, there are multiple ISP outages upstream of us currently being worked through. These external issues may impact connectivity for some users depending on their path to our services.

The Game Control Panel is now reporting as online and panel functionality appears to be restored. We have moved this incident into a monitoring state as we continue to assess stability.

Game servers remained online throughout the incident. While there was a brief disruption to our Chicago-based servers that may have caused temporary player disconnections, we have not received any further reports of connectivity issues.

We will continue to monitor and provide updates as new information becomes available.

Our Panel Team is actively working to resolve the current issue affecting the Game Control Panel. Game servers remain online and operational; however, panel functionality such as power actions, file access, and console viewing is currently unavailable.

Additionally, our Chicago-based game servers experienced a brief disruption, which may have caused temporary player disconnections. At this time, we have not received any further reports of ongoing connection issues.