HostVenom & WinterNode - Notice history

We have swapped the PSU on this node as we suspect this was the cause of the outage. We'll continue to monitor this node, but as of now, it is back online.

No further tickets have been raised regarding this issue and no Panel Team intervention has been noticed. Marking as resolved.

No further alerts have been triggered and we have not seen any support requests filed. This incident has been resolved.

Unfortunately, we're back to the "Identified" stage. Our Panel Team is actively looking into this issue.

Note: Game Servers are not affected and remain online if previously turned on prior to this incident.

A fix has been applied by the Panel Team and our automated alerts are now showing the Game Panel as online again. We will be monitoring the result and support queues to ensure there are no lingering issues.

Our automated alerts have been triggered for the WinterNode Game Panel reporting "Error 1016 Origin DNS error". Our Panel Team acknowledges this issue and are currently working on a fix.

Our Panel Team has implemented a temporary workaround and we are currently monitoring the result. We have noticed a reduction in support requests regarding this issue.

No further issues have been reported and our automated alerts are no longer triggering for downtime.

No further issues have been reported. Marking as Resolved.

Our upstream service provider has identified and resolved the root cause of this disruption. The issue was traced to a failed line card in their core Coventry router, which triggered a reboot of the remaining cards in the system. The faulty hardware has since been removed, and the remaining components have been inspected and restarted.

Our uptime monitoring confirms that services are coming back online, and we are seeing restoration across the network. We will continue to closely monitor the situation to ensure stability.

If you are still experiencing any connectivity issues, please don't hesitate to reach out to our Support Team.

We are actively tracking an open incident with our service provider regarding this outage.

Customers on MC.LON5 will likely be facing issues connecting and managing their game servers.

No further issues have been reported. Marking as Resolved.

The migration has been completed. The node is now back online. We will continue to monitor for any additional alerts.

No further issues have been reported. Marking as Resolved.

If you are experiencing any kind of issue, please reach out to us.

We have received an update from the upstream that the issue should be resolved.
All all carrier connections have come online without issues.

The Game Control Panel is now reporting fully as online and panel functionality is restored. Connections to the servers should have stabilized back to normal now.

Game servers power shouldn't affected by the incident, if you are still experiencing any issues, please reach to us. Chicago-based servers should be back to normal state.
We are continuing to monitor the issue for any further issues for the next few hours.

We are currently observing a major outage impacting Zayo, which is affecting both halves of our fiber ring into 603 Discovery as of approximately 15 minutes ago. This issue appears to stem from a dark fiber and ISP-level disruption impacting the backbone infrastructure used by multiple carriers traversing that path.

While our network is designed with redundancy in place, there are multiple ISP outages upstream of us currently being worked through. These external issues may impact connectivity for some users depending on their path to our services.

The Game Control Panel is now reporting as online and panel functionality appears to be restored. We have moved this incident into a monitoring state as we continue to assess stability.

Game servers remained online throughout the incident. While there was a brief disruption to our Chicago-based servers that may have caused temporary player disconnections, we have not received any further reports of connectivity issues.

We will continue to monitor and provide updates as new information becomes available.

Our Panel Team is actively working to resolve the current issue affecting the Game Control Panel. Game servers remain online and operational; however, panel functionality such as power actions, file access, and console viewing is currently unavailable.

Additionally, our Chicago-based game servers experienced a brief disruption, which may have caused temporary player disconnections. At this time, we have not received any further reports of ongoing connection issues.

We have successfully completed the migration of customer data and databases from the GRA3 node to our new GRA4 node (although same provider to avoid IP changes).

However, we want to be clear: this does not mean the GRA3 hardware issues have been resolved. In fact, during the migration process, GRA3 experienced three additional outages. For one of the "longer" incidents, our provider's interventions were limited to issuing a "hard reboot" - not the comprehensive hardware replacement we had requested. We share in your frustration and are deeply disappointed with our provider's handling of this situation.

Key Details:

• IP Address: We were able to migrate the IP block alongside your servers. As a result, your IP address and ports should remain unchanged.

• Databases: If you are using our sql databases, please double-check your configurations as the machine IP has changed during the migration for GRA4. You will not be able to login/use the former database server on GRA3 as it has been turned off.

Ongoing Monitoring:

We will continue to closely monitor the environment for any further issues. Should any new concerns arise, we are prepared to escalate internally and externally as needed to ensure stability and performance.

Compensation:

We understand how disruptive this has been and sincerely apologize for the inconvenience. Impacted customers are encouraged to reach out to us to request an account credit for the downtime and related issues.

Thank you for your patience and understanding as we worked to restore stability. We remain committed to providing better reliability moving forward.

Brief update: We're working on migrating customers to another node right now. We'll be powering down any servers currently online shortly.

This is a hardware issue, and our provider dropped the ball on their previous statements to us, however, we're taking all action in our power to get our customers back on a stable machine ASAP.

Impacted customers should reach out to us at any point to request an account credit for these issues.

We briefly experienced another unexpected outage at 7:51pm Pacific Time and our monitoring is now reporting an online status.

An automatic intervention ticket has been opened and we have not received any updates from our previous tickets, we will keep everyone posted.

The node is back online.

We are awaiting additional information from OVH. They appear to have performed testing instead of replacing hardware.

The server replacement should begin shortly.

Our uptime monitors are now reporting our Chicago services as online again at 12:01 PM Pacific Time. We will be posting a RFO as soon as we have one available.