HostVenom & WinterNode - Notice history

The router chassis has been rebooted and the emergency maintenance by our service provider has been marked as completed. Our automated monitoring has not been tripped since the reboot.

All affected services are fully operational and stable.

This incident is now considered closed. If you experience any further issues, please reach out to our support team at https://winternode.com/support .

The issue has been identified as an issue with the routing engine at the Los Angeles data center. Our service provider has begun an emergency maintenance window between 16:30 to 21:30 Eastern to resolve this issue.

Affected services may experience intermittent connectivity during this time.

The router in LA1 will be power cycled to fully resolve firmware related issues identified in a prior network event. While the maintenance window is set for 5 hours, we only expect roughly 15 - 20 minutes of network unavailability during this time. Additional time is taken into account in case of unexpected issues.

Further updates will follow.

We implemented a fix and are currently monitoring the result. We'll continue to monitor before closing this incident out.

This incident has been resolved.Customers can start and manage their services again from gcp.winternode.com

Local DDoS filtering was preventing our panel and daemon from communicating after the DDoS incident at this provider. DDoS update to be provided and tracked in separate issue.

This incident has been resolved. We will create a new incident if necessary, however, changes have been made to better protect our Node IP.

Please reach out over Discord or Client Area Tickets if you are experiencing any issues.

We observed some servers were still having connectivity issues on mc.lon5 and have applied an additional fix.

If you need any assistance with your server please reach out to our support team.

We are continuing to monitor, but as this update everything should be online and functional.

We saw another attack that impacted this box, and we have made an adjustment in discussion with our upstream provider. Panel server management should be restored again.
We are continuing to monitor this node.

If you would like to move to another node (at our other provider) to avoid this issue, we're happy to transfer you. Please open a ticket.

One of our IP addresses was temporarily unavailable due to a large external DDoS attack. Service has since been restored, and we are actively monitoring the situation to determine whether further action is needed.

We are in contact with the provider and monitoring the situation. They have shared some options if attacks continue.

Our service provider has booted our machine into rescue mode and the technician that looked into it was unable to resolve the issue. They have returned our machine back into the normal operating system without networking still.

Our ticket has now been assigned to the network team with our service provider, although we might be waiting longer than normal for a reply as it is a specific team.

We have rebooted the node at the suggestion of the service provider. This has not resolved the network issue. We have provided more information to the service provider to see if this is a configuration issue on their end.

Our automated alerts have been triggered again and we are receiving support tickets regarding services on MC.LON5 being inaccessible.

We have reviewed the server console and gameservers are still running, however, logging onto the server may be affected. We have re-opened our ticket with our service provider for more details and will post updates as they come in.

We've received an update from our upstream provider. Similar to the November Chicago Data Center incident, they reported high-volume "carpet bomb" DDoS activity that temporarily placed a large number of IPs into mitigation.

We have not received any customer reports of disconnections. The provider advised that only ICMP (ping) may have been filtered during mitigation, which can trigger false positives on our uptime monitors. Regular TCP and UDP traffic was not impacted, and our IP addresses have now been removed from mitigation.

This incident has been resolved and all customers we have checked in with are reporting they are able to connect and manage their service.

We have restarted the network interface and confirmed that the IPs serving our game server customers are properly bound to the node. After starting multiple game servers, we can confirm that they are now coming online successfully.

We ask all customers to start their servers and report any issues. We are moving this incident to the monitoring status and will be marked as resolved in one hour if no further problems are observed.

A member of our management team has reviewed the server console and confirmed that the machine experienced a kernel panic. The server has been hard reset and has successfully rebooted to the operating system login screen.

After attempting to start a game server, customers may encounter an error similar to:

(HTTP code 500) server error - failed to set up container networking: driver failed programming external connectivity on endpoint ... (...): failed to bind host port for .../tcp: cannot assign requested address

We are actively investigating this issue.

No further alerts have been triggered and we have not received any new support tickets regarding this issue. This incident has been marked as resolved.

Our quick tests have come back clear. We're returning this machine back to operational status and will monitor for an hour before closing this incident.

If you are experiencing any issues, please reach out to our Support Team.

We have received some reports from customers that they are unable to access their game servers on MC.CHI34.

We are continuing to investigate and are running some tests before bringing this machine back online. We expect to provide another update in 20 minutes.