HostVenom & WinterNode - Notice history

No further prolonged outages have occurred since our last update. We'll be marking this as resolved for the time being.

As a friendly reminder, issues with our Panel do not impact the accessibility of your game servers as they are hosted on separate infrastructure.

We open similar incidents for all our services as well to keep you informed. You may view them on https://status.winternode.com/history/1 and https://resources.winternode.com for uptime as reported by our monitoring software.

If you are getting the error "We're having some trouble connecting to your server, please wait…", please open a ticket in our Discord Server, as we may need to restart the daemon which interfaces between the Game Panel and your server. The easiest way to check, is to visit the Web File Manager.

We've received acknowledgement of the incident from our Panel Provider at 1:44 AM PT. A temporary fixed has been put into place at 2:30AM PT and an investigation into what occurred is underway by the Panel Team. The Game Panel is now online again.

We're getting a bit tired of formal messages… We know the Panel is down again 🤷‍♂️.

Game servers hosted by us are not affected. It's been a rough couple of days for our Panel Provider. 😐 Feel free to take a look at the previous post-mortem provided to learn more about the current infrastructure setup.

A post-mortem was provided by our Panel Team - https://gamepanel.notion.site/11-10-2023-0d4aaef1626f4494b6238ef7044de522

In summary: Currently all traffic is diverted to a small set of webserver Point of Presence (POPs) (mostly nl-ams-web01), but in the upcoming days 3 new POPs will be introduced and ready to serve you our Game Panel with lower latency and resiliency.

We experienced another small blip around 11:14PM PT for a few minutes, but did not result in another major outage of our CHI services.

A daemon restart was required and fulfilled for all CHI nodes around 9:50PM PT to re-establish connectivity with our Panel.

Follow-ups have been scheduled for the morning, we'll be marking this incident as resolved for now and release a Reason For Outage if available.

If you are experiencing any lingering issues, please reach out to our Support Team on Discord.

No further issues have been reported from any of the affected nodes.

This incident was caused by a PDU failure which resulted in the loss of power for the 14 affected nodes.

We were unable to bring MC.CHI8 up with the rest of the nodes due to a drive failure. We immediately began our restoration process, rolling back game server files to our 2pm Central Time backup. We are happy to report MC.CHI8 is back in service. Please note, at this time MySQL Databases are still not backed up, therefore, data stored in the MC.CHI8 database was lost.

If you require any further assistance, our Support Team in Discord will be more than happy to help you.

All affected nodes except MC.CHI8 are online again according to our monitoring software. Clients will need to manually power on their server again.

We'll be monitoring over the hour and will continue to actively work on restoring access to MC.CHI8.

Please reach out to our Support Team on Discord if you need any assistance.

We are still investigating this issue. In the meantime, we've re-scoped this incident with the correct nodes affected according to our monitoring software.