HostVenom & WinterNode - Network Connectivity Issues with MC.LON5 – Incident details

Network Connectivity Issues with MC.LON5

Resolved
Major outage
Started 8 days agoLasted about 22 hours

Affected

WinterNode

Degraded performance from 3:12 AM to 4:04 AM, Operational from 4:04 AM to 7:37 AM, Major outage from 7:37 AM to 1:00 AM

London Game Servers

Degraded performance from 3:12 AM to 4:04 AM, Operational from 4:04 AM to 7:37 AM, Major outage from 7:37 AM to 1:00 AM

mc.lon5.winternode.com

Degraded performance from 3:12 AM to 4:04 AM, Operational from 4:04 AM to 7:37 AM, Major outage from 7:37 AM to 1:00 AM

Updates
  • Resolved
    Resolved

    This incident has been resolved. We will create a new incident if necessary, however, changes have been made to better protect our Node IP.

    Please reach out over Discord or Client Area Tickets if you are experiencing any issues.

  • Update
    Update

    We observed some servers were still having connectivity issues on mc.lon5 and have applied an additional fix.

    If you need any assistance with your server please reach out to our support team.

    We are continuing to monitor, but as this update everything should be online and functional.

  • Update
    Update

    We saw another attack that impacted this box, and we have made an adjustment in discussion with our upstream provider. Panel server management should be restored again.
    We are continuing to monitor this node.

    If you would like to move to another node (at our other provider) to avoid this issue, we're happy to transfer you. Please open a ticket.

  • Monitoring
    Monitoring

    One of our IP addresses was temporarily unavailable due to a large external DDoS attack. Service has since been restored, and we are actively monitoring the situation to determine whether further action is needed.

    We are in contact with the provider and monitoring the situation. They have shared some options if attacks continue.

  • Identified
    Identified

    Our service provider has booted our machine into rescue mode and the technician that looked into it was unable to resolve the issue. They have returned our machine back into the normal operating system without networking still.

    Our ticket has now been assigned to the network team with our service provider, although we might be waiting longer than normal for a reply as it is a specific team.

  • Investigating
    Investigating

    We have rebooted the node at the suggestion of the service provider. This has not resolved the network issue. We have provided more information to the service provider to see if this is a configuration issue on their end.

  • Identified
    Identified

    Our automated alerts have been triggered again and we are receiving support tickets regarding services on MC.LON5 being inaccessible.

    We have reviewed the server console and gameservers are still running, however, logging onto the server may be affected. We have re-opened our ticket with our service provider for more details and will post updates as they come in.

  • Resolved
    Resolved

    We've received an update from our upstream provider. Similar to the November Chicago Data Center incident, they reported high-volume "carpet bomb" DDoS activity that temporarily placed a large number of IPs into mitigation.

    We have not received any customer reports of disconnections. The provider advised that only ICMP (ping) may have been filtered during mitigation, which can trigger false positives on our uptime monitors. Regular TCP and UDP traffic was not impacted, and our IP addresses have now been removed from mitigation.

  • Investigating
    Investigating

    We have received a few automated alerts regarding network connectivity issues to MC.LON5. We have confirmed with additional MTR tests that some clients might be unable to join servers hosted on MC.LON5.

    A ticket has been filed and we are awaiting further information from our service provider.