HostVenom & WinterNode - Chicago Data Center Network Blips – Incident details

Chicago Data Center Network Blips

Identified
Degraded performance
Started 3 days ago

Affected

HostVenom

Degraded performance from 11:47 PM to 9:09 AM, Operational from 11:47 PM to 7:58 PM, Degraded performance from 9:09 AM to 4:19 PM, Operational from 4:19 PM to 8:30 PM, Degraded performance from 8:30 PM to 12:00 AM

Billing Area

Operational from 11:47 PM to 7:58 PM, Degraded performance from 7:58 PM to 4:19 PM, Operational from 4:19 PM to 8:30 PM, Degraded performance from 8:30 PM to 12:00 AM

Dedicated Servers

Degraded performance from 11:47 PM to 4:19 PM, Operational from 4:19 PM to 8:30 PM, Degraded performance from 8:30 PM to 12:00 AM

Colocation

Degraded performance from 11:47 PM to 4:19 PM, Operational from 4:19 PM to 8:30 PM, Degraded performance from 8:30 PM to 12:00 AM

WinterNode

Degraded performance from 11:47 PM to 9:09 AM, Operational from 11:47 PM to 7:58 PM, Degraded performance from 9:09 AM to 4:19 PM, Operational from 4:19 PM to 8:30 PM, Degraded performance from 8:30 PM to 12:00 AM

Control Panels

Degraded performance from 11:47 PM to 4:19 PM, Operational from 4:19 PM to 8:30 PM, Degraded performance from 8:30 PM to 12:00 AM

Updates
  • Identified
    Identified

    An automated alert has been triggered by our monitoring system regrading the temporarily connectivity loss. From our alerts, it lasts ~1 minute during each network blip.

    The data center engineering team is actively working to mitigate the issue and we will continue to monitor and communicate with them to resolve the issue.

    We completely understand how frustrating this can be and appreciate everyone's patience while the data center continues to work toward full resolution. We will post another update as soon as more information becomes available.

  • Monitoring
    Monitoring

    As of 4 PM Central Time yesterday, no additional automated alerts have been triggered. We will continue to monitor the fix.

  • Update
    Update

    Our data center has identified most of the target prefixes related to the ongoing DDoS event and is shifting network announcements away from primary carriers to help reduce impact.

    While this work is still in progress, these mitigation efforts are expected to improve network stability as adjustments take effect.

    To reduce notification spam, we've consolidated all WinterNode Server notices under a new component titled Chicago Data Center for the duration of this incident. It is recommended you subscribe to this incident directly to receive the latest updates regardless of component selected.

    Thank you for your continued patience and understanding while this work is in progress.

  • Update
    Update

    We have continued to observe intermittent periods of packet loss and short network interruptions affecting our Chicago Data Center. The data center engineering team has confirmed that these disruptions are related to an ongoing large-scale DDoS event. While the attacks are not targeting HostVenom directly, they are impacting connectivity across several upstream carriers and other providers in the facility.

    The data center engineering team is actively working to mitigate the issue and has implemented multiple measures to isolate the affected network paths. The scale and distributed nature of the attacks have made mitigation challenging, but we are seeing some periods of stability between brief interruptions.

    We completely understand how frustrating this can be and appreciate everyone's patience while the data center continues to work toward full resolution. We will post another update as soon as more information becomes available.

  • Update
    Update

    We have continued to observe intermittent network blips at our Chicago Data Center. Multiple automated alerts have been triggered throughout the day, each indicating brief connectivity losses lasting approximately one minute.

    Our data center has confirmed that they have observed additional short inbound traffic spikes affecting specific upstream carriers (GTT, Lumen, and Comcast). They are continuing to investigate and have made routing adjustments to help isolate the source of the issue.

    At this time, we are continuing to monitor uptime closely and remain in contact with the data center team for further updates. We will post the next update once more information becomes available or if the situation changes.

  • Update
    Update

    The data center has confirmed they are see some sporadic issues as well on the network and are currently investigating this matter. Some of our team members are also noticing periods of slowness accessing our Client/Billing Areas and connections between our VPS/Dedicated Control Panel and our Client/Billing Area systems.

    No further alerts for loss of connectivity have been triggered since our last update.

    We will continue to update this incident as we receive more information. The next expected update likely between the morning and afternoon.

  • Identified
    Identified

    An automated alert has been triggered by our monitoring system and our team has been notified. We have filed a ticket with our data center to take a closer look.

    From our alerts, there is temporarily connectivity loss of ~3 minutes during each network blip.