HostVenom & WinterNode - Historial de avisos

We have swapped the PSU on this node as we suspect this was the cause of the outage. We'll continue to monitor this node, but as of now, it is back online.

No further tickets have been raised regarding this issue and no Panel Team intervention has been noticed. Marking as resolved.

No further alerts have been triggered and we have not seen any support requests filed. This incident has been resolved.

Unfortunately, we're back to the "Identified" stage. Our Panel Team is actively looking into this issue.

Note: Game Servers are not affected and remain online if previously turned on prior to this incident.

A fix has been applied by the Panel Team and our automated alerts are now showing the Game Panel as online again. We will be monitoring the result and support queues to ensure there are no lingering issues.

Our automated alerts have been triggered for the WinterNode Game Panel reporting "Error 1016 Origin DNS error". Our Panel Team acknowledges this issue and are currently working on a fix.

Our Panel Team has implemented a temporary workaround and we are currently monitoring the result. We have noticed a reduction in support requests regarding this issue.

No further issues have been reported and our automated alerts are no longer triggering for downtime.

No se han reportado más problemas. Marcado como resuelto.

Our upstream service provider has identified and resolved the root cause of this disruption. The issue was traced to a failed line card in their core Coventry router, which triggered a reboot of the remaining cards in the system. The faulty hardware has since been removed, and the remaining components have been inspected and restarted.

Our uptime monitoring confirms that services are coming back online, and we are seeing restoration across the network. We will continue to closely monitor the situation to ensure stability.

If you are still experiencing any connectivity issues, please don't hesitate to reach out to our Support Team.

We are actively tracking an open incident with our service provider regarding this outage.

Customers on MC.LON5 will likely be facing issues connecting and managing their game servers.

No se han reportado más problemas. Marcado como resuelto.

La migración se ha completado. El nodo ya está en línea. Seguiremos monitoreando cualquier alerta adicional.

No se han reportado más problemas. Marcado como resuelto.

Si tiene algún tipo de problema, comuníquese con nosotros.

Hemos recibido una actualización desde arriba indicando que el problema debería estar resuelto.
Todas las conexiones de todos los operadores se han puesto en línea sin problemas.

El Panel de Control del Juego ya informa de la conexión y la funcionalidad del panel se ha restablecido. Las conexiones a los servidores deberían haberse normalizado.

El suministro de energía de los servidores del juego no debería verse afectado por el incidente. Si el problema persiste, contáctanos. Los servidores de Chicago deberían haber vuelto a la normalidad.
Continuaremos monitoreando el problema para detectar cualquier otro inconveniente durante las próximas horas.

We are currently observing a major outage impacting Zayo, which is affecting both halves of our fiber ring into 603 Discovery as of approximately 15 minutes ago. This issue appears to stem from a dark fiber and ISP-level disruption impacting the backbone infrastructure used by multiple carriers traversing that path.

While our network is designed with redundancy in place, there are multiple ISP outages upstream of us currently being worked through. These external issues may impact connectivity for some users depending on their path to our services.

The Game Control Panel is now reporting as online and panel functionality appears to be restored. We have moved this incident into a monitoring state as we continue to assess stability.

Game servers remained online throughout the incident. While there was a brief disruption to our Chicago-based servers that may have caused temporary player disconnections, we have not received any further reports of connectivity issues.

We will continue to monitor and provide updates as new information becomes available.

Our Panel Team is actively working to resolve the current issue affecting the Game Control Panel. Game servers remain online and operational; however, panel functionality such as power actions, file access, and console viewing is currently unavailable.

Additionally, our Chicago-based game servers experienced a brief disruption, which may have caused temporary player disconnections. At this time, we have not received any further reports of ongoing connection issues.

We have successfully completed the migration of customer data and databases from the GRA3 node to our new GRA4 node (although same provider to avoid IP changes).

However, we want to be clear: this does not mean the GRA3 hardware issues have been resolved. In fact, during the migration process, GRA3 experienced three additional outages. For one of the "longer" incidents, our provider's interventions were limited to issuing a "hard reboot" - not the comprehensive hardware replacement we had requested. We share in your frustration and are deeply disappointed with our provider's handling of this situation.

Key Details:

• IP Address: We were able to migrate the IP block alongside your servers. As a result, your IP address and ports should remain unchanged.

• Databases: If you are using our sql databases, please double-check your configurations as the machine IP has changed during the migration for GRA4. You will not be able to login/use the former database server on GRA3 as it has been turned off.

Ongoing Monitoring:

We will continue to closely monitor the environment for any further issues. Should any new concerns arise, we are prepared to escalate internally and externally as needed to ensure stability and performance.

Compensation:

We understand how disruptive this has been and sincerely apologize for the inconvenience. Impacted customers are encouraged to reach out to us to request an account credit for the downtime and related issues.

Thank you for your patience and understanding as we worked to restore stability. We remain committed to providing better reliability moving forward.

Actualización breve: Estamos trabajando en la migración de clientes a otro nodo. En breve, desconectaremos los servidores que estén en línea.

Se trata de un problema de hardware y nuestro proveedor no cumplió con sus declaraciones anteriores; sin embargo, estamos tomando todas las medidas a nuestro alcance para que nuestros clientes vuelvan a tener una máquina estable lo antes posible.

Los clientes afectados deben comunicarse con nosotros en cualquier momento para solicitar un crédito en la cuenta por estos problemas.

We briefly experienced another unexpected outage at 7:51pm Pacific Time and our monitoring is now reporting an online status.

An automatic intervention ticket has been opened and we have not received any updates from our previous tickets, we will keep everyone posted.

El nodo está nuevamente en línea.

Estamos esperando información adicional de OVH. Parece que realizaron pruebas en lugar de reemplazar el hardware.

El reemplazo del servidor debería comenzar en breve.

Nuestros monitores de disponibilidad informan que nuestros servicios de Chicago están nuevamente en línea a las 12:01 p. m., hora del Pacífico. Publicaremos una solicitud de oferta tan pronto como la tengamos disponible.