Kinetic Hosting - We're looking into an issue with some nodes in our Texas location – Incident details

We're looking into an issue with some nodes in our Texas location

Resolved
Operational
Started 22 days agoLasted about 10 hours

Affected

Game Servers: North America

Operational from 11:21 AM to 9:08 PM

Game Servers: USA - Texas

Operational from 11:21 AM to 9:08 PM

Updates
  • Resolved
    Resolved

    All systems can now be reached via the panel.

    We'll keep an eye over the next few hours to make sure everything is working as normal

  • Monitoring
    Monitoring

    We're still getting panel connection issues on:

    USC-TX-P1-A2-S
    USC-TX-P2-A2-S
    USC-TX-P3-A2-S
    USC-TX-P5-A2-S
    USC-TX-P6-A2-S
    USC-TX-P7-A2-S
    USC-TX-P8-A2-S

    Our team is looking into the error

  • Update
    Update
  • Resolved
    Resolved

    The systems are now back online, and responding to ping.

    The outage was caused by an error with out data centre provider. We have a run down of the issue here.

  • Update
    Update

    It looks like the data centre still has the original data on the drives. We're currently waiting for their team to reinstall the drives into the servers to restore service.

    We'll publish a detailed write-up about this incident on our blog once all systems are fully back online.

  • Identified
    Identified

    We had a number of systems on order with our Texas data centre partner. Due to delays, we sourced these systems from another partner and requested that our Texas partners cancel any pending orders.

    Unfortunately, their billing team mistakenly terminated the wrong servers, removing systems already actively in use rather than those pending cancellation.

    As soon as we identified this mistake, we contacted our representative with the provider. They're currently assisting us alongside their billing team to restore access to the affected systems.

    Because these servers were mistakenly reclaimed, it's likely the drives have been wiped. However, we maintain backups of all our systems, so we'll begin restoring data immediately once the servers are back in our control.

  • Investigating
    Investigating

    We're looking into reports that some nodes in our Texas location can't be reached