BreezeTech - Network issues in Dallas Texas – Incident details

Network issues in Dallas Texas

Resolved
Major outage
Started 1 day agoLasted about 24 hours

Affected

Data Centers

Major outage from 3:31 PM to 3:35 PM, Operational from 3:35 PM to 1:28 AM, Major outage from 1:28 AM to 4:17 AM, Operational from 4:17 AM to 3:11 PM

DFW01 - Dallas TX

Major outage from 3:31 PM to 3:35 PM, Operational from 3:35 PM to 1:28 AM, Major outage from 1:28 AM to 4:17 AM, Operational from 4:17 AM to 3:11 PM

Virtual Private Servers

Major outage from 3:31 PM to 3:35 PM, Operational from 3:35 PM to 1:28 AM, Major outage from 1:28 AM to 4:17 AM, Operational from 4:17 AM to 3:11 PM

DFW VPS Infrastructure

Major outage from 3:31 PM to 3:35 PM, Operational from 3:35 PM to 1:28 AM, Major outage from 1:28 AM to 4:17 AM, Operational from 4:17 AM to 3:11 PM

DFW-KVM-1

Major outage from 3:31 PM to 3:35 PM, Operational from 3:35 PM to 1:28 AM, Major outage from 1:28 AM to 4:17 AM, Operational from 4:17 AM to 3:11 PM

DFW-KVM-2

Major outage from 3:31 PM to 3:35 PM, Operational from 3:35 PM to 1:28 AM, Major outage from 1:28 AM to 4:17 AM, Operational from 4:17 AM to 3:11 PM

Updates
  • Resolved
    Resolved

    All customers should be back online at this time. If any other customers are experiencing issues please notify us via support ticket or chat. At this time, we are closing this window and we will be scheduling a maintenance to address the failed hardware at a later date. Thank you for your patience.

  • Monitoring
    Monitoring

    At this time we have failed over to a different switch that we will be using temporarily until we replace the failed switch. We most customers online (waiting on remaining 1 customer that is a colo customer) to be moved by local datacenter team.

    We will continue to monitor this and will schedule a replacement in the upcoming days to go back to our original setup. We apologize greatly for the issues today. If you have any outstanding issues, please contact support.

  • Update
    Update

    We are still working on this at the highest priority. We will continue to provide updates as soon as possible.

  • Update
    Update

    The datacenter has located the faulty switch and has been instructed to reboot it. Our team is awaiting for them to do this, at which time we will enter into a recovery mode on the switch to work on bringing it back up. Updates will be provided as soon as possible.

  • Update
    Update

    While preparing the files for the firmware updates to be performed on the impacted switch, we lost connection again and while our remote console is still working to the switch, the switch has locked up and is not showing any activity. We have requested the datacenter team to place eyes on this switch to help us determine the issue. We will provide another update once we are able to do so.

  • Identified
    Identified

    We have identified the repeat offending issue in our Dallas market specifically to a core distribution switch. We are establishing an emergency maintenance window to attempt repairs and may result in hardware replacement. Please stay tuned to this notification channel for updates.

  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring

    We implemented a fix and have recovered services. We are monitoring this fix and will advise of any further issues.

  • Identified
    Identified

    We have identified an issue with our core router in Dallas TX and working to resolve the issues.