BreezeTech - Charlotte, NC Network Issues – Incident details

All systems operational

Charlotte, NC Network Issues

Resolved
Degraded performance
Started 28 days agoLasted 6 days

Affected

Data Centers

Degraded performance from 3:25 AM to 9:01 PM, Under maintenance from 9:01 PM to 8:33 PM

CLT01 - Charlotte, NC

Degraded performance from 3:25 AM to 9:01 PM, Under maintenance from 9:01 PM to 8:33 PM

Updates
  • Resolved
    Resolved

    We have completed the maintenance and have tested a failure on our hardware to ensure proper VRRP and failover. We are happy to report that this is a success and as such, we are ending the maintenance. In addition, we have not see any further hardware/software issues on the new router after the failed one was replaced. At this time we are confident that we have completed this and closing the maintenance. As always, our NOC will continue monitoring our network health and advise if any changes in the network. Thank you for your patience!

  • Identified
    Identified

    We will be working on the final configurations of the replacement router starting at 3:00 PM EST. You may experience intermittent drops and issues during this time period.

    After the maintenance, this should conclude the changes for Charlotte. We will provide further updates as needed.

  • Monitoring
    Monitoring

    Malfunctioning Router has been replaced. Our networking team will be running through configurations and ensuring everything is operating as planned this evening. While we do not expect any long term outages there may be short periods of lost connections while we complete the config updates.

  • Identified
    Identified

    We have identified an issue with one of the core routers kernel panicking and rebooting.  Most customers are unaffected since we have redundantly on the new stack. However there are a few outliers that are having some issues.  We have already ordered replacement hardware to be replaced Thursday or Friday and will be working with our networking team to minimize any outages till then by migrating those affected to a different core switch.  

    As with any large scale upgrades there are bound to be a few hiccups along the way and we appreciate the patience.  Once we complete ironing out these last few issues we expect to see consistent uptimes and are already seeing much lower latency’s across the network.

    We will send updates as needed, and as we go on with the replacement.