BreezeTech - Notice history

All systems operational

100% - uptime

Notice history

Mar 2025

Dallas Emergency Network Maintenance
  • Completed
    March 18, 2025 at 8:39 PM
    Completed
    March 18, 2025 at 8:39 PM

    Maintenance has been completed! If you have any outstanding issues, please as always contact support.

  • In progress
    March 18, 2025 at 7:48 PM
    In progress
    March 18, 2025 at 7:48 PM

    We are in the process of performing an emergency network maintenance in our Dallas, TX location. While we do not expect any drops in traffic, we wanted to alert our customer base that changes were taking place in case any issues arise.

    Should you have any questions, please contact support.

Charlotte NC Segra circuit upgrade
  • Completed
    March 26, 2025 at 8:34 PM
    Completed
    March 26, 2025 at 8:34 PM

    The maintenance has been completed with success. We are closing this task out now. If you have any issues, please contact support.

  • Update
    March 26, 2025 at 12:12 PM
    In progress
    March 26, 2025 at 12:12 PM

    Segra was unable to finish the upgrade last week and is wrapping this up today, March 26th - Segra and our team will be performing a "hot cut" starting at 10AM which involves swapping optics and moving connections from multiple connection points. We will advise once this maintenance is completed, but please be advised you may see intermittent drops this morning and until we are complete as a result. We will do our best to keep this down to a minimum. We will advise once it is completed.

  • Update
    March 17, 2025 at 5:32 PM
    In progress
    March 17, 2025 at 5:32 PM

    Segra NOC and field ops are still working on this upgrade and our team is continuing to remain on the call to address any issues that may arise. We will advise once the maintenance is completed

  • In progress
    March 17, 2025 at 4:00 PM
    In progress
    March 17, 2025 at 4:00 PM
    Maintenance is now in progress
  • Planned
    March 17, 2025 at 4:00 PM
    Planned
    March 17, 2025 at 4:00 PM

    At 12PM EST today, we will be upgrading our Segra circuit with Segra NOC and Field Ops teams. There may be a brief drop in connectivity, depending on your routes, but should restore automatically. We will advise once this has been completed. If you have any questions please reach out to support.

Dallas Issue
  • Resolved
    Resolved
    This incident has been resolved.
  • Investigating
    Investigating
    We are currently investigating this incident.
Charlotte, NC Network Issues
  • Resolved
    Resolved

    We have completed the maintenance and have tested a failure on our hardware to ensure proper VRRP and failover. We are happy to report that this is a success and as such, we are ending the maintenance. In addition, we have not see any further hardware/software issues on the new router after the failed one was replaced. At this time we are confident that we have completed this and closing the maintenance. As always, our NOC will continue monitoring our network health and advise if any changes in the network. Thank you for your patience!

  • Identified
    Identified

    We will be working on the final configurations of the replacement router starting at 3:00 PM EST. You may experience intermittent drops and issues during this time period.

    After the maintenance, this should conclude the changes for Charlotte. We will provide further updates as needed.

  • Monitoring
    Monitoring

    Malfunctioning Router has been replaced. Our networking team will be running through configurations and ensuring everything is operating as planned this evening. While we do not expect any long term outages there may be short periods of lost connections while we complete the config updates.

  • Identified
    Identified

    We have identified an issue with one of the core routers kernel panicking and rebooting.  Most customers are unaffected since we have redundantly on the new stack. However there are a few outliers that are having some issues.  We have already ordered replacement hardware to be replaced Thursday or Friday and will be working with our networking team to minimize any outages till then by migrating those affected to a different core switch.  

    As with any large scale upgrades there are bound to be a few hiccups along the way and we appreciate the patience.  Once we complete ironing out these last few issues we expect to see consistent uptimes and are already seeing much lower latency’s across the network.

    We will send updates as needed, and as we go on with the replacement.

Charlotte, NC Emergency Network Maintenance
  • Resolved
    Resolved

    The emergency maintenance has been completed and we are hands off for the night. If anyone is experiencing any issues, please contact support by creating a support ticket.

  • Investigating
    Investigating

    We are deploying an emergency network maintenance to address the continued issues on the Charlotte network that occurred a while ago. We are committed to continuing to harden and implement more fixes with the new network deployment. While each issue is different and presents its own challenges, we see the importance of providing a good quality service for all of our customers. We deeply apologize for the issues that this has caused all of our customers. If you have any questions regarding this, please reach out to support via support ticket.

Feb 2025

Charlotte Configuration
  • Completed
    March 01, 2025 at 1:02 AM
    Completed
    March 01, 2025 at 1:02 AM

    Work is completed on this.

  • In progress
    February 28, 2025 at 10:30 PM
    In progress
    February 28, 2025 at 10:30 PM
    Maintenance is now in progress
  • Planned
    February 28, 2025 at 10:30 PM
    Planned
    February 28, 2025 at 10:30 PM

    We are working on some routing changes and config updates to better route traffic to our new network stacks in Charlotte. We are not expecting any downtime but there may be some small outages are traffic is redirected to the new stack.

Charlotte Maintenance
  • Completed
    February 27, 2025 at 3:27 PM
    Completed
    February 27, 2025 at 3:27 PM

    We are now issuing the all clear for Charlotte maintenance after a quite night and solid connectivity. We do have a few configuration items still pending to complete but are not expecting any downtime to be associated with those changes.

  • Update
    February 26, 2025 at 5:32 PM
    In progress
    February 26, 2025 at 5:32 PM

    Good afternoon,

    We are wrapping up this maintenance today. We will be doing service impacting events in the next few minutes and you may experience short drops. We will advise once we have completed this activity.

  • Update
    February 26, 2025 at 5:43 AM
    In progress
    February 26, 2025 at 5:43 AM

    We are hands off for the night. Any users still experiencing issues, will need to create a support ticket. We will finish wrapping up loose ends tomorrow morning. We appreciate all of your patience. Most users have experienced less than 20 mins of downtime, while the average was less than 5 minutes! We will send out another update tomorrow morning as we are getting started.

  • Update
    February 26, 2025 at 4:02 AM
    In progress
    February 26, 2025 at 4:02 AM

    We are working on our carrier blends in Charlotte. Certain ranges may see intermittent drops at this time. We will notify once we are hands off for the night.

  • Update
    February 25, 2025 at 11:02 PM
    In progress
    February 25, 2025 at 11:02 PM

    D05 portion of the service impact has been completed. If you are in D05 and still experiencing issues, please contact support. We will update as needed.

  • Update
    February 25, 2025 at 10:54 PM
    In progress
    February 25, 2025 at 10:54 PM

    We are about to start the service impacting portion of D05. We will advise once D05 has been completed.

  • Update
    February 25, 2025 at 10:12 PM
    In progress
    February 25, 2025 at 10:12 PM

    We have completed the D01 service impacting portion of the maintenance. We are working on the upstream portion of the maintenance now (SEGRA blend first). You may see quick blips as routes reconverge across the network. Please contact support if you are offline and we will investigate.

  • Update
    February 25, 2025 at 8:46 PM
    In progress
    February 25, 2025 at 8:46 PM

    We are about to start the service impacting work for D01. We will advise once we are completed. We will try to keep downtime as minimal as possible during this process.

  • Update
    February 25, 2025 at 6:56 PM
    In progress
    February 25, 2025 at 6:56 PM

    All impacting work is completed in D02. If you’re having issues, please contact support. We will notify everyone once we are starting the next rack impact work.

  • Update
    February 25, 2025 at 4:37 PM
    In progress
    February 25, 2025 at 4:37 PM

    Good morning - We are beginning the service impacting portion of the maintenance shortly. We will try to keep the impacts as minimal as possible. We will advise once we are cleared. We are starting impact maintenance for all customers with services that begin in "D02..."

  • Update
    February 24, 2025 at 9:19 PM
    In progress
    February 24, 2025 at 9:19 PM

    We are done with this round of impacting changes. If you're down as of right now, please let us know.

  • Update
    February 24, 2025 at 7:17 PM
    In progress
    February 24, 2025 at 7:17 PM

    We will beginning work that may impact connectivity now.

  • In progress
    February 24, 2025 at 3:00 PM
    In progress
    February 24, 2025 at 3:00 PM
    Maintenance is now in progress
  • Planned
    February 24, 2025 at 3:00 PM
    Planned
    February 24, 2025 at 3:00 PM

    We will be performing scheduled maintenance at our Charlotte, NC data center to upgrade and replace our network stack and reconfigure redundant systems for improved stability and performance.

    Maintenance Window:
    📅 Start: February 24, 2025, at 9:00 AM (EST)
    📅 End: February 26, 2025, at approximately 6:00 PM (EST)
    **While we try to perform all major maintenance during off hours this one requires normal business hours due to communication with our network team and upstreams.

    During this period, you may experience intermittent connectivity disruptions as we implement these improvements. Our team will work diligently to minimize downtime and ensure a smooth transition.

    We appreciate your patience and understanding. If you have any questions or concerns, please reach out to our support team.

    Thank you for your continued trust in us.

Dropping Routes
  • Resolved
    Resolved

    This incident has been resolved. It appears to have been a. Dropped BGP by one upstream. Our team will get with them to determine the root cause and prevent further occurrences.

  • Investigating
    Investigating

    We are are of an intermittent issue affecting routing in our LAX location. We are currently investigating this incident.

Possible issues with NC and TX
  • Resolved
    Resolved

    The network has been stable since this morning and we are closing this incident. Should you experience any issues, please notify support by creating a support ticket.

  • Update
    Update

    We have received the all clear from Cosmic our DDoS provider, and we are still monitoring the situation from our NOC. We have provided a timeline of the events to detail below and wanted to be transparent in our response.

    11:40AM EST  - We were notified of network issues across multiple locations

    11:41AM EST - An emergency bridge call was started with BreezeTech staff to triage the issue

    11:42AM EST - Charlotte network determined partial outage

    11:43AM EST - Chicago network determined unimpacted

    11:44AM EST - Dallas network determined partial outage

    11:46AM EST - Outage determined related to Cosmic. Cosmic was contacted to check on issues

    11:48AM EST - Cosmic confirmed outage

    11:49AM EST - Charlotte Cosmic routes shifted to primary/secondary ISP carriers. Confirmed solution

    11:52AM EST - Dallas Cosmic routes shifted to primary/secondary ISP carriers. Confirmed solution

    11:53AM EST - Los Angeles network determined partial outage

    11:54AM EST - Los Angeles Cosmic routes shifted to primary/secondary ISP carriers. Confirmed solution

    We are continuing to monitor the situation but all services should be online at this time. If you are experiencing any issues, please reach out to support directly.

  • Monitoring
    Monitoring

    The temporary fix has been implemented, and we are actively monitoring the situation.

  • Identified
    Identified

    We've identified the issue as originating from Cosmic Guard ranges. A temporary fix is currently being implemented.

  • Investigating
    Investigating
    We are currently investigating this incident.
Charlotte, NC Reduced Network Capacity
  • Resolved
    Resolved

    As of 2:30AM EST, our Cogent port has recovered and we have shifted traffic back to normal routes. If you have any outstanding issues, please contact our support department by emailing [email protected]. We will continue to monitor for any network changes, but as of this time we believe the issue has been resolved. We apologize for any inconvenience and we appreciate your patience and understanding.

  • Monitoring
    Monitoring

    At 2:16AM EST, we were alerted to a drop of our Cogent ISP Connection with our Charlotte, NC facility.

    At 2:23AM EST, we gained access into the router and began investigating.

    At 2:24AM EST, we started failing over to our backup connection.

    We are monitoring the situation and working diligently to bring back the Cogent circuit. We apologize for any inconvenience and we will advise once we are fully back up and running at normal capacity.

Jan 2025

Los Angeles, CA Fires
  • Resolved
    Resolved

    This incident has been resolved and all services are back restored. We thank you all for your patience and are keeping everyone affected in our prayers. Hardware can be replaced but lives cannot! Stay safe and pray for some rain.

  • Update
    Update

    Our team has been trying to reach the site all day and have been turned back due to road closures and evac zones. We are still making attempts to locate an open path to the facility.

  • Update
    Update

    We have received word that the HVAC has been restored. Our techs have been dispatched and will be on site shortly to inspect and begin powering units back online.

  • Monitoring
    Monitoring

    At this time, we have entered into a full facility shutdown state. We have powered off our critical infrastructure in this facility to prevent hardware damage.

    We are continuing to monitor the fire situation and you can also monitor Cogent's status page located here: https://ecogent.cogentco.com/network-status
    "Customers impacted due to services provided from the Cogent Data Center located at 2947 Bradley Street in Pasadena, Ca area may be experiencing service interruption. Government News has issued a state of emergency and thousands of people have been evacuated and also police are NOT allowing access for our Field technicians to attempt to resolve at this time. Cogent NOC is monitoring at this time. There is no ettr at this time. The master case is HD304552047."

  • Identified
    Identified

    We have started losing communications with LAX facility around 2 hours ago. We are working on reaching our datacenter team, however with mandatory evacuations, we have been unable to reach the facility.

    We are doing everything we can in the meantime to isolate and mitigate risks. We will advise shortly once we have an update.

  • Monitoring
    Monitoring

    Our hearts go out to all of those who are impacted with the LAX area fires. We are currently monitoring the Eaton Fire, which is in Pasadena, the small town where our LAX facility is. We are working with our local datacenter team and are monitoring the situation closely. While our facility is hardened against these types of events, other external factors can play a role in uptime like power and fiber communications staying up.

    Should we experience any issues, we will notify you via this notification. Otherwise, we will keep this open until we have received word that the fire is contained.

Jan 2025 to Mar 2025

Next