Affects
Under maintenance from 4:47 PM to 2:04 PM
Under maintenance from 4:47 PM to 2:04 PM
- CompletedJune 08, 2024 at 2:04 PMCompletedJune 08, 2024 at 2:04 PM
This has been completed and all services are back to normal operations. We have been running on full capacity once again since about 8AM EST. We appreciate your cooperation and look forward to continuing to service all of our customers. Should you experience any issues, please reach out to support.
- UpdateJune 08, 2024 at 1:46 AMIn progressJune 08, 2024 at 1:46 AM
We have been notified by NOC staff that some hardware has been replaced and is being configured by the engineering team to be ready for tonight's maintenance window. We continue to monitor for stability and routing traffic as we can to maintain the network integrity.
Best case scenario is we will see no loss of connectivity, however things can quickly change and services can be impacted.
- In progressJune 07, 2024 at 4:47 PMIn progressJune 07, 2024 at 4:47 PM
Please read the full details of this emergency maintenance, if you are in the Charlotte, NC datacenter.
At approximately 12:50AM EST on 6/7/24, we started receiving alarms that our backup/overflow internet circuit in our Charlotte, NC facility was reporting high errors and the BGP with the provider had dropped. Initial investigation showed that it had recovered at 12:56AM EST.
We continued monitoring the circuit overnight and it was fine until 3:58AM EST. At this time, I immediately reached out to our datacenter provider, our local hands to be on standby and the NOC for the internet circuit. The datacenter advised all looked good on their end, but the NOC advised there was an unannounced emergency maintenance performed by a third party that was not relayed to them. This caused us to not be aware of the situation and be unable to announce this to our customers.
We suffered a good deal of packet loss between 3:38AM and 4:18AM. I made the decision to disable the routes that were coming through on this backup connection. Because we have a redundant path for our Charlotte circuits (one built to Charlotte, NC, the other built back to Columbia, SC), this has allowed us to remain online for the time being. However at this time, we are in a "degraded" status.
The carrier has just informed us of several plans forward, from medium risk to high risk. Because of this, we are entering into a prolonged emergency maintenance status for at least the next 24 hours. We have been informed that we can expect to be in this "degraded" status through tonight, with some risk of going into a hard down state for a period of time, depending on which plan succeeds. The carrier has advised they are in a 70%/30% state, meaning the circuit that is working is part of the 70%, whereas the one that's not is part of the 30%. There is some risks involved that could take down the other 70% or cause packet loss as they are working through solutions.
Here is a snippet provided by our carrier, explaining the trouble:
~~~~~~~~~~
At this time we have a situation where one of our core transport devices is in a state where ~30% of the services are not working. A good portion, approx. 70% of the box is up and running with no issues. Some of your services fall into both of those areas and at this time is about where we have been for the past several hours. The emergency maintenance that was scheduled for this morning to alleviate the instability did not resolve the issue and has actually caused more impact. All attempts to resolve the impacted services during the maintenance window were attempted and unfortunately created a broader impact until we were able to get back to the 70% operational state.
~~~~~~~~~~
I will continue to provide updates as they become available to us. At this time, we recommend all business customers that are able to do so, failover to other locations, until the end of this outage. If you have any questions in the meantime or we can be of any assistance, please reach out to our support department and we will do our best to assist however we can.