Resolved -
We have the incident fully resolved, all systems are stabilized.
Oct 21, 21:54 MDT
Update -
Systems are back to normal. Team are doing clean ups, expected to mark the incident to "Resolved" by the end of today EST.
Oct 21, 09:24 MDT
Update -
SMC and reporting panel is fully operational. Search, Indexing, and autocomplete fully back to normal state.
We will be monitoring in the next 24 hours before we close this incident.
Oct 20, 20:40 MDT
Update -
Our Engineering team has brought Indexing Service back.
We are still seeing reporting panel fail to render, we are working with our backend team and vendor to bring it back.
We will be updating the individual service status in the next couple hours as services stabilize.
Oct 20, 18:19 MDT
Update -
We are seeing Search/Autocomplete fully recovered.
SMC is mostly recovered, experiencing intermittent issue with reporting (not being able to render reports properly).
Our Engineering team is still actively working on restoring our indexing services.
Oct 20, 17:32 MDT
Update -
Reporting in SMC is currently experiencing issues due to the same ongoing AWS outage. We are working on bringing it back.
Oct 20, 16:59 MDT
Update -
Management Console is back to Operational
Oct 20, 16:30 MDT
Update -
Our engineering team has taken Management Console out of maintenance mode. We are still working on bringing indexing service back.
Oct 20, 16:23 MDT
Update -
Search API/Autocomplete API are fully recovered.
Engineering team is still working on fixes to bring Indexing service back.
Oct 20, 16:08 MDT
Update -
Search and Autocomplete is being recovered. Team is working on a fix to bring Indexing and Management back.
Oct 20, 15:24 MDT
Update -
We continue to observe some instability on AWS side as well as our vendor who's hosted on AWS. Team is still working on restoring the services, while monitoring the restoring effort of AWS.
Oct 20, 14:30 MDT
Monitoring -
We are monitoring the results for Search, Autocomplete, while waiting for AWS fix to propagate
Oct 20, 12:10 MDT
Update -
Search API is now back to normal level - we are monitoring
Oct 20, 12:10 MDT
Update -
Part of the services are brought back to normal level. We are still waiting on another AWS-dependent vendor to come back to normal before we take Management Console back)
Oct 20, 12:05 MDT
Update -
Updating impact: we have confirmed the impact is confined to the same region (us-east) as AWS has identified. Traffic on us-west and other regions are still fine.
Oct 20, 10:37 MDT
Update -
In order to protect our system and make sure we get services back as quickly as possible once AWS outage is recovered, we have to put our Management Console into maintenance mode. Users are blocked from making any changes as of now. Thanks for your understanding during this difficult time!
Oct 20, 10:28 MDT
Update -
We have isolated the problem to be fully related to AWS outages. Indexing, Search functionality and also management console are impacted.
Oct 20, 09:41 MDT
Identified -
We have identified an incident that our Search API is currently down, post AWS outage. The team is currently investigating, and identified a potential problem.
Oct 20, 09:12 MDT
