Amazon Having Issues with S3

Incident Report for Searchspring Public

Postmortem

This morning Amazon Web Services began having issues with their Simple Storage Service. SearchSpring uses this service to store web assets for our content delivery network (CDN). This resulted in some customers having issues loading SearchSpring's JavaScript and CSS files, including slow loading or resources not loading at all. This also affected SearchSpring's indexing capabilities for some customers.

As of 2:12PM MST Amazon reported that this service was again fully operational for reads, meaning customers could now load SearchSpring's files without any issues. At 3:08PM MST Amazon stated that the service was fully operational, including the writing of files, which allows customers to update their SearchSpring settings and resolves any indexing issues experienced by our customers.

At this time we are still monitoring the recovery and do not expect to see any further issues. We will have a meeting to discuss this outage and how to prevent similar outages in the future. We will update you with our findings.

Posted Feb 28, 2017 - 15:53 MST

Resolved

This issue has subsided. We are actively monitoring the situation in case it changes, but all services are fully operational.

Posted Feb 28, 2017 - 15:31 MST

Monitoring

SearchSpring's CDN assets (javascript, css, images) are returning to normal delivery. Overall platform traffic is returning to normal levels. Indexing updates are paused until S3 fully recovers.

Posted Feb 28, 2017 - 14:40 MST

Update

This issue has improved, but is not resolved yet. Amazon has stated that they expect the issue to be improved further soon. We are also actively working on a workaround.

Posted Feb 28, 2017 - 14:10 MST

Investigating

Amazon is currently experiencing issues with the S3 service. A number of our process utilize this service.

Posted Feb 28, 2017 - 10:58 MST