Written by Founder, Anupama Vaid
As promised I’m following up on our note from last week after ParentSquare experienced an unplanned outage on Thursday, August 13th at 5pm pacific for about an hour — our worst outage in many years.
Again, please accept our humble, sincere apologies and assurances that the entire team is working hard to ensure ParentSquare meets everyone’s expectations.
Here’s What Happened
We have been working to improve ParentSquare’s performance and reliability. To that end, we made some changes to how we cache results on our main posts page. We intended to reduce how often the cache needs to be refreshed, and to reduce unnecessary load on the database when doing so.
When the changes were deployed, we immediately noticed that the load on our caching server was much too high. As we attempted to roll back the change, we experienced a cascading failure and were unable to roll it back quickly. Since the caching server was overloaded, many customers experienced an error page during that time.
As a result, we are reexamining our deployment procedures and plan to make them more robust and fault-tolerant. We’ve since deployed a fixed version of the caching update without any interruption to customers.
Follow Up
Since the outage the customer success and technical teams have worked closely to audit all technical and support processes to identify shortfalls and required adjustments. We’re confident that the following actions will improve resiliency and stability of the application, as well as our processes, moving forward.
Technical Processes
-
We updated our software deployment policy and procedures and added controls to identify potential issues quickly and accurately.
-
We are improving our roll back procedures to minimize potential downtimes and get you back up and running ASAP.
Communication & Support Processes
-
We are instituting a ParentSquare Systems Status webpage and will follow up within a few weeks regarding the status page.
-
We reviewed and updated our internal severity definitions, troubleshooting and communication protocols to ensure all procedures are clearly stated and followed across our growing team.
Whether you’ve been working with ParentSquare for years or have just recently become a ParentSquare customer, please know that we take enormous pride in the level of service we provide to our customers, our response times and the quality of the experience when using the ParentSquare application. We will continue to improve upon this philosophy.
Thank you for your continued support, understanding and patience as we grow from this experience and work together to re-open our schools safely and effectively.
Anu Vaid
Founder and President
ParentSquare, Inc.
