Letterhead temporarily down
Incident Report for Letterhead
Postmortem

We have completed our initial analysis of the Letterhead outage that occurred on October 24, 2023 from approximately 7 a.m. ET to approximately 9 a.m. ET, and are taking necessary steps to prevent future service interruptions. We’ve identified the root cause as an outage with our database hosting provider. We apologize for the incident and any impact it may have had on your business.

As our engineers worked with our provider to restore service, our customer experience team reached out to affected customers to alert them to the issue. After restoring service, our team double-checked scheduled emails in Letterhead and ensured that they were sent as quickly as possible. No data was lost in the course of this outage.

We are implementing the following actions over the next 90 days to enhance the reliability and performance of our systems and prevent future occurrence of this issue or similar issues. 

  1. Implement additional database backups: We will create and configure additional backups and fallback measures for our databases to allow us to get databases back up and running more quickly in the event of a failure.
  2. Improve data architecture and infrastructure: We have already begun an engineering initiative to improve our data architecture in Letterhead. These improvements will also make services more resilient and faster to restore.

Our commitment to providing the best possible service remains our top priority. We value your trust and appreciate your patience during this incident. As always, our customer experience team is ready to help you with any questions or concerns in our Help Center. To stay informed about any future updates, please check and subscribe to alerts on our Status Page.

Posted Oct 24, 2023 - 12:57 EDT

Resolved
Our team has resolved the partial outage and Letterhead services are now fully operational. We are continuing to monitor the incident and all services for any lingering effects. We apologize for the inconvenience this incident has caused and we appreciate your patience while we worked to resolve it.

Letterhead is committed to providing the best possible service, and we continuously strive to improve. If you need assistance, please contact support at https://help.tryletterhead.com. Please continue to check our status page and subscribe to alerts for the most up-to-date information and a final report on this incident: https://letterhead.statuspage.io/
Posted Oct 24, 2023 - 09:31 EDT
Monitoring
Our team has resolved the partial outage and Letterhead services are now fully operational. We are continuing to monitor the incident and all services for any lingering effects. We apologize for the inconvenience this incident has caused and we appreciate your patience while we worked to resolve it.

Letterhead is committed to providing the best possible service, and we continuously strive to improve. If you need assistance, please contact support at https://help.tryletterhead.com. Please continue to check our status page and subscribe to alerts for the most up-to-date information and a final report on this incident: https://letterhead.statuspage.io/
Posted Oct 24, 2023 - 09:30 EDT
Update
Service has been fully restored for Promotions API, Archives and Ticketing. Other features and functions remain partially out of service, due to an outage with our cloud host. Our team is working to restore service as quickly as possible. We will continue to provide timely updates as this incident develops.
Posted Oct 24, 2023 - 09:21 EDT
Update
Partial service has been restored to the Letterhead app, but some features and functions remain out of service, due to an outage with our cloud host. Our team is working to restore service as quickly as possible. We will provide another update within an hour.

Please check our status page and subscribe to alerts for the most up-to-date information: https://letterhead.statuspage.io/
Posted Oct 24, 2023 - 09:11 EDT
Update
We’re currently experiencing a service outage due to an issue with our server host. Our team is committed to providing you with excellent service and we are working with our provider to restore service as quickly as possible. We will provide an update within an hour.

Please continue to check our status page and subscribe to alerts for the most up-to-date information: https://letterhead.statuspage.io
Posted Oct 24, 2023 - 07:32 EDT
Identified
We identified the issue with an upstream provider and we are currently working with them to restore service.
Posted Oct 24, 2023 - 07:24 EDT
Investigating
We are currently investigating this issue.
Posted Oct 24, 2023 - 07:12 EDT
This incident affected: Applications (Archives, Audience, Promotions, Newsletters) and Services (Email Service Provider, Promotions API, Ticketing, Webhooks).