February 2017

Ongoing Firebase Outage
Okay, they came back up again. Seems like we were only down for a minute.
Feb 13, 14:22-14:24 PST

January 2017

Brief execution downtime
Do to user error, code execution was offline for about 7:10pm PST, and was back up by 7:15pm. We're sorry, we'll try not to do that again.
Jan 17, 19:20 PST

December 2016

No incidents reported for this month.

November 2016

Service degraded due to Firebase downtime
We appear to have recovered, systems are now functioning normally again.
Nov 17, 11:41-11:46 PST

October 2016

Problems with code execution
Our cluster was using a allocation strategy that kept breaking - we've since moved to a more simple one.
Oct 6, 18:15-18:47 PST
Code execution unavailable 8:25am - 8:35am PST
In the course of rolling out our new cluster architecture for code execution, we hit a snag on a file descriptor limit. We've since fixed the issue but this resulted in a brief outage. Apologies!
Oct 6, 13:02 PST

September 2016

Problems with code execution
We've recovered the service and are investigating root cause.
Sep 13, 16:44-17:09 PST

August 2016

Problems with code execution
We've identified some issues that may have caused the performance problems seen today and have applied fixes.
Aug 26, 11:02-14:16 PST

July 2016

No incidents reported for this month.

June 2016

Brief Firebase Outage
At approximately 3:30pm today, CoderPad suffered a ~1 minute outage due to Firebase connectivity issues. Full functionality has returned.
Jun 15, 15:36 PST
API downtime
From approximately 1:40am PDT to 1:40pm PDT on Thursday, June 9th, many API calls to CoderPad were failing. Specifically the pads#show route. This was due to a combination of Google Hangouts API change and a code push on our end to address it. Compounding the issue was the fact that our alarming was not properly setup for API failure. This was a large outage and we are very sorry for the downtime. The problem should now be resolved.
Jun 9, 13:48 PST

May 2016

No incidents reported for this month.

April 2016

Site down due to Firebase outage
Okay, Firebase seems to be back up.
Apr 11, 19:14-19:33 PST

March 2016

Execution downtime
This incident has been resolved.
Mar 1, 12:27-20:13 PST

February 2016

Downtime due to Firebase outage
Okay, we should be back up, Firebase fixed its networking issue: https://twitter.com/FirebaseStatus/status/694767601868627969
Feb 2, 21:24-22:23 PST
Scheduled Maintenance
Donezo!
Feb 2, 03:23-03:25 PST

January 2016

Minor execution downtime
From Midnight to 1am PST on January 22, code execution was unavailable on CoderPad due to a botched deploy. Services are recovered now.
Jan 22
Extended execution downtime
From 12am to 10am on January 19th, our execution service was down. This was due to a logistical problem after recovering from a previous incident (http://coderpad.statuspage.io/incidents/jmmdmcmfkhbf). We're very sorry for the extended downtime, and are working on improving both our paging and redundancy in these areas. Synchronization of code editing during the interview was not effected.
Jan 19, 10:06 PST
Execution downtime
From about 2:32pm to 2:50pm today, CoderPad experienced downtime on its code execution environment. It appears the Docker daemon got stuck. We're back online and will investigate ways to insulate ourselves from this in the future.
Jan 18, 14:55 PST

December 2015

No incidents reported for this month.

November 2015

SSL Certificate Expired
Okay, we got a new cert. Sorry everyone!
Nov 8, 16:16-16:32 PST
Code execution offline
We've identified candidate problems and will address root cause in the coming week.
Nov 8, 12:36-16:15 PST

October 2015

Firebase latency causing degraded performance
And they're back: https://twitter.com/FirebaseStatus/status/659124800334331904 We're looking into insulating ourselves better from this.
Oct 27, 13:51-14:53 PST
Web tier unreachable
For a 15 minute period ending at Tuesday, 4:08pm PST, the CoderPad website was unreachable. We were seeing connection timeouts in Heroku. We still aren't certain why this was happening, but have resolved the incident with a restart for now. We will investigate further immediately.
Oct 13, 16:09 PST

September 2015

No incidents reported for this month.

August 2015

Elevated Firebase Latency
Firebase latency appears to have returned to normal. We'll continue to monitor the situation.
Aug 21, 15:45-15:57 PST

July 2015

Firebase Offline
Firebase has recovered https://twitter.com/FirebaseStatus/status/624416044556234752
Jul 23, 20:02-20:15 PST

June 2015

CoderPad Downtime due to Firebase outage
From approximately 11:10 to 11:30 today, most CoderPad functionality was offline due to a Firebase outage. We apologize for the inconvenience.
Jun 15, 13:28 PST
CoderPad.io unreachable
A dyno becoming unresponsive lead to our website being unreachable for 8m today, starting at 3:19pm PST. We have since increased redundancy and restored service.
Jun 5, 15:37 PST

May 2015

Firebase is down
Firebase has recovered.
May 14, 14:32-14:42 PST

April 2015

No incidents reported for this month.

March 2015

No incidents reported for this month.

February 2015

Firebase Downtime
Firebase has recovered.
Feb 8, 09:43-10:02 PST

January 2015

Firebase Outage
This incident has been resolved.
Jan 22, 20:37-20:47 PST
Code Execution Downtime
From 3:30 to 3:39 pm PST, code execution on CoderPad was unavailable. The root cause was collectd log files consuming all available space on disk. Specifically, the interaction between docker virtual interfaces caused a very large number of metrics to be tracked. We are now curbing these extraneous metrics. Service has returned to normal.
Jan 21, 15:56 PST
Pad UI impacted
For one hour, between 4am and 5am PST, a bad CSS rule was cached on our web frontend. This resulted in an unusable UI. Luckily, few interviews were impacted, but we are very sorry for the interruption.
Jan 12, 06:12 PST

December 2014

No incidents reported for this month.