Intermittent Firebase flakiness

Incident Report for CoderPad

Resolved

It looks like this was due to a large customer performing a large number of deletes, which overwhelmed one shard. We will return this shard to rotation and increase our overall number of shards.

Posted Feb 04, 2019 - 15:15 PST

Update

We've additionally removed this shards for new pads. New pads created will not hit this problem from this moment forward (1:47pm PST).

Posted Feb 04, 2019 - 13:47 PST

Investigating

It seems that one of our Firebase shards has been intermittently failing. Users may see failure to synchronize or mutate some pads (approximately one fifth). If you create a pad and are unable to load it, you may try creating another pad and emailing support (support@coderpad.io) for a quota refund.

We are working on escalating this problem with Google.

Posted Feb 04, 2019 - 13:40 PST

This incident affected: Web Tier.