Your developer just pushed a "quick fix" to production at 2 AM. Without review. Without tests. Without telling anyone.
Now your site is down. Your payment processor is broken. Your customers are emailing support asking if you got hacked.
Welcome to the Burned Dev Patch—the single most common way startups accidentally destroy their own infrastructure.
What Happened
Here's the pattern:
- Something breaks (bug report, customer complaint, broken feature)
- Dev sees it late at night (scrolling Slack before bed)
- "I can fix this in 5 minutes" (narrator: they cannot)
- Pushes directly to main (skips PR, skips CI, skips testing)
- Goes to bed feeling heroic
- Wakes up to Slack on fire
The "quick fix" introduced three new bugs. One broke authentication. One corrupted the database. One disabled error logging so nobody could tell what was happening.
And because it was pushed at 2 AM, nobody was awake to catch it.
Why Developers Do This
It comes from a good place:
- They care about the product
- They want to be responsive
- They see a simple fix and think "I can save time"
- They don't want to bother anyone with a "tiny change"
But the Burned Dev Patch isn't about bad intent. It's about bad process.
Specifically:
- No mandatory code review
- No staging environment testing
- No deployment checklist
- No rollback plan
- No "don't deploy after 10 PM" rule
Real Examples (Names Changed)
Startup A: Dev pushed a "one-line CSS fix" that accidentally changed a CSS class used by the payment form. Payments stopped working for 6 hours before anyone noticed (it was a weekend). Lost $15K in revenue.
Startup B: Dev pushed a database migration at midnight to "clean up old user data." Migration had a typo. Deleted active user accounts instead of inactive ones. No backups. Company died.
Startup C: Dev pushed a hotfix for a broken API endpoint. Didn't test locally. Turns out the fix worked on their machine but broke in production (environment variable mismatch). Site down for 4 hours during peak traffic.
The Five Rules
If you want to avoid the Burned Dev Patch:
1. No Deploys After 10 PM (Unless P0 Emergency)
If it can wait until morning, it should.
Exception: The site is literally down right now and customers can't access it. Then yes, deploy at 2 AM.
Not an exception: "This bug is annoying" or "I want to ship this feature today."
2. All Production Deploys Require Review
Even "one-line fixes."
Especially one-line fixes (those are the ones that break everything).
Minimum: One other engineer reviews the PR before merge.
3. Staging Environment That Mirrors Production
If you don't have a staging server where you can test changes before production, build one.
Cost: $20–$50/month (cheap insurance)
Benefit: Catching breaking changes before customers see them (priceless)
4. Automated Tests + CI
Every PR should:
- Run your test suite automatically
- Block merge if tests fail
- Require passing CI before deploy
If you don't have tests yet, start with smoke tests:
- Does the site load?
- Can users log in?
- Can users check out?
5. Easy Rollback
Every deploy should be easy to undo.
Use:
- Git tags (so you can revert to last known good commit)
- Feature flags (so you can disable new code without redeploying)
- Database migrations that are reversible (never write a migration that can't be undone)
If rolling back takes more than 5 minutes, your deploy process is broken.
What to Do When It Happens
Because it will.
- Immediate rollback (don't try to "fix forward" in the moment)
- Assess damage (what broke, how many users affected, data loss?)
- Communicate with customers (status page, email, social)
- Post-mortem (blameless, focus on process not people)
- Update process (what rule would've prevented this?)
The Culture Piece
The hardest part isn't technical. It's cultural.
You need to create an environment where:
- Developers feel safe saying "I'm not sure, let me test this"
- Nobody is rewarded for "moving fast and breaking things" (that's not a virtue at 2 AM)
- Deployment speed matters, but not more than reliability
Because the Burned Dev Patch isn't a technical failure. It's a judgment failure.
And judgment failures come from bad incentives.
The Bottom Line
Your 2 AM hero developer isn't the problem.
Your lack of guardrails is.
Build the process. Enforce the rules. Sleep peacefully.
END OF ALL 8 ARTICLES
