A show about getting your best ideas into the world and seeing what happens. We talk about code, ops, infrastructure, and the people that make it happen. Gerhard Lazu and friends explore all things DevOps, infra, and running apps in production. Whether you’re cloud native, Kubernetes curious, a pro SRE, or just operating a VPS… you’ll love coming along for the ride. Ship It honors the makers, the shippers, and the visionaries that see it through. Some people search for ShipIt or ShipItFM and can’t find the show, so now the strings ShipIt and ShipItFM are in our description too.
Kaizen! Five incidents later
This is our second Kaizen episode, where Adam, Jerod & Gerhard talk about changelog.com improvements since episode 10. OK, so Gerhard deleted the DNS API token. Not only did he take the time to understand how that happened, so that he could actually learn from his mistake, but now we have a system in place so that we can share learnings from incidents. By the way, these are publicly available in our #incidents Slack channel.
A great and unexpected thing that happened since we recorded this, is Jerod fixing 99% of all the errors that were happening in prod. The top error was the broken Twitter auth - sorry Matt - which was a result of us upgrading to OTP 24 a few months back. Episode 3 show notes include a YouTube stream which captures it all.
We wrap up this episode by each of us sharing the improvements that we would like to do until our next Kaizen. You heard it from Adam first: Ship It Driven Development
Join Changelog++ to support our work, get closer to the metal, and make the ads disappear!
Sponsors
- Fly.io – Deploy your apps and databases close to your users. In minutes you can run your Ruby, Go, Node, Deno, Python, or Elixir app (and databases!) all over the world. No ops required. Learn more at fly.io/changelog and check out the speedrun in their docs.
- PlanetScale – PlanetScale is the only serverless database platform you can start in an instant and scale indefinitely with unlimited connections. Never think about database servers again. Everything you want to control is available through the beautifully designed PlanetScale CLI. Learn more and start your database in seconds at planetscale.com
- Honeycomb – Guess less, know more. When production is running slow, it’s hard to know where problems originate: is it your application code, users, or the underlying systems? With Honeycomb you get a fast, unified, and clear understanding of the one thing driving your business: production. Join the swarm and try Honeycomb free today at honeycomb.io/changelog
- Fastly – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com
Featuring
- Adam Stacoviak – Twitter, GitHub, LinkedIn, Website
- Jerod Santo – Twitter, GitHub
- Gerhard Lazu – Twitter, GitHub, Website
Notes and Links
- changelog.com PR#378
- Oban: Robust job processing for Elixir
- changelog.slack.com #incidents
- Incident.io: Our incidents platform
- DNSimple: Our DNS provider
- Fastly: Our content delivery network
- Linode - Connectivity Issue - US-East (Newark)
- Linode - Service Issue - Linode Kubernetes Engine
- Linode Global Infrastructure
- Sentry: Our error tracking
- Jerod: This is what happens when @gerhardlazu…
- Founderstalk #80: Iterating to globally distributed apps and databases
- 🎬 Firecracker VMs on Metal, Oh My! - Kurt Mackey, March 2021
- 🎬 Live Debugging the Changelog’s Production Kubernetes | Rawkode Live, September 2021
