Five years ago this would have been a controversial post. Today we have battle-tested operators, decent storage classes and a community that has finally agreed on backup tooling. Here is the stack we landed on for a fleet of around forty clusters.
Continuous profiling sounds great until you turn it on in a fleet of three hundred pods and your p99 latency doubles. Here is how we instrumented pprof endpoints behind a feature flag, sampled at one percent and shipped flame graphs straight to Grafana.
Internal tools tend to rot the moment they leave the founding team. We rebuilt our admin from scratch around three principles: every action should be auditable, nothing should require a tribal-knowledge runbook, and there should be exactly one button that resolves the on-call page.
Sometimes the right answer is not Hugo or Astro but a file you can read in one sitting. We walked through every line in a guild lunch and somehow shipped a new docs site by the end of the day.
There is no faster way to grow as an engineer than spending an evening with a debugger and a project you depend on every day. Here are the libraries that taught us the most and the exact commits worth bookmarking.
Most database migrations fail not because the SQL is wrong but because nobody tested them against a realistic dataset. We share the rollout checklist we use for every schema change, including the awkward question of when to lock a table and when to live with the inconsistency.
Every team eventually discovers the joy of shipping behind a flag. Fewer teams discover the misery of cleaning up flags two years later when the original author has long since left the company. A short rant and a longer guide.