Fragment

Observability Engineering Notes - 1

(Majors et al., 2026)

I am trying to organize my thoughts around the book as I read. Although not groundbreaking, the first 3 chapters offer good, concise wording for concepts and practices that were present in my work experience, but not always had tight wording around them.

So I will take this chance to go through what I noted as remarkable, so I can easily refer to them at a later point.

First a sentence that I think summarizes the whole thing well:

Done means fully in production

General notes

  • Software can be durable or disposable. Once disposable graduates to durable, it becomes a contract with your users and stakeholders. This means proper process, instrumentation and practices around it become essential
  • Not all software that is disposable must become durable. Criticality and risk (as in cost of failure) should be the main criteria
  • It is important to test before production, but "done" means fully in production. Including seeing it work there. Production is part of the lifecycle, and therefore part of the technical staff responsibilities
  • Instrumentation has to do with the dev's intent. A code change should carry an intention. Agents can be leveraged to see if intent graduates into a fact (assuming instrumentation is there).

On deploying

  • Team should optimize for merging code ASAP, without downtime and without immediate user impact
  • This means use CI/CD for deployment and feature flags for releases
  • Feature flags allow this decoupling, and put full control in the hands of whoever releases features and communicates with stakeholder
  • Feature flags also allows for instant rollback of features (which vibes well with "test in production")
  • Feature flags are also no enough. Other things help on that:
    • Blue-green deployments
    • Canaries (route users progressively to the server with the new code)
    • Do capture/replay (capture a snapshot, capture 24h of traffic, replay these 24h from the captured snapshot and log the differences)
    • Traffic splitters (do the same request to two endpoints following the same contracts and record the differences)
    • Strangler fig (slowly replace old stuff with new stuff, by implementing and migrating little by little)