From firefighting to foresight: an observability overhaul that dramatically cut production incidents
How I introduced observability practices across Gridware’s core platforms as Engineering Manager — so problems were found by dashboards instead of customers, incidents became dramatically rarer, and on-call went from chaotic to calm.
The challenge
Gridware builds grid-monitoring technology, and the cloud-native and mobile-first platforms my teams ran sat in the critical path for external customers. As those platforms scaled, production issues were too often discovered the worst possible way: a customer noticed before our dashboards did.
Each incident pulled engineers off the roadmap and into firefighting. Without shared visibility into what the systems were actually doing, diagnosis was slow, fixes were stressful, and reliability felt like luck rather than a property the team controlled.
My role
As Engineering Manager (via Remotebase) I owned both delivery and production stability across critical systems — designing serverless solutions on AWS using microservices and SOA with NestJS, streamlining delivery with GraphQL, AWS Amplify, and CI/CD pipelines, and working directly with C-level stakeholders and external customers on roadmap alignment. Incident management was mine end to end: if production broke, the buck stopped with me.
The approach
- Instrument firstBefore changing any process, we made the systems observable: metrics, logs, and traces flowing from the core platforms into dashboards the whole team could read — the standard toolkit of the Datadog / Grafana / Prometheus / Sentry / CloudWatch class. You can’t fix what you can’t see.
- Define what “healthy” meansWe set SLO-style thresholds for the things users actually feel, and wired alerting to user impact rather than raw noise — so a page meant something real was wrong, and quiet meant the platforms were genuinely fine.
- Make incidents a process, not a panicWe introduced structured incident command: clear severity levels, a single owner per incident, predictable communication, and blameless post-incident reviews that turned every failure into a concrete improvement instead of a scar.
- Harden the pipelineReliability also depends on how fast you can ship a fix. CI/CD discipline across the GraphQL and AWS Amplify stack meant remediations and preventative work rolled out quickly and safely, with the serverless architecture keeping blast radius small.
The results
- Dramatically fewer production incidents across the core platforms — the firefighting that had been eating roadmap time largely disappeared.
- Issues caught before customers noticed — dashboards and impact-based alerts surfaced problems while they were still cheap to fix.
- A real learning loop — blameless post-incident reviews fed directly into the backlog, so each incident made the system stronger instead of just older.
- Stability across critical systems — calm, structured on-call and a reliability story C-level stakeholders and customers could trust.
What made it work
Observability is a product decision, not an ops chore. We treated instrumentation, dashboards, and alerting as part of the platform itself — planned, owned, and prioritized like any feature — instead of something bolted on after the next outage.
SLOs turn reliability debates into data. Once “healthy” was defined in terms of user impact, conversations with C-level stakeholders stopped being about anecdotes and started being about thresholds — which made it far easier to balance roadmap work against reliability work.
The goal is boring on-call. Structure, severity levels, and blameless reviews exist so that pages are rare, response is calm, and engineers keep their energy for building. Boring on-call is what a reliable platform feels like from the inside.
“Anas’ leadership and commitment to getting the job done was a force that always moved the needle for us.”
Is reliability slowing your roadmap?
I help product organizations make production stability a managed outcome — observability, SLOs, and incident practices that let teams ship fast without fear — as an engineering leader, fractional CTO, or consultant.