When a destination becomes slow or unavailable, new requests can continue accumulating until threads, connections, and memory are exhausted. A circuit breaker temporarily stops calls after a defined failure pattern, gives the dependency time to recover, and then probes cautiously to see whether traffic can resume.
How do the three circuit-breaker states work?
In closed state, requests proceed and outcomes are measured. When a failure threshold is met, the breaker becomes open and rejects new calls quickly. After a period, half-open allows a small number of probes. Enough success closes the breaker; another failure opens it again.
The state names are simple, but thresholds must reflect the dependency. A validation error is not a service outage. Timeouts, connection failures, selected 5xx responses, and high latency may be relevant. Agree on classification with the retry policy.
Which parameters need design?
| Parameter | Question |
|---|---|
| Window | How many requests or how much time is evaluated? |
| Minimum volume | Can one failure open it? |
| Threshold | Which error ratio or latency? |
| Open duration | How long before probing? |
| Half-open limit | How many probes are safe? |
Overly sensitive configuration rejects healthy services; a loose breaker allows cascading failure. Use historical data and disruption tests rather than copying example numbers as universal defaults.
What happens while the circuit is open?
Options include failing fast, returning age-labeled cache data, accepting asynchronous work, using an alternative service, or asking a user to try later. A fallback must preserve meaning. An old price may be acceptable for browsing, while a stale payment balance is unsafe for settlement.
State whether the result is stale, estimated, or pending. Never convert a failure into false success just to keep a dashboard green.
Read about false success.
What scope should a breaker use?
A global breaker is simple, but one bad operation may block everything. Per-host, endpoint, tenant, or operation-class scope can be more precise but needs enough volume for meaningful signals. Separate reads from writes when their impact and latency differ.
In a distributed deployment, local breakers on each instance may have different states. A centralized breaker adds coordination and becomes a dependency. Choose scope by blast radius and capacity objective, then document whether operators see one global state or several local states.
What should be observed and tested?
- State changes with reason and time.
- Fast-rejected requests versus actual failures.
- Latency, error ratio, and window volume.
- Successful and failed half-open probes.
- Fallback impact on business outcomes.
- Behavior when many instances recover together.
Test slow service, 5xx responses, timeout, invalid response, and gradual recovery. Add manual procedures to the incident runbook.
When is a circuit breaker not the answer?
Do not use a breaker for permanent business errors, invalid input, or local operations with no dependency. Do not add one before timeouts and resource limits are clear. A breaker provides no rollback for an effect that already occurred and does not remove the need for idempotency after an uncertain result.
This article describes a general pattern and does not claim NitroNet implements it. If the pattern is needed around a workflow, validate it through a threat model and timeout budget. Evaluate NitroNet only from public capabilities and test evidence in your environment.
Exercise a simulated dependency so no user is affected. Increase latency and errors gradually, then restore the service slowly. Observe state, fallback, resources, and user experience across the complete cycle rather than only whether the breaker opened.
- Begin below minimum volume so one failure does not incorrectly open the circuit.
- Increase timeout ratio to the threshold and make the state-change reason auditable.
- Ensure requests during open fail fast or use a fallback with approved meaning.
- Test stale cache with an age label and prevent users from treating it as final.
- Allow few half-open probes and stop all instances from flooding the dependency together.
- Restore service partially and observe whether thresholds cause repeated open-close movement.
- Test a write with uncertain outcome and ensure the breaker promises no rollback.
- Preserve a timeline of metrics, operator decisions, and configuration changes after the exercise.
A sound breaker limits blast radius without concealing failure. If fallback creates a wrong decision, fast failure may be safer. Review parameters against real volume and latency. Remove a breaker that lacks ownership or observability because hidden state can complicate diagnosis more than the original dependency.