Health Checks Lab
An Orders service runs four instances behind a service mesh, with a sidecar proxy beside each one and a shared database behind them. At 0:10 one instance gets stuck: its process is up, but every request it gets fails. At 1:00 the database fails over to its standby. Choose the probes and the mesh settings, then play the two minutes and see which ones protect the traffic and which ones cause an outage of their own.
Background: Health Check API, Service Mesh, Sidecar.
- Events will appear here.
Run these in order
Run with no real checks
Liveness only asks whether the process is running, there is no readiness probe, and the mesh just balances load. At 0:10 instance C's connection pool leaks: the process is fine, but every request it gets fails. Liveness keeps passing, so C keeps a quarter of the traffic and fails 100 requests a second until the end. At 1:00 the database fails over and every request fails for 20 seconds. No health check can prevent that part.
Check the database in liveness
Make liveness query the database. C fails it and is restarted at 0:20, which fixes the leak. But at 1:00 every instance fails the same check, and at 1:10 all four are restarted at once. The database is back at 1:20, but the instances are still loading their caches until 1:40: a 40-second outage instead of 20. Set the interval to 10 s: three failures now take 30 seconds, longer than this failover, so nothing restarts. A longer failover would still take everything down.
Add a readiness probe
Keep liveness to the process, and add a readiness probe that checks the instance's own connection pool. C fails it three times and leaves the rotation at 0:20, without a restart. The database failover passes without a single restart. Try probing every second with a threshold of 1: C is out almost at once. Readiness protects the traffic, but C is still stuck, and someone has to restart it.
Let the mesh eject it
Turn on outlier detection: the sidecar proxies eject an instance after 5 errors in a row. The proxies see real requests, not a probe every few seconds, so C is ejected within a second, after 5 failed requests instead of about 1,000. Watch the failover: every instance returns errors, and the mesh ejects a healthy one too. It cannot tell a shared failure from a sick instance, which is why meshes cap ejections, here at one instance at a time.
Retry in the mesh
Drop the checks and let the mesh retry each failed request up to three times on another instance. Users now see a fraction of C's errors, so nothing looks wrong, but C never leaves the rotation and every request costs a third more work. During the failover each request is tried four times. When the new database primary comes up with a cold cache, the retries keep it overloaded and requests keep failing for 11 seconds after it is back. Retries hide problems. Health checks and ejection remove them.
This lab simplifies reality on purpose. Traffic is flat, every probe runs at the same moment on every instance, restarts always take the same time, and the mesh balances load evenly. Real orchestrators also have startup probes and back off after repeated restarts, real meshes can eject on success rate instead of errors in a row, and retry budgets cap how much extra load retries may add. Whether readiness should check shared dependencies at all is a judgement call that depends on what the instance can do without them.