The Plural of Mutex Is Deadlock
Every good idea, every software pattern is dangerous if used ex extremious. Null will eventually become NullPointerException. Inheritance will refactor itself into legacy code. Refactoring becomes broken code. Mutexes will become deadlock. Seriously.
I want to take you on this deep-dive into everyone’s favorite concurrency primitive, the mutex. A primitive, a nice little lock with the downside potential to completely ruin your day.
Nothing is Friendly in Software Development
A mutex is innocent enough. Protect shared state in a multithreaded environment by bringing this baby to the party, and you’re going to be good until the end of time.
You’re doing the right thing if you wrap a function in a mutex. You can breath a sigh of relief and go back to actual work feeling smart, safe and clever.
That feeling won’t last.
Scaling Up the Chaos
One mutex is a cute solution, and a pattern to solve a well-known problem.
Two? It’s tense.
Three? Welcome to the chaos.
The problem is, as the number of mutexes grows, so does the possibility of deadlock. You can imagine it as a group of well-meaning threads each holding one key and needing the key another thread has. Forever.
Here’s how that plays out in reality:
Thread A locks mutex 1.
Thread B locks mutex 2.
Thread A wants mutex 2.
Thread B wants mutex 1.
They stare at each other across the battlefield.
No one wins.
Production hangs. Your pager explodes.
The plural of mutex is deadlock. And it’s always your fault. You’ll want to cry in the incident call.
If you’ve never tried debugging a deadlock in a production incident, I envy you. If you have, you already know it goes something like this:
1. The app is frozen.
2. Nothing in the logs. Zero errors. It just stops.
3. You add logging and… it never reproduces.
4. You remove logging. Deadlock is back.
5. You start believing in curses.
This is the developer version of paranormal activity.
Advice You’ll Ignore
There’s no shortage of advice out there:
Always lock in a consistent order
Minimize the use of shared state
Use higher-level concurrency tools like semaphores, channels, or actor models
Great advice. Advice you’ll remember after you ship the feature. After the late-night Slack from QA. After you realize that one tiny mutex in your utility function was actually accessed by five different services and now your system is a very expensive spinlock.
Alternatives That Don’t Suck
Here’s what I actually recommend:
Don’t share state if you can help it. Functional programming folks might be smug, but they’re also not woken up by deadlocks
Use atomic structures if possible. They’re boring, but safe
Consider task queues, event-driven models, or letting the whole thing burn and pretending you work in marketing now
Honestly, most of the time, we use mutexes because we didn’t design things well enough up front. I say that as someone who will absolutely use one again tomorrow.
Threads
I’m going to be generous here. You’re not a bad developer, and neither am I. I’m saying that as concurrency is hard and our computers simply function as chaos machines.
You’re juggling knives in the dark, and if you find yourself in this position you’ll feel like your team is throwing in more knives periodically. You’re being paid for the privilege of being blamed when it all goes sideways (because after all AI wrote the code, didn’t it?).
Conclusion
You need to add further tools in your armory.
Having a mutex as your go-to is an excellent move, well done. But when using them you should ask yourself whether this is absolutely necessary. Just because you have a knife doesn’t mean that it’s the most suitable weapon for a gunfight.
You’ll go ahead and use a mutex anyway. Because you’re on a deadline and everything else is already broken. Because this is software development, and that is how things roll, right?
About The Author
Professional Software Developer “The Secret Developer” can be found on Twitter @TheSDeveloper.
The Secret Developer says deadlock isn’t caused by having multiple mutexes, but by cyclic lock acquisition without ordering.