Sometimes people ask me what they should learn to become a better programmer. I feel like the default recommendation here is usually an obscure programming language or a textbook on some high-powered machinery like ML. So I always feel a little bit embarrassed and boring when I instead suggest going really deep on what you already know: your main programming language, web framework, object-relational mapper, UI library, version control system, database, Unix tools, etc. It’s not shiny or esoteric, but for me, building a detailed mental model of those (and how they compare to alternatives) might be the learning that’s contributed most to my effectiveness as an engineer.
A coworker coined the phrase “blub studies” to refer to this sort of mundane, ultra-specific-seeming knowledge. “Blub” comes from a Paul Graham essay, Beating the Averages, in which Blub is a hypothetical middlebrow language whose programmers get defensive when Graham asserts that Lisp is superior. Blub studies is the study of what goes on in the guts of these boring, everyday systems—not the kind you get tenure for inventing, but the kind people actually use.
Blub studies is a never-ending treadmill of engineering know-how. It’s the fiddly technical details of how Git stores data, or how Postgres locking semantics caused your migration to bring down prod, or why
pip install failed this time. It’s what goes on inside the boiler rooms of your computer. There’s a seemingly infinite amount of it, full of bespoke details for you to stumble over, and that makes it, often, unbelievably frustrating. Experts in shiny fields like machine learning write shiny-sounding articles like A theory of the learnable; experts in blub studies emit screeds like The Law of Leaky Abstractions and Programming Sucks.
In short, if you’re in search of generalizable knowledge that compounds exponentially over time, then blub studies looks like the crap you have to wade through to get to the good stuff. So it’s easy to see why people give up on understanding all the blub they’re surrounded by, except what they need to get the job done.
But for me, the opposite attitude has been more productive. Computers can be understood—even if it’s hard and takes a while. Blub studies is more generalizable than it seems, and has its own way of compounding over time, too. That makes it a lot more useful than you’d expect.✻ Of course, there are useless parts of blub studies: if this essay gets you excited to memorize a bunch of command-line flags, consider reversing this advice. But in my experience, it’s more common to neglect the useful parts of blubs, than to over-index on trivia.
The most straightforward benefit of blub expertise is that it saves you time. “You can’t apply those brilliant insights you learned from SICP if you don’t have the knowledge base and emotional fortitude to fight through
pip install first.” If you know how Git’s internal model works, you can get your repository out of its borked state without spending hours on Stack Overflow.
This effect is larger than it might seem. If you’re working with a system you don’t understand, you’re limited to debugging via guess-and-check, which can be arbitrarily slow. A more efficient method would be to get as much information as possible about your program’s execution and then use that information to exclude most of the hypothesis space. But this requires a good understanding of both the system, and the tools available for inspecting it. If you’re tracking down, say, a networking problem, staring at some
tcpdump output will often get you most of the way there, but only if you know how to interpret it and what to look for.
If you spend half your programming time debugging, and being a blub expert lets you debug twice as fast, then just the speed gain from blub expertise will let you increase your output by a third.†
If you think “half of programming time debugging” sounds high, imagine how much faster you’d be if all your code worked the first time.
Doubling debugging speed is probably a conservative estimate—you can save pretty much unlimited time via things like “hmm, 40 milliseconds sounds like the timeout for Nagle’s algorithm, try setting
TCP_NODELAY”. I somewhat frequently debug tricky things 5x+ faster than coworkers, just because I’ve been working with our stack for a long time, so I know where to look for problems and how to quickly test hypotheses. That justifies a lot of time staring at
tcpdump output! But there are also more subtle reasons I’ve gotten so much from blub studies. It’s both more general, lasts longer, and has more of a compounding effect, than I expected.
Blub studies are surprisingly broadly applicable because, even if you’re learning about the details of some specific blubby system, that system’s design will contain a juicy non-blubby core of extractible general principles. Unlike many “general principles” people try to teach you, the ones you learn via blub studies are guaranteed to be important to at least one real-world system (the one you’re learning about). And you’ll see them realized in all their messy detail, which academic presentations often leave out.
Suppose your blub of choice is React. You might worry that learning the gory details will be useless if you ever move to a different part of the stack, or even a different web framework. And, yes, some of them will. But the core idea of React—writing pure render functions, using reconciliation to make updates fast—is extremely powerful and general. In fact, it’s now been copied by the next generation of UI frameworks on both iOS (SwiftUI) and Android (Jetpack Compose). Learning the principles behind React makes it easier to learn those other frameworks. In fact, it can even be a useful source of ideas to “import” from one to the other. At Wave, for instance, we’ve gotten a lot of mileage out of importing ideas from Relay into our mobile apps.
This is a good example of an idea that, as far as I know, you can only learn about through blub studies. Academia didn’t give much attention to React-style UI programming. In fact, it doesn’t seem to view user-interface programming paradigms as a particularly interesting object of study at all. People do sometimes publish on it but, for instance, I couldn’t find any courses on it in MIT’s extensive course catalog.‡ You could argue that this is because UI programming is “too applied” and one shouldn’t expect it to be covered in an academic curriculum. But computer science covers many other equally-“applied” areas, like networking, databases, operating systems, and graphics.
Blub studies also compound more than you’d naively expect, in two ways. First, knowing about one blub makes it easier to learn about alternative blubs that serve the same purpose—like the React/SwiftUI example above. Second, knowing more about one blub helps you learn blubs in adjacent parts of the stack more quickly.
Once, while pair programming with a more junior coworker, we were writing a complicated SQLAlchemy query. My coworker used
name field of an object stored in the
user variable) instead of
name field of the class
User) and was wondering why her query gave the wrong results. I tried to explain the “magic” by which
User.name was an instance of
user.name was a simple
str. I went around in circles for a little while until I eventually explained Python’s descriptor protocol to her (the language feature SQLAlchemy uses to enable the “declarative” ORM syntax). At that point, everything clicked—and I realized that Python’s
__dunder__ methods are the key to decoding quite a lot of “magical” seeming code. If you learn the Python language features well, lots of complicated libraries will become a lot easier to understand.
I had a similar experience myself with Kubernetes. The first time I tried to learn it, it was a bewildering morass of jargon—all those namespaces and containers and Pods and Deployments and Services and Ingresses just to get a simple HTTP server running! Then I read a networking textbook and everything made much more sense. The (arguably) most complicated parts of Kubernetes exist to solve networking-related problems—allowing hundreds of containers to talk to each other independently while hosted on a much smaller set of computers—so the networking textbook gave me a schema onto which I could hang all my Kubernetes factoids. Once I knew how Linux’s IP routing, iptables, and network namespaces worked, it was much easier for me to understand what exactly something like “kube-proxy” was doing.
If you know enough different blubs, you can end up at the point where you don’t even need to look things up to figure out how they’re (probably) implemented. An experienced Python programmer can guess immediately how SQLAlchemy’s “declarative” ORM works under the hood. That’s the point when your blub expertise will really start compounding—almost as soon as you start working with something new, you’ll start figuring out how it works and extracting the kernel of generally-interesting ideas.
Because of this compounding effect, the most important step toward becoming a blub master is to kickstart your “blub flywheel”—the virtuous cycle of blub accumulation—however you can. That means starting with whichever blubs are the easiest or most motivating to learn, and branching out from there. For me, the easiest place to start has been with blubs I’m already using at my day job. I have a couple strategies for getting the most out of those.
First, I’ll try to go deeper than necessary. If I really want to ship something, it’s easy to give into temptation to, say, Google an error message, copy-paste a fix from Stack Overflow, and move on with my day. But it often doesn’t take that much longer to actually read the error message, understand what it means, and try to figure out why that Stack Overflow answer fixed my problem. Similarly, if I’m stuck in a tricky yak shave, I’ll bias against “guess-and-check” style debugging in favor of getting a better understanding of the system I’m trying to debug. It doesn’t always feel worth it to, e.g., dive into the docs of
iptables rules to track down my weird one-off networking issue—but over time I’ve run into enough “weird one-off networking issues” that it’s paid off many times over.
The second part of my blub flywheel is to pay attention to magic. Whenever I’m working with something new, I try to continuously update my best-guess mental model of how it’s implemented. “Okay, the docs are telling me to create an
Ingress, I guess this is probably the widget that provisions a load balancer to talk to my backend containers?” If I realize I’m wrong, I’ll dig in and update. “Hmm, I can’t ping those pods from outside the cluster, so how could the load balancer be talking to them? Aha—it’s talking to the nodes, and there’s a NodePort Service as a second layer of indirection.” If I have no idea at all how something could work, that usually means it’s time to read a book.
One thing I wish other engineers would do to make blub studies easier is to produce more and better “advanced” documentation of their software. Most mature and widely-used libraries have great tutorials and introductory content, but far fewer make it easy to get past the “copy-pasting examples” stage. I think this is mostly a matter of organization and navigation; if I comb through, say, the SQLAlchemy docs, I can actually find enough information to piece together a good mental model of the system, but it’s scattered among many different pages and it’s not clear what order I should read them in. The React docs are probably the best I know of here, but still leave a lot out.
Over time, by consistently exploring the guts of anything I’m working with that seems magical, I’ve built up a broad base of knowledge about how various technical systems work. This helps me in tons of different ways. It makes it easier to track down tricky bugs across many layers of the stack. I can learn new languages and libraries quickly by pattern-matching them to what I already know. It gives me better ideas for software designs, by imitating other systems I’ve seen, or by reusing ideas or tools I’ve heard of in a different context. Maybe most importantly, it gives me the confidence that, if I run into a tricky problem, I can learn enough to solve it, instead of feeling like I’m at the mercy of a system too complex to hope to understand.
So if you’re looking to learn something that will make you a better, and happier, programmer, ask yourself which parts of your most-used blub seem magical to you, and try to understand how they work. You’ll learn more than you think!
Thanks to David Golden, Awais Hussain, Will Larson, Dan Luu, Kevin Lynagh, Caleb Ontiveros, and Lincoln Quirk for commenting on a draft of this post.
This matches my experience with learning Scala, a programming language that builds Functional Programming (FP) on top of the JVM.
I thought I had to understand Monads, a complex FP concept with many bad metaphors (the most famous example being “Monads are burritos” article).
But really, I got way more mileage learning the 15+ usages of the underscore character ("_") https://stackoverflow.com/questions/8000903/what-are-all-the-uses-of-an-underscore-in-scala
And understanding the intricacies of all 15+ usages helped me understand evaluation, currying, and all the other highbrow FP concepts
Heads-up that your footnote system is broken on mobile. I expect that tapping the asterisk for a footnote should expand its corresponding footnote, but instead, every single asterisk for every single footnote, when clicked, expands only the first footnote on the page.
Whoops, thanks for reporting! A refactor attempt yesterday went badly. Should be fixed now.
I have a feeling this is a concept already known but discussed with different language or examples. And you’re talking about “real world experience” or real world knowledge. i.e. domain specific expertise. or contextual knowledge.
I strongly disagree with this article and will explain why.
These millions of lines of obscure and poorly documented systems are precisely what esoteric should mean. APL isn’t esoteric, but learning the quirks of some poorly-designed UNIX program that hasn’t been fixed in half a century and never will be is.
I would rather liken this Blub studies to a cult of the status quo. When so much is useless, unnecessary, and endlessly building upon like things, it’s easy to see it everywhere, and pretend it’s useful. I prefer pursuing ideals. It’s better to have layers which actually work, rather than endlessly layering broken interfaces over broken interfaces.
Regarding The Law of Leaky Abstractions, I’ve written a rebuttal to that nihilistic sophistry here: http://verisimilitudes.net/2020-08-24
This is, amusingly, a proper use of the word factoid here. Consider why those containers need to exist at all. Alan Kay apparently had children writing massively-parallel programs with simple objects; why shouldn’t adults get to play with such nice toys?
I’ll never understand this mindset. The most any programmer should need the Internet for is to find documentation and standards and whatnot, and to talk with others, but to use these crutches to the point where many argue surely everyone uses them is an obscenity. I never use StackOverflow, Google, or similar such things, because I use properly-documented and well-designed systems. It’s very easy for there to be many questions and answers for something eternally broken, and not for something which isn’t.
Perhaps ironically, this is an ideal, which I agree with, but the poorly-documented get poorer, and the richly-understood get richer.
Some people practice magic to feel like wizards, and pale in comparison to scientists who actually understand their work. The only solution to this Hellscape is to break free of it, which is what I seek to do by slowly writing software for myself, pursuing ideals I’ve very carefully considered, until I can one day breathe without inhaling ash and being told it’s fine.
You “prefer pursuing ideals”—fair, and if I shared your preferences I’d agree that rebuilding broken things is better than trying to use them. I don’t share your preferences because I have a different goal, which is to build things that are useful to people, and it seems to be easier to do this (i.e. you can build more useful things, for a larger number of people) by building on existing technology than by first developing your own “properly-documented and well-designed system” for each of the many tasks where none exists today.
If you will “never understand this mindset,” fair, but that’s a fact about you, and not about the mindset :)
PS: I can’t help but notice that your blog advertises a TCP library you wrote, which makes me think that either we must have different definitions of “properly-documented and well-designed” or you’re not being serious about only using such systems.
I document my code so that others may use it, but I won’t pretend I’m not the intended primary recipient of my work.
I won’t disagree that it’s, in some ways, easier. There’s a great benefit in owning and understanding the entirety of the system, however, and poring over millions of lines of code one didn’t write isn’t the way to achieve such.
I’ve gained a great deal of understanding by doing what I do, and it’s easy to disregard the common advice when it’s realized as idiocy. As an example, don’t roll your own crypto is a common phrase, but it’s clear that the people who write crypto anyway are generally incompetent or subverted. I won’t relinquish anything to a class of charlatans.
I find it important to criticize, even if only so that someone recognizes there are people who don’t do that. In all things, it’s typically in the interests of people who don’t want to improve for them to portray anything beyond their abilities as unreasonable and unachievable in practice. This isn’t meant as a personal attack, but as a more general comment.
It’s telling that the one library explicitly marked as an experiment was singled out here. The character sets library I wrote and the Gopher library are both very bare and not much to look at as well, but the experiment was chosen for criticism.
I’m to understand that Smalltalk systems at Xerox Parc could very easily make network requests as simply as sending any other message. Why shouldn’t UNIX, ostensibly file-based (despite lacking locks, record files, and everything else), allow me to easily create a network connection through the file system? I don’t care what Plan 9 is doing. No, instead I’m forced to write bindings to a C interface, BSD Sockets, designed at a university I’ll leave it at with this quote:
The obvious reason some things would be delegated to the file system and not others is because there’s no true overarching design principle. The only reason C persists is because UNIX tries its damnedest to enforce it. Now, yes, I don’t write network programs yet, because I’ve not yet built interfaces to TCP and UDP in the languages I use which satisfy me; I’ve read the IETF RFCs, and have ideals I’ll pursue, so that when I leave UNIX, only the internal side of the interfaces need to change. I also need a DNS library, and I’ve read some of the standards, but I’ve not read enough for me to be comfortable designing a library yet, with the benefit of hindsight, because I don’t want to write something which merely works, but works well, and being easily extended is part of that. In any case, when I do finish my DNS library, it will work entirely divorced from the network interfaces, operating on discrete messages, and this is particularly suited to DNS, and its various transporting mechanisms commonly used.
Now, it’s common to link to Ncurses when wanting to control a terminal device, despite effectively all modern terminals being compatible with the ECMA-48 standard, because it’s in this UNIX and Blub tradition to stack interfaces neverending, rather than build one good one. Instead, I read the relevant standards and wrote two libraries for doing this, in pure Common Lisp, and my program is better because of this; it could be called a waste of time, but I learned, albeit rather useless knowledge.
The program I write these and other libraries for is a machine code tool, because I decided I don’t like assemblers. Most people don’t think twice about assemblers, but I realized they’re stupid anachronisms. It’s in the interests of the magicians to pretend writing assembler language programs is hard, but it’s really not in any way, and my tool makes it much easier than an assembler ever can. I wrote one version of this program that worked poorly, so I reimplemented it, and now I’m working on a second rewriting, because flaws encountered with the first shouldn’t be possible at all, and a better internal representation can achieve this; I’ll rewrite it again if I realize I should. I don’t want to throw around links to my website obnoxiously, so I’ll leave it at this: http://verisimilitudes.net/2017-07-07
So, rather than encouraging people to throw themselves to the sharks, it’s better to encourage them to recognize that none of this needs to be the way it is, and to experiment. It’s actually very easy to do something not done before, if one merely ventures where others aren’t.
I want to achieve more than the current paradigm will allow, and defending Blub studies is just building an altar to false gods.
I disagree about whether you gain knowledge of useful algorithms / patterns by digging into blub. You use React as an example, but I knew that functional + caching was good just from watching 30 minutes of youtube about React, without reading any error messages. You can get core algorithms of a lot of libraries by watching talks, without wading throught the code.
Sure, I agree that reading source code is mostly not a very effective way of learning about blub and would recommend other things like talks, blog posts, docs, etc. wherever possible!