I’ve written several partial versions of this post in various emails and Slack posts, and finally decided I should just put it on the blog.
The tech landscape is complex and picking the right tool is hard, but the vast majority of problems can be solved in a “good enough” way using a wide variety of tools. The best choice is usually the one you know well already. So I tend to think most developers should have a “default tech stack” that they use for most things, only switching when the problem constraints or early experience dictate otherwise.
And here’s mine! This is the list of tools I usually start with, and use most frequently in production. I will frequently adjust some part of this list for any given project, but I find these are usually useful choices. I don’t expect any of these to be very surprising, but I think there’s some value in writing them down.
Programming language: Python. It’s only about the third best language for anything, but it also tends to be the third best language for everything. I’ll sometimes swap to another language for production, but my prototype is nearly always in Python.
Data file format: SQLite. It has huge benefits once you start doing anything even moderately complex, and most languages have SQLite support so I don’t need to write a parser. (Notable exception: for numerical data I will often reach directly for HDF5.)
Config file format: YAML. I like TOML better but YAML is well-supported everywhere. Again, I don’t like writing parsers. ?
Database: PostgreSQL. Easy to deploy, easy to use, works really well.
Internal RPC: gRPC. Being able to pass around neatly serialized protobufs for everything is really nice.
External RPC: If I’m exposing it to the world I’ll use a REST API with JSON documents like everyone else. It’s hard to convince other people to pay attention to your protobuf defs.
HTTP framework: Bottle. It’s a simple, easy-to-use Python framework that I find very handy for REST APIs. If I had to write more complex web apps frequently I might use something more fully featured, but for my usual needs Bottle is plenty.
Booting cloud hosts: Terraform. It’s the standard at this point and I find it very handy for defining some set of cloud resources and then instantiating them. I do find that the provider-specific tools are often better in some respects, but I work on projects on multiple providers enough that I prefer to stay in one tool.
Booting bare-metal hosts: MAAS. It’s easy to deploy and I can generally talk someone through how to use it. There’s better support for building your own images with Packer these days too, which removes my biggest pain point. I wish it had better support for diskless booting, though.
Configuration management: Ansible. The wealth of modules, push model, and lack of infrastructure to set up make it a really excellent deployment and configuration tool. Though I do find that over a certain scale you have to restart re-inventing the pull model.
Linux distribution: Ubuntu LTS. I used to strongly prefer CentOS, but (a) the CentOS project direction has changed unexpectedly; and (b) I’ve done enough work on data science infra lately that I appreciate Ubuntu’s better support in that area. (I also support fewer commercial apps than I used to!) I still like RPM build tools better than DEB ones, but I’ll live.
Single-host service deployment: I’ll usually package my service as an OS package (DEB or RPM), write a unit file, and just install it. Sometimes the old ways are still the best.
Multi-host service deployment: Kubernetes. I have concerns about it getting over-complex, but like Excel, most people use at least some of that complexity — what varies is which parts! I’ll still usually build an OS package for my app, but for k8s I’ll also install it in a container image for deployment.