How to build closed-source software

Teams building closed-source software benefit from working in a high-coordination, high-obligation environment. They can plan, delegate, and execute tasks efficiently by relying on the structure and continuity provided by their organization. What software engineering practices will make such a team as effective as possible? And how do those practices differ from the ones used in open-source software?

In most cases, closed-source development is optimized with a combination of small PRs, fast code reviews, and automated testing. These three practices form a reciprocal frame; they are mutually-supporting and reinforcing. Together, they allow a coordinated team to build software both quickly and flexibly.

Small PRs are easier to write correctly and review thoroughly than large ones. They reduce the size and frequency of merge conflicts but require fast reviews to keep throughput high. Fast reviews rely on automated testing: to review a higher volume of changes efficiently, reviewers need automatic assurance of the code’s logical validity. Tests must then be provided along with every change to support the overall process.

The absence of any one of these practices makes the others unsustainable. Reviewers cannot be expected to review large changes or changes without tests quickly since both require significant time investment which interrupts the reviewer’s own work. Pragmatically, reviewers also know that large reviews are more burdensome and will therefore be inclined to procrastinate them. Slow reviews then incentivize authors to make PRs as large as possible in order to limit the impact of the review’s overhead.

This synergistic trifecta of practices is uniquely suited to closed-source development. In open-source projects, the relative lack of coordination and obligation between contributors and maintainers typically makes larger changes more efficient. Open-source maintainers are incentivized to merge completed features (which inherently means larger PRs) since they cannot count on contributors to follow up on in-progress work. Likewise, contributors cannot typically count on prompt reviews and are therefore motivated to package work in larger units. In contrast, a corporate environment creates a coordination structure that can ensure continued attention to a long-runing task. It is perfectly fine to break a large feature up into dozens of smaller changes (utilizing feature flags where necessary to hide new functionality).

Besides the first-order benefit of improved development speed, these practices create second-order improvements in build health and release cadence. A commitment to automated testing means that releases can be validated programmatically and shipped without human oversight. With small PRs, this means that new code can be deployed to production many times per day. More incremental releases reduce the impact of any individual change, so when regressions do happen they are less severe. Small PRs are also easier to roll back than larger ones, which allows build breakages and production issues to be addressed quickly and easily (and at the extreme, in an automated way).

It is an unfortunate consequence of Git and GitHub’s success that many closed-source teams naturally gravitate towards the practices encouraged by those tools. Git’s federated model is awesome for open-source, but makes it hard for closed-source teams to work as productively as possible. While the practices above don’t mesh perfectly with Git’s model, they can still work (and work well with additional tooling). In other areas, Git’s model creates more significant obstacles for closed-source teams.

For example, Git struggles with large repository sizes. This encourages a many-repo (vs. monorepo) setup and a reliance on package managers to install dependencies. For closed-source teams, a monorepo is (at least in principle) a superior solution because it enables easy code sharing within an organization and eliminates an entire class of cross-repo dependency and deployment issues. The ability to check all dependencies into source control encourages healthier dependency practices but balloons repo size, creating Git performance problems for teams whose first-party codebase is otherwise too small to be a problem.

Despite all of this, a healthy and growing ecosystem of tooling provides some options for teams who want to fight the current of Git-based tooling. Jujutsu is a new Git-based VCS that provides some helpful new functionality that would otherwise be at odds with Git’s model. On the repository side, tools like Sapling are emerging from large companies with the resources to scale Git monorepos across tens of thousands of developers. With developer sentiment slowly-but-surely turning against GitHub seven years post-acquisition, we may even eventually see a rival platform provide a first-class experience for newer and closed-source-friendly ways of working.