7 years of development: discipline in software engineering

June 7th 2025 marks 7 long years of development on Sandpolis, an attempt to build the ultimate remote administration/management tool for sysadmins like you and me.

For 7 years I thought about and/or worked on this project almost daily, and yet it's still nowhere near being finished, not even MVP-level. Am I the worst software developer ever to touch a keyboard?

Unlikely :)

Quite a few things happened in the last 7 years: college, a job, a wife, a house, a child, a full rewrite from Java to Rust...

... is what I would have said, but I now realize those are shallow excuses.

What's slowing down this project is a critical quality that's becoming as scarce as good taste in software engineering: discipline.

Discipline in the software engineering discipline

If your project is fueled by necessity, curiosity, or excitement alone, it's unlikely to reach 100%.

That's primarily what motivated these last 7 years of development on Sandpolis. I'd have a satisfying streak of progress in some interesting area until I got stuck. And since solving hard problems is hard, I'd jump to somewhere else and repeat.

Unlike in other crafts, in software engineering, it truly is possible to build the roof before you build the walls. And, inconveniently, it's also likely that floating roof won't even line up with the walls once they're finally done (ever happened to anyone in Minecraft?).

Nevertheless, at one point in 2019, I had an application that technically worked. The server side was entirely Java (~50K lines) with a cute little iOS frontend app in Swift (~10K lines). Let's admire it for a second:

Back then, I thought Java was quite alright. Modern features made the language decently comfortable. I don't remember exactly what prompted it, but at some point, I decided to rewrite everything in Rust - probably a case of cosmic rays in my brain, much like in my code.

So, while my curiosity took me down an exciting new path with the most hyped language of the decade, I no longer had a working application (a grievous mistake). Rewrites are extremely costly and rarely the right answer. By that point, I was experienced enough to know better, but I let the decision make itself instead of taking a strategic path.

Since this was just a side project, I usually worked on the fun stuff over the hard stuff. After a while, what you're left with is a project full of holes. Parts of it are certainly nicely done, but if it's not possible to unite those pieces into a working whole, you can't get off the launchpad.

Discipline is what unites a project into a working whole. It's what allows you to solve the hard problems. It's what keeps you on a path going forward when temptations arise.

Surely there are some people out there with a side project that they enjoy coding just for the sake of coding, but I always aspired to make my programs actually do something useful. Maybe even something that no one else has ever done before.

AI development tools

Now that AI development tools are here to stay, discipline in software engineering is more important than ever. Consider these related questions:

• Why remember directions when the GPS is always right?
• Why learn long division when everyone has a calculator within arms reach?
• Why practice handwriting when we type almost everything?
• ...
• Why write mundane code when the AI can do it in seconds?

Doing all of these things leads to understanding (sometimes of things you didn't even intend). AI development tools are shortcuts that can rob you of understanding if you're not careful.

Discipline is what causes you to do the things that lead to long-term understanding, even when a shortcut is right in front of you.

Of course that doesn't mean you should always do things the "hard way" to maximize gained understanding. You get diminishing returns on understanding as you repeat a task more and more. Now that I've drilled long division problems enough times to "get it", I can let the computer do it a billion times more efficiently than I can.

Similarly, there are many paths in programming that I've trodden many times. By this point, I understand these things pretty well. So, I let the AI take over when the experience gain is less valuable than the time and effort I might save.

Identifying these tradeoffs is a skill we have to develop.

How to practice discipline when coding

In general, do the highest priority thing until it's as close to 100% done as possible, even when it's not the most enjoyable part.

Don't let problems on the critical path leave your L1 cache

For example, the data model in Sandpolis is critical for everything else to work. When I got stuck on this hard problem early on, I diverted my attention to other aspects that weren't as important like how user passwords are handled to avoid hash shucking attacks, for example.

Sometimes when I'm stuck on a problem, I can work on something else in the meantime and a solution to my original problem suddenly presents itself, as if my unconscious brain was thinking about it the whole time. Once a problem leaves your L1 cache because you haven't thought about it in a while, you're no longer making progress on it. In fact, it's the opposite of progress because you slowly start to lose context which will take effort to regain.

That's why it's best to mostly stay on the critical path than to hop around. It's OK to do some side quests here and there, but don't let them subvert the main questline.

Minimize the length of time the software is broken

To make real improvements in software, you usually have to break things first. The longer it takes to get your application compiling or running again, the more costly the improvement becomes. Once a change starts taking weeks, the full context becomes hard to keep in your L1 cache and the probability of refuctoring increases.

I'm a repeat offender when it comes to this. I've made many fundamental improvements very low in the stack that affects basically everything, but I've failed to propagate that change completely for a long time, usually due to some difficult edge case that I didn't think about initially.

As a corollary, don't get tempted to rewrite so easily. Rewriting is the ultimate form of breaking your software. My favorite discussion on why rewriting is bad is Spolsky's Things You Should Never Do. Part I.

Pay now or pay later

Usually, both in software and in life, it's better to pay now rather than pay later. It's going to cost much more time and energy to change a function prototype after it has hundreds of call sites than when it has none or just a few. In other words, do it right or do it again.

It takes experience to predict what's going to be worth paying your time into in the future and what will be a waste when it becomes quickly irrelevant. Of course, it's still important to have a healthy amount of YAGNI (you ain't gonna need it) to stay working on the things that actually matter.

As an aside, you can combine "pay later" with the witty "later is never" quip to derive: "pay never", which sounds like a sweet deal until you realize that it turns your software into concrete over time - impossible to change without a jackhammer.

Conclusion

Although Sandpolis could still very well end up a failure, I recognize there's nothing else to blame other than myself. The only thing worse than failing is failing and blaming something else for it.