For those of us who have ever worked on modernizing a legacy system, you know it can feel as much like archeology as computer science. We dig through old documents, code comments, and interview whoever we can find from when it was built to piece it all together. We’re not asking how the system works, we want to know why it works the way it does. As systems age two things happen at the same time: the system improves (but usually gets more complex), and fewer people remain who were there when it was started. As those folks move on, maintenance and modernization gets much more risky and expensive.

We have to step back a minute to understand how this phenomenon unfolds. You see, when people spend a lot of time together, they develop a sense of what the other person knows. This is the foundation behind a theoretical model developed by Daniel Wegner in the mid-80’s. An excerpt from a paper he co-wrote in 1991 illustrates the idea:

One partner may not know where to find candles around the house, for instance, but may still be able to find them in a blackout by asking the other partner where the candles are. […] Such knowledge of one another’s memory areas takes time and practice to develop, but the result is that close couples have an implicit structure for carrying out the pair’s memory tasks. With this structure in place, couples in close relationships have a transactive memory that is greater than either of their individual memories.¹

This happens at work as well as at home. We develop this same sense with our colleagues. When you scale this out to larger groups, you start to get into a whole area of study called Organizational Cognition. Work in that field has been steadily expanding since the early 90’s when the US Academy of Management established a formal Interest Group for it.². A subset of this field deals specifically with the concept of Organizational Memory.

Tacit Knowledge

When I wrote my graduate thesis on the impacts of turnover on software teams, this was one of the frameworks that help us understand how turnover hurts knowledge work teams. Some amount of the knowledge of the group gets written down as formal documentation, and for teams that produce artifacts like databases or software code, still more of that knowledge gets transferred into the code. However, it’s impossible to capture everything a person knows. Therefore, when a person leaves a company we know there is some amount of tacit knowledge that goes with them and cannot be replaced or recovered.

What is not clear yet is the size of the gap between what we can retrieve from those artifacts and what is actually lost, and what LLMs could change about that.

What does our Code Preserve?

Way back in 2012 before LLMs were commonly available, a couple of researchers found they could use basic NLP to extract information about the domain areas of a program by parsing the source code directly.³ Since the explosion of LLMs, research in this area has been rapidly expanding. The most recent and comprehensive survey I found on the topic had 173 references in it.⁴

While it’s amazing that an LLM can now read most types of source code (they struggle with some languages) and tell us how the code works, I think the tacit knowledge lost by turnover usually has more to do with why the code works the way it does. Business rules become fossilized in source code. Policies become if statements, regulations become validation logic, and historical compromises become code branches and special cases.

Let’s consider some representative variable names:

• DeferredRevenuePreASC606
• legacyTaxMultiplierNYPre2018
• manualOverrideForHospitalAccounts
• isMaterialityException

Variable names like these have historical context. They may reflect aspects of the state of risk tolerance or the interpretation of accounting standards. Characteristics like these are not static. Software code becomes a tiny window into the climate of when it was written. Like studying ice core samples or diaries.⁵

Procedural Memory in Practice

Consider this block of code:

if (isHealthcareClient && contractDate < ACA_CUTOFF) {
    applyLegacyBillingRules();
}

From this short snippet we learn quite a bit:

• Healthcare customers are special
• Contracts around a cutoff date behave differently
• The business lived through a regulatory change
• Backward compatibility mattered more than elegance

The principle here is that organizational memories often survive as symbols in software code or other artifacts, but the meaning behind those symbols often cannot be gleaned from the artifact alone. They are memories that are preserved procedurally, but not semantically.

The Value of Historical Context

That matters more than you might think. When it comes time to inevitably replace that code snippet above, the new team will have questions about why the old code was written that way. They will look up the names of long departed engineers in vain and be left with uncertainty and doubt. Did we apply legacy rules that way because other systems also had to call that same block of code at the time? Did we use those if statements because of regulatory rules, or because of system performance issues that had to be mitigated?

Studies support the importance of historical context on the modernization process. One study showed 37% faster implementation and 42% fewer defects when a formal knowledge transfer framework was followed.⁶ An IBM case study of modernizing a Cobol system called out knowledge of the old system as a critical component of success.⁷ So did a multi-case study of legacy systems in European public administration tools.⁸ There are many more references like these available.

So knowing why systems work a certain way is not just a nice-to-have. I’ve already written about the practical limitations of trying to “teach” an LLM with current tools. Current-gen LLMs are great at explaining how most software code works, but so far it doesn’t seem possible to get much of the why.

The Value of a Software Engineer

I think about this question a lot. The industry tends to think of engineers primarily as code production machines. That’s why there was so much hype early on about LLMs replacing software developers - they both write code, so what’s the difference? Managers usually recognize that part of an engineer’s value is the domain areas they become experts on. Indeed, at least one study found that it was just as important to know where to find expertise on a software team as the expertise itself.⁹ I’d argue, though, that we don’t hire Senior Engineers at higher salaries than Jr. levels because of their prior domain knowledge - the value proposition starts at the type, amount, and quality of code they can write. The promise of how quickly they can learn and teach has historically been secondary.

As code production becomes cheaper thanks to LLMs, the scarce skill will be the ability to bind technical systems to shared organizational meaning. That work is not automation-resistant because it is complex; it is automation-resistant because it is contextual, historical, and social. In this way, modernization is not just a technical process, it’s an act of historical interpretation. As with archeology, once the context is lost we’re stuck guessing about what the artifacts are telling us.