Get With the Program

My first experience with programming occurred at a Cub Scout meeting. To build our visions of the future, our den leader brought in parents to talk about the work that they did. My father came in with a set of 3×5 cards. We sat in a circle and he handed each of us a card. Following the instructions on the card, we were able to perform an operation of binary logic on two numbers. Believe me, we didn’t have a clue what we were doing, but the insights we gained into the nature of computing sure made us feel smart.

Perhaps forty-five years later, I now go by the airy title of “Software Engineer.” This is actually a fraud. By law, “engineer” is a designation reserved for technologists that have passed a rigorous assessment of capability. No such test exists for software. As one consequence, the lack of reliable practices for predicting cost and schedule for software development is a scandal. At one point, surveys of software projects reported that almost half failed.

Given the importance of software, a number of serious efforts have been made to attack the problem, spanning the full spectrum from machine design all the way up to the executive suites. Methods and tools, team dynamics, and project management were all brought under scrutiny and overhauled. Nothing has worked.

Looking back at that Cub Scout meeting, I must admit to some bemusement, because in fact what goes on in computers is exactly what we did with our 3×5 cards, except on a much larger scale and at incomprehensibly faster pace. It’s just a bunch of 0’s and 1’s flying around, with some conversion going on to allow people to digest the results of the calculation. Why should it be so hard to get a handle on the process?

Among the reasons are those that we might hope would be resolved in the foreseeable future. First is the enormously rapid pace of change in the industry. Moore’s Law meant that the difficulty of the problems that we could address grew by a factor of two every eighteen months over a span of thirty years – compounding the growth, that becomes a million times! In data storage and access, the factors are even larger. No other technology discipline comes even close to boasting these kinds of numbers. The rapid pace of change means that what was important and interesting five years ago is meaningless today. So how can we certify practices at the forefront of technology?

Of course, as has been reported in the technology press, Moore’s Law has progressed to the extent that fundamental laws of physics will impose limits on computing power in the foreseeable future. That may allow the test designers some time to catch up with progress.

A second cause for hope is that pattern processing algorithms are becoming sophisticated enough to effectively interpret human behavior. Some of those algorithms are used for gesture and speech recognition. Some are used to recognize our habits and track our schedule, allowing digital assistants to interpret our gestures and speech as specific instructions appropriate to the context. The near-term impact of this capability will be that more and more of us will become programmers, because the barrier to entry – learning how to “speak” languages understood by computers – will be removed.

But there are certain aspects of computer programming that will remain intractable, the foremost of them being trying to visualize clearly the outcome of a program’s execution.

My father Karl was given a consulting assignment in the ‘60s at a large defense contractor. The president had committed to automation of production operations, and the project was nearing completion. But while the developers were confident that the software would perform as specified, nobody had performed a comprehensive assessment of the impact on current operations. That was the task presented to Karl.

Of particular concern was the production scheduling office. The process transformed his understanding of programming. The walls of the large room were covered with clipboards hanging on hooks. Colored sheets were used to identify time-sensitive assemblies. Written manuals defined the procedures for movement of the clipboards on the walls. In many respects, it resembled the process he guided the Cub Scouts through a few years later. It was a program, and there was no way that the production automation system would be able to replicate its level of sophistication. When presented with the assessment, the president chose to force the scheduling team to use the new software, and production collapsed.

Fundamentally, software is just manipulations of zeros and ones. It is useful only to the extent that we can use the zeros and ones to represent the real world. For most of the history of computing, that meant that the principle role of the software developer was to translate the language of the human expert into terms that could be executed by a computer. When that knowledge was packaged and distributed, it meant that every local expert was replaced by a world’s expert, improving the decisions made by others, who then commissioned software solutions that stimulated improvements elsewhere, all at an ever increasing pace that frustrated the attempts by managers to actually control the end result.

What Karl realized in the scheduling office was that programs exist all around us. Some of them we call “habits”, others we call “software”, others we call “symphonic scores.” To realize the intentions of the people that commission the creation of the program, the programmer must describe the actions to be taken in terms understood by the executor of the program – whether a mechanic, a computer or a musician. What is common to each of those situations, however, is the fact that the program only specifies a pattern of execution. A spark plug can be changed in a driveway or in a repair shop. Microsoft Word can be used to write a love letter or a subpoena. A symphony can be performed on period or modern instruments. The result in each case may be subtly or grossly different, but the pattern of the execution is the same.

And there was no good way of representing such patterns. It wasn’t science, and it wasn’t engineering. “Programming” was as good a word as any to use.

Being Boxed In

The management guru fad of the ‘80s wound itself up just at the turn of the millennium. This may have been due, in part, to the rise of information technology that shifted analytical emphasis away from the personality of the leader toward those directly involved in creating value. That was evident in the last book I read on leadership, which warned managers that knowledge workers would simply walk away from organizations that did not adopt collaborative management strategies.

I find myself in such a situation at this point. When I started my current job, I sat through meetings that devolved into pitched shouting matches. Such altercations were a daily event between some of my peers. When I intervened with the HR staff to bring this to an end, I exposed patterns that dated back to the formation of the company by people that used competition to maintain control of knowledge workers. Circulating to upper management an essay on triangulation and its consequences did not make me a popular person. When I put a copy of “Breaking The Fear Barrier” on my desk, they just stopped coming into my office.

They kept me on because I sat in my office and did what I do best: create a garden in an overgrown software jungle. The application that I manage, which has always been a critical part of the user’s experience of our products, has gone from being something hidden until the sale is complete to an essential part of the sales process. At the same time I have created component libraries and leveraged them to build new applications, algorithms and regression test suites. Having gone eight years without a pay raise, however, it’s time for me to move on to a place that understands and appreciates the principles that I use to understand the needs of my customers and create success for them.

So I’ve been working on my resume and trying to visualize the kind of place that I would find success in. I have a certain sympathy for Microsoft, and went out to the Research site on Saturday to see whether I could put up a resume. The site indicates that researchers are expected to be recognized experts in their field with a substantial body of publications. Well, shoot, most of my career has been spent in top secret facilities or in small companies that use trade secrets to protect their intellectual property. Microsoft Research does have a category for applications developers, but that link led me through to the main Microsoft site.

And so I find myself where I was eight years ago: a really smart guy who was told by an honest recruiter that “if I knew someone starting an R&D program, you’re the guy I would recommend, but you just don’t market well as either a manager or a software developer.” Where most developers focus on coding efficiency and the arcane syntax, most of my effort is invested in understanding the application domain so that I write the right code. Whether it’s produced in JavaScript or C++ or C# is pretty irrelevant – once I’ve established the design, I evaluate and select an implementation platform, read a book, start writing code, and use the web to find answers to arcane syntax questions when they come up.

Most hiring managers don’t have a clue how to evaluate a person like me. They want a known quantity – a specific skill set that will allow someone to come in and deliver value on day one. They’re not willing to take a risk on learning, and don’t know how to evaluate that capability by probing past experience to determine whether a resume represents individual contribution, or simply takes credit for work done by others.

So I’m resolved to find my own opportunities this time – searching the web for job openings and pushing my resume through directly. I’m making culture an explicit issue by declaring that my loyalties lie first and foremost with the customer.

To those of you who have been following this blog, this probably resonates. I’m hard to pin down because I don’t specialize narrowly. Rather, I examine relationships (personal, institutional and intellectual), and advocate for deepening them where others pick a side. But, damn it, I don’t want to be in a conveniently packaged box that can be moved easily around (itinerancy being the biggest problem faced by software companies). I want a home, and I’m willing to meet people where they are.