17. Promise Theory

There's an unspoken bond between researchers. You don't criticize the tenets of the field in which you work; you don't try to do anything to undermine the fragile status quo. It's sometimes a hard promise to keep. It's the opposite of what scientists should do, if they keep their promise to science, but it's the human face of it. If you break this covenant, you will no longer be welcome within the tribe.

This thinking contrasts with the self-image that scientists have of themselves as liberal thinkers who can entertain new any ideas. It couldn't be farther from the truth. "New" has nothing to do with it. Like all the cliques of society, academia is most often about bending the knee and kissing the ring of some mafia don to acquiesce before the alpha.

Promise Theory is such an idea I had, which became a story about my running away from this orthodoxy, in search of my own peace of mind.

I mentioned my dalliance with graph theory. Over the years, I had use for its methods on many occasions and in many different ways. It's one thing to study existing graphs and networks, but where do they come from? What feature of these connected spaces could join together and unify all the independent points. And how, specifically, could I use them to connect all the ideas I'd pulled together about computers so far? The list was long: autonomy of agents, disagreements about semantics, dynamics, policy, change, fixed points, flows, dynamics, forces, and equilibria. Something had to connect all these pieces in a simple way.

What eventually came out of five years of trying to capture the ideas of computer behaviour, across a variety of scales, was a simple realization: the dominant quantitative methods of physics were incomplete — and the thing missing was the role of semantics. This is where graphs and computer science could help.

The years following the millennium were a cynical time. I'd struggled to get certain papers published for three or four years because they didn't fit into the existing cliques. Some established professors in the US, in particular, did everything to oppose the publication of work that didn't defer to them.

As my research was unfolding, some special strategic government funds became available for the Masters degree that I'd started. The university was able to create a professorship in computer science, with the aim of working towards awarding students PhD status in this area.

Norway was in the process of upgrading its staff to put them on a par with the international system. They needed professors. It became policy that researchers who were Associate Professors could upgrade themselves by applying to a committee of colleagues (not partial at all!) to judge themselves worthy of the title of full professor. This would be an advantage in the European system, where a PhD programme needed a full professor to support it. In this new scheme of upgrading, the basic idea reduced to this: if you had published approximately 3 PhD's worth of material, you qualified to be full professor!

Having made some progress in a unique field, the university encouraged me to reapply under this new remit, but the local committee refused to consider my application, because half my work was in physics. They couldn't classify me as physics and they couldn't classify me as computer science. My Norwegian colleague who had come after me and done far less did qualify!

Although I was initially rejected by the Norwegian computer science board, because of my cross-disciplinary background, in this specialist area there was no one who could argue with me. I was, after all, "the only show in town".

As luck would have it, my efforts in establishing a "completely new" academic field, and publishing at least some significant research, as well as starting an International Masters Degree did warrant some kind of merit. The university acquired some funds as part of a satsingsområde or a focus area for the department. That came with some money and a professorship appointment specific to that area. It wasn't a professorship in the new model, but in the old model. So this time it would have to be an international committee of peers who decided whether the applicant was worthy or not. Now my luck was in. Having defined the whole field of research, I was the only possible applicant for the job! I could define it to be a cross disciplinary field and suddenly I could stand a chance at being judged accordingly.

The university bent over backwards to accommodate an appointment based on the old system of professorship, in which professors are nominated by a jury of their peers. I went through a gruelling international committee process of printing out a sample of my published papers and providing four copies of every book and publication I'd stood for. They say that a professorship is supposed to be the equivalent to about three PhDs, in terms of publications. That's quite a lot of material to print and send to a committee.

The international committee consisted of a Norwegian professor, Pål Spilling, who had brought the Internet to Norway as a physicist, and was thus something of a cross disciplinary man himself, Stephanie Forrest from Santa Fe Institute and University of New Mexico (whom I'd whiffed rumours of through Computer Immunology), and one other person I didn't know. It took several years, but finally I was accepted as a Professor (not of physics but) of Computer Science. It seemed like a hollow victory, but I was happy for a modest pay increase and a bigger office. For this, I got two PhD positions allocated, and we hired two PhD students under my supervision.

Our initial successes, with the educational side of Network and System Administration, led to my starting a Workshop at the LISA conference — to discuss how computer engineering should be taught. I'd written a definitive (if not wildly best selling) textbook, which received limited interest, so there was that. Yet, there was a backlash against the academization of the field amongst practitioners, who didn't want to see their hard-won practical knowledge devalued by some smalmy academic pipsqueak. Thankfully, there was a also a small core of people who welcomed an academic legitimacy. It made sense for there to be some international standardization; a degree is international in scope, and while there were plenty of people interested in the topic, no one else had yet managed to push through a programme at a university at Masters level. Ours was the first Masters level degree in the world in this area.

The workshop led to a lot of interesting discussions, but little standardization. That was fine by me. After one workshop, I was approached by a group from Amsterdam University. They had the idea to run a similar but shorter programme. I received an email from Jan Bergstra, who was a professor in Computer Science and Logic at the University of Amsterdam, and had been given the task of establishing the degree. He invited me to the Netherlands to present our course and join an advisory committee for their own proposal. He laughed, "I looked up people doing research into System Administration on the Internet, and Mark Burgess was the only show in town."

The idea for the degree was the brainchild of Karst Koymans, a friendly and charming lecturer who had learned of me through the LISA conferences. He began by warning me that he thought I wouldn't get along with Jan. Clearly the two of them didn't see eye to eye. But Jan and Karst both welcomed me into their homes, and I met their charming families. I liked both of them very much. In the end, it was Jan that I became closer to. Karst was offended that I supported the position of having theoretical content in the university degree. After all, I had put heart and soul into trying to create a field of study around computing as a service, and universities did not award degrees as apprenticeships. No matter what the informal community wanted, universities had their own set of standards for awarding degrees.

Like the wizard practitioners from the LISA conference, Karst quietly despised the idea of a theoretical basis for the field. You were supposed to fly by the seat of your pants. It was an apprenticeship, a mentorship, not a degree course. Their programme leader was clearly having similar conflicts in Amsterdam, but they wanted to have the degree. There were willing students and willing lecturers.

Jan and I became great friends. We went on the develop my idea about Promise Theory. He has been one of the few researchers that I've continued to collaborate with over the years. Karst — I think, disappointed — eventually pulled away from me and my efforts at the LISA Workshops. I stopped hosting these after a few years and left it for others to pick up. The populists were people who enjoyed the magic of system administration. They didn't want to see that taken away by some killjoy mathematicians.

Meeting Jan proved to be an important turn. Just as I was labouring to flesh out some of the formal requirements for a theory of systems in terms of agents, in a rough set of notes based on the idea of graphs of promises, Jan Bergstra came to visit the department at Oslo. We discussed some of my ideas, and I showed him the notes I'd been writing up on promise graphs. He was intrigued, but graphs were not his language. Jan was an expert in logic, and was sympathetic to my failures to find usefulness in symbolic logics. Indeed, he handed me a book (a "Handbook") on Process Algebras that he had co-edited. It was a thousand pages on the topic of networks and logics for describing digital processes — from Petri Nets to Pi Calculus. He opened a doorway to a new world of research on the modelling processes, which I had (almost) completely ignorant of. "In fifty years of research into Modal Logic, the field has really accomplished nothing," he told me.

Although I didn't fully appreciate it then, my ideas about promises were just one side of a much broader and deeper theory of processes. It would eventually prove to be related not only to the process algebras but also to the strange behaviours in Quantum Mechanics, and the dichotomy between evolution and measurement. It's a long story for another occasion.

At this stage, I had in mind only some basic principles about the importance of treating every agent in a system as being autonomous. Autonomous meant two things: causally independent, and independent in all decisions. Agents couldn't be forced by other agents. This was something that had come straight out of my experiences with networks at the university — where every kid is special, and can't be expected to play by anyone's rules. Such agents did coexist and formed stable computer societies, in spite.

I also had some ideas about a calculus or an algebra to express the outcomes of decisions and behaviours, i.e. the intentions of the agents, in terms of sets. Agents, things, people — or whatever we want to call them — are basically autonomous in their natural state, and that means one can't just impose on them with a guarantee of success. In fact, there was a field of study called Multi-Agent Systems, which is quite large. It's about robots and control systems, but amazingly not about autonomous systems. Computer science wants everything to be about control: guarantees and requirements — through the imposition of rules and obligations. But trying to force them to obey commands is often an exercise in futility. People, things, and software may be unable or unwilling (or both) to comply. I knew that the voluntary nature of cooperation meant that only voluntary choices were important.

This all sounds like a simple truth, but in philosophical and engineering terms, it was still pure heresy. The overwhelming bulk of literature in philosophy was (and still is) based on the total primacy of obligations. Deontic logic is the study of what MUST BE. Agents MUST do was they are commanded. The same is true of the Newtonian era physics of particles and rigid bodies acting under forces.

To me, however, the idea of imposition felt like nonsense, and the whole field of Multi-Agent systems and control theory for such complex networks felt misguided. Tradition be damned. Being able to influence another agent with certainty would violate locality in physics, and it wouldn't agree with the contrary human behaviours I know in any sense either. Control was merely wishful thinking — probably originating in the Biblical ideas of Moses and the amous commandments from god.

Jan suggested that we should try to collaborate on producing an academic book on the research. This fitted well with the EMANICS Europrean Network of Excellence aims too. We proposed the Elsevier Handbook of Network and System Administration, which became one of my responsibilities in the network. The collaborators each wrote chapters for the book, and I edited it together into a massive tome — the likes of which no human hand could ever hold without mechanical assistance.

The field of Network and System Administration had two related parts, and yet there was a gulf between them. The EMANICS EU project experience taught me a simple truth. The Networking industry was the domain of the telecoms industry. The culture of telecoms was very different from that of computer management. These two parts needed each other vitally, but both lived separate lives. Networking was dominated by industry types. They saw every problem as a database problem. There was extensive use of semantic webs or knowledge graphs using RDF and OWL (the web ontology language) in this field. The concept of ontologies was seen as a way to get beyond the simplistic relational data models of SNMP/MIB and CIM (The Common Information Model). I got to know a researcher called John Strassner who championed this idea as a way to get to a logical model of management. I loved John, but I thought the idea was doomed to failure. At the time, there were 120,000 different device models in MIB, but only 2000 were ever in use. This amortization of effort was the direct result of demanding strict logical types for every kind of entity in a system. It was unsustainable.

In my mind, I believed that there was a better way of managing systems without detailed data model specifications — something more like what I'd designed intuitively into CFEngine. In practice, all that was needed was to be able to discuss a few relevant behavioural properties of the agents in a problem. All these things, their capabilities, their intrinsic properties, their behaviours and expectations, could all be summarized by arrows in a network. The arrows weren't forces or wires; they weren't commands or rules. The only word I could come up with that approximately summarized their role was that they were promises to each other.

Computer Science is stuck on its model of determinism. It's a belief that comes from its mathematical heritage. The language I picked up on was graph theory, but, in its most basic form, that didn't cover the semantics of IT either. There had to be something else to bridge the world of dynamics with semantics. The answer seemed to lie in melding dynamics with Information Theory. Information Theory also avoids the issue of semantics, as noted in the essay by Weaver in the original Shannon and Weaver book. But functional meaning could be built around the alphabet of symbols as in the Chomsky machine linguistic hierarchy. If that all sounds a bit technical, then sure enough it is, and subtle to boot. Often the simplest of ideas are the hardest to imagine. That's the same story in biology, but there there is no formalization except in bioinformatics, which is too narrow.

I looked at all of these things, and poked around — and ended up with the idea that the arrows between the different agents in a network, whether hardware, software, or even human were essentially promises. This confused a few people. They couldn't see how a machine could make a promise. After all a machine doesn't have free will. But that wasn't the point. Who came up with the intention was irrelevant. A table can promise to be strong, no matter whose idea it was originally. What mattered was who or what upheld that intention, as an independent agent, whether by proxy or by free will. Never mind obligations.

Indeed, I wanted to eliminate the idea of such obligations entirely, but Jan later pushed for the need for impositions as a form of obligation (he even suggested the word). Understanding impositions in depth turned out to be a crucial aspect of applying Promise Theory to both physics and to social interactions.

After thinking it through, I agreed that this was an important thing to address, and I was able to show how many voluntary promises were actually needed to support the idea of obligations, using the algebra I'd developed. Jan was impressed by this tiny bit of basic algebra. We decided that there was no obvious logic to be found around promises, but that promises were something else. Something more interesting and flexible.

In the literature, even promises were considered to be a stepping stone to obligations — the promise induces the promissory obligation to keep the promise. Although initially sceptical himself, as a logician, Jan quickly realized that there was something completely radical about the idea and he did an about turn. After thinking through the concept of autonomy and voluntary cooperation, he decided that promises were something completely different to obligations.

Jan Bergstra took one look at my mathematical presentation and persuaded me to write about the idea, before diving into an obscure formalization. That was good advice for selling it, as it turned out. Mathematics is not anyone's strength in networking or computer engineering. I stripped down my notes to bare minimum, and outfitted it with an introduction about the failures of logic, as pointed out by Kripke and his many worlds arguments. It was a long shot, but I didn't have much time to write a paper for the conference.

I'd become used to having ideas rejected out of hand by colleagues, but with Promise Theory there was an instant interest from the conference attendees. From the moment I presented it at a DSOM Conference in Barcelona, in 2005 it garnered interest. In my slides, I slipped in a tantalizing phrase at the last moment: I decided to call the model "Promise Theory". As I said it, I could see people nodding. Someone even said "I like that!" On a spontaneous whim, I decided to call it Promise Theory and it stuck.

Unexpectedly, the paper at DSOM 2005 was voted the best paper and most important idea of the conference in an informal vote prompted by German researcher Jürgen Schönwalder. Bert Wijnen, a Dutch attendee pointed immediately to my talk. Even the top dog from IBM was said he like the idea — high praise indeed.

Later, Jan surprised me by announcing that he considered the concept of a promise (in the way I'd defined it) to be a more elementary concept than logic or obligation. This is the opposite — upside down — from what classical philosophers believed. Before our discussions, obligation had been the only story in town about intentionality. There was John Searle, a famous philosopher, trying to break away from it, but his writing is wrapped up too much in philosophical argumentation and the idea of speech acts in language. He wasn't able to cut through the words with the precision of a mathematical model.

Jan and I had many adventures driving around the Netherlands, and hanging out at cafes as we discussed everything we could think of. Did you know the Netherlands has a pyramid? The story of Promise Theory is still unfolding, but we can be happy that these details contributed to a new way of understanding a variety of problems from first principles.

Something about the voluntary nature of cooperation appealed both to people's intuition and to their moral beliefs — but its first application was in security.

By this time, CFEngine had become pretty well known all over the world. Its magical abilities gave a direct example of how promises could work, through an assembly of completely independent software agents. Just because agents are naturally independent, doesn't mean they can't forego their independence to cooperate — that's where policy comes in. The Promise Theory at least explained the ingredients necessary for that to happen.

For physicists, the idea of autonomy occupies a strange place too. The idea is firmly ensconced in the notion of locality in physics. But, in a blatant contradiction, locality is associated with continuum physics of "manifolds and vector spaces", where physical effects are continuously connected with rigid determinism, veiled by some probabilistic seasoning. Physics has struggled with the inherent contradictions of continuum versus locality for a century.

In a similar way, computer science was stuck in the analogous mindspace of push-button control thinking. Virtually all computer to computer interactions taught in college assume the ballistic model of information transfer. If you can hit a device with a command, it must respond predictably. Messages are pushed or "imposed" onto other computers, which are them expected to catch them and process them as slaves. They are assumed to have no choice. But, my observation was that this violates their natural state of autonomy. There were all kinds of reasons why a computer might fail to react predictably to being hit by a command. One of these was access control.

You really don't want devices to be completely susceptible to external commands. This model was already big trouble in computer security, as viruses and worms ravaged networks, due to this peculiar design assumption. Ironically, engineers started to build artificial boundaries around computers called firewalls — but these were exceptional hacks. Nothing changed in the way computer scientists modelled behaviour — in denial or even defiance of what empirical facts were telling them.

I had an idea for protecting against this. The theoretical aspects of Promise Theory were also a practical solution to the security question of how to make oneself immune to remote exploitation. I could see how to design an interaction protocol based on message passing, in which it would be impossible (by design) to force one computer to do what another asked of it. Autonomy is deafness to exploitation. Requests for cooperation could be exchanged, but these would always be validated and scrutinized before being acted upon. No direct buttons to push.

I made a rough implementation of the idea using the agent code I already had in CFEngine 2, and called it the Voluntary Remote Procedure Call (vRPC), after Sun Microsystem's famous RPC protocol. I was able to arrange for the execution of "peer to peer" tasks without the idea of commands. The idea worked well enough, but we didn't really have a killer application for it, so it languished for a time.

Later, when my PhD student Kyrre Begnum joined the department, I wanted to bring him along to the LISA conference, so we wrote a joint paper about the Voluntary RPC, performing some tests that we could present as a paper. I didn't expect the technology to be useful without something much faster at the bottom of it, but at least we could stamp our names of the idea. A Masters student Matt Disney also made measurements on the speed of transmission by voluntary cooperation — like a game of Chinese whispers.

In my slow protocol version, built into CFEngine 2, the idea was to implement as a Remote Procedure Call, with some handshaking. It was a fully asynchronous protocol for pull-based interaction in which each side could completely control the conditions under which it interacted with the other. But it was very slow, and criticized for that. Matt Disney tested the rate at which information could propagate using this method. The behaviour was peculiar, and counter-intuitive, and it needed more study than a short masters project could allow. CFEngine was only designed to run every half hour at the time, meaning (by Nyquist frequency arguments) that it could only complete a operation reliably on the timescale of an hour or more.

The Voluntary RPC was ridiculed initially by the most engineers at the LISA conference, but as always people were only seeing to the end of their noses. The potential was greater than the toy model we'd put together. Today, the idea has been quietly formalized for web interactions, and also as part of something called publish-subscribe methods. These work at a much faster rate. An entire version of Internet called NDN (Name Data Networking) is based on it. Web transactions increasingly decouple using these asynchronous methods. A simpler, one-way version of it has also been used for scaling proprietary information in CDNs or Content Delivery Networks since the 1990s.

At the LISA 2005 conference, Kyrre Begnum made friends with an innovative thinker from the US called John Sechrest. John came to visit us in Oslo and look over our masters degree setup. Kyrre fell in love with the idea of virtualization when the Xen virtual Linux was released. John Sechrest saw the commercial potential of virtualization, as it was already being exploited by Amazon in his part of the US. We talked about how the model of promises could be used to scale cooperation for cloud services and wrote a paper between us. I was sceptical of how interesting virtualization was until we wrote down how it made management of computers easier. Only then did I realize how it could be an important (albeit wasteful) tool for management, if some basic improvements could be made. The VMware company was then selling virtual machines to run under Windows for sandboxing. They were slow and inefficient, which offended my sensibility, but I could see that they were cleverly made. And suddenly there was chip support for virtualization in the x86 chips, making the whole thing plausible. Amazon already knew.

The ability to see how to connect semantics and dynamics together came just at the right moment. Universities were starting to feel out of date. The real challenges were being faced by industry, and that balance was only going to be tipped further in their favour. The idea that IT could be used for business purpose and that there was an optimization issue strongly appealed to me. My friend, Claudio Bartolini and a Canadian-Brazilian researcher Jacques Sauve, whom I met through the EMANICS project piqued my interest in the idea of aligning IT systems with user needs. He called it Business Driven IT Management. I began to try to apply Promise Theory to answer questions about this. I took the workshop to LISA too to broaden the interest. Carolyn Rowland and I hosted had some fun hosting some tutorials on the idea. They were quite popular. We co-wrote a booklet on A Sysadmin's Guide to Navigating the Business World.

Promise Theory attracted a lot of interest from very different groups, particularly in the social rather than the technological area, as they struggled against old top down management methods, especially in the US. I also wanted to understand human interactions more clearly, but I was wary of the political aspect that made autonomy an attractive position.

The first years of the new millennium saw the rise of something new: e-commerce. The huge expansion in jobs for IT workers (and not the sex variety) meant that it was no longer confined to a coven of old wizards. Computer programming — specifically web programming — went mainstream. A new generation of programmers, who knew almost nothing about how computers were built, began to dominate the management of systems. Agile methods and DevOps were born, and social aspects of IT management began to dominate as the new touchy-feely generation wanted to be recognized and appreciated.

Certain forces were at work that wanted to seize upon Promise Theory as "scientific proof" of a kind of anarchy, of the superiority of leaving people alone to get on with their jobs, and to use their expertise creatively. There was a transformation taking place amongst the millennials. As the demand for programmers was driven by market forces, developers usurped control of organizations and an entire social movement grew up, initially under the auspices of oppression by evil corporate management forces, but later as a way of taking charge of career development.

Promise Theory was clearly in danger of being hijacked for a political purpose. All I could do was to double down on the work and stay true to the science. I wrote many books and papers applying it to different topics. A lot of work went into these, but it's all slowly paying off. Perhaps in a hundred years, everyone will be able to see what I can see through its funny diagrams and simple-minded algebra.

Well, no promises, I suppose.