Carnets de bord

I'd be the first one to recognize that I am playing way out of my league here, no question about it, but it seems to me that Computer Science has been neglecting, if not abusing, the notion of state in ways that seem to be counter productive. I understand that the work of Church and Turing was all about "Computing" and not surprisingly when we started to look for better abstractions to write programs the foundation of λ-calculus was never reassessed but it is difficult today to not ask the question as to whether Abstract Data Types are at the foundation of the General Theory of Software Engineering (GTSE).

I spent the last couple of weeks pounding on the idea that there is more to Barbara Liskov statement about the fundamental role of programs than Abstract Data Types: 

I believe that the fundamental role of programs is to modify state not just the bits

Indeed, when you look at it closely, the role of "computers" today is far less about "computing" than it is about managing state, even though computing is required to transition from state to state properly.

After defining BOLT, and looking at extensions in the problem domain, I found that the idea of defining a programming model based on Abstract State Types could shed some new light on what a programming language truly is.

Surprisingly, there is very little literature about ASTs as I define them. It seems that ASTs fit the model of Coalgebras, but again, I am playing way out of my league here. 

In the paper referenced below, I define an AST as a simple data structure (key value pairs) which all share the same lifecycle (yes, states have a lifecycle too!). A state lifecycle is triggered by an interaction between two (or more) states. States are immutable and new states (of the same type or a different type) are created at the end of an interaction. I define 3 composition mechanisms (containment, extension, connection). Interestingly, there is no ASM (Abstract State Machine) behind ASTs. What we see as a "State Machine" is actually the occurence of repeatable interactions, so all the complexity associated to modeling state machines goes away.

If anyone feels there is a strong analogy between ASTs and QED, well, it is nearly by design, but it really came about from me staring at UML State Diagrams, linking them to business process execution, then to business strategy and then all the way back to problem definition and today programming model. 

This paper is far from complete, a very rough draft at best. I publish it as is for three reasons:

a) I strongly believe that the fundamental idea behind ASTs will not change

b) It would take me an infinite amount of time, which I don't have, to bring it to maturation

c) This is really a call for help, if someone would like to jump in as a coauthor, to further the idea

I don't see ASTs as being the next gen programming concept, ADTs are just fine, but it was hard to resist not extending the BOLT conceptual framework to the programming model level, especially in the context of Barbara Liskov's statement on the role of programs.

Here is the paper, as usual any feedback is welcomed, especially if someone has already worked on that idea. My next step is really to put this idea in practice to explore it further. In particular the paper does not speak at all of the general orchestration behind state interactions (mechanisms such as addressing, activation, ...) but this is by design because I feel it is totally orthogonal.

Again, I welcome the contribution of one or two co-authors, otherwise I'll continue working on this paper in an open content mode.

I have an infinite respect of Prof. Liskov, for her, personally, and her work. A lot of what she has discovered has guided my own work and the work of millions more for the past four decades and her insight on the role of programs has litterally transformed and shaped the way I think:

I believe that the fundamental role of programs is to modify state not just the bits

But, what if her fundamental assumptions were off? After all they are assumptions and assumptions need to be revisited periodically, they are (somewhat) artificial, often unconscious, constraints that we believe make it easier to find the solution of a given problem. Just ask Richard Feynman.

After reading, and rereading her seminal paper on "Programming with Abstract Data Types", it is difficult not to question her assumptions. 

For centuries, we have created valuable abstractions that make it easier to solve problems, very complex problems. As we all understand the value of high level programming languages, it would be hard to disagree, for instance, with her introductory statement:

Unfortunately, it is very difficult for a designer to select in advance all the abstractions which the users of his language might need. If a language is to be used at all, it is likely to be used to solve problems which its designer did not envision, and for which the abstractions embedded in the language are not sufficient.

Yet, that statement creates an unfortunate bias that somehow the language designer and the solution developer must always be two different people. It also creates another unfortunate bias, as it, somehow, suggests that every a team of developers should use the same programming language, maybe within a tier, if not for the entire solution stack.

Actually, it is probably very difficult to not count these two assumptions as mere facts. My career started pretty nicely with the development of advanced semi-conductor deposition systems. I used NeXTStep to build these systems: ADTs, OOP, and Model-Driven development tools (Interface Builder) offered a delightful way to create a model of these systems. These abstractions were so powerful that I ended up building a Natural Language Processor on top of the control systems (using Objective-C's reflection mechanisms). Back in 1996, this control system was jaw dropping, but to be fair, NeXTStep, ADTs, i.e. abstractions, combined with some elbow finger grease was all that was needed to create that kind of magic. Retrospectivelly, I trully think this was magic, and I could have never build such a system by myself without all the good work that was done at that time. 

In 1997, when Steve Jobs returned to Apple and killed NeXT, I had to leave this wonderfully abstracted world and decided to join the ranks of Information Technology professionals. IT was very different from the world of industrial process control: none of these abstractions make any sense in IT. When I say any, I really mean any. Everywhere I turned, ADTs didn't help: BPM (process), SOA (service), XML (eXtensible Data Structures), HTTP (Protocols), SQL or NoSQL (Data).  Worse, ADTs seemed to make things far more complex than one would have originally thought.

So why not aiming at a completely different direction? 

• What if every development team would be allowed to build (and have control over) their programming language?
• What if every developer in that team would have the flexibility to create their own programming language, if it made sense? 
• What if these programming languages could even be solution specific, again if it made sense? 

I know, this is a radical idea. I just ask you to consider the following proposal. The work of a team of developers would look like something like this: 

Ultimately Metaprogramming delivers 3 key benefits:

1) As the project progresses, you end up mostly editing the solution model. Writing code at this level is usually very compact and very stable. This means that you can make major functional changes up to the last minute, if you need to

2) You can refactor the code generator fairly easily, for that app, the code generator is about 10k LOC, while the generated code is slightly above 100k LOC. As you refactor the technology, this is about 90% less effort

3) QA costs go way down, again as you are in effect stabilizing far less amounts of code

This post is a synthesis of two posts I originally published on my other blog "unRelated".

One of the key foundations and most attractive principle of Agile or Lean methodologies is that  "Everyone can help each other remain focused on the highest possible business value per unit of time".

I am certainly a strong supporter of that principle. However, value is often difficult to assess, I would actually argue that it is easier to identify what has less or little value, but what we think as valuable can potentially lead to many false positive, or simply be "business-as-usual" and hide the broader structure of the solution. 

"User Stories" are the corner stone of identifying and delivering value:

An argument can be made that the user story is the most important artifact in agile development, because it is the container that primarily carries the value stream to the user, and agile development is all about rapid value delivery.

In practice, very few people focus on the benefits part of a user story. All user stories I see are either what we used to call "requirements" (just phrased slightly differently but isomorphically) or "tasks" needed to advance the state of the project.

However, there is a fundamental flaw in the construction of user stories, even when they are properly written, because they somehow make an assumption about the shape of the solution, and drive the author to turn almost immediately in solution mode, leaving no room for creative and out-of-the-box thinking.

Let's compare the metamodel of a User Story and to the formal definition of a Problem. The metamodel of a User Story looks like that (using the BOLT notation):

As a <role> I want to <action> so that <benefit>

I define a problem formally as a non existing transition between two known states [1],  the metamodel of a problem looks like that:

A solution is a way to transition between these two states. Please note that both the actors and the actions are part of the solution:

This is where the problem lies when using User Stories, you are specifying the requirements with the solution in mind. There is, of course, a general relationship between some of the actors and entities of the system with the "start" and "end" states of the problem. The problem states are always defined in terms of their respective states (possibly as a composite state), but it is a mistake to think that the actors and entities that perform the actions, as part of the solution, are always the same as the actors and entities related to the (problem) states.

Hence, an action is solution centric and should not be part of the problem definition. As soon as you pick one, you have put a stake in the ground towards the direction you are going to take to solve the underlying problem. The other issue is that the start and end states are never clearly identified in a user story leading to confusion in the in the solutioning and verification process, since the problem is not defined with enough precision. Benefits could sometimes align with the target/desirable state, but the definition is often too fluffy and more goal centric, not effectively representing that (problem) state.

Ultimately, the relationship between problems and solutions is a graph (states, transitions as problems, actions as solutions), and this is where the coupling between the problem space and the solution space at the User Story level becomes unfortunate. This means that User stories cannot be effectively nested and clearly cannot fit in hierarchical structures (which is common to most Agile tools I know). This problem is quite accute as teams struggle to connect business level user stories and system level or solution level user stories. The concept of having a single parent directly conflicts with the possibility of having multiple possible transitions into a single state and decomposition principles where the same problem appears in the decomposition of several higher level problems. 

I feel that distinction is profound because we can now clearly articulate:

a) the problem statements with respect to each other (as a graph of states and transitions)

b) we can articulate the solution in relation to the problem statements

c) we can articulate the verification (BDD) in relation to the problem and solution [2]

d) we can actually articulate the Business Strategy [3], the Problem Statement, the Solution and the Verification with the same conceptual framework

e) derive the Organizational IQ from the problems being solved on an every day basis

To the best of my knowledge none of these articulations have been suggested before and no one has ever provided a unified framework that spans such a broad conceptual view from the Business Strategy to the Verification. In the proposed framework the business strategy is simply a higher level and specialized view of the problem and solution domains, but using the exact same semantics (which are described here). In other words the enterprise is a solution to a problem, which is a composition of smaller problems and more fine grained solutions, etc. This has an extremely important implication for the execution of the strategy because now both the Strategy and its Execution are perfectly aligned, at the semantic level: the strategy, problem, solution and verification graph represent a map that everyone in the organization can refer to. 

To take advantage of this new conceptual framework. I suggest that we make a very simple and easy change to Agile and replace "user stories" by "problem statements". Each problem must be "solutioned", either by decomposing it into simpler problems or solutioning it directly. Value can still be used to prioritize which problems are addressed first, that part of the Agile and Lean movement is very valuable, so too speak, but the focus on problems and solutions opens a new flexibility in how we handle the long range aspects of the solution while enabling the highest level of creativity and ultimately a direct articulation with the IQ of the organization. 

As problems are decomposed, we will eventually reach a point where the subproblems will be close to or isomorphically related to the solution. But it would be a mistake to not clearly delineate the problems from solutions, simply because at the lowest level, they appear isomorphic. 

If we start drawing some BOLT diagrams, a problem lifecycle can be defined as:

The fact that the lifecycle is pretty much identical as the one of a user story enables most of the Agile processes and tools to work nearly unchanged.

You may want to know "How do I write a Problem Statement?". Personally, I don't like canned approaches. Oviously here, the mere definition of the two states (low value and high value) is enough to describe the problem. If a solution already exists (i.e. it is possible to transition between these two states) you may want to describe some characteristics of the new solution. I googled "How to write a Problem Statement?" and I felt there was already a good alignment betweent the results and the abstract definition provided above. For instance:

We want all of our software releases to go to production seamlessly, without defects, where everyone is aware and informed of the outcomes and status. (Vision)

Today we have too many release failures that result in too many rollback failures. If we ignore this problem; resources will need to increase to handle the cascading problems, and we may miss critical customer deadlines which could result in lost revenue, SLA penalties, lost business, and further damage to our quality reputation. (Issue Statement)

Here we see two states for the releases: initial state (low value) tested, and the high value state (in production). There is also an undesirable state (failure) that the new solution will prevent reaching. For me the most important thing is that the problem statement must avoid at all cost to refer to the solution. Even if the people specifying the problem statement have an idea about the solution, they should capture it separately.

This new focus on problem & solution provides a rich conceptual framework to effectively organize the work of a team. After all, we have been innovating, i.e. creating solutions to problems, for thousands of years, so it is no surprise that our vocabulary is quite rich. Here are a few concepts that could be used:

Goal: a goal is not a problem, but you often need to solve problems to reach goals, so it's important to keep them in mind

Fact: a fact often constrains the solution, so they need to be clearly surfaced and accounted for

Assumption: assumptions are very important because they also constrain the solution, but in a more flexible way. Assumptions can be changed, facts generally cannot.

Statement: the problem statement is what physically replaces the user story.

Hurdle: During the decomposition of a problem, hurdles might be identified, they are not a problem per say, but they impact the solution. It could be for instance that a resource is not available in time to meet the deadline.

Snag: A problem can be downgraded to a snag as the solution is obvious to the team and represent a low level of effort. It can also be a small unexpected issue, that need to be quickly resolved.

Dilemma: A problem can be upgraded to a dilemma, when several solutions are possible and it is not clear which one to chose

Setback: The team can suffer a setback when it thought it had found the solution but it didn't, or could not find a solution and need to reassess either the problem or the approach

On the solution side, we can also capture different elements and stages of the solutioning process:

Answer: Findings related to a question raised in the problem statement.

Result: A validation that the solution conforms to a Fact

Resolution: The choice made after reaching a dilemma

Fix: a temporary solution to a problem or a snag to make progress towards the solution to the greater problem

Development: An element of the solution, usually the solution to a subproblem or a snag

Breakthrough: The solution found after reaching a setback

Way out: A solution was not found, nevertheless, the project reached a satisfactory state to meet some or all of the initial goals

...

From a management perspective. The Solution or Delivery Manager can escape the bureaucracy that Agile has created. Ironically, moving stickers around is a zero value activity, with zero impact on the organizational IQ. The solution manager can and should be responsible for the IQ of the project, which rolls up and benefits from the IQ of the organization. It should keep track of the elements that are incorporated in the solution as problems are solved. It should encourage team members to be creative when necessary and to shamelessly adopt existing solutions when it makes sense. It should help resolve dilemmas and push for breakthroughs.

The PMO organization becomes the steward of the Organization's IQ.

As we define problems and solutions in terms of entities, state, transitions and actions, the BOLT methodology provides a unified conceptual framework that spans from Business Strategy to Problem and Solution Domains to Verification (BDD).

To summarize,

1) We have provided a formal model of a problem and a solution, and how they relate to each other

2) This formal model offers the ability to compose problems and solutions at any scale, over the scope of the enterprise

3) Problems and Solutions can be composed from Business Strategy down to Verification

4) We suggest that Agile methodologies replace User Stories by Problem Statements

5) With the renewed focus on "problems", we can also integrate the work of Prof. Knott on Organizational IQ in the whole framework

Last, but not least, decoupling problem definition and solution yields a tremendous benefit in the sense that both can evolve independently during the construction process. 

_______________________________________________________________________________

[1] For instance, you build a vehicle, obviously you want to car to transition to the "in motion" state. Different "actions" will lead to the vehicle to reach that state (a horse pulling, an engine, transmission and wheels, a fan, ...).

[2] BDD Metamodel (Scenario):

[3] Living Social Business Strategy mapped using the same conceptual framework (Source: B = mc2)

I explained last month that the Web Style never took off since Tim Bray's post which predicted the End of SOA with a great dose of exaggeration. Instead, developers quickly pivoted towards a traditional, albeit HTTP-based, API style. I am not sure who could have expected a different outcome... 

Over all these years, the process of defining an API style has been quite chaotic as people swung from resource orientation to service orientation and back and, today, there is no particular "standard" or "sytle" which people can confidently rely on to design their API. Every developer decides artistically (and often inconsistenly) how he wants to use HTTP to marshal a particular API call.

I was doing some research on a few APIs this week-end and took another look at Swagger. I really like what they do. Incidentally, if anybody doubts that the Web Style is dead, I would take a peak at their petshop demo... That being said, that petshop API is terrible. 

When it comes to Resource Orientation, Service Orientation, Event Orientation... I actually like them all: I am a polyadic programmer (a.k.a metaprogrammer) and if anyone doubts it, I would encourage you to look at a framework I built in 2007, WSPER ("whisper"), which stands for Web Service, Process, Event, Resource.  

So, after so many years, should we continue opposing Resource and Service orientation? I would like to show here that indeed we can weave them together and as a result we can surface resource orientation in a much cleaner way while retaining all the benefits of service orientation in the process.

Services have this nice property of being exposed via an endpoint which creates a natural boundary of deployment, access and management. Enpoints truly enable an interface to be totally decoupled from its implementation. I cannot emphasize enough how important that decoupling is to distributed computing.

Service orientation brought another important innovation by moving the focus from objects to views. Services hide, just enough, the model behind the Service Interface: you understand that you are dealing with a customer view, but you never have to understand the extend of a customer class. Actually I would argue that a "true" customer class cannot be built, you always interact with a particular resource in a given context (Sales, Billing, Support...). If you doubt it, please take a look at this presentation from Daniel Jacobson. To be fair, however, as a side effect of this approach, queries or (partial) updates are notoriously painful when you need to define them in a WSDL, no argument there. I get that. But is that a reason to give up altogether on Service Orientation? 

On the other hand the RESTafarians have grossly misunderstood Service Orientation and at the same time, never really understood that REST brought back distributed objects into the programming model. Worse, because of its sole focus on resources, REST forces you to expose every single detail of the model to the resource consumer. It shouldn't be a surprise to anyone that in the end all they can deliver is CRUD. 

One thing that people often forget in SOA is that a service implementation can and should expose multiple service contracts, this is a bit harder to do with APIs because resource paths are not polymorphic by design and they are often hard wired to the implementation. Yes, I know, I could use yet another HTTP header or even concatenate an nth parameter to the Content-type but that is inelegant because you lose the natural boundary offered by a "service" endpoint. The real elegance I see behind Resource orientation (that none of the Web APIs I know implement, including the so-called RESTful ones) is when a path, say /customers/1234, truly points to the same resource regardless of which service I use to access it. 

What I mean here is when you have a customer service and a purchase order service, you should be able to fetch the same customer information (albeit with a different view) using the same path. Yes, when you query an order GET /orders/345 and it returns a path (not a link) to /customers/1234 the PO Service could implement that path and return a view of the customer in the context of a purchase order and if you invoke the same path on the customer service, you would get a more complete view (assuming you have the access rights). 

When you marshal the service name within the path (and the version for that matter), you cannot achieve that elegant behavior without a lot of out of band conventions and string manipulations. That's why I suggest that you define the service/API boundaries using subdomains rather than paths. For instance we would have two subdomains:

http://v1.customers.api.ebpml.org

http://v2.orders.api.ebpml.org

Each subdomain would provide access to its own resource collections, including overlapping collections. Both API calls would return a view of the same customer:

[GET] http://v1.customers.api.ebpml.org/customers/1234

[GET] http://v2.orders.api.ebpml.org/customers/1234

Imagine a "link" that works across services and versions? Many times I have mentioned that REST couples access and identity. Here we have a clear separation between access (subdomain endpoint) and identity (at the path level). 

This API style is capable of supporting the traditional Query/Create/Update/Delete operations quite efficiently, in a way that could look attractive to Web developers. However, I would like to emphasize again that HTTP alone is not a good query language. If you doubt it,please take look at the MongoDB API. Any serious service implementation will have to follow their lead, especially for complex queries and partial updates.

Actions are another point of misalignment between resource and service orientation. Many of the RESTafarians have dismissed the need for actions and advised to use a simple update instead. In the real world this is of course a completely bogus approach (validation, authorization, logging, throttling, versioning, duplication of the update logic on the consumer side...) and Web API designers have been prompt to use action verbs in their paths, wired to specific method calls.

I have also explained many times that every resource has a lifecycle. Some resource (such as a Tweet or Status) have a CUD lifecycle (Create -> Update* -> Delete) which maps well to the lifecycle of a Web page, and hence the HTTP verbs, but again in the real world, lifecycles are complex and each transition maps to an action which you need to express one way or another. A verb is what everyone understands. If you want to use a noun for a verb, be my guest, but semantically you have not changed the meaning of the call, you are still triggering an action, which when successful will transition the resource from one state to the next. 

So there is no reason to be afraid (and do like everyone else): use verbs as needed, either behind a resource when they apply to a resource instance or at the collection level when you initiate a lifecycle.

[POST] http://v2.orders.api.ebpml.org/orders/create

[POST] http://v2.orders.api.ebpml.org/orders/1234/submit

[POST] http://v2.orders.api.ebpml.org/orders/1234/pay

[POST] http://v2.orders.api.ebpml.org/orders/1234/ship

[POST] http://v2.orders.api.ebpml.org/orders/1234/deliver

[POST] http://v2.orders.api.ebpml.org/orders/1234/return

...

Composite calls (which fire several actions that transition to multiple states on different resources) should also be considered.

You will notice that the use of a subdomain as a service boundary greatly simplifies the migration from one major version to the next since API signatures and identities are not polluted by the version number like it is often the case in traditional API designs.

That's it, I believe that approach defines a clean Web API style which brings together the best of Resource Orientation and Service Orientation:

• It achieves a true decoupling between resource access and identity, which in turns allow service consumers to use these identities across multiple services and supports a polymorphic resource access model (at the service level) which encourages the usage of views on the model, rather than a direct access to the model
• It introduces a clean API boundary which can be much more efficiently managed (and authorized)
• It provides a clear and clean way to define actions which map directly to the resource lifecycle
• It simplifies version migrations

I came up with these ideas while building a Swagger stencil for Omnigraffle. So here it is.