This post is a synthesis of two posts I originally published on my other blog "unRelated".
One of the key foundations and most attractive principle of Agile or Lean methodologies is that "Everyone can help each other remain focused on the highest possible business value per unit of time".
I am certainly a strong supporter of that principle. However, value is often difficult to assess, I would actually argue that it is easier to identify what has less or little value, but what we think as valuable can potentially lead to many false positive, or simply be "business-as-usual" and hide the broader structure of the solution.
"User Stories" are the corner stone of identifying and delivering value:
An argument can be made that the user story is the most important artifact in agile development, because it is the container that primarily carries the value stream to the user, and agile development is all about rapid value delivery.
In practice, very few people focus on the benefits part of a user story. All user stories I see are either what we used to call "requirements" (just phrased slightly differently but isomorphically) or "tasks" needed to advance the state of the project.
However, there is a fundamental flaw in the construction of user stories, even when they are properly written, because they somehow make an assumption about the shape of the solution, and drive the author to turn almost immediately in solution mode, leaving no room for creative and out-of-the-box thinking.
Let's compare the metamodel of a User Story and to the formal definition of a Problem. The metamodel of a User Story looks like that (using the BOLT notation):
As a <role> I want to <action> so that <benefit>
I define a problem formally as a non existing transition between two known states [1], the metamodel of a problem looks like that:
A solution is a way to transition between these two states. Please note that both the actors and the actions are part of the solution:
This is where the problem lies when using User Stories, you are specifying the requirements with the solution in mind. There is, of course, a general relationship between some of the actors and entities of the system with the "start" and "end" states of the problem. The problem states are always defined in terms of their respective states (possibly as a composite state), but it is a mistake to think that the actors and entities that perform the actions, as part of the solution, are always the same as the actors and entities related to the (problem) states.
Hence, an action is solution centric and should not be part of the problem definition. As soon as you pick one, you have put a stake in the ground towards the direction you are going to take to solve the underlying problem. The other issue is that the start and end states are never clearly identified in a user story leading to confusion in the in the solutioning and verification process, since the problem is not defined with enough precision. Benefits could sometimes align with the target/desirable state, but the definition is often too fluffy and more goal centric, not effectively representing that (problem) state.
Ultimately, the relationship between problems and solutions is a graph (states, transitions as problems, actions as solutions), and this is where the coupling between the problem space and the solution space at the User Story level becomes unfortunate. This means that User stories cannot be effectively nested and clearly cannot fit in hierarchical structures (which is common to most Agile tools I know). This problem is quite accute as teams struggle to connect business level user stories and system level or solution level user stories. The concept of having a single parent directly conflicts with the possibility of having multiple possible transitions into a single state and decomposition principles where the same problem appears in the decomposition of several higher level problems.
I feel that distinction is profound because we can now clearly articulate:
a) the problem statements with respect to each other (as a graph of states and transitions)
b) we can articulate the solution in relation to the problem statements
c) we can articulate the verification (BDD) in relation to the problem and solution [2]
d) we can actually articulate the Business Strategy [3], the Problem Statement, the Solution and the Verification with the same conceptual framework
e) derive the Organizational IQ from the problems being solved on an every day basis
To the best of my knowledge none of these articulations have been suggested before and no one has ever provided a unified framework that spans such a broad conceptual view from the Business Strategy to the Verification. In the proposed framework the business strategy is simply a higher level and specialized view of the problem and solution domains, but using the exact same semantics (which are described here). In other words the enterprise is a solution to a problem, which is a composition of smaller problems and more fine grained solutions, etc. This has an extremely important implication for the execution of the strategy because now both the Strategy and its Execution are perfectly aligned, at the semantic level: the strategy, problem, solution and verification graph represent a map that everyone in the organization can refer to.
To take advantage of this new conceptual framework. I suggest that we make a very simple and easy change to Agile and replace "user stories" by "problem statements". Each problem must be "solutioned", either by decomposing it into simpler problems or solutioning it directly. Value can still be used to prioritize which problems are addressed first, that part of the Agile and Lean movement is very valuable, so too speak, but the focus on problems and solutions opens a new flexibility in how we handle the long range aspects of the solution while enabling the highest level of creativity and ultimately a direct articulation with the IQ of the organization.
As problems are decomposed, we will eventually reach a point where the subproblems will be close to or isomorphically related to the solution. But it would be a mistake to not clearly delineate the problems from solutions, simply because at the lowest level, they appear isomorphic.
If we start drawing some BOLT diagrams, a problem lifecycle can be defined as:
The fact that the lifecycle is pretty much identical as the one of a user story enables most of the Agile processes and tools to work nearly unchanged.
You may want to know "How do I write a Problem Statement?". Personally, I don't like canned approaches. Oviously here, the mere definition of the two states (low value and high value) is enough to describe the problem. If a solution already exists (i.e. it is possible to transition between these two states) you may want to describe some characteristics of the new solution. I googled "How to write a Problem Statement?" and I felt there was already a good alignment betweent the results and the abstract definition provided above. For instance:
We want all of our software releases to go to production seamlessly, without defects, where everyone is aware and informed of the outcomes and status. (Vision)
Today we have too many release failures that result in too many rollback failures. If we ignore this problem; resources will need to increase to handle the cascading problems, and we may miss critical customer deadlines which could result in lost revenue, SLA penalties, lost business, and further damage to our quality reputation. (Issue Statement)
Here we see two states for the releases: initial state (low value) tested, and the high value state (in production). There is also an undesirable state (failure) that the new solution will prevent reaching. For me the most important thing is that the problem statement must avoid at all cost to refer to the solution. Even if the people specifying the problem statement have an idea about the solution, they should capture it separately.
This new focus on problem & solution provides a rich conceptual framework to effectively organize the work of a team. After all, we have been innovating, i.e. creating solutions to problems, for thousands of years, so it is no surprise that our vocabulary is quite rich. Here are a few concepts that could be used:
Goal: a goal is not a problem, but you often need to solve problems to reach goals, so it's important to keep them in mind
Fact: a fact often constrains the solution, so they need to be clearly surfaced and accounted for
Assumption: assumptions are very important because they also constrain the solution, but in a more flexible way. Assumptions can be changed, facts generally cannot.
Statement: the problem statement is what physically replaces the user story.
Hurdle: During the decomposition of a problem, hurdles might be identified, they are not a problem per say, but they impact the solution. It could be for instance that a resource is not available in time to meet the deadline.
Snag: A problem can be downgraded to a snag as the solution is obvious to the team and represent a low level of effort. It can also be a small unexpected issue, that need to be quickly resolved.
Dilemma: A problem can be upgraded to a dilemma, when several solutions are possible and it is not clear which one to chose
Setback: The team can suffer a setback when it thought it had found the solution but it didn't, or could not find a solution and need to reassess either the problem or the approach
On the solution side, we can also capture different elements and stages of the solutioning process:
Answer: Findings related to a question raised in the problem statement.
Result: A validation that the solution conforms to a Fact
Resolution: The choice made after reaching a dilemma
Fix: a temporary solution to a problem or a snag to make progress towards the solution to the greater problem
Development: An element of the solution, usually the solution to a subproblem or a snag
Breakthrough: The solution found after reaching a setback
Way out: A solution was not found, nevertheless, the project reached a satisfactory state to meet some or all of the initial goals
...
From a management perspective. The Solution or Delivery Manager can escape the bureaucracy that Agile has created. Ironically, moving stickers around is a zero value activity, with zero impact on the organizational IQ. The solution manager can and should be responsible for the IQ of the project, which rolls up and benefits from the IQ of the organization. It should keep track of the elements that are incorporated in the solution as problems are solved. It should encourage team members to be creative when necessary and to shamelessly adopt existing solutions when it makes sense. It should help resolve dilemmas and push for breakthroughs.
The PMO organization becomes the steward of the Organization's IQ.
As we define problems and solutions in terms of entities, state, transitions and actions, the BOLT methodology provides a unified conceptual framework that spans from Business Strategy to Problem and Solution Domains to Verification (BDD).
To summarize,
1) We have provided a formal model of a problem and a solution, and how they relate to each other
2) This formal model offers the ability to compose problems and solutions at any scale, over the scope of the enterprise
3) Problems and Solutions can be composed from Business Strategy down to Verification
4) We suggest that Agile methodologies replace User Stories by Problem Statements
5) With the renewed focus on "problems", we can also integrate the work of Prof. Knott on Organizational IQ in the whole framework
Last, but not least, decoupling problem definition and solution yields a tremendous benefit in the sense that both can evolve independently during the construction process.
_______________________________________________________________________________
[1] For instance, you build a vehicle, obviously you want to car to transition to the "in motion" state. Different "actions" will lead to the vehicle to reach that state (a horse pulling, an engine, transmission and wheels, a fan, ...).
[2] BDD Metamodel (Scenario):
[3] Living Social Business Strategy mapped using the same conceptual framework (Source: B = mc2)
I have an infinite respect of Prof. Liskov, for her, personally, and her work. A lot of what she has discovered has guided my own work and the work of millions more for the past four decades and her insight on the role of programs has litterally transformed and shaped the way I think:
I believe that the fundamental role of programs is to modify state not just the bits
But, what if her fundamental assumptions were off? After all they are assumptions and assumptions need to be revisited periodically, they are (somewhat) artificial, often unconscious, constraints that we believe make it easier to find the solution of a given problem. Just ask Richard Feynman.
After reading, and rereading her seminal paper on "Programming with Abstract Data Types", it is difficult not to question her assumptions.
For centuries, we have created valuable abstractions that make it easier to solve problems, very complex problems. As we all understand the value of high level programming languages, it would be hard to disagree, for instance, with her introductory statement:
Unfortunately, it is very difficult for a designer to select in advance all the abstractions which the users of his language might need. If a language is to be used at all, it is likely to be used to solve problems which its designer did not envision, and for which the abstractions embedded in the language are not sufficient.
Yet, that statement creates an unfortunate bias that somehow the language designer and the solution developer must always be two different people. It also creates another unfortunate bias, as it, somehow, suggests that every a team of developers should use the same programming language, maybe within a tier, if not for the entire solution stack.
Actually, it is probably very difficult to not count these two assumptions as mere facts. My career started pretty nicely with the development of advanced semi-conductor deposition systems. I used NeXTStep to build these systems: ADTs, OOP, and Model-Driven development tools (Interface Builder) offered a delightful way to create a model of these systems. These abstractions were so powerful that I ended up building a Natural Language Processor on top of the control systems (using Objective-C's reflection mechanisms). Back in 1996, this control system was jaw dropping, but to be fair, NeXTStep, ADTs, i.e. abstractions, combined with some elbow finger grease was all that was needed to create that kind of magic. Retrospectivelly, I trully think this was magic, and I could have never build such a system by myself without all the good work that was done at that time.
In 1997, when Steve Jobs returned to Apple and killed NeXT, I had to leave this wonderfully abstracted world and decided to join the ranks of Information Technology professionals. IT was very different from the world of industrial process control: none of these abstractions make any sense in IT. When I say any, I really mean any. Everywhere I turned, ADTs didn't help: BPM (process), SOA (service), XML (eXtensible Data Structures), HTTP (Protocols), SQL or NoSQL (Data). Worse, ADTs seemed to make things far more complex than one would have originally thought.
So why not aiming at a completely different direction?
- What if every development team would be allowed to build (and have control over) their programming language?
- What if every developer in that team would have the flexibility to create their own programming language, if it made sense?
- What if these programming languages could even be solution specific, again if it made sense?
I know, this is a radical idea. I just ask you to consider the following proposal. The work of a team of developers would look like something like this:
04/13/13
Formalizing the API Style
I explained last month that the Web Style never took off since Tim Bray's post which predicted the End of SOA with a great dose of exaggeration. Instead, developers quickly pivoted towards a traditional, albeit HTTP-based, API style. I am not sure who could have expected a different outcome...
Over all these years, the process of defining an API style has been quite chaotic as people swung from resource orientation to service orientation and back and, today, there is no particular "standard" or "sytle" which people can confidently rely on to design their API. Every developer decides artistically (and often inconsistenly) how he wants to use HTTP to marshal a particular API call.
I was doing some research on a few APIs this week-end and took another look at Swagger. I really like what they do. Incidentally, if anybody doubts that the Web Style is dead, I would take a peak at their petshop demo... That being said, that petshop API is terrible.
When it comes to Resource Orientation, Service Orientation, Event Orientation... I actually like them all: I am a polyadic programmer (a.k.a metaprogrammer) and if anyone doubts it, I would encourage you to look at a framework I built in 2007, WSPER ("whisper"), which stands for Web Service, Process, Event, Resource.
So, after so many years, should we continue opposing Resource and Service orientation? I would like to show here that indeed we can weave them together and as a result we can surface resource orientation in a much cleaner way while retaining all the benefits of service orientation in the process.
Services have this nice property of being exposed via an endpoint which creates a natural boundary of deployment, access and management. Enpoints truly enable an interface to be totally decoupled from its implementation. I cannot emphasize enough how important that decoupling is to distributed computing.
Service orientation brought another important innovation by moving the focus from objects to views. Services hide, just enough, the model behind the Service Interface: you understand that you are dealing with a customer view, but you never have to understand the extend of a customer class. Actually I would argue that a "true" customer class cannot be built, you always interact with a particular resource in a given context (Sales, Billing, Support...). If you doubt it, please take a look at this presentation from Daniel Jacobson. To be fair, however, as a side effect of this approach, queries or (partial) updates are notoriously painful when you need to define them in a WSDL, no argument there. I get that. But is that a reason to give up altogether on Service Orientation?
On the other hand the RESTafarians have grossly misunderstood Service Orientation and at the same time, never really understood that REST brought back distributed objects into the programming model. Worse, because of its sole focus on resources, REST forces you to expose every single detail of the model to the resource consumer. It shouldn't be a surprise to anyone that in the end all they can deliver is CRUD.
One thing that people often forget in SOA is that a service implementation can and should expose multiple service contracts, this is a bit harder to do with APIs because resource paths are not polymorphic by design and they are often hard wired to the implementation. Yes, I know, I could use yet another HTTP header or even concatenate an nth parameter to the Content-type but that is inelegant because you lose the natural boundary offered by a "service" endpoint. The real elegance I see behind Resource orientation (that none of the Web APIs I know implement, including the so-called RESTful ones) is when a path, say /customers/1234, truly points to the same resource regardless of which service I use to access it.
What I mean here is when you have a customer service and a purchase order service, you should be able to fetch the same customer information (albeit with a different view) using the same path. Yes, when you query an order GET /orders/345 and it returns a path (not a link) to /customers/1234 the PO Service could implement that path and return a view of the customer in the context of a purchase order and if you invoke the same path on the customer service, you would get a more complete view (assuming you have the access rights).
When you marshal the service name within the path (and the version for that matter), you cannot achieve that elegant behavior without a lot of out of band conventions and string manipulations. That's why I suggest that you define the service/API boundaries using subdomains rather than paths. For instance we would have two subdomains:
http://v1.customers.api.ebpml.org
http://v2.orders.api.ebpml.org
Each subdomain would provide access to its own resource collections, including overlapping collections. Both API calls would return a view of the same customer:
[GET] http://v1.customers.api.ebpml.org/customers/1234
[GET] http://v2.orders.api.ebpml.org/customers/1234
Imagine a "link" that works across services and versions? Many times I have mentioned that REST couples access and identity. Here we have a clear separation between access (subdomain endpoint) and identity (at the path level).
This API style is capable of supporting the traditional Query/Create/Update/Delete operations quite efficiently, in a way that could look attractive to Web developers. However, I would like to emphasize again that HTTP alone is not a good query language. If you doubt it,please take look at the MongoDB API. Any serious service implementation will have to follow their lead, especially for complex queries and partial updates.
Actions are another point of misalignment between resource and service orientation. Many of the RESTafarians have dismissed the need for actions and advised to use a simple update instead. In the real world this is of course a completely bogus approach (validation, authorization, logging, throttling, versioning, duplication of the update logic on the consumer side...) and Web API designers have been prompt to use action verbs in their paths, wired to specific method calls.
I have also explained many times that every resource has a lifecycle. Some resource (such as a Tweet or Status) have a CUD lifecycle (Create -> Update* -> Delete) which maps well to the lifecycle of a Web page, and hence the HTTP verbs, but again in the real world, lifecycles are complex and each transition maps to an action which you need to express one way or another. A verb is what everyone understands. If you want to use a noun for a verb, be my guest, but semantically you have not changed the meaning of the call, you are still triggering an action, which when successful will transition the resource from one state to the next.
So there is no reason to be afraid (and do like everyone else): use verbs as needed, either behind a resource when they apply to a resource instance or at the collection level when you initiate a lifecycle.
[POST] http://v2.orders.api.ebpml.org/orders/create
[POST] http://v2.orders.api.ebpml.org/orders/1234/submit
[POST] http://v2.orders.api.ebpml.org/orders/1234/pay
[POST] http://v2.orders.api.ebpml.org/orders/1234/ship
[POST] http://v2.orders.api.ebpml.org/orders/1234/deliver
[POST] http://v2.orders.api.ebpml.org/orders/1234/return
...
Composite calls (which fire several actions that transition to multiple states on different resources) should also be considered.
You will notice that the use of a subdomain as a service boundary greatly simplifies the migration from one major version to the next since API signatures and identities are not polluted by the version number like it is often the case in traditional API designs.
That's it, I believe that approach defines a clean Web API style which brings together the best of Resource Orientation and Service Orientation:
- It achieves a true decoupling between resource access and identity, which in turns allow service consumers to use these identities across multiple services and supports a polymorphic resource access model (at the service level) which encourages the usage of views on the model, rather than a direct access to the model
- It introduces a clean API boundary which can be much more efficiently managed (and authorized)
- It provides a clear and clean way to define actions which map directly to the resource lifecycle
- It simplifies version migrations
I came up with these ideas while building a Swagger stencil for Omnigraffle. So here it is.
About 4 years ago I fired up my first Cloud server on Rackspace. The service has been great but Rackspace's pricing strategy has been abysmal. In 4 years, my bill stayed constant, so did the specs of my "servers" (a modest 10 Gb / 256 Mb).
While Amazon customers enjoyed something like 26 price drops in that timeframe, I got nothing. Worse, for the kind of servers I use, the price went up 50%, earlier this year: Rackspace no longer offers a micro server (i.e. a slice of a server), for anything I do, I now have to buy a small instance at a 50% premium (after the 30% "drop").
So I just got a reserved Amazon's micro instance, a whopping 600 Mb "machine", substantially faster (my guesstimate is about 3X) for about the same price as a now defunct micro Rackspace instance. I can easily run 2 micro Rackspace instances in a single not-so-micro Amazon instance.
Don't get me wrong, I am sad to switch. When I told Robert Scoble that Rackspace's pricing Strategy sucks, he responded and got the President, Lew Moorman and the CTO to reply to me personally. But for me, the cloud is all about slices: temporal slices, or tenancy slices, and clearly Amazon understands that game far better than any other Cloud provider. I am not in the business of sponsoring Rackspace, I expect them to innovate at a reasonable rate and pass that innovation to their customers, the way Amazon does it, or almost any business. Even Apple gets that, the iPhone 4 is 30% cheaper than when it came out, the iPad 2 is 25% cheaper ... and so on.
The migration was uneventful, a simple file transfer was enough since my workload is stateless. At first glance security seems a lot tighter.
So long Rackspace.
03/20/13
The End of the Web (Style)
Seven years ago, next month, Tim Bray predicted the End of SOA and concluded his commentary with:
"SOA isn’t the future, Web style is."
That post opened the door to a big circus and gave pundits the "opportunity" to predict with more accuracy than any oracle the Death of SOA. Anne Thomas Manes explained to us in 2009, from the comfort of her cublicle at Delphi Gartner, that "SOA [was] a great failed experiment". Another "analyst", David Chappell, added to the circus by asking tong-in-cheek "if SOA was failing", and went as far as calling SOA: "Suckered Once Again". Then there was Jim Webber's "Same Old Atrocity" which is one of the most profound and graph-ical presentation IT had ever seen.
How many SOA initiatives were stopped or slowed down by "The End of SOA"? Mine was. My manager at the time told me that Tim Bray's post was circulating in our IT department and he didn't know where to start to craft an answer to his management. His team had built their own ESB at a time when hardly anyone had heard of XML and several years of hard work and constantly rising transaction volumes (a respectable 10 M requests/day at the time, in early 2007) were now jeopardized by a single paragraph written by Tim Bray.
Seven years later, the Programmable Web's directory has close to 9000 registered public APIs (yes APIs, as in Ancient Programming Interface). That number is growing exponentially. Out of 9000 APIs, my best estimate is that less than 1% are following Tim Bray's Web Style. They are all but following an "API" style, short of RPC.
If we just take a sample of the newest APIs on the ProgrammableWeb (as of March 18), we have:
- Ask Ziggy which offers the ability (sic) to define "actions" (such as Play, NextSong, Previous Song, Shuffle...)
- WhatLanguage explains that you can use, as you wish, a GET (if your request is less than 7500 chars) or a POST to the same URI to detect the language of a given string
- Do.com actually seem to be offering a Web Style API, but it does not do much, it is simply CRUDing 5 resources (tasks, project, users,...)
- SkyBuffer is also following the Web Style, but just like DO.com, this is just some CRUD on a couple of entities
- MaShape which is a "Cloud API hub" is very interesting because they offer a better way for developers to consumer APIs. How do they do that? They invite developers to "Learn how to describe your API on Mashape to autogenerate client libraries and documentation". Yes you heard right, after years of bashing, developers start talking about client library code generation.
