One of the key foundations and most attractive principle of Agile or Lean methodologies is that "Everyone can help each other remain focused on the highest possible business value per unit of time".
I am certainly a strong supporter of that principle. However, value is often difficult to assess, I would actually argue that it is easier to identify what has less or little value, but what we think as valuable can potentially lead to many false positive, or simply be "business-as-usual" and hide the broader structure of the solution.
"User Stories" are the corner stone of identifying and delivering value:
An argument can be made that the user story is the most important artifact in agile development, because it is the container that primarily carries the value stream to the user, and agile development is all about rapid value delivery.
In practice, very few people focus on the benefits part of a user story. All user stories I see are either what we used to call "requirements" (just phrased slightly differently but isomorphically) or "tasks" needed to advance the state of the project.
However, there is a fundamental flaw in the construction of user stories, even when they are properly written, because they somehow make an assumption about the shape of the solution, and drive the author to turn almost immediately in solution mode, leaving no room for creative and out-of-the-box thinking.
Let's compare the metamodel of a User Story and to the formal definition of a Problem. The metamodel of a User Story looks like that (using the BOLT notation):
As a <role> I want to <action> so that <benefit>
I define a problem formally as a non existing transition between two known states , the metamodel of a problem looks like that:
A solution is a way to transition between these two states. Please note that both the actors and the actions are part of the solution:
This is where the problem lies when using User Stories, you are specifying the requirements with the solution in mind. There is, of course, a general relationship between some of the actors and entities of the system with the "start" and "end" states of the problem. The problem states are always defined in terms of their respective states (possibly as a composite state), but it is a mistake to think that the actors and entities that perform the actions, as part of the solution, are always the same as the actors and entities related to the (problem) states.
Hence, an action is solution centric and should not be part of the problem definition. As soon as you pick one, you have put a stake in the ground towards the direction you are going to take to solve the underlying problem. The other issue is that the start and end states are never clearly identified in a user story leading to confusion in the in the solutioning and verification process, since the problem is not defined with enough precision. Benefits could sometimes align with the target/desirable state, but the definition is often too fluffy and more goal centric, not effectively representing that (problem) state.
Ultimately, the relationship between problems and solutions is a graph (states, transitions as problems, actions as solutions), and this is where the coupling between the problem space and the solution space at the User Story level becomes unfortunate. This means that User stories cannot be effectively nested and clearly cannot fit in hierarchical structures (which is common to most Agile tools I know). This problem is quite accute as teams struggle to connect business level user stories and system level or solution level user stories. The concept of having a single parent directly conflicts with the possibility of having multiple possible transitions into a single state and decomposition principles where the same problem appears in the decomposition of several higher level problems.
I feel that distinction is profound because we can now clearly articulate:
a) the problem statements with respect to each other (as a graph of states and transitions)
b) we can articulate the solution in relation to the problem statements
c) we can articulate the verification (BDD) in relation to the problem and solution 
d) we can actually articulate the Business Strategy , the Problem Statement, the Solution and the Verification with the same conceptual framework
e) derive the Organizational IQ from the problems being solved on an every day basis
To the best of my knowledge none of these articulations have been suggested before and no one has ever provided a unified framework that spans such a broad conceptual view from the Business Strategy to the Verification. In the proposed framework the business strategy is simply a higher level and specialized view of the problem and solution domains, but using the exact same semantics (which are described here). In other words the enterprise is a solution to a problem, which is a composition of smaller problems and more fine grained solutions, etc. This has an extremely important implication for the execution of the strategy because now both the Strategy and its Execution are perfectly aligned, at the semantic level: the strategy, problem, solution and verification graph represent a map that everyone in the organization can refer to.
To take advantage of this new conceptual framework. I suggest that we make a very simple and easy change to Agile and replace "user stories" by "problem statements". Each problem must be "solutioned", either by decomposing it into simpler problems or solutioning it directly. Value can still be used to prioritize which problems are addressed first, that part of the Agile and Lean movement is very valuable, so too speak, but the focus on problems and solutions opens a new flexibility in how we handle the long range aspects of the solution while enabling the highest level of creativity and ultimately a direct articulation with the IQ of the organization.
As problems are decomposed, we will eventually reach a point where the subproblems will be close to or isomorphically related to the solution. But it would be a mistake to not clearly delineate the problems from solutions, simply because at the lowest level, they appear isomorphic.
The fact that the lifecycle is pretty much identical as the one of a user story enables most of the Agile processes and tools to work nearly unchanged.
You may want to know "How do I write a Problem Statement?". Personally, I don't like canned approaches. Oviously here, the mere definition of the two states (low value and high value) is enough to describe the problem. If a solution already exists (i.e. it is possible to transition between these two states) you may want to describe some characteristics of the new solution. I googled "How to write a Problem Statement?" and I felt there was already a good alignment betweent the results and the abstract definition provided above. For instance:
We want all of our software releases to go to production seamlessly, without defects, where everyone is aware and informed of the outcomes and status. (Vision)
Today we have too many release failures that result in too many rollback failures. If we ignore this problem; resources will need to increase to handle the cascading problems, and we may miss critical customer deadlines which could result in lost revenue, SLA penalties, lost business, and further damage to our quality reputation. (Issue Statement)
Here we see two states for the releases: initial state (low value) tested, and the high value state (in production). There is also an undesirable state (failure) that the new solution will prevent reaching. For me the most important thing is that the problem statement must avoid at all cost to refer to the solution. Even if the people specifying the problem statement have an idea about the solution, they should capture it separately.
This new focus on problem & solution provides a rich conceptual framework to effectively organize the work of a team. After all, we have been innovating, i.e. creating solutions to problems, for thousands of years, so it is no surprise that our vocabulary is quite rich. Here are a few concepts that could be used:
Goal: a goal is not a problem, but you often need to solve problems to reach goals, so it's important to keep them in mind
Fact: a fact often constrains the solution, so they need to be clearly surfaced and accounted for
Assumption: assumptions are very important because they also constrain the solution, but in a more flexible way. Assumptions can be changed, facts generally cannot.
Statement: the problem statement is what physically replaces the user story.
Hurdle: During the decomposition of a problem, hurdles might be identified, they are not a problem per say, but they impact the solution. It could be for instance that a resource is not available in time to meet the deadline.
Snag: A problem can be downgraded to a snag as the solution is obvious to the team and represent a low level of effort. It can also be a small unexpected issue, that need to be quickly resolved.
Dilemma: A problem can be upgraded to a dilemma, when several solutions are possible and it is not clear which one to chose
Setback: The team can suffer a setback when it thought it had found the solution but it didn't, or could not find a solution and need to reassess either the problem or the approach
On the solution side, we can also capture different elements and stages of the solutioning process:
Answer: Findings related to a question raised in the problem statement.
Result: A validation that the solution conforms to a Fact
Resolution: The choice made after reaching a dilemma
Fix: a temporary solution to a problem or a snag to make progress towards the solution to the greater problem
Development: An element of the solution, usually the solution to a subproblem or a snag
Breakthrough: The solution found after reaching a setback
Way out: A solution was not found, nevertheless, the project reached a satisfactory state to meet some or all of the initial goals
From a management perspective. The Solution or Delivery Manager can escape the bureaucracy that Agile has created. Ironically, moving stickers around is a zero value activity, with zero impact on the organizational IQ. The solution manager can and should be responsible for the IQ of the project, which rolls up and benefits from the IQ of the organization. It should keep track of the elements that are incorporated in the solution as problems are solved. It should encourage team members to be creative when necessary and to shamelessly adopt existing solutions when it makes sense. It should help resolve dilemmas and push for breakthroughs.
The PMO organization becomes the steward of the Organization's IQ.
As we define problems and solutions in terms of entities, state, transitions and actions, the BOLT methodology provides a unified conceptual framework that spans from Business Strategy to Problem and Solution Domains to Verification (BDD).
1) We have provided a formal model of a problem and a solution, and how they relate to each other
2) This formal model offers the ability to compose problems and solutions at any scale, over the scope of the enterprise
3) Problems and Solutions can be composed from Business Strategy down to Verification
4) We suggest that Agile methodologies replace User Stories by Problem Statements
5) With the renewed focus on "problems", we can also integrate the work of Prof. Knott on Organizational IQ in the whole framework
Last, but not least, decoupling problem definition and solution yields a tremendous benefit in the sense that both can evolve independently during the construction process.
 For instance, you build a vehicle, obviously you want to car to transition to the "in motion" state. Different "actions" will lead to the vehicle to reach that state (a horse pulling, an engine, transmission and wheels, a fan, ...).
 BDD Metamodel (Scenario):
 Living Social Business Strategy mapped using the same conceptual framework (Source: B = mc2)
Last week Subbu complained about Web APIs: "they are a pain". Really? Subbu gives four reasons:
1. Interfaces are bound to tradeoff special requirements of their consumers and try to stick to a common denominator
I don't get it, REST was sold to us as "easy". You can't actually give a talk or attend a talk without a guy in the attendance chanting "REST is easy". Why would you stick to a "common denominator" when you can push resource representations at will?
2. Writing client code is slow, repetitive and verbose
Subbu adds: "In one of the Java implementations I looked at, the code was over 300 lines long – that was after excluding the request making and response parsing code which was already factored into into supporting libraries. If the client needs to implement ten such use cases, you got 3000 lines of code just doing data retrieval."
I don't get it. The industry was sold on REST because, we could curl up calls to endpoints, god forbids, even GETting a resource representation directly from a browser's address field, in other words, REST is easy. We were even told that human readable documentation was enough. How in the world could we endup there?
3. API changes that the clients need don’t happen immediately. Sometimes never. The reality is that teams have their own priorities.
I don't get it. REST was sold to us because it had a "uniform interface" and good APIs URLs don't change. HATEOAS was once touted as the end of all evil, links everywhere, autodiscovery, and voila, you could fetch all you need with ease. What happened? why would you need "changes"?
4. Requests are chatty. Three hundred lines of code may not be a big deal, but making so many HTTP requests is.
I don't get it. Resource representations and the uniform interface were touted as what we would ever need and now we can only build "chatty" interactions.
Subbu is by far the most decent RESTafarian I know. Not only does he really knows what he is talking about, probably by a factor of 10, when compared to any other RESTafarians, but he is also genuinely looking for a better way to do things. But my questions to you Subbu, are:
a) was not this totally predictable 4 years ago? Who could imagine that by adding some annotations on top of Java (JAX-RS), you would end up anywhere else?
b) Isn't it time that we honestly and objectively look at (the other) "REST" (not Roy's REST)?
c) now that we are in this mess ocean of technical debt, what is the path forward ?
I stepped up to shake hands with Steve Vinoski at QCon last week, as I was getting ready to detail the program of the "Web API" track. That was quite a moment: sorry guys, Pizza is a not vegetable, no matter how many people say it or even when the US Congress says so.
We will showcase Canappi during an AT&T Webinar on "Building Multi-Platform Native Applications" on Wednesday 9/15.
Canappi is a Mobile Application Development Platform that generates iOS or Android code from a DSL. It also generates back-end services (MySQL/PHP and WSO2 Stratos Data Services).
Please register here to view the webinar.
Since 2007, the industry was told by a handful of people that the "Web" could teach us everything about distributed computing, and that everything we had done before was simply wrong. Four years later, we can only see that the emperor is completely naked (what else could we expect?).
REST was designed with an end-user in mind, operating a user agent: the browser (I know Roy is going to get mad if he reads this). Everything that has been written to apply REST principles to a software agent-to-server scenario is simply wrong. To quote Roy himself:
A REST API should be entered with no prior knowledge beyond the initial URI (bookmark) and set of standardized media types that are appropriate for the intended audience (i.e., expected to be understood by any client that might use the API).
A REST API should never have “typed” resources that are significant to the client. ... The only types that are significant to a client are the current representation’s media type and standardized relation names.
A REST API must not define fixed resource names or hierarchies (an obvious coupling of client and server). Servers must have the freedom to control their own namespace.
How many Web APIs are actually designed that way? It is time to move on from all this mind bending serendipitous complexity that brings absolutely nothing, except more lost productivity by forcing to everyone to hand code everything we got for free in previous distributed computing paradigms.
So I encourage everyone to unREST with pride. No offense to Roy, REST is brilliant. It is just not applicable to the problem at hand and we should give him some REST in return, for all that he has given to us. It's cool to be RESTless. Let's not waste any more time.
So what does it mean to unREST?
a) Define a uniform interface that suits your domain
There are many uniform interfaces to choose from: Send/Receive is one, GET/PUT/POST/DELETE is the "REST" one, CRUD has been used for 30 years world widely, ...
Don't be shy, forget the 4 little HTTP verbs, you can even define your own: Query/Do/Notify/Synchronize is a perfectly good one. It means that all methods in your Web API will be of type query, action, notification or sync request.
One could even imagine a 3-legged uniform interface, well suited to support modern converged / composite applications.
In fact, REST's so-called "uniform interface" is quite broken outside of the Web as some queries must be implemented with a POST and lots of people struggle to use PUT, POST and DELETE meaningfully. I have seen a standard using DELETE to implement the "cancel" payment operation. (I bet that's what some would like to do with their CDS transactions... )
A uniform interface is essential to a healthy API, pick one that fits your domain and stick to it. Let your infrastructure know about it from a security , scalability ... perspective.
b) Specify messages
when two agents communicate, they exchange messages. Duh? These messages rely on a shared understanding of some sort be it a standard type or semantic. Define your API messages loud an clear. This is a major step in becoming unREST. The syntax of your message doesn't matter, who cares about XML, JSON, Protocol Buffers? What is essential is that your syntax must be extensible. This is what allows forward versioning.
Explicit message definitions which can be explicitely versioned is essential to a healthy API definition. Message Extensibility is the property that makes distributed computing work.
c) Specify contracts with a clear versioning strategy
There has been many "IDLs" defined over the years, few were truly message based. Think of a relationship between software agents as forming a machine readable "Contract" that defines the rules upon which messages are exchanged between them. A contract is something you break only when you have no other choice. A Contract is most often bi-directional and asynchronous (anyone is receiving any notification on their phone?). Thinking of every relationship as being synchronous and of type client/server, is pure nonsense. Convincing people that they could do without a contract, just with some "human readable documentation" (HRD) is also pure nonsense. How many millions of hours have been wasted worldwide the day we decided to drop WSDL for "HRDs". A uniform interface is not enough to form a contract, it is only the substrate on which a contract is defined. That contract is not uniform, by any stretch of imagination.
Machine readable contracts are the foundation on which to build a healthy ecosystem of Web API consumers. Versioning is essential to evolution and reuse, and hence a healthy API.
Please note that, reuse, in distributed computing, happens the other way around when compared to traditional software engineering. It is the old "clients" of a Web API that "reuse" the new versions of a service (without breaking). Who would expect that anyone could design an API that 3 years down the line would be "reusable" by any 3rd party consumer? It is far more valuable to enable existing client to "reuse" the new features of your API, at their pace, without breaking them, for every tweak you make into your message formats.
That's it, that's all you need to create a successful API nothing else, nothing more. Sure, it would be great if we converge on a handful of uniform interfaces and versionable message /contract specifications. I am not sure we can find "the one" that works for everyone.
I have participated in crafting web-based protocols for most of the last 12 years, starting with ebXML in 1999. The big difference between today and then, is that we have thousands of openly available data points teaching us, we, the "standard" guys, how people actually create APIs, for what purpose. Maybe for once, they could teach us a thing or two. That would avoid picking just a couple of use cases convenient to a vendor's product and forcing the REST of the world fit under that petty umbrella.
You are now free to unREST and laugh quietly at anyone who asks you to marvel at an HTTP header, a "link", use a 7 letter acronym, or tells you with the authority of a story teller that you don't get it. REST was just a (bad) dream, unREST is what you want to do.
The core of the presentation is on two slides (10 & 11). They focus on two important dimensions of mobile apps: the information dimension and the actions dimension (there are more such as the Converged dimension or the UX dimension).
Mobility is a major paradigm shift because mobile apps are based on a 3-legged information model: Product-Place-People, unlike the tethered Web that has no actionable location information. Amazon's business model, for instance, relies solely on linking products to people and I would not be surprised if a startup coming out of nowhere disrupts Amazon's eCommerce franchise.
Why? because location is not just about finding disruptive way to link products and people, it is also weaving time (i.e. events) into the business model and, most importantly, it makes the world "actionable" (taking actions at a distance doesn't always work well...).
This new relationship between people and products (via places, events and action) gives the opportunity to create a virtuous circle (slide 11): people, places and events enable advanced advertising/offers models and as people take actions on goods, services, media, knowledge ... (such as purchase, consume, invoke, favor, link, collect ...) it gives the ability to capture favorites and historical data (which would be impossible to tie to a location or event without a mobile device) thereby providing a formidable feedback loop to advertising models and offers in very innovative ways.
Hope you like the presentation and join us in Redmond if you can !
Boris posted yet another article on SOAP vs REST on InfoQ yesterday. I had demonstrated last year that REST had brought nothing new. This is year it is even more crying. Nobody cares any longer about being RESTful, the industry even came up with a new term to celebrate this transition: "Web APIs". "Service" is the new "Distributed Object". I am actually ready to give $100 to the first person who proves me that there are more than %1 of Web APIs on the Programmable Web that are "RESTful" (i.e. at least 33 with the current count of 3300+). What I mean by RESTful is that 100% of the provider's Web API follows REST principles (URL path, correct use of HTTP verbs, use only HTTP verbs + nouns, HATEOAS, media types ...), you know the kind of things that the RESTafarians sold to the industry way back when in 2007. (I am dead serious, send me your list and I'll publish it and send you a check).
The problem that REST has created (that my good friend Jim Webber would simply qualify as the Same Old Atrocity) is that now any API designer gets a free license to be a distributed computing guru, artistically deciding where to put things like version and credentials, and encode actions and queries as they please amongst a wealth of headers, tokens, query parameters, bodies and what not. If you ask me, REST has produced the software industry WOrst Atrocity ever.
But, let's move on, because it actually doesn't matter: we now live in a POST REST/SOAP world. Converged (Mobile) Applications have pushed the envelope of what distributed computing should do and how it does it. Never HTTP or SOAP were really prepared for 3-legged authentication, respect privacy and support monetization at the scale of several billion API calls/month. Actually SOAP was a bit more prepared since a message could have a number of security tokens representing different principals, but who cares, we were told that HTTP was the answer, because it was "RESTful" so now we struggle with all kinds of three-legged authentication mechanisms.
A converged app is an app that is leveraging a combination of 21st century capabilities like voice, messaging, data, location, video, payments ... in a secure and privacy-friendly way. A typical converged app would be for instance an app that is able to call FourSquare Web API to get my list of friends nearby, lookup their phone numbers from a "My Contacts" API (Local or Remote), then call Yelp to find the places where we could meet or EventFul for the events we could go to and finally call Twilio's SendSMS API, and see who responds. The app collects the votes, and everyone meets at the place that has the most votes. After the party, the same app would allow me to share pictures and videos with all my (FB) friends. When using a converged app, end users don't want any API provider (and marketer) to know what they are doing, this is called privacy, something the good old Web made a business model of, on your back. Web advertising has become creepy lately because I visit sites I never visited before who advertise for products I searched on Google last week. That is really creepy, it turns me off completely.
Converged applications are changing everything because the ability to mashup APIs (and no longer just establishing a one-to-one consumer/provider relationship) becomes the most important focus of the application, in a 3 legged scenario where the user, the application and the Web API provider(s) are in a completely different relationship than the monolithic relationships we were so used to, in the mainframe, client-server and Web app days, where basically the enduser "logs in" to a monolithic app and that app only knows what the user does with it.
What's a "contract" in this new world of Converged Applications? after all the industry is only selling billions of them every year. Shouldn't we care a bit? As a starter, if you are an API provider you better get ready to manage the "apps" that consume your API and start issuing API keys. Is there a standard for that? uh? sorry, the standard bodies missed that. API keys are now the key to your endpoints. Shouldn't you also try to avoid the embarrassment of a Facebook which leaks its enduser identifiers in its API in 3-legged scenarios, seriously and totally compromising the privacy of their members. uh? is there a standard for securely and privately identifying users (and other key identifiers) across converged apps? uh? can REST do that? SOAP? wake up... guys the world if moving at lightning speed.
The old "consumer/provider" relationship established by SOAP/WSDL and later "enhanced" with HATEOAS is completely overwhelmed by this new world. Just take the simple problem of mapping data identifiers in a mashup (PK/FK). We need a new "contract" concept much more consumer centric, that helps build and maintain converged applications. At canappi, I have started to come up with such contract definition. If we wait for another year to come up with a decent Web API versioning strategy will create "for-ever" APIs or a massive break-down in all the converged apps. We also need much more sophisticated security standards, where a service provider needs to enforce that an authenticated application, used by an authenticated user does what it is supposed to do while understanding a thing or two about privacy and monetization (Converged App users vote with their wallet, not "clicks"). Finally, I contend that HTTP is completely inefficient to serve Gazillions of API calls per year and that we need a much better API protocol.
We also need standards that are (client) developer friendly not Google or Facebook friendly, that let them write apps that are (3-legged) secure, privacy-aware and monetizable (unlike HTTP or SOAP). It's already hard enough to write a mobile app, we don't need any kind of additional crap because it is convenient to X,Y or Z API provider. We are in the digital economy (if you doubt it, just check how many swords your phone carrier sold last year): we need the standards to support it. Will the industry really have the courage to take that on? will my good friends in the standard bodies, for once, try to be ahead or at the very least in line with their time? or will the RESTafarians one more time force us to hand code every f**ing API calls? I doubt we'll do the right thing, but I'd love to be wrong.
The Web, as it was designed 20+ years ago, is old, way too old to move forward in the Converged Apps world. Anyone who tries to convince you otherwise is emotionally attached to this great but rapidly aging platform.