This post is a synthesis of two posts I originally published on my other blog "unRelated".

One of the key foundations and most attractive principle of Agile or Lean methodologies is that  "Everyone can help each other remain focused on the highest possible business value per unit of time".

I am certainly a strong supporter of that principle. However, value is often difficult to assess, I would actually argue that it is easier to identify what has less or little value, but what we think as valuable can potentially lead to many false positive, or simply be "business-as-usual" and hide the broader structure of the solution. 

"User Stories" are the corner stone of identifying and delivering value:

An argument can be made that the user story is the most important artifact in agile development, because it is the container that primarily carries the value stream to the user, and agile development is all about rapid value delivery.

In practice, very few people focus on the benefits part of a user story. All user stories I see are either what we used to call "requirements" (just phrased slightly differently but isomorphically) or "tasks" needed to advance the state of the project.

However, there is a fundamental flaw in the construction of user stories, even when they are properly written, because they somehow make an assumption about the shape of the solution, and drive the author to turn almost immediately in solution mode, leaving no room for creative and out-of-the-box thinking.

Let's compare the metamodel of a User Story and to the formal definition of a Problem. The metamodel of a User Story looks like that (using the BOLT notation):

As a <role> I want to <action> so that <benefit>

us-model

I define a problem formally as a non existing transition between two known states [1],  the metamodel of a problem looks like that:

pb-model2

 

A solution is a way to transition between these two states. Please note that both the actors and the actions are part of the solution:

sol-model

 

This is where the problem lies when using User Stories, you are specifying the requirements with the solution in mind. There is, of course, a general relationship between some of the actors and entities of the system with the "start" and "end" states of the problem. The problem states are always defined in terms of their respective states (possibly as a composite state), but it is a mistake to think that the actors and entities that perform the actions, as part of the solution, are always the same as the actors and entities related to the (problem) states.

Hence, an action is solution centric and should not be part of the problem definition. As soon as you pick one, you have put a stake in the ground towards the direction you are going to take to solve the underlying problem. The other issue is that the start and end states are never clearly identified in a user story leading to confusion in the in the solutioning and verification process, since the problem is not defined with enough precision. Benefits could sometimes align with the target/desirable state, but the definition is often too fluffy and more goal centric, not effectively representing that (problem) state.

Ultimately, the relationship between problems and solutions is a graph (states, transitions as problems, actions as solutions), and this is where the coupling between the problem space and the solution space at the User Story level becomes unfortunate. This means that User stories cannot be effectively nested and clearly cannot fit in hierarchical structures (which is common to most Agile tools I know). This problem is quite accute as teams struggle to connect business level user stories and system level or solution level user stories. The concept of having a single parent directly conflicts with the possibility of having multiple possible transitions into a single state and decomposition principles where the same problem appears in the decomposition of several higher level problems. 

I feel that distinction is profound because we can now clearly articulate:

a) the problem statements with respect to each other (as a graph of states and transitions)

b) we can articulate the solution in relation to the problem statements

c) we can articulate the verification (BDD) in relation to the problem and solution [2]

d) we can actually articulate the Business Strategy [3], the Problem Statement, the Solution and the Verification with the same conceptual framework

e) derive the Organizational IQ from the problems being solved on an every day basis

To the best of my knowledge none of these articulations have been suggested before and no one has ever provided a unified framework that spans such a broad conceptual view from the Business Strategy to the Verification. In the proposed framework the business strategy is simply a higher level and specialized view of the problem and solution domains, but using the exact same semantics (which are described here). In other words the enterprise is a solution to a problem, which is a composition of smaller problems and more fine grained solutions, etc. This has an extremely important implication for the execution of the strategy because now both the Strategy and its Execution are perfectly aligned, at the semantic level: the strategy, problem, solution and verification graph represent a map that everyone in the organization can refer to. 

To take advantage of this new conceptual framework. I suggest that we make a very simple and easy change to Agile and replace "user stories" by "problem statements". Each problem must be "solutioned", either by decomposing it into simpler problems or solutioning it directly. Value can still be used to prioritize which problems are addressed first, that part of the Agile and Lean movement is very valuable, so too speak, but the focus on problems and solutions opens a new flexibility in how we handle the long range aspects of the solution while enabling the highest level of creativity and ultimately a direct articulation with the IQ of the organization. 

As problems are decomposed, we will eventually reach a point where the subproblems will be close to or isomorphically related to the solution. But it would be a mistake to not clearly delineate the problems from solutions, simply because at the lowest level, they appear isomorphic. 

If we start drawing some BOLT diagrams, a problem lifecycle can be defined as:
problem-lc

The fact that the lifecycle is pretty much identical as the one of a user story enables most of the Agile processes and tools to work nearly unchanged.

You may want to know "How do I write a Problem Statement?". Personally, I don't like canned approaches. Oviously here, the mere definition of the two states (low value and high value) is enough to describe the problem. If a solution already exists (i.e. it is possible to transition between these two states) you may want to describe some characteristics of the new solution. I googled "How to write a Problem Statement?" and I felt there was already a good alignment betweent the results and the abstract definition provided above. For instance:

We want all of our software releases to go to production seamlessly, without defects, where everyone is aware and informed of the outcomes and status. (Vision)

Today we have too many release failures that result in too many rollback failures. If we ignore this problem; resources will need to increase to handle the cascading problems, and we may miss critical customer deadlines which could result in lost revenue, SLA penalties, lost business, and further damage to our quality reputation. (Issue Statement)

Here we see two states for the releases: initial state (low value) tested, and the high value state (in production). There is also an undesirable state (failure) that the new solution will prevent reaching. For me the most important thing is that the problem statement must avoid at all cost to refer to the solution. Even if the people specifying the problem statement have an idea about the solution, they should capture it separately.

This new focus on problem & solution provides a rich conceptual framework to effectively organize the work of a team. After all, we have been innovating, i.e. creating solutions to problems, for thousands of years, so it is no surprise that our vocabulary is quite rich. Here are a few concepts that could be used:

Goal: a goal is not a problem, but you often need to solve problems to reach goals, so it's important to keep them in mind

Fact: a fact often constrains the solution, so they need to be clearly surfaced and accounted for

Assumption: assumptions are very important because they also constrain the solution, but in a more flexible way. Assumptions can be changed, facts generally cannot.

Statement: the problem statement is what physically replaces the user story.

Hurdle: During the decomposition of a problem, hurdles might be identified, they are not a problem per say, but they impact the solution. It could be for instance that a resource is not available in time to meet the deadline.

Snag: A problem can be downgraded to a snag as the solution is obvious to the team and represent a low level of effort. It can also be a small unexpected issue, that need to be quickly resolved.

Dilemma: A problem can be upgraded to a dilemma, when several solutions are possible and it is not clear which one to chose

Setback: The team can suffer a setback when it thought it had found the solution but it didn't, or could not find a solution and need to reassess either the problem or the approach

On the solution side, we can also capture different elements and stages of the solutioning process:

Answer: Findings related to a question raised in the problem statement.

Result: A validation that the solution conforms to a Fact

Resolution: The choice made after reaching a dilemma

Fix: a temporary solution to a problem or a snag to make progress towards the solution to the greater problem

Development: An element of the solution, usually the solution to a subproblem or a snag

Breakthrough: The solution found after reaching a setback

Way out: A solution was not found, nevertheless, the project reached a satisfactory state to meet some or all of the initial goals

...

From a management perspective. The Solution or Delivery Manager can escape the bureaucracy that Agile has created. Ironically, moving stickers around is a zero value activity, with zero impact on the organizational IQ. The solution manager can and should be responsible for the IQ of the project, which rolls up and benefits from the IQ of the organization. It should keep track of the elements that are incorporated in the solution as problems are solved. It should encourage team members to be creative when necessary and to shamelessly adopt existing solutions when it makes sense. It should help resolve dilemmas and push for breakthroughs.

The PMO organization becomes the steward of the Organization's IQ.

As we define problems and solutions in terms of entities, state, transitions and actions, the BOLT methodology provides a unified conceptual framework that spans from Business Strategy to Problem and Solution Domains to Verification (BDD).

To summarize,

1) We have provided a formal model of a problem and a solution, and how they relate to each other

2) This formal model offers the ability to compose problems and solutions at any scale, over the scope of the enterprise

3) Problems and Solutions can be composed from Business Strategy down to Verification

4) We suggest that Agile methodologies replace User Stories by Problem Statements

5) With the renewed focus on "problems", we can also integrate the work of Prof. Knott on Organizational IQ in the whole framework

Last, but not least, decoupling problem definition and solution yields a tremendous benefit in the sense that both can evolve independently during the construction process. 

_______________________________________________________________________________

[1] For instance, you build a vehicle, obviously you want to car to transition to the "in motion" state. Different "actions" will lead to the vehicle to reach that state (a horse pulling, an engine, transmission and wheels, a fan, ...).

[2] BDD Metamodel (Scenario):

bdd-model

 

[3] Living Social Business Strategy mapped using the same conceptual framework (Source: B = mc2)

lifecycle

I have been asked recently to create an estimate of the costs of versioning APIs (or Web Services). I wanted to share this estimate because I feel a lot of people still don't understand the cost implications of API/Service versioning.

We found in this study that the cost of building APIs can vary greatly based on the approach / pattern used to deliver subsequent versions of that API (or Web Service)

The key point to understand is that even if the cost to your consumers may look small to you, it is not just a pure cost, it is risks, disrupted project plans, unavailable budgets... with changes that often have no immediate business value to an existing consumer who was not expecting any change to API.

When it comes to API versioning, I feel that there are three types of patterns that are relevant:

A) The Knot

All API consumers are tied to a single version of the API, when that API changes, all consumers have to change, in essence creating a massive ripple effect across the entire set of consumers / ecosystem.

 

The cost of creating new versions of the service is best represented by that formula:

 

where:

S = cost of building an API/Service

V = cost of building a version (update to API/Service)

U = cost of updating a group of API consumers

T = cost of testing each group of consumers works properly

 

 

B) Point-to-Point

Every service version is left running in production and consumers are required to migrate on their own, when they need to. The maintenance costs increase as the number of version in production increases.

The cost of creating new versions of the API/Service is best represented by that formula:

where:

S = cost of building an API/Service (as an update to API/Service)

U = cost of updating a group of API consumer

T = cost of testing each consumer works properly

 There is also a hidden assumption in this formula which does not take into account the cost of operating more than one API version (which is not the case of Pattern 1 or 3). My assumption is that, U, the cost of updating a group of API consumer (which may not happen) represent the cost of operating a service, when it is not updated to catch up with newer versions. 

C) Compatible Versioning

 All clients talk to the same compatible API/Service version.

 

The cost of creating new versions of the API/Service is best represented by that formula:

where:

S = cost of building an API/Service (as an update to the API/Service)

U = cost of updating a group of API consumers

C = cost of certifying that each group of API consumers will work properly

A = cost of preparing consumers and clients for compatible versioning 

Approximations & Assumptions

Let's do some approximations to be able to compare the relative cost of each pattern to the entire ecosystem (consumers and services).

Results

The results seem to speak for themselves: a single version forcing every consumer to upgrade when the API changes is the most expensive to the ecosystem. A multiplicity of versions that need to be maintained is better, but still quite costly when you try to keep upgrading each version or alternatively operating older versions. A compatible versioning strategy seem to offer the best efficiency. 

Of course in reality not every version can be compatible and an API with major (incompatible) version and (minor) compatible versions will be a combination of pattern 2 and 3. In any case, you should never put the consumers in the position to have to upgrade, unless this is a case of Force Majeure

In this figure, the Y axis represents the costs of creating API/Service versions, based on the approximations and assumptions stated above, the X axis is the version number of the API/Service (1..10).

In an effort to illustrate how Metaprogramming works, I have decided to open source Canappi (there is no license attached, feel free to use it whichever way you want, that includes building mobile development tools).

Canappi itself is a model-driven mobile application development tool. Canappi can be compared to an Interface Builder that works for both iOS and Android, understands application flows, business logic and back-end APIs. I built Canappi with a Language Workbench called Xtext. Canappi's programming language is called "mdsl" (as in mobile domain specific language). Canappi generates native iOS and Android code from an mdsl file which specifies the solution. Canappi is textual, not visual, however, I have build code generators that consume Apple's IB files that generate mdsl and deliver a graphical design experience that way (not bad, you can use IB to design Android GUIs). Canappi's code generator itself resides in the Cloud, running on Amazon EC2.

Metaprogramming is a new approach to writing software which decouples and articulates properly three important aspects of the software construction process:

  • Conceptual (What is a house? - according to me the builder )
  • Logical (What does this house look like? - surely not exactly the same others I have built)
  • Physical (How do I build a house, today? which may change over time, or depend on geography)

In traditional software engineering, the Conceptual aspect is made of the programming language + software development kit, which are typically under the control of a vendor. Even though it looks appealing to rely on a not so stable conceptual aspect, I strongly believe, after nearly 35 years of programming, that it is the root of all software engineering problems. It forces developers to create a virtual conceptual aspect in their mind that they constantly translate into the vendor's conceptual aspect -which we all know is pristine and never changes. Imagine now if you had to refactor the developer conceptual aspect...

Development teams should, instead, have full control over these three levels and a programmer should be allowed to augment or refactor the (now explicit) conceptual aspect at any time, continously, as the solution is being built. That ... is metaprogramming. The conceptual aspect should also be independent of any technology: technology should be dealt with in the physical aspect, exclusively. Very few solutions need to be cross-platform, unless you are a vendor. Even though it is easier to create a cross platform conceptual aspect using that approach, I am not sure today it is that desirable, especially in the context of highly differentiated user experiences (iOS, Android, Windows 8). 

With all that being said, if you truly want to become a metaprogrammer you should not be building tools. It is tempting to design the conceptual and physical aspects as a tool such that it could be reused in different solutions, and even try to sell it as such, but this would not be metaprogramming, and to the best of my knowledge that kind of initiative is most likely to fail like thousands before it. You should look at the code released here as the starting point to build a mobile solution following the principles of metaprogramming, not as a mobile development tool itself. 

Metaprogramming is not about building "libraries" or "wrappers", the conceptual + physical aspects are the (solution specific) libraries and wrappers. Why would your client-side team follow the same principles as your SOA team? as your package apps team? not a chance. Metaprogramming is both freedom and structure. 

This approach was not possible until the advent of Language Workbenches. I don't believe in visual programming, I like code and I am glad that Language Workbenches are textual. I don't believe in Internal DSLs either, even though they support metaprogramming well, they are technology dependent. Metaprogramming is about minimizing technology dependencies as much as possible. 

So today, I published Canappi's Conceptual and Logical aspects to bitbucket (thank you Jordi for the tip). These two aspects can be achieved with a single file ... MDsl.xtext and ... of course, Xtext, which can create an Eclipse plugin from the programming model specified in that file. In general the syntax of the programming model can be adapted to look like familiar syntaxes. The plugin allows you to create the logical model of the solution based on the programming model you defined at the conceptual level. 

Canappi uses Xtext 1.0.2 (it will not work without some cleanup with the latest version of Xtext).

Even though I use Xtext, I am not tying Metaprogramming to Xtext. I use it because I became familiar enough with it, not because it is the best tool. I would recommend you do your homework and find a language workbench that is best suited to your needs. 

The code can be found here. I will release the code generators over the next couple of weeks. It is now complete, you can find all the components of Canappi:

  • the MDSL language
  • the iOS, Android, ... code generators
  • Balsamiq -> MDSL generator
  • Interface Builder -> MDSL generator
  • MDSL code samples

I'd be the first one to recognize that I am playing way out of my league here, no question about it, but it seems to me that Computer Science has been neglecting, if not abusing, the notion of state in ways that seem to be counter productive. I understand that the work of Church and Turing was all about "Computing" and not surprisingly when we started to look for better abstractions to write programs the foundation of λ-calculus was never reassessed but it is difficult today to not ask the question as to whether Abstract Data Types are at the foundation of the General Theory of Software Engineering (GTSE).

I spent the last couple of weeks pounding on the idea that there is more to Barbara Liskov statement about the fundamental role of programs than Abstract Data Types

I believe that the fundamental role of programs is to modify state not just the bits

Indeed, when you look at it closely, the role of "computers" today is far less about "computing" than it is about managing state, even though computing is required to transition from state to state properly.

After defining BOLT, and looking at extensions in the problem domain, I found that the idea of defining a programming model based on Abstract State Types could shed some new light on what a programming language truly is.

Surprisingly, there is very little literature about ASTs as I define them. It seems that ASTs fit the model of Coalgebras, but again, I am playing way out of my league here. 

In the paper referenced below, I define an AST as a simple data structure (key value pairs) which all share the same lifecycle (yes, states have a lifecycle too!). A state lifecycle is triggered by an interaction between two (or more) states. States are immutable and new states (of the same type or a different type) are created at the end of an interaction. I define 3 composition mechanisms (containment, extension, connection). Interestingly, there is no ASM (Abstract State Machine) behind ASTs. What we see as a "State Machine" is actually the occurence of repeatable interactions, so all the complexity associated to modeling state machines goes away.

If anyone feels there is a strong analogy between ASTs and QED, well, it is nearly by design, but it really came about from me staring at UML State Diagrams, linking them to business process execution, then to business strategy and then all the way back to problem definition and today programming model. 

This paper is far from complete, a very rough draft at best. I publish it as is for three reasons:

a) I strongly believe that the fundamental idea behind ASTs will not change

b) It would take me an infinite amount of time, which I don't have, to bring it to maturation

c) This is really a call for help, if someone would like to jump in as a coauthor, to further the idea

I don't see ASTs as being the next gen programming concept, ... well who knows, but it was hard to resist not extending the BOLT conceptual framework to the programming model level, especially in the context of Barbara Liskov's statement on the role of programs.

Here is the paper, as usual any feedback is welcomed, especially if someone has already worked on that idea. My next step is really to put this idea in practice to explore it further. In particular the paper does not speak at all of the general orchestration behind state interactions (mechanisms such as addressing, activation, ...) but this is by design because I feel it is totally orthogonal.

Again, I welcome the contribution of one or two co-authors, otherwise I'll continue working on this paper in an open content mode.

 

I have an infinite respect of Prof. Liskov, for her, personally, and her work. A lot of what she has discovered has guided my own work and the work of millions more for the past four decades and her insight on the role of programs has litterally transformed and shaped the way I think:

I believe that the fundamental role of programs is to modify state not just the bits

But, what if her fundamental assumptions were off? After all they are assumptions and assumptions need to be revisited periodically, they are (somewhat) artificial, often unconscious, constraints that we believe make it easier to find the solution of a given problem. Just ask Richard Feynman.

After reading, and rereading her seminal paper on "Programming with Abstract Data Types", it is difficult not to question her assumptions. 

For centuries, we have created valuable abstractions that make it easier to solve problems, very complex problems. As we all understand the value of high level programming languages, it would be hard to disagree, for instance, with her introductory statement:

Unfortunately, it is very difficult for a designer to select in advance all the abstractions which the users of his language might need. If a language is to be used at all, it is likely to be used to solve problems which its designer did not envision, and for which the abstractions embedded in the language are not sufficient.

Yet, that statement creates an unfortunate bias that somehow the language designer and the solution developer must always be two different people. It also creates another unfortunate bias, as it, somehow, suggests that every team of developers should use the same programming language, maybe within a tier, if not for the entire solution stack.

Actually, it is probably very difficult to not count these two assumptions as mere facts. My career started pretty nicely with the development of advanced semi-conductor deposition systems. I used NeXTStep to build these systems: ADTs, OOP, and Model-Driven development tools (Interface Builder) offered a delightful way to create a model of these systems. These abstractions were so powerful that I ended up building a Natural Language Processor on top of the control systems (using Objective-C's reflection mechanisms). Back in 1996, this control system was jaw dropping, but to be fair, NeXTStep, ADTs, i.e. abstractions, combined with some elbow finger grease was all that was needed to create that kind of magic. Retrospectivelly, I trully think this was magic, and I could have never build such a system by myself without all the good work that was done at that time. 

In 1997, when Steve Jobs returned to Apple and killed NeXT, I had to leave this wonderfully abstracted world and decided to join the ranks of Information Technology professionals. IT was very different from the world of industrial process control: none of these abstractions make any sense in IT. When I say any, I really mean any. Everywhere I turned, ADTs didn't help: BPM (process), SOA (service), XML (eXtensible Data Structures), HTTP (Protocols and integration), SQL or NoSQL (query languages and datasets).  Worse, ADTs seemed to make things far more complex than one would have originally thought.

So why not aiming at a completely different direction? 

  • What if every development team would be allowed to build (and have control over) their programming language?
  • What if every developer in that team would have the flexibility to create their own programming language, if it made sense? 
  • What if these programming languages could even be solution specific, again if it made sense? 

I know, this is a radical idea. I just ask you to consider the following proposal. The work of a team of developers would look like something like this: 

 
What's different here, compared to traditional Model Driven Software Development is that the developer writes code in three dimensions:
- the programming model, i.e. the abstractions that are best suited to create the solution
- the solution model, a.k.a the metadata, expressed with the abstractions of the programming model
- the code generator that generates the code from the solution model
 
For every feature, the developer has now the choice to change or enhance the programming model, add more metadata or change the code generator. The problem of round tripping mostly disappears as we never need to touch the generated code directly (it happens in the code generator). 
 
 
 
In many ways this is how developers think already. Their thougth process is already structured along the lines of a conceptual programming model, sometimes well aligned with the one of the underlying technology, and sometimes not. The code that they write is then projected in the underlying technology using well defined or ad hoc patterns that truly amount to a specific programming model. Actually, I think that anyone who has worked long enough in our industry would agree that the greatest risk to any project is actually when developers think that the underlying technology is the conceptual programming model of the solution they are building. So why not make all these abstractions explicit and technology independent? Technologies do change and after all, architecture refactoring is one of the hardest thing to do. 
 
MDE, MDSD, MDA... failed because just like for traditional programming language you simply cannot design abstractions in a distance. That is a myth. You can't have a group or a vendor produce the abractions and the developer figure out how to use them to solve the problem at end. This is even more clear in MDE as the model is by definition more rigid than in a 3GL. The development of the programming model has to be a continous process, along side the construction of the solution. Over time, the programming model and the code generator may stabilize, for a given activity, a type of solutions or even an entire company, but this should be a non goal. The goal of metaprogramming is to create the best abstractions to solve the problem at hand, well, because, this is what (good) developers do anyway, but for most of us, in a very crooked way.
 
Canappi (a tool I developed over the last 2 years, allows me to do just that). I built this beautiful next generation social app for 6IX Inc.
 
Metaprogramming allowed us to try different UX and quickly converge to what we felt was optimal. All the MongoDB and S3 client code is generated and modeled at the programming model level. 
Ultimately Metaprogramming delivers 3 key benefits to a project:
1) As the project progresses, you end up mostly editing the solution model. Writing code at this level is usually very compact and very stable. This means that you can make major functional changes up to the last minute, if you need to
2) You can refactor the code generator fairly easily, for that app, the code generator is about 10k LOC, while the generated code is slightly above 100k LOC. As you refactor the technology, this is about 90% less effort (for instance when the iPhone 5 came out last September, it took me 2 hours, to retrofit the programming model and the code generator to support it, after a few more hours of design we modified the solution model of the application and we had a side-by-side iPhone 4/5 the day after the iPhone 5 came out).
3) QA costs go way down, again as you are in effect stabilizing far less amounts of code
 
I use Eclipse Xtext with Xtenda Language Workbench, to develop the programming model, the corresponding Eclipse Plugins and the code generator, but other choices are possible with MPS , Essential or Spoofax. I can assure you that this is far easier than you may think.
 
It is hard to believe that in the late 70s people would have thought that Language Workbenches were something that was even remotely possible, at least not to the level of Xtext, Xtend and Xbase. I sure would be really curious to ask that question to Barbara one day. 
 
It is key to understand that metaprogramming is fully anchored in the solution domain (not the problem domain). I respect the fact that some solutions may be built directly from the specification of the problem, but after writing this post on User Stories vs Problem Statements, I now trust even less our ability to create full fledge solutions from a problem specification. I am not saying it is always impossible, I am just saying it is generally much harder and delivers less compeling solutions. 
 
(All pictures are from Wikimedia)

1 2 3 4 5 6 7 8 9 10 11 ... 21 >>

Search

 Subscribe







blog engine