• ebpml
• wsper
• infoq
• Feed
• del.icio.us

07/08/08 :: [SOA] Hint: Programming Models Matter in BPM too  [permalink]

Fred Cummins asked the question last week: Why Should BPMN Support Choreography?

I may be wrong but I believe that BPDM started to support the concept of choreography under the momentum of Antoine Lonjon who was actively participating in the UN/CEFACT BPSS working group in the early 2000s.

Having been one of the editor of the OASIS ebBP 2.0.4 (and UN/CEFACT BPSS 1.1) and worked on BPSS from 1999 to 2005, I may have an opinion or two on the question. Of course I have already clearly explained my opinions here on the relationship between BPMN, BPEL and Choreographies.

I generally highly respect the work coming out of the OMG, BPDM is no exception. This is the best "standards" organization and a lot of other standards organization could learn a thing or two on specification development processes. That being said, I am not sure the BPDM working group is really clear about what they are attempting to do.

Requirements for BPDM were very ambitious (way too ambitious?):

• Provide a common basis for all process oriented models
• Provide support for the service oriented world
• Integrate rules within processes
• Ensure Execution Interoperability of process models
• Use BPMN as the standard notation for processes
• Leverage other “process” knowledge : UML, BPMN, PSL

Unfortunately, the work around BPDM is rooted in corny understanding of the Business Process Management field. At the time when Antoine left the BPSS group the industry was still talking about "public" and "private" processes and of course, the corniest idea of all, that somehow there was a "compilation path" between a "business process model (BPMN)" and "orchestration" (BPEL) popularized by BPMI and BizTalk Server.

I have news for you, programming models matter in BPM too. It is not exactly new since I published this in 2002, but orchestration, choreography, B2B and workflow can participate in a unified programming model. I apologize to have to repeat myself, but these concepts seem to be so foreign to the BPM intelligentsia that it is not even funny to discuss them with them. I am actually quite disturbed how people can be closed to any discussion in that field too.

So one more time, the center of this unified programming model is the "resource lifecycle", I will say. Resources lifecycles interact and to events they publish and subscribe (not the processes, this is a myopic vision of BPM, an event is the occurrence of a state). Workflow of activities represent the work that is performed to advance the state of resource lifecycles. This model is trivial to understand, relatively trivial to implement with orchestration languages such as BPEL, BizTalk XLang, or Workflow Foundation. If you understand this programming model then to the following conclusions, you can come:

a) Business Processes "do not exist", a workflow, specified in BPMN for instance, is merely a possible set of activities to achieve a particular goal, it is not "THE set of activities". Do they represent the only possibility to achieve this goal? no usually the combinatorial possibilities are such that you can hardly define (with precision) a complete workflow when taking into account all states and transitions and all activities. Is BPMN useful, of course, it is important to use it for defining "best practices", communicate with the business, understand behavior and bottlenecks, but it is completely stupid to ever think someone will make use of BPMN to directly build entire solutions. The pundits seem to argue that they never said that, but as one of the post I read said, they kind of sell it that way even if they don't say it as clearly.

b) The distinction between private and public processes do not exist. A B2B boundary is an arbitrary legal boundary, you can label some of the interactions between resource lifecycles as being B2B but from a programming model perspective they are absolutely no different from a "private" interaction. Intuitively you should think, that if a manufacturing company buys its supplier there should be "no difference" in the information systems. The only thing that changes is the visibility (and control of course) but from a programming model perspective we should be able to express the continuity that truly exist.

c) Choreographies exist but the concept of Assemblies, I prefer (This is one and the same concept). Sure in B2B, you need an explicit contract, this is what OASIS ebBP offers today in a very sophisticated way. Now, I think that at the programming model level, WS-CDL and SCA's assembly specification should somehow merge. This would greatly improve SCA's assembly (only based on interface signatures today) and it will clearly show the value of WS-CDL in this programming model.

So, just as with the remoting bunch, the BPM bunch will err for decades until they finally understand the unified programming model behind their work (if ever). Interestingly (but not surprisingly) a tight coupling between the BPM programming model and the interactions that can be "remoted", there is.

You can see just by looking at the kinds of things that the remoting bunch (REST/RPC) and the BPM bunch (BPDM/BPMN) are doing and talking about that they will not connect for another 50 years. It is kind of ironic because the OMG wagged these two dogs for years, no to mention the concept of "State and Action" which are so central to UML. You would think someone at the OMG serendipitously uncover this programming model?

In all fairness, it was attempted with BOCA (Business Object Component Architecture) under the leadership of Jeff Sutherland amongst others. They were damn close but the bad press of CORBA and the simplistic views of some of the BOCA working group members such as Cory Casanave killed that initiative and prevented it to grow in this unified programming model. What a loss.

Does the OMG hold the key to that programming model? Sure, if it wants to, it has all the cards in hand, including, the SOA Consortium. IT is in dire need of this programming model, the question is who will have the guts to...?

 

07/08/08 :: [SOA] Programming Models Matter  [permalink]

The latest column of Steve Vinoski generated quite a stir.

I would like to comment on one point that I feel is essential -and not surprisingly missing from Steve's argumentation: Programming Models Matter.

The grand goal of Remoting was to make a given programming construct both remotely accessible and appear as local. RPC is an example of remoting. Remoting implies extreme coupling and requires the highest levels of cohesion by definition. But more importantly, remoting also implies a particular composition mechanism imposed by the programming model from which a particular remoted construct originates. To illustrate this point, I'd like to use this quote from John Hughes coming from his seminal paper on functional programming. 

When writing a modular program to solve a problem, one first divides the problem into subproblems, then solves the sub-problems and combines the solutions. The ways in which one can divide up the original problem depend directly on the ways in which one can glue solutions together.

Remoting in general, and RPC in particular, exacerbate the lack of appropriate decomposition/recomposition mechanisms (of current programming models) suited for building connected systems. Specifically, remoting technologies have never enforced any particular factoring that would match the characteristics of the network. This is the core of all problems and this is why patterns like the DTO pattern emerged and to a certain extend this is why Resource Representation interchanges could be viewed as a better approach, specially with notions such as "safe" methods.

The reason why a large number of developers can't write connected systems is simply and solely because the factoring implied by traditional programming models and techniques is not suited to "glue [connected] solutions together". As long as the remoting bunch will continue to believe that there must be a technical way to make remoting work (i.e. any call can be worth remoting) we will remain in the deep black hole where we are today. Arbitrary remoting does not work, only a certain type of interactions can flow across the network.

Most recently, the remoting bunch has set its eyes on REST/HTTP. Insidiously, they hi-jacked and disrupted an otherwise outstanding piece of work established on a solid rationale for a complete different purpose, the Web. The Web IS-A giant piece of middleware, not: the Web is an information system that happens to use some middleware components. Roy's REST modeled the Web as an information system and derived REST based on the model of this particular information system. Do all information systems look like the Web? no, does the Web share similarities with other information system types (such as document centric, data centric, or metadata centric information systems), sure. But the remoting bunch does not care about these petty considerations, they salivate at  something they could have only dreamed of, just a few years ago: a gianganticus middleware infrastructurus (world-wide scale) and self-expanding, if you please. It doesn't matter to them that Roy, or Tim for that matter, never thought about the Web that way. They will take no prisoners in their quest to conquer what they consider myopically the wholly grail of middleware.

Incidentally, Roy's REST offers an incredibly innovative "ways in which one can divide up the original problem depend directly on the ways in which one can glue solutions together", HATEOS offers composition capabilities unheard of before (ok, there was Apple's hypercards), and again it has to be noted that the programming model and new programming constructs (hypermedia) enable this capability. Remoting alone has nothing to do with that.

That being said, the remoting bunch wants to make people believe that somehow because REST/HTTP is programming language independent there is no underlying programming constructs being remoted and REST/HTTP is not remoting. The remoting bunch has been very quiet on the implementation details of what they would consider "RESTful". When Stefan finally told us how he was implementing RESTful "calls" it became crystal clear. This is what he said:

I force my application semantics to adhere to the common HTTP semantics, which means that for every operation, I have to decide whether it's a derivative of PUT, POST, DELETE, or GET. POST is the one with the least meaning. So in the end, I mind end up with a mixture of mapping, some matching [message passing], some matching [CRUD].

After years of discussions, this is the only thing I could get out of Stefan to explain how exactly a REST implementation looked like. Not a single time in his articles does he talk about how this is implemented. Of course, explaining how a REST implementation looks like (a "minor" detail) would immediately uncover what REST-as-a-middleware infrastructure is about. 

You have to understand that in the (other) REST, programming model constructs (of your choice) are hard-wired on both sides (client and server) to your code "opportunistically". This is not as "transparent" for the developer as previous remoting approaches since you have to go through the "mapping ritual" of remoting calls to HTTP and down to the resources, without forgetting the error codes. The mapping happens as you go (and as you can) based on patterns, tricks and tips exchanged over blogs. The benefit? there is a benefit over traditional remoting technologies like WCF, CORBA, RMI or EJBs: pretty much anyone on the planet can write a "REST" client as long as they can open an HTTP connection, better, it is potentially accessible from anywhere in the world. This is not bad, but it is only a trade-off.

So you probably going to ask me at this point "what is not remoting?". You are not doing remoting when your "glue" supports:

• Interactions, limited to:
  • events
  • actions (in the state machine sense)
  • queries
• Loose coupling at several levels
  • communication protocols
  • security
  • business protocol
  • activation
• Forwards compatible versioning schemes (to be clear when a consumer of version N can still consume a service upgraded to version N+1)
• Peer-to-peer interactions
• Asynchronous interactions
• An assembly mechanism

REST/HTTP is a remoting technology because:

1) it cannot support a versioning scheme satisfactorily since there is nothing to version.

2) offers no loose coupling capabilities. By definition, it is HTTP and its security mechanism, but more importantly REST/HTTP offers no capabilities to achieve business protocol loose coupling.  Activation is somewhat independent of the the approach since it is a service implementation capability, it is however important to include it in this list. REST/HTTP does fare well in this capability because of the architecture of Web Servers.

3) it does not support events and actions, it is limited to a set of predefined actions. REST/HTTP supports queries.

4) it is inherently client/server and synchronous.

5) It does not have an assembly mechanism.

When you look at that list, you realize that we are actually very close to having the capabilities required to build robust connected systems if only we would care to look. Indeed WS-* does not fare too bad, but you also realize how far away REST/HTTP is.

It should now be clear to you that the programming models in use today exhibit a terrible mismatch between the capabilities that are needed to assemble a connected system and their programming constructs, and no "mapping ritual" can bridge that.

So at the end of the day, it should be no surprise that 96% of developers and architects can't build a decent connected system easily. They simply have no chance, no matter how smart they are or how many patterns they use. You simply can't build serendipitously the capabilities on that list, be it on top of RPC, CORBA, J2EE, WCF or REST/HTTP for that matter.

As soon as you understand and agree on what's needed to build connected systems, you must change the programming model, there is simply no other way around. We could spend another 50 years circling as Stu suggested (actually, I have no problem to believe that the remoting bunch could circle for another 50 years and every 5 years or so claim that they were wrong again). On the other hand we could also consider that maybe, just maybe, 40 years is enough. I personally don't have that much time left, at best 20-25 years.

So, I repeat, the corny Synchronous CRUD-oriented Client/Server programming model that has been in use for the last 40 years is going to be marginalized (not disappear). An Asynchronous Inter-Action Oriented Peer-to-Peer programming model is going to be the dominant programming model for building connected systems.

The (other) REST is a fraud and everyone knows it.

 

07/08/08 :: [SOA] the (other) REST is a Fraud and Everyone knows it.  [permalink]

|

Stu responded to my post yesterday showing that I did not quote Matt properly. He was correct. However, after reviewing Matt's response, it is not that clear that he disagrees with Eric's view of REST since several of his points simply corroborate Eric's quote. So until he clarifies, I'll stand by what I wrote.

Stu also took issue when I questioned the integrity of some of the members of the (other) REST community.

Let me start by saying that IT is dying. No, Nick Carr did not kill it. No, it is not dying because "It does not matter". It is dying because IT can't do its job. IT can't do it's job because ever since we left the mainframe era IT has been served sub-optimal technologies, developed in a hurry by people largely incompetent (on the business side), sometimes simply greedy, that didn't give a damn about what they were chartered to do. 90% of the people I met on the vendor side match this description. This system was fueled by an ever lower barrier of entry: everyone and their brother could and did build "solutions", "frameworks" and "tools". Let's face it, vendors, small, medium or large, have always been concerned by the big bucks rather than delivering value. I would argue that the only reason why the business did not get rid of IT altogether is "Core Data". Core Data provides enough visibility into the business and this is the last thread which keeps IT within the walls of corporate. There are of course organizational issues within IT which contribute to its lack of value, but believe me, these would be minor annoyances if IT could deliver what the business needs. IT should not matter, the business only matters.

But, here we are in 2008: as a result of IT's debacle, G2000 companies are actually failing at an alarming rate. The French Société Générale lost 7 billion because of IT and sloppy information system architectures (IT does matter). How many more of these do you need to open your eyes wide shut? The (other) REST community is about to axe the hand that fed it (and us) for so many years.

The situation is dire. The (other) REST is the ultimate moose trap: the (other) REST costs nothing, you don't even need to learn it, as you know it already, your developers are going to love it... The reality is that the (other) REST community who simply and only dreams of a universal middleware technology (who said again the business needs that?) could very well be the final blow to IT by creating "uniformly unmanageable information systems".

Now, when you say with your polished language that:

I think anyone claiming that contracts and governance are unnecessary complexities are sorely mistaken ... I also note have never seen any reputable member of the REST community suggest any such thing. I have seen advocacy for using more general and uniform contracts, but not avoiding contracts altogether.

I want to throw up. How could someone not question someone's integrity after this? The contract question was discussed for years. How many more years do we need until you guys stop spreading deeply and totally flawed arguments in the industry? A "uniform" contract is not a contract. Ron Schmelzer said in 2006:

The real challenge to applying REST as a means to implement loosely-coupled Services is that REST has no explicit contract to define behavior, which means that all behaviors are implied. In REST, the data pumped through a GET, PUT, or POST operation contain all the semantics necessary to understand how a Service behaves.

Therefore, REST isn’t appropriate for organizations that wish to take a contract-first development approach aimed at maximizing reuse and composition, in addition to loose coupling. And that’s the rub with REST – SOA aims to reuse not just the resource, but also the operation and the composition as well.

Any CompSci 101 student should be able to understand these two paragraphs, this discussion could have been closed years ago, yet the (other) REST community -that includes you- is trying desperately to keep it open and deceive scores of developers.

It took a comment by Roy himself to finalize the debate on the uniform interface vs contracts. Now did the (other) REST community advertized the results of this final discussion? no. This went right under the carpet. How convenient. This is integrity at its best.

I have personally spent countless hours to get Stefan or Steve to articulate the rationale behind their assertions that ultimately proved to be deeply and totally flawed. Did they talk about it? no. This the kind of things that matters to me. I have discussed REST versioning with Stefan several times, did he ever mention the discussions I had with Pete Williams? no, this is the kind of things that matters to me. I did however notice that as a result Steve is not talking about "uniform" interfaces anymore, and Tim Bray toned down significantly his rhetoric on "the end of SOA" or "WS-*". I have shown over and over that I was capable of a logical discussion be it with Teo, John Heintz, Subbu, yourself or Pete Williams. Can you honestly say that the leaders of the (other) REST community are capable of such logical discussions?

Stu do you honestly think that any other engineering discipline (civil, aerospace, electrical, automotive, chemical...) would tolerate this level of discussion? no !! bridges would collapse, aircraft would crash, cars would roll over, people would get electrocuted,... I let you contemplate the Societe Generale 7 billion loss. Maybe after all we should grow up as an engineering discipline, because there won't be much discipline left in a few years.

The (other) REST is a fraud, a major fraud like so many before it (Business Process Execution Languages comes to mind as another example). You know very well that you will have to somehow recreate the REST-* stack. It will take years, and enormous resources while delaying the emergence of a new programming model so critically needed in IT. You know that, you can't deny it. You are doing the exact same coup d'état that WS-* did to ebXML and we lost 5 years there. How many years do you think IT is ready to loose on disserting on the merits of a "contract"? The (other) REST as it stands today is complete BS.

Now, Steve complains of being "stalked". But who is stalking who? (Roy's) REST has nothing to do with WS-*. They have completely different domains of application. What is the (other) REST community doing all the time? they are criticizing everything that is not "RESTful" (though Roy has to remind them what RESTful really means). Remember the SimpleDB saga? how about this post I wrote on InfoQ? How childish were the comments from the (other) REST community on an otherwise scientific review of REST vs WS-*? How many times the (other) REST community went after WS-*, SOA? without any justification, with completely flawed arguments. How many times?

Stu, if you are who I think you are, you would not be talking like this. It is time to face these issues (contracts, versioning, loose coupling, resource centric programming model, security, reliability, transactions, management... that are so critical to SOA and the renaissance of IT) and clean up your own community. Both you and I (and many others) know where the path lies, there is no question about it, so why not walking it with a solid rationale  for a change? Would you dare?

07/07/08 :: [SOA] Pioneer of Computer Business Analysis Dies [permalink]

I caught this news while I was on vacation reading the paper. I felt sorry for David Caminer

... who as an employee of a legendary chain of British tea shops found the earliest ways to use a computer for business purposes, including standardizing flavorful, cost-effective cups of tea.

Lyons was the first company in the world to computerize its commercial operations, partly because it had so many of them: It had more than 200 teahouses in London and its suburbs, with each Lyons Corner House daily generating thousands of paper receipts and needing scores of fresh-baked items.

The result was LEO, its name derived from Lyons Electronic Office. The Economist magazine called it "the first dedicated business machine to operate on the 'stored program principle,' meaning that it could be quickly reconfigured to perform different tasks by loading a new program."

"LEO's early success owed less to its hardware than to its highly innovative systems-oriented approach to programming, devised and led by David Caminer," Computer Weekly said last year.

LEO performed its first calculation on Nov. 17, 1951, running a program to evaluate costs, prices and margins of that week's baked output. At that moment, Lyons was years ahead of IBM and the other computer giants that eventually overtook it.

The finished LEO, which had less than 100,000th the power of a current PC, could calculate an employee's pay in 1.5 seconds, a job that took an experienced clerk eight minutes.

Sadly, as I mentioned in previous posts, computers were first used to compute, we never really left this legacy behind. Storage came as an "add-on", a value-add to computing. In 2008 we are still in the "Programming Technology" era, nothing has really changed, except for "more cycles, users, bandwidth and bytes". Nobody has ever designed an "Information System". I bet that most of the programming concepts in use today can be traced back to David's work.

 

07/07/08 :: [REST] The real Face of REST [permalink]

Matt McKnight argued on one of my posts at InfoQ that:

Yet REST and now the WOA crowd continue to argue that SOA, contracts and governance are just unnecessary complexities. The whole web works off REST so why not every-one's IT shop? Just mash everything up without any contracts (stuff like security, transactions and service levels) and let it evolve.

People don't need governance to re-use a service- they need documentation and sample code.

A WSDL doesn't tell me much semantically, it's true. That's why they invented documentation a few years ago. Not everything has to be done spewed out in XML.

What an outstanding summary of the current thinking of the (other) REST community. (Again, nothing that I say applies to Roy's REST).

Any volunteer to test these proposals? Common on, they are so tempting.

Before you volunteer, you will note that not a single time the (other) REST community could point to a successful REST-based implementation of an information system. But that just a detail. All their arguments are as hot as the air of the REST bubble. You would also think that some people would have had the intellectual integrity to relate some elements of the definitive arguments that I have made about REST and contracts. I guess that simply prove who they are and what we can expect from them. I am sure CEOs are going to be thrilled by the value proposition of letting their IT "evolve". Rabelais would find our times fascinating, how can it be that easy to make people jump off the cliff?   

The real issue with SOA is that developers and architects, at large, are not prepared to deal with a "contract-first" approach which is the essence of Services. Pretty much everything in current CRUD-oriented Synchronous Client/Server programming model is pushing for implicit inter-actions buried at the controller(s) level. SOA has to undo 40 years of damage to the programming model which has been dominated naively by the programming languages and middleware people.

SOA is about the emergence of an information and process centric programming model. No wonder a lot of people can't deal with it or feel they will be disrupted by it.  Make no mistake, REST is by far the most stupid direction in which IT can go today. If this quote from Mark cannot convince you of it, what will?

Matt, good luck with your REST implementation. Let's talk about it in a couple of years !

 

07/05/08 :: [SOA] Remoting is Dead [permalink]

Steve Vinoski killed remoting in his latest column. He finally understands that programming constructs such as function or method definitions are not semantically adequate to define inter-action contracts across a connected system. This is why everybody abandoned these constructs almost a decade ago and focused on a "contract-first" approach when implementing their Service Oriented Architectures. This is where the value of XML technologies, including XML Schemas, Web Services, including WS-Policies, come to play. Do they represent the perfect contract definition tool box? Hell no, but because of the remoting bunch (that includes Steve) we still don't have a reasonable commonly agreed inter-action contract definition language. ebXML had a decent one, and we could have adopted similar principles years ago in the Web Services space, if only the remoting bunch had any clue about what they were doing instead of focusing on "interoperability" only.

It is because of people like Steve that we had to bring the baggage of RPC into the realm of SOA. It is because of people like Steve (and other Remoting gurus that could not think outside of the CORBA or DCOM black hole) that we are where we are today, still stuck in a synchronous CRUD-oriented client/server programming model.

And what Steve has to offer today? Nothing at all, he is still confused about what "inter-action" means. He is getting closer: he gets that "asynchrony" is better than "synchrony". A uniform contract? No, surprisingly this word seems to have disappeared from his language. Even his rhetoric around Web Services and RPC has disappeared. That must be hard not to write about his favorite piece of boloney.

So, what did he use instead? Steve's advice is that:

[you] are better off with Message Queuing or RESTful HTTP, depending on the nature of [your] application.

How pathetic. People would be better off choosing between strings or shoe laces. In 2008, this is all the remoting bunch can do. They have miserably failed and have no where to go.  They simply have no clue. no clue at all.

Steve, the day you will understand that an inter-action is leveraging a message inter-change capability you will have made a giant leap forward from the "contract-less" world that you would like us to jump into. It is not because the semantics of the contract were not adequate that you are better off without a contract. ebXML or Web Services delivered that capability but because of you and people like you we could not get standards, frameworks and products that fully expressed this idea.

There is simply no other way forward than replacing a corny Synchronous CRUD-Oriented Client/Server programming model by a modern Asynchronous Inter-Action Oriented Peer-to-Peer programming model. 

06/19/08 :: [OTHER] Interesting Economic News [permalink]

I always suspected that the main reason behind the wars in Iraq and Afghanistan were to keep a strong hand on oil production and distribution (A pipeline to deliver oil from Russia to Asia runs through Afghanistan) and basically control the world's oil prices by controlling a small fraction of the oil production and distribution. Well, well, well, today Iraq announced that it was ready to award contracts to western oil companies. The faucet is going to open up a notch... Of course it is totally coincidental that Iraq's oil production has been so crippled for the last 5 years (and actually beyond that with the sanctions).

Now the other very interesting news I found watching the French TV tonight was an explanation of the speculation on raw materials. It's clear to me that productivity gains of the last 20 years have created an accumulation of wealth that is quite "un-common" (if you know what I mean). For me money represents "activity", if you can't buy "activity" because it takes too little time/activity to produce anything you need, then you are left out with a bunch of money that you don't know what to do with, this is when inflation moves in. There are only so many Porsches, Mansions or Yachts you can own, not that I own any, but at some point even the wealthy don't know what to do with their money. The capitalistic system (which I truly support) would normally use this "extra" activity to innovate and fulfill new needs by creating new products and markets. But it looks like that the current generation of entrepreneurs got lazy and instead of innovating, it invested all this "un-common" wealth in good old "raw materials". That's a great way to park wealth. This apparently was the only way wealth was parked before the industrial revolution. I personally know a much better way to park wealth: build schools and hospitals, improve the world's population quality of life and the environment. But hey, that's boring.

Of course, government could tax this "new" wealth to bring it back into the system precisely as infrastructure. This is actually what Bill Gates, Sr is arguing. But it seems that the wealthy have gone really smart at that too. They were reporting that in France 150 of our millionaires did not pay any taxes last year (legally).

The question of course is where are they going to park the excess wealth generated on the speculation for raw materials? (How many wealthy idiots are working on a Space Tourism project?)

06/15/08 :: [SOA] the Remoting Bunch [permalink]

Jim Webber pointed out (a better) conversation he has had with Herbjörn Wilhelmsen at the end of last year. The level is a bit higher here. Interestingly he starts talking about "states", maybe another year and he will realize how important the concept of "state" is to constructing reusable IT assets assembled in composite information systems. He is still cohesively stuck on the concept of "one process" equals "one service" and mixing up loose coupling and cohesion. But hey, what are the kinds of ideas that humor can't make you swallow: if you can "joke" about it, it must be right.  I crack jokes therefore I am is the 21st century philosophy.

At the end of the day, you have a bunch of guys who have been remoting anything they could -probably because it was a "cool" idea. Most of their focus has been on making remoting "seamless" (or even serendipitous). Ultimately the remoting bunch (any one remembers this movie "My Name is Nobody"? -it's a spaghetti western that features the Wild Bunch), ultimately the remoting bunch  did not find many applications for their work, sure enough "scaling out", "fail over" or "Distributed transactions" were direct applications of distributed computing but non content of these major achievements they kept shooting at everything they could.

SOA is an architecture that let's you build solutions from reusable IT assets. Sure enough that smells remoting, that feels remoting but not the kind of remoting that Jim, Steve or Stefan are talking about. They focus only, and exclusively, on calling a process from another process as much as possible (not even reliably). Over all these years, the only thing they dropped is the making remoting "seamless", they got that right. In a reverse of fortune, they now want to make remoting "explicit", subsuming programming models. They see the Web not as resources (don't be fooled, they can't care less about the resources) but as a massive self-expanding remoting infrastructure. Wow, could they have ever dreamed of something like that? REST and MEST allied to the same cause.

The question I have and will always have for this kind of reasoning is: Do you really think that by using HTTP you are automatically creating reusable assets? This is the fundamental question. Do you give a money back guarantee to your statements? What Jim tells you in the small print (and without jokes) is that there the "smarts" are pushed on the "edges". Yeah,  ... yawn ... this is how ESB's have been implemented for years now. Jim, you need to change your slides and come up with new jokes. The big "in the middle" infrastructure is a thing of the past that initially came out precisely of the limitations of HTTP as a transport for Web Services. The "common" information model and the "common" transport of EAI have long disappeared. Sure JBI perpetuated that idea with the "Normalized Message Router" but who is using JBI "in the middle" today? not many people or vendors.

An ESB is a "Service Container", you can have more than one ESB type in your infrastructure. Service Containers can even be nested. The primary purpose of a Service Container is to provide "loose coupling" between the service implementation and the interface it is exposed with. This is what you call "smart" on the edges, in the fine prints. Now you can hand-code these "smarts" with lots of people or you can use technologies (and tools) that make it easier to "transform", "route" (behind the service interface to the appropriate service implementation), ...

I would even argue that when it comes to "ESBs" (as Service Containers), two can prove better than one. You need a service container on the consumer side and one on the provider side. Of course, I am a "peer-to-peer" guy, so the notion of "consumer" and "provider" is kind of weak. The only distinction is "who fired the first message first". At the end of the day, a Service container is in charge of bridging the internal interface with the external interface from many different aspects (Transport, Security, Protocol, Message exchange pattern, ...). If you push these "smarts" in your code and hand code them because you think "HTTP is enough" you are in for a very big disillusion.

Now, if you think just by throwing an ESB in your IT organization you will create reusable assets? you would be wrong too. Creating reusable assets is not easy (it is not really hard to), what I mean by that is that it is not serendipitous. Reusable assets are reusable because you spent a bit of time on governing them, you have a rock solid versioning strategy (I would argue 99% of the people don't get that) and because you have an ESB to help you inject some code without breaking or touching your service implementation. A guerilla approach to SOA will sent you directly to the inextricable jungle where Che ended his life. I am a lot closer to Jean-Jacques Rousseau than the Che, it is not with "killings" from the "remoting bunch" that we will restore IT's ability to deliver value, far from it. There is a lot more to learn in Rousseau, for instance in the Contrat Social, when building your SOA than in the Che: 

• understanding the tension that seems to exist between liberalism and communitarianism
• following the general will allows for individual diversity and freedom
• Simply having power is not sufficient for that power to be morally legitimate
• There is often a great deal of difference between the will of all and the general will

I think that I have overwhelmingly proven that thinking "remoting" is the worst thing you can do when building a Service Oriented Architecture. You should focus on creating software agent "interactions" and then, only then use middleware to realize these interactions. Now, don't get me wrong I don't recommend that you start a massive SOA project, each project within IT should be an opportunity to create reusable assets. But this is not "Guerilla SOA", this is another instance of think "Globally" act "Locally".

If you ever wonder what happened to the Wild Bunch, they got all shot -like the Che, after messing up all they could- by Jack Beauregard (with the help of Nobody - Terence Hill).

 

06/06/08 :: [Computing] RoadRunner: Who Moved My MIPS? [permalink]

|

The world's first hybrid supercomputer has broken through the "petaflop barrier" of 1,000 trillion operations per second...

the machine was designed by IBM and uses Cell Broadband Engine chips—originally developed for video game platforms—in conjunction with x86 processors from AMD...

Roadrunner is twice as fast as the world-leading Blue Gene, which is itself three times more powerful than the remaining contenders on the industry's Top500 list of supercomputers. ... [it] has the computing power of 100,000 of today's most powerful laptops

Their capability improves by a factor of 1000 every decade, propelling the global economy as more business sectors begin to ride the industry's advantageous price/performance curve.

The three pillars Computer Science -as I see them- include: Computing, Information Systems and Middleware (as in any "wares" in the middle of computing nodes or information systems - soft & hard). All systems are made of any number of these 3 constituents. These constituents are atomic, they cannot be further decomposed, only recombined.

I am not sure everyone makes this distinction. I am not sure everyone understands that computing parallelism is different from distributed Information Systems. I am not sure everyone understands the difference between middleware protocols and interactions.

If you want to see the consequence of people mixing up these concepts look no further than Mark Little's latest post. Yeah, we are in a real mess. Wait until Erlang is thrown at SOA because people are mixing up parallel "processing" with business "process". This is bound to happen. Another month or two and an Erlang "furious" will come up with this great idea, claim that BPEL is a piece of crap and all business processes in the world can -of course- be executed in a single Erlang engine, because it is so scalable. I can't wait until someone write a posts along these lines.

If you want further proof that people mix all these concepts, look no further than the reference to Anne's post when she claims:

.. the most important thing to understand from a technical perspective is that a service should support access via multiple styles of interface. It should enable an application to interact with it using whatever means it wants:

• A method-oriented interface (e.g., SOAP)
• A message-oriented interface (e.g., JMS)
• A resource-oriented interface (e.g., HTTP)

If we agree on the most important point which is SOA is technology independent, in other words, no technology has the monopoly on building a successful SOA, it is clear from Anne's words that she does not make any distinction between middleware protocols and connected information systems interactions. As long as these kinds of confusion (distributed vs connected and message exchange vs interaction patterns) will happen, we will loose ourselves in infinite conjectures and more importantly drive scores of architects and developers in the wrong direction.

So no Anne, any of these technologies can be used to exchange messages within a connected system, but the interaction patterns have to be layered on top of it. Services do need (inter)actions, events and resources whether you use SOAP, HTTP or JMS.  The middleware protocol should be selected based on non-functional requirements such as scalability, security,... but the use of the middleware protocol should not constrain you to a particular type of inter-action (or lack there of).

 

06/06/08 :: [SOA] Kung Fu SOA [permalink]

You probably need to see the movie to get these few lines... Po is a fat apprentice noodle maker who served noodles his entire life. He keeps dreaming about being a great martial arts hero like his idols the Furious Five. But can Po really be a Dragon Warrior? Can he really beat Tai Lung and return peace and prosperity to his village? (Yes he can and he does) Kung Fu SOA is about a journey of faith, faith in the Dragon Scroll and the journey it leads you on.

 

I watched Jim and Martin's presentation tonight before I went with the kids see Kung Fu Panda. The two furious have given a presentation that's got to be by far the worst presentation on SOA I have ever seen, it is not even pathetic, it reached the level of Sadness. If you wanted to turn off customers you couldn't do it better. I just can't believe that one of the co-author of WS-CAF and the father of Dependency Injection could give such a shitty presentation. They make fun of pretty much everything they can with corny undergrad jokes, even joking about a 12 century bridge that stood up for 500 years and that's still half standing 900 years later, not to mention Sir Berners-Lee. They'd be lucky if the information systems that ThoughtWorks builds following their recommendations (Guerilla SOA and MEST) are still running in 5 years.

They conclude that all you need is the Web used as a middleware platform (not even as REST). They don't understand a thing about both loose coupling and exposing legacy assets as services. How can the Web as a Middleware platform do that?

Posts from Mark Masterson are rare, and always appreciated. I find it quite remarkable that all the industry pundits deploy all this energy to extract the substance of extremely sophisticated thinking (e.g. Jenny's talk entitled "Structural Holes & the Space between the Tools") when they would not consider something as simple as an object lifecycle or can't figure how REST really works in integration scenarios.

I had an interesting discussion with yet another furious five. Sandy I don't mean this in a disrespectful way (just trying to keep the movie thread going), I enjoy reading your posts and respect your knowledge of the BPM field, just like Bruce. I do however think you could consider alternatives to some of the things you have been saying for several years.

So for Jim, Martin, Mark M., Stefan, Steve, Sandy, Bruce and probably many other "furious" who have dispensed their mastery to the apprentice noodle makers over the last ten years or more, I'd like to tell you this:

For almost ten years, everyone has banged their head on the wall trying to figure out how to build composite information systems (and return peace and prosperity to the business). Very few have succeeded, Amazon comes to mind. However, despite all your mastery, noodle makers are still making noodles.

You would think after 10 years, somebody really smart in academia or in a research lab would have come out with a solution to the problem. When that kind of thing happens, it is probably worth to reconsidering the problem and see if it cannot be formulated in a different way, where the solution is more obvious.

I have great news for everybody, the Dragon Scroll was published (fell from the dragon mouth) last week by Ksenia (Ryndina) Wahler and her colleagues from IBM Research in Zurich. Yeap, it is quite simple -almost unsophisticated- and yeap any fat apprentice noodle maker can use it to become a Dragon Warrior. Yeap, all these silly discussions about REST, ESB, BPM... can all be resolved once and for all.

Will you have the wisdom to read the Dragon Scroll, and more importantly understand its teachings? And... no Mark it is never too late to figure out the meaning of the scroll (nice try).

"You just need to believe..."

06/01/08 :: [SOA] Full Circle [permalink]

I'd be curious to know what the SOA thought leaders are going to talk about in 2015. It sounds like the only constant is the lack of imagination. Today, Dave Linthicum asked "Is BPEL irrelevant?". Dave how many times did you ask this question in the last 5 years?

Dave argues:

in many instances BPEL is just shelfware

now get this, all wrapped in one convenient paragraph.

Core issues with BPEL include the fact that it is very synchronous in its approach and has a few programmer-level issues including limitations around request/reply exchanges in a heterogeneous architecture, exception handling, failure recover, multi-programming model support, and a few other issues which I view as minor. However, what's core to the issue with BPEL is that it's not very good at adding a human as part of the process and as SOA moves forward, I'm finding that composites and workflows are more applicable than simple service binding and extending. Moreover, there is not a lot of BPEL on-demand, and many SOAs are heading the way of the Web, or Web Oriented Architecture (WOA).

"Very synchronous", he says. I think Dave means it is not "event driven" enough. As a matter of fact if a resource lifecycle reaches a particular state, BPEL is not equipped to emit a message event. Yes, that could be improved.

Some clarification on the other points would be helpful, so I'll on the comments. I do love the "multi-programming model support" issue. For the readers who don't get it, here is a tip: it's a lot clearer when you take the square root of this statement. 

My favorite is as usual the "BPEL is ... not very good at adding a human as part of the process". Just as if BPEL had anything to do with humans.

Now, I have explained many times, there is an emerging inter-action oriented, asynchronous, peer-to-peer programming model that is relying on concepts such as resources, assemblies, bi-directional interfaces, orchestration. It's actually quite elegant, it solves in one big scoop all the endless discussions that we are having, REST, BPEL, BPMN, "a process is a service", SOA RM ... I am no Roy Fielding (by far) by I did spend quite a bit of time last summer to write a book about it. Strangely enough no one came on that turf. You would think someone would think that it is quite insane that all these discussions are so rough, that all these technologies don't quite fit. You could think that programming concepts invented 40 years, don't apply in a connected world.

You could think on the other hand with all these discussions gone all these people would have nothing to do. You would think that having a better ROI to build solutions in IT would not hurt that much.

I know, you are thinking that this guy is about to throw a conspiracy theory. Well I wish, then we could uncover it. No I think the truth is a lot simpler. It is best summarized (as always) by Calvin telling Hobbes a few years ago:

05/30/08 :: [REST] See it for yourself [permalink]

Now that REST is being used by more and more people, it is becoming very difficult to hide behind a few general statements and hand waving. If you want to take all the measure of Stefan's comment on how application semantics are marshaled in REST:

I force my application semantics to adhere to the common HTTP semantics, which means that for every operation, I have to decide whether it's a derivative of PUT, POST, DELETE, or GET. POST is the one with the least meaning. So in the end, I mind end up with a mixture of mapping, some matching your alternative a), some matching b).

Look no further than the good old Wf-XML now dubbed Wf-XML-R for REST of course. What wouldn't you do to get a little bit of attention?

So after the easy stuff, GET /workflows, GET /activities... comes the "actions". With  resources such as "Process Definitions", "Process Instances", "Activities"... you can't really avoid actions such as start/stop, pause/resume, deploy, activate... It happens they all have well established lifecycle (a.k.a state machine).

Remote Application needs to be able to start (and stop) a process instance

You bet ! So how do they  map all these actions to HTTP? You may proceed directly to the 5.2 section of the document (get ready for a good laugh).

To create a new process resource, the Remote Application will need to use the HTTP POST operation.

...

POST /wfcs/workflows/{name}/definitions/{version}/processes

Thank god, Pete Williams started to solve the API versioning problem (they did version Wf-XML once), otherwise we would have had 2 {version} elements in there, one for the "Start" element, the other one for the process definition version.

They also re-invent WS-Security (not):

We will show later how to execute a similar workflow without requiring the user to type his user name/password in the request (which is not very user friendly)

Now, how do you Pause/Resume a process? Eeeeasy of course, you PUT something in it:

Remote Application updates a particular process (halt/resume)

PUT /wfcs/processes/51.atom

How do you Stop a process? even eeeasier, you DELETE it:

Remote Application deletes a particular process (stop)

deletes a particular process (stop) DELETE /wfcs/processes/51.atom

Don't miss the CRUD in section 5.3.1

Workflow Creation, Update, Delete

The Grand Finale is section 7.0: Event Notification

The workflow engine may have the need to notify users when resources change, processes are started and completed...

Well the remainder of the section is .... blank.

You can't secure this type of crap with robust authorization rules, you can't build tools, you can't assemble components...

I think at this point, I just want to throw up altogether. How can our industry have any respect for this kind of work, for the people that recommend going that way?

How can you ever expect that a set of technologies developed for a very particular usage pattern (Human/Browser/Server interactions and Feeds management) is going to work serendipitously for something else such as information system construction, connected systems...? After 20 years or more, you guys still don't understand the fundamentals of information system construction.

05/27/08 :: [SOA] Architecture of a $7B Loss [permalink]

I wrote this news for InfoQ last night. When you see the picture of the trader's P&L you realize how far a little CRUD can go.

He didn't know a thing about SQL injection, but I bet that over the years the systems "opened up" one modification at a time to eventually let anyone CRUD what ever they needed to make the system(s) of record look like what ever they wanted.

I admit that trading application are a bit extreme and time-to-market is usually prioritized over security or QA. But at the end of the day, this is what happens in CRUD-oriented synchronous client/server model.

In a COSC/S the developer can only guess what the user is doing when a particular line of code is executed: Human tasks  and Resource Lifecycles (state machines) are not explicit. Nothing can really be audited or reported on unless you put a lot of energy into it. Authorization mechanisms are ad hoc.

If you take one of the technique the trader used: the "technical" transaction of entering a provision for the purpose of simulation, you realize how a simple lifecycle would have easily caught the problem. "Expirations" are annoying to code, you don't have a user waiting to see things that expire, so you need a notification system (the cheapest being an email), not to mention that you have to harvest periodically the things that expire. This is the prototypical example of how bidirectional interfaces and orchestration can help implement this type of functionality. This is just a few lines of BPEL when in a COSC/S this could translate into hundreds of LOC, at best.

Same thing is true of the 947 fake operations the trader entered in the system. I don't know much about the world of finance, but you would think that these fake operations have counterparts and therefore, if, as a resource, an operation could send event notifications to a coordinator which job is to make sure that all transactions have counterparts, nothing of that sort could have happened. A COSC/S programming model requires that you build such mechanism from scratch all the time: everything that's outbound, asynchronous and long running is such a pain to write that you implement these elements of functionality when you can afford it or have enough time.

Our world is bidirectional, asynchronous and long running, as long as we will continue using a programming model that does not take these aspects into account we will expose ourselves to this kind of CRUD.

05/26/08 :: [SOA] Web Services is an RPC-Oriented Technology [permalink]

|

I remember when I was a young graduate student in the late eighties thinking about IEEE as some kind of powerful temple of knowledge and source of wisdom. Having anything published in an IEEE journal was a distant dream of mine (that never materialized).

Then comes Steve Vinoski, a senior member of the IEEE and a member of the ACM.

He and Stefan skillfully keep avoiding the question about how actions are modeled (and implemented) in REST. In this excellent interview (great job Markus), Stefan shows how knowledgeable about HTTP and REST he is. But when Markus asked him, "Stefan, tell us how you implement "Submit Order" and "Cancel Order" in REST", he starts answering how to "Create a resource" with a POST and with great skills, quickly changes subject before answering the real question, how are actions modeled in REST? We know better today they are not.

In his latest column, Steve goes one step further in the misinformation, after a summary of the benefits of REST and HTTP, this is what he claims:

RPC-oriented technologies such as Web services are intended primarily to extend programming language idioms and patterns across the network.

By doing so, they hope to make the developer's job easier; unfortunately, this comes at the cost of significantly reduced scale, greater client-server coupling, and more difficult system modification and maintenance.

Boy, isn't the review process broken at IEEE? I would not want to publish anything there as long as this kind of boloney can pass through publication.

I'll pass on the section that Steve calls "Specialized Data Issues", this section was written to  comfort the REST followers in their beliefs. Steve uses a very well know technique: You talk about the problems that Web Services took on to solve (and for the most part solves) and you play them as scare-crows.

Then, you take the great things about REST, and you play them in raw fashion as incantations (BTW, half of them WS-* can do). That's when the followers cheer. Then you close, with modesty, saying, there a still a few issues with REST but we are working on it, or you claim "the world can't be perfect".

Now, these two guys are really smart, I would even argue a lot smarter than me (and I mean a lot). But at this point, how could we trust what they say? The kinds of things that Steve (such as the quote above) or the kinds of convenient omissions [1] are no longer "honest mistakes", they are meant to deceive.

That being said, I find Steve's quote really symptomatic of what has happened over the last 50 years (though I still don't think this is an honest mistake [1]). My theory is that "computers" were, well..., originally ... computers. Meaning they "computed". Programming languages were invented to crunch numbers nothing less, nothing more. Having been the first guy to hack an implementation of "Factor Analysis" on a 640k PC in the late 80s, I can safely say that. The "processor" is deep at the roots of all programming languages. All our programming abstractions were invented to make it easier to shift bits around. And yes, Steve is right, there was once a group of people in the industry that focused on "extending programming language idioms and patterns across the network". He was part of them, and we all know how that ended. Insidiously, he is trying to use REST to keep the CRUD-oriented Synchronous Client/Server programming model alive for a few more years.

The disconnect at the programming language level comes from the fact that computers were not the systems of record. It is not until the advent of mass storage that the "computer" slowly evolved into an "information system". In the most ironic fashion we talk about "Information Technology" when we should be talking about "Programming Technology". None of the programming languages evolved during this transition and this is the root cause of most of our problems today. They remained for the most part unchanged since the early 60s. I believe that the core rationale for keeping them unchanged was that both actions and functions could be modeled with the same programming idioms. The advent of SQL in the early 80s only comforted this design decision by relegating all information related stuff to constructing a "string" and calling an RPC "exec" to the RDBMS.

This seemingly harmless and simple design proved to be a tragic mistake. We lost states and transitions in the process: they never made it to the programming model even though they were core to information systems (but totally useless for "computing"). We also pushed the responsibility of sorting actions from functions on the developer which eventually did not make any clear distinctions between the two leading to the "distributed mess" Steve is talking about. OO was no better, it was a "programming technology" deceptively disguised in an "information technology" which only sealed the problem by creating bubbles of data over a relational information sea.

The bottom line is that Information systems never got a chance.

The reason why I am extremely upset at Stefan, Steve and Tim (I do think all the other people in the REST community are genuinely looking for answers not followers) is that the software industry is very close to have a programming paradigm that would constitute a new and solid foundation for information systems using idioms such as resources, inter-actions, state, transitions, trans-actions... and new programming constructs such as a bidirectional interface, orchestration... all built on top of an asynchronous programming model. We could have been there for nearly 10 years at least if only interoperability nuts would have left us alone instead of "extend programming language idioms and patterns across the network" using XML and HTTP, now some REST nuts are keeping us behind by pretending there is no need for actions, you can CRUD your way in anything you want. It is a bit harder for the REST nuts to get away with it because they have a "resource" at the core of their programming model, so it is going to be impossible to keep pretending there is no "actions".

Steve, you are going to kill both REST and Web Services by making the claim that Web Services is an RPC-Oriented technology and forcing people to use REST in a CRUD-oriented way. You setting us back 40 years, to your world. This is ubuesque.  

The need for this new programming model is exacerbated today by the emergence of "composite information systems" where no single software agent can be the sole system of record. How do you think that a programming model invented in the 60s is going to survive that?

---

[1] Before anyone flames me, please read this section of Steve's paper:

Resource representations contain hyperlinks to help applications know how to perform application state transitions. For example, a resource designating a list of employees might return a representation containing a list of hyperlinks, each referring to a separate resource for each employee in the list. An application looking for information about a given employee need only follow the relevant hyperlinks using data and metadata within the representation as a guide.

This section means that Steve (like Stefan) understands very well (too well) the question of "state" and "transition" and how an action can cause a transition from one state to another. It would sure be a bit more trustworthy if they would explain clearly how an application (not a human) "need only follow the relevant hyperlinks".

 

05/24/08 :: [SOA] Seattle's SOA Users Group [permalink]

I created a Ning Social Network (the "soag") for Seattle's Developers and Architects who want to exchange ideas around their Service Oriented Architecture's implementation or simply learn more about Enterprise SOA.

I will set up an evening meeting if there is enough interest.

My goal is to discuss a broad set of questions from Governance, Reference Architecture, Methodology, ESB, WS-*, SCA, to BPM and service oriented, process centric, model driven programming models.

If people are interested I could talk about Praxeme, an open source SOA enterprise methodology invented in France and currently being translated in English. There is really a lot to learn from it.

05/23/08 :: [BPM] BP what? [permalink]

So, Nick and I never see eye-to-eye, this is no secret and no different today.  So he gives us a "wild oversimplification of a process" as a sample to show how requirements can be captured along with the process. The little sample embodies really well all that has been wrong with BPM in the last 10 years.

So what's wrong? 

Do you think that "the process" really describes what the user is going to do step by step? huh? Have you ever been in a call center? have you ever tried to capture (not "enter") data with someone on the phone telling you what to do?

Rule 1: As many activities should happen without a definite flow

That's because you can't predict the flow. You want to know how bad this can get you into? The report that details how a trader at a French bank blew 6 billion dollars last January just came out today. It explains that the trader was able to take a position of 49 billion euros (~75 billion USD) simply because he was allowed to "CRUD" the system.

JK utilisait la possibilité, normalement réservée aux assistants traders (mais sans blocage informatique pour les traders) pour corriger des biais de modélisation, de saisir des provisions positives ou négatives venant modifier la valorisation calculée par le système front office.

Which I translate: "The trader (JK) used the ability to correct model biases, normally reserved to trader-assistants, to enter positive or negative provisions  to modify the calculated value [of a position] by the front-office system".

I bet some Einstein developer at the Societe Generale was tasked one day to change the system to give the possibility to enter these provision updates. I bet the system was initially modeled a-la-Nick and implemented with a CRUD-oriented synchronous client/server programming model: i.e. you generally have no idea when an activity is performed, and often who (which roles) performs it, because there are no explicit activity boundaries   in the programming model. So our Einstein guy decided to add a menu item, without any particular authorization assertion, and voila, you could CRUD your way to a 75 billion dollar position without being detected.

Rule 2a: Every data capture is an "activity" that starts and ends, with pre and post conditions and with a clear authorization assertion.

Rule 2b: A message is passed to the system of record as an action requested by the activity

Nick, "Select a Business Type from list of possible business types" is not an activity, this is a post-condition of an activity associated to creating a new license (if I understand your process). Within an activity you can select a business type from a list of value. That also means that your "activity", "Enter existing licensing ID or request new license" is not "one" activity but "two".

Rule 3: Use events wherever possible (an event is the occurrence of a state).

Nick, you don't end your processes by "Submit Contact Info Change Request", no, once the "contact" business entity reaches the state "Contact Updated" and event message is generated. Other activities might subscribe to that event, such as "Create Contact Info Change Request".

Rule 4: Exceptions are enforced at the system of record level by prevent state changes (invalid transition) or content changes (could not pass some validation rules)

Nick, I don't see much exception in your sample, I guess it is for the sake of simplicity, yet, if you were to represent ALL THE EXCEPTIONS on this little bit of process, you will realize that this is not the correct model.

Nick, you model your processes with a corny programming model in mind. Worse, you suggest to capture requirements with the same programming model in mind. How many requirements do you expect to miss? 60%? (conservatively).

One day, after years of failed projects, countless disgruntled users -frustrated by the useless "flows" you cooked up for them-, and managers who can't sleep at night because of all the holes you had to poke in the system after the fact to let the users CRUD around, you'll realize that the programming model needs to change, it is not a DSL, it is not a new methodology, it is not XQuery which will help in any way.

This is hopeless. I mean you would think, no one, I mean not a single company in the world has transitioned all (or even a fraction) of its processes into a BPMS since these products have been on the market (roughly 10 years). And architects are happily trying it again and again and again.

05/23/08 :: [SOA] The fundamental achievement of SOA [permalink]

Many pundits have criticized SOA for not delivering the kind of reuse that was once promised. Creating reusable IT assets is not easy, yet it is the fundamental goal and therefore achievement of a Service Oriented Architecure. There are at least 3 things you need to do right to achieve measurable reuse:

• Governance
• Versioning
• Loose coupling

Governance will only get you so far. It's good to bring every potential service consumer at the specification table, but you can rarely predict the kinds of scenario a service will support beyond 12 months. There is also a cost associated to governance (in analysis, design and implementation) that's important to access and minimize.

I talked quite a bit about versioning with Pete Williams. It sounds like we were able to proceed with logical arguments. What I call forwards compatibility is very important to achieve -this is not easy again, and easily overlooked. Forwards compatibility is when a service consumer of version m.n can consume the versions of a service from m.n to m.n+p (m represents the major version, n the minor version and p an arbitrary increment of the service). Versioning is key to reuse because as new consumers come on board, they will require minor changes. No other technology before web services (that I know of) ever supported forwards compatibility. You really need it because you never want to tell an existing consumer that they have to change anything when a new consumer comes on board nor do you want to support a version of the service for every single consumer.

Last but not least, loose coupling. Neither governance or versioning are perfect. You want to build your reusable assets within containers that support loose coupling (for both the consumer and the provider), just in case.  I provided some comments here on how an ESB and SCA can complement each other very nicely in that role.

So if you fail to establish an efficient governance, if you fail to establish a versioning strategy that can tolerate changes to the service without propagating those changes to the existing consumers and if you fail to understand the principles of loose coupling and choose a programming model that allows you to assemble consumers and providers efficiently, you will automatically fail your SOA implementation, will experience little or no reuse and conclude that SOA is just vendor markitecture. The irony, is that very few of the vendor white papers would tell you what I just wrote.

In the near future I will talk about how you can connect governance, service design and logical data models. This is an important aspect of achieving reuse as well as it helps get service message types right. 

05/18/08 :: [REST] Answer to my Letter [permalink]

I have spent an extensive amount of time on the REST vs WS-* debate since 2003. Stefan and I were already arguing back then (I was also arguing with Jim and Savas  on SSDL). So you can imagine my excitement when Stefan, Teo and Subbu replied to my open letter. I wrote this letter after reading Roy's comment 10740 on the REST yahoo board. In this letter, I asked Stefan, Teo and Subbu to clarify how the "actions" from order management service were implemented. Stefan had defined two actions: Submit Order and Cancel Order, but had not given any explanation on how they were implemented in a RESTful way.

In particular, Stefan had made claims that:

The statelessness constraint isolates the client against changes on the server as it is not dependent on talking to the same server in two consecutive requests.

Stefan asked that I do not paraphrase what he says so I am simply going to copy and paste some sections from a 34 email thread between all of us.

The core of my argument is that you can interact with a system in two ways:

a) ask the system to do something for you (and as a result the system changes state), but you have no control over the state of the system
b) directly change the state of the system, to reflect what you think it should be

I don't want to say that one is bad and the other one is good, it is really a question of trade-off.

As I said, the problem that I have with the REST-based programming model is that somehow they eliminated "actions" from their programming model. They focused on the resource. I don't know who did that or why, it seems to me that Roy disagrees with this paradigm and that b) is not a viable approach to build loosely coupled systems.

I also provided the definition of a message from Wikipedia and asked both Stefan and Teo to explain their position with respect to this definition:

"Message passing is a form of communication used in concurrent and parallel computing, object-oriented programming, and interprocess communication, where communication is made by sending messages to recipients. In a related use of this sense of a message, in object-oriented programming languages such as Smalltalk or Java, a message is sent to an object, specifying a request for action."

Stefan replied:

I force my application semantics to adhere to the common HTTP semantics, which means that for every operation, I have to decide whether it's a derivative of PUT, POST, DELETE, or GET. POST is the one with the least meaning. So in the end, I mind end up with a mixture of mapping, some matching your alternative a), some matching b).

Teo replied:

Do you think about the possibility of using b) , but prohibit "direct"
change of the system state. Instead, hypermedia is used to guide and
control the flow of state change, to decide when a state can/cannot be
changed based on a particular application context.

So I wrote this little piece of code to illustrate how the client code of the decision service would look like:

class Decision
{
      ...
      public static boolean check(Application a)
      {
           // make a decision
      }
      public static synchronized void onApplication(Applicaiton a)
      {
          boolean decision = check(a);
          //now assuming there is two links in the Application representation
          if (decision) a.navigate(a.acceptLink());
          else a.navigate(a.rejectLink());
      }
}

I then asked Teo to clarify what he meant by: "hypermedia is used to guide and
control the flow of state change"

At that point Teo replied:

I agree with you. When I realized this issue recently, I began to
understand these action/control/navigational (or whatever we should
call them) information in hypertext documents are the public APIs.
Developers must agree on them before their applications can work
together.

Web community build & agree on these public APIs on top of Internet
Media Types.

If you notice, in atom-protocol mailing list, what they are doing most
of the time is to propose and agree on new public APIs to support new
features that most developers need. Enough momentum will push the
proposed API as published standards, e.g. RFC5023 (AtomPub), RFC5005
(Feed Paging & Archiving) etc. These RFCs helps to promote the reuse
of public APIs.. hoping for more adoption to allow more applications
to work together.

So to sum up, I had asked 3 simple questions that have never been really clarified:

Question a): how to write some client code and potentially some service code that result in transitioning (in a RESTful way) the Job Application resource into a rejected state

Answer: Teo and Stefan's responses indicate that the client code would look like the code I provided above. The actions will be encoded, in some ways, over HTTP. Teo provided a great sample on how to do this encoding with APP. Stefan declined to provide a sample, but his answer was explicit enough.

Question b):how once the lifecycle has ended, no further changes can be made to the resource

Answer: No reply were specifically provided for this question, though it looks like they would do it the normal way, by preventing an action to operate based on the state of the resource.

Question c): how can I add an operation on the job application service and demonstrate that the decision service client is isolated from this change

Answer: You can't

This discussion makes this claim totally, absolutely, definitely and forever BOGUS:

The statelessness constraint isolates the client against changes on the server as it is not dependent on talking to the same server in two consecutive requests.

Anybody who would be using it today would be misleading other people.

This discussion debunks once and for all the utopia of the uniform interface when dealing with resource-to-resource interactions. Either your interface is uniform and your are CRUDing your resources creating a terrible coupling between resources and resource consumers, either you use REST to pass messages to your resource(s), if that's the case, your interface is not uniform.

My conclusion, and it is definitive, is that REST as a programming model for resource-to-resource inter-actions brings very little compared to WS-* (considering that you are on your own for so many things that WS-* provides out of the box).

I am not here to tell you which stack to use WS-* or REST/APP. Personally I don't care, this is "on the wire", and frankly, I don't think many people care. I have expressed what I like about REST. You might also want to use this paper to help you decide which to use when. Contrary to what some have said, this comparison is fair and care to help people make passionless decisions.

REST as defined by Roy Fielding (not by the other REST community) brought us the Web. THAT REST is brilliant, I have said it many many time, let me say it one more time before I close this debate forever. I have never, a single time, attacked REST as defined by Roy.

I apologize to all members of the (other) REST community for my "rudeness". I hope they will understand that, after five years, these answers were critical to get as they impact hundreds of thousands of architects that have to deal with this very question (REST or WS-*). You will also understand that we could  have had the exact same discussion last November, plain and simple.

I don't expect that the members of the REST community will advertize this post and this discussion, nor will they stop their silly comments on WS-* and SOA. This is their call, at least we know better today.

Yet, the REST vs WS-* debate is o-v-e-r. Now I understand.

05/18/08 :: [REST] Versioning [permalink]

I like to argue that "Versioning" is the most important concept in SOA. Your SOA is doomed to fail if you are not careful about your versioning strategy.

Why? the reason is quite simple. A Service is a reusable IT asset, a SOA is an architecture that enables you to build solutions from reusable assets. As services gets reused, their ability to change becomes seriously impacted.  A service cannot easily change because not every consumer can evolve at the same time, they don't have the money, the resources, they are not ready to take the risk...

Service versioning is also key because "Governance" is by definition imperfect. Don't get me wrong, Governing your services is the right thing to do. However, you can only do so much with Governance to predict what future consumers will need to consume based on the service that you need to build now. Ultimately, and most likely, you will version your services to increase the consumer base. Do you plan on operating one service (version) per consumer? hopefully not, SOA would be useless if this was the case.

Web Services (as in WS-*) support a very sophisticated versioning strategy fully aligned with service reuse. This versioning strategy relies mostly on the extensibility mechanism of XML and XSD, so one would think that REST would be able to benefit just as well.

Stefan has avoided discussing "versioning" because indeed REST does not have a great versioning strategy. I friendly reminded him last fall that encoding the version in the URL was not viable. Peter agrees.. At least, that's something I understand. Peter provides some tips and tricks that amount to an alternative solution based on Media Type negotiation (and the capabilities of XML and XSD). Unfortunately, Peter, your approach to versioning is still incomplete.

Let's look at why. The reason is because you need a service versioning strategy that creates both forwards compatible (Peter, this is called forwards compatible, not backwards compatible) services and backwards compatible versioning and you also need a versioning strategy that decouples endpoints and versions for operational reasons.

The difference between forwards and backwards compatibility has been explained extensively. You need both, though one might argue that forwards compatibility is a tiny bit more important (the kind that Peter mislabels as backwards compatibility). Backwards compatibility is when the consumer comes in with a "newer" request version than the service provider can provide. This is common when a consumer uses different providers for the same type of service. So ultimately, you need to provide some room to define the version of both the consumer and the version of the service provider that it is targeting. Your mechanism only supports "one version".  

The other problem I see is that ANY REST versioning strategy couples, by definition, the endpoint and the version. This can be a problem. This is actually a general flaw in REST (as a programming model for SOA, Roy designed REST for completely different purposes). Let's say you built a service Sv1 and you serve 1000 requests per day. Now, a new consumer comes along and wants you to serve 1 M requests per day. This might require a completely different infrastructure. I know this is not an insurmountable problem and I am sure some router can use the media type to route the request, but now you really have broken the "global scope" of a URI, I mean a single URI does not point to the same thing, the media type is part of resolving the real endpoint.

Another flaw of your versioning strategy is that URIs are by default part of the versioning strategy. I have often pointed out that "Query-by-examples" are encoded by members of the REST community (MORCs) in a URI syntax, for instance:

/customer/{id}/PurchaseOrders ...

Peter, how do you express that a particular QBE belongs to one version and not to the other?

Peter will soon start realizing that resources do have a state that is independent of the "application" since by definition a resource can participate in multiple "applications". This is the essence of SOA, i.e. the essence of reuse. At least he is one of the most lucid MORCs I know of:

A URI that is constructed by a client constitutes a permanent, potentially huge, commitment by the server. Any resource that may be addressed by the constructed URIs must forever live on that particular server (or set of servers) and the URI patterns must be supported forever.

Effectively, you are trading a small one time development cost on the client side for an ongoing, and ever increasing, maintenance cost on the server side. When it is stated like that it becomes obvious that URI construction introduces an almost absurd level of coupling between the client and server.

Yes, Peter, we agree, a URI is an "Identifier", nothing else, nothing more. You conclude:

Using hypermedia as the engine of application state has the effect of preserving the reversible for a huge number of decisions that are irreversible in most other architectural styles.

which is of course absolutely true in Roy's world -the Web- but completely "absurd" in SOA. It is not surprising that you find media types as the key to REST versioning. This is the essence of WSDL contracts, add some operations and you would have reinvented WSDL.

Peter, you seem to be close to understanding that resources interact with each other. Since you seem to have quite an open mind,  have you ever considered these two scenarios (interactions and resource-side state?).

If you consider these two scenarios you will be so close to WS-* that there won't be any noticeable difference between how to use REST in SOA and WS-*.  (Again, REST, the way Roy defines it, is a completely different thing and was never designed to be applied to SOA).

So sure, MORCs can keep adding semantics (I really mean tips and tricks) to Roy's REST but ultimately they'll remain tips and tricks, i.e. particular point solutions interpreted slightly differently by different people.

At least, some of the MORCs could have the courtesy to acknowledge that they are indeed building a programming model on top of REST, that this programming model needs clear semantics and that these semantics are not intrinsically part of REST (nor always RESTful).

 

05/10/08 :: [SOA] BEA's Epitaph [permalink]

Stu is arguably one of the brightest and most knowledgeable person I know. He published BEA's epitaph a couple days ago. I too remember the "Tanga" days and I have also been quite privileged to experience an era that started in 1995 (when the first application server was released, NeXT's webObjects) till now.

IMHO, BEA's failure is not just BEA's failure, it goes far beyond BEA, it represents the failure of the industry at large -including OSS- to create a modern programming model to build 21st century class information systems. It also represents the failure of the standardization process (not standards) that delivers large collections of politically charged, character tainted, poorly aligned, lowest common denominator... specifications.

Of course Stu is a member of the REST-based programming model community, and he points out the Architecture of the World Wide Web document, as the solution moving forward. 

Somehow it looks like this solution came to him after realizing that:

... I had stopped believing that SOA would make anyone's life any easier, and reading some of the ITIL v2 material that was guiding our efforts also really just seemed to reinforce that we were following in the grand tradition of "smart people building skyscrapers to nowhere".

Stu, I work for a BEA shop and my management had asked me to elaborate on all the SOA domains that you guys have defined. I can assure you that you guys were millions of miles away from providing guidance that would deliver successful SOAs. You simply did not get it.

Today, your analysis is that returning to the principles of the Web is somehow going to give us this magical programming model that everyone is looking for. What's the evidence? I am desperately seeking that evidence. Even though some people don't like my style, I am genuinely open to anything that works. I have no technology or product interest, I work for an IT organization. I do expect that people would be just as open as I am, but that's not always true.

Could there be a possibility that one more time you would be going in the wrong direction? Have you ever considered that your experience and training conditioned you to pretty much create the same programming model over and over? from EJB to Web Services to REST. Behind the bytes, isn't it the same thing?

Now, isn't it clear that the Web, even though it is the largest information system by far, has a very limited programming model unsuitable for building enterprise class information systems?

Let's just take an example, don't you realize that for instance, and because it lacks a robust programming model (for good reasons of course, and this is precisely why it has been so successful), you are actually diverting the very concept of a URI

The choice of syntax for global identifiers is somewhat arbitrary; it is their global scope that is important.

 to hand code all kinds of programming semantics behind it.

I don't mean to piss any one off, because Stefan, Subbu and Teo have agreed to tell me how they would implement resource lifecycles in a RESTful way, but I just want to point out that BEA's death is somewhat the death of an era and a programming model.

Stu, the way I view it, when you say the "Web is enough" you are simply rejecting the all the vendors (small or large, from Redmond to Waldorf) as well as the standards and technologies they produce, rather than convincingly provide a solution to the problem.

05/09/08 :: [REST] Answer to My Open Letter [permalink]

Stefan and Subbu have kindly offered to help me understand how REST can be used to manage the lifecycle of resources. So let's go back to where we were last November and if it is ok with you, here is the example that we could pick (I don't think there is anything particular to this example, and if you'd like you could pick any other one).

This is the beginning of a job application lifecycle. We could focus on implementing the "RejectApplication" operation, i.e. the Job Application Resource transitions from a "submitted" state to an end state (Rejected).

Please note that the red boxes represent some code that would typically invoke the DAL as part of the action implementation. The blue rounded boxes represent states, typically a message event would be sent upon reaching a particular state. But unless you insist, let's forget about events for now. Let's keep it simple.

The trick in this scenario is that once the application has been rejected, the Job Application resource cannot accept any other state changes. Its lifecycle has ended. That would be interesting to show how that can happen based on the implementation that you choose (a, b or c as explained in my previous post).

Ideally, I would like to see both some "client code" and some "service code" (if any). You can pick any programming language -of course. Let's assume that the client is another service -an automated service- that was notified by an ApplicationSubmittedEvent (not represented on the picture) and that service is capable of processing automatically the application to decide to reject the application or not. Upon rejection, the decision service invokes the RejectApplication operation, otherwise, the job application simply waits for interviewers to submit their interviews. In the past, we used to process these applications manually, but James Taylor taught us how we could use a decision service instead, so it's all automated now. SOA really worked for us because we did not have to change a thing on the Job Application Service to make that transition.

Now, I am a lousy analyst and we realized once all this is in production that we can't give a clear status of the application until an interview is submitted (a rejection could still potentially come). What we should have done is to have the decision service tell us also whether the application has passed its criteria. So now, I need to add a new action "AcceptApplication" to my Job Application Service. Could you please explain how this change of the service impacts the "client", i.e. the Job Application Decision Service:

The statelessness constraint isolates the client against changes on the server as it is not dependent on talking to the same server in two consecutive requests.

So to sum up, if you could show me:

a) how to write some client code and potentially some service code that result in transitioning (in a RESTful way) the Job Application resource into a rejected state

b) how once the lifecycle has ended, no further changes can be made to the resource

c) how can I add an operation on the job application service and demonstrate that the decision service client is isolated from this change

I'd be infinitely grateful (and apologetic for all the trouble that I have caused).

Of course, my favorite implementation for the Job Application Service would be to use an SCA composite with a BPEL component to manage the lifecycle, wired to some Java components that implement the resource's DAL. If you allow me, I will publish that Sample code too.

05/08/08 :: [SOA] Open Letter to the (other) Rest Community  [permalink]

I know it is far easier to tell me that "You don't understand" than to have an intelligent discussion about the kind of things you dare publishing.

When someone is capable of writing :

The statelessness constraint isolates the client against changes on the server as it is not dependent on talking to the same server in two consecutive requests.

I don't know who does not understand what. Why don't we run a scenario to validate your point? Again, in Roy's world, I see exactly how this works, but when two software agents communicate (say on behalf of two resources inter-acting - a PO and an Invoice), that is pure rubbish. I don't know who convince you of this, but this does not make any sense if unqualified.

I have spent an extensive amount of time reading about your claims and in particular I read your bible, the RESTful Web Services book, front to back. All they explain in this arrogant book is how they created a DAL using REST, and with that they claim victory. Incidentally, to build a decent DAL, the concept of "collection" was missing  in (Roy's) REST, so APP was put together to plug that inconvenient hole after years of efforts.

Let me repeat it, your understanding of SOA is pre-2004, what you say about SOA and Web Services might have been true at some point, but things have evolved and it is antiquated today. As Udi pointed out, this is not SOA 2.0, but the picture that we have of SOA today is a lot clearer for a lot more people than it was before 2004.

You know, or course (but that would be too easy) that the best way to convince me (and many other people) is to show me something that does not look like a DAL, so far all the examples I have seen from you or anybody in your community are DAL-oriented:

Yes, you understood that my questions are relevant so you sugar coated your interface with "SubmitOrder" or "Cancel Order" (did I say interface? did you see the interface? the actions? does not look that uniform to me !).  You proceed to explain (smartly not in the article, but way down as an answer to a reader's question):

a) one is to PUT a new state to the resource, effectively changing its internal e.g. from booked to shipped.

b) Another way is to do a logical move of the resource from one collection (of booked orders) to another (of shipped orders).

c) A third way is to represent the state change as a resource in itself, e.g. by POSTing it to the order where it becomes a sub-resource. This way, you have a history of changes. The mapping to CRUD is not 1:1 -- a POST can create new resources, or simply process something and return a result. In case a POST is used to create a new resource, the server chooses the URI.

So you would easily admit:

a) is an "UPDATE" and you also understand the coupling it creates between the consumer and provider.

b) is an "INSERT" in a collection, but it is not really practical because you'd have to "search" all collections to figure out in which state the resource is. (I assume that when you say a "logical" move, you are not doing a physical move so other links to this resource would not be broken, right?)

c) is funny, now you "INSERT" another resource to the initial resource, of course that's not CRUDy? How does the consumer gets notified that the order submission failed or succeeded? 

And, all the consumer wanted to do is express his or her intent to "submit an order". Do you really think this is a lot simpler than non-CRUD oriented approaches? Who are you fooling? 

The reality of everything that you explain is a "build-as-you-go" CRUDing language that can deal with the complex cases supported by SQL or XQuery and which relies on a million different conventions decided in a common document between the developers of the consumer and the provider. This is not coupling, this is called mortar. These conventions will be done and interpreted differently by different people. So what have you gained? absolutely nothing, we just created a bigger mess than before. Sure, some people can now get back to their familiar CRUD-Oriented Synchronous Client/Server architecture and continue CRUDing happily. They don't realize that COSC/S is the core of the problem in IT today. Some people call that "“Full control without much complexity.” Yeah, that works for me.

But of course "I don't understand". How easy! Why don't you show us the code? Why not? why wait for questions from the readers to even touch that very central question? (BTW, I was not the one who asked the question in case you wonder, I always use my true identity - but of course, I am lonely).

At the end of the day, I have simply never seen any evidence that the (other) REST community would use (Roy's) REST for anything other than a DAL and CRUD around with it. When you talk to Steve Vinoski, the action "interface" does not even exist. Few people are that "extreme", the non-sectarian REST lovers use POSTs and define an action interface with WADL. If I understand Roy correctly, that's considered RESTful not harmful.

You have one way to stop this huge waste of time: why don't you show us a non-trivial scenario implemented with REST principles? Pick your process, insurance, banking, procurement, reserve a tennis court… whatever. Maybe, just maybe, you will realize that you can't build connected systems with just a DAL and a CRUD mentality. You might also realize that even this principle will not work: "REST is limited to the client being told what to do next by the current state of where they are now". The way Roy frames REST is counter to SOA principles. It assumes an "application" boundary. It assumes shackles between the client and the server. SOA is about Peer-to-Peer software agents performing units of work. There is no client, there is no server, there is no-single agent telling everybody else what to do. There are autonomous agents which work Coordination (and not cohesion) is a central concept of SOA. Go read WS-CAF if you don't understand what I mean.

Yes, I have great hopes for the future, I start seeing that there is a strong movement emerging behind SOA. It is built along the lines of SOA-a-la-WSDL with a bi-directional interface asynchronous mechanism, with orchestration at the core, coordination at the center, with resources too !  SCA and the SCA's Widget capability will kill this silly RESTy CRUDing, have no doubt about it.

I think at this point you guys are just wasting everyone's time. Other "You don't understand" won't cut it much longer... hey, you can use it as much as you want, but, funny enough, there is only one thing that I care, it is precisely to "understand" like the most of us. So why don't you explain? No you won't...

 

05/03/08 :: [SOA] Roy's Post  [permalink]

Subbu pointed to a really interesting post from Roy Fielding.

There seems to be a common thread with most posts here. People have been busy modeling everything as a resource and now they want to know how to do everything in a PUT or DELETE instead of any of the other HTTP methods. That is wrong.

REST is not limited to GET, PUT, and DELETE. Anyone who says so is just making things up as they go along.

REST is limited to the client being told what to do next by the current state of where they are now, aside from the entry point(s) we call a bookmark.

That is feasible because the set of methods is uniform, not because it is limited to CRUD.

If you read the (other) REST community bible, the RESTful Web Services book, all they recommend is to use REST to create a Data Access Layer, so they mostly GET, PUT and DELETE. Unless I am mistaken, I also heard most of the people in the (other) REST community talk about how bad "POST" was, not to mention how a WADLization of the resources would be a tragic design mistake.

I have heard so many times that there is no "actions" in REST that you simply "PUT" resource representations back or new resources in collections at will.

I must admit that your post is refreshing, though I would still need some clarification about the "Client is being told what to do next". As I have explained many times, this makes perfect sense when the client is a browser operated by a human. When the client is a (mundane) piece of code, I would be really curious to understand your opinion on how much "coupling" is needed in that case, and how this coupling would materialize (in relation to the uniform interface).

05/03/08 :: [SOA] SOA Data Services  [permalink]

Eric is definitely the kind of people I like (apologies for not tracking your blog earlier). 

Yet when I review many SOA enterprise designs, I find that creating specific entity services are most often designed and implemented incorrectly. What many are building is simple CRUD service access to data that provides little business value and causes technical problems with data integrity, performance and tight-coupling with the underlying data structures – see CRUD anti-pattern.

What you want from an entity service is business orientation (e.g. an order or customer entity vs. CRUD like operations), federation (aggregating disparate but related data), high performance (such as caching federated data) and abstraction (data structure and location hidden from the services).

Actually REST (Roy's Theory of the Web) does has something to teach us in the Data Services world. When Data is spread across autonomous systems of record (e.g. services), you need to consider 3 levels of federation:

• navigational (from customer to order)
• attribute level (some customer attributes are in two separate systems)
• record level (customer records are in different systems)

Of course an information system often involves all 3 levels of federation simultaneously. The role of federation is to manage seamlessly the relationships (as in ER) across disparate systems. So for instance, in the first type, a purchase order systems is obviously going to have a bit of customer information embedded in it. Each customer will have an identifier and the goal of a federation system is to make sure that when you navigate from a customer to an order, the correct identifier mapping is applied to get to its orders.

If you use a URI as an identifier (and not as a QBE with this silly URI mapping ritual), REST provides an identifier mechanism that can avoid complex foreign key mappings.

At the attribute level, REST (Roy's Theory of the Web) has also something to teach us. If we go back to the cohesion discussion and the point I was making about "Cohesive Information Models", REST can also help to solve quite elegantly the problem of providing just-in-time information. The purchase order would hold a URI to the customer's contact information (which for a business can change quite often). It is not until you physically need to consume this information (i.e. call the contact, or at least display this information) that you would actually invoke the corresponding GET operation.

I cannot stress enough the importance of Data Federation concepts in a Service Oriented Architecture, precisely because cohesive boundaries are virtually impossible to define.

The only issue I have with the REST community (not Roy) is that they probably see the same benefits as I do, and they go all the way, they say, REST works for everything else, yet what they end up doing is CRUDing because they absolutely refuse to consider how important the concept of Inter-Action is (which is missing from Roy's Theory).

I have nothing against theories, they are obviously the right thing to do, but so many times in my career have I seen theories applied outside the context from which they were developed (no less than the Web here). The first week into my PhD, my advisor asked me to review some of his calculations. I smelled something fishy, but I was not familiar with the theory, and I told him. Remember it was a time where you could not "Google" anything. It was a time where you had to order books from a librarian and wait sometimes several weeks before you could get it. Sure enough during my second PhD (I have technically 1.5 PhD, I dropped out the second), I attended a class where they reviewed the theory that my advisor had applied and he had pretty much violated all assumptions that you needed to validate before you could apply the theory). Similarly, I was working as part of a 50 M DARPA project at HRL, and again a Professor from CU demonstrated that one of the key goal of this project could not be achieved based on the theory of non-linear systems. Yet, our system had only a few discontinuities and was "linear-enough" otherwise to get the job done with linear system controls. So if you allow me, I am quite suspicious when people (PhDs or not) apply "theories" well outside their boundaries without validating (and understanding) the context in which the theories apply.

Roy's developed his theory by looking at the Web as a collection of "resources". IMHO, he has built his theory without  considering the concept of "Resource Lifecycles" (for good reasons). He made an approximation, like every scientist does. Here is why, the lifecycle of a Web Page is very basic:

You can question whether "update" is a good thing to do for a Web page, but, whatever the lifecycle is, it is basic and most importantly, it is UNIFORM for all web pages. This uniformity is the reason for the Uniform Interface, there is nothing intrinsic to the "uniform interface" (I think Roy goes a bit overboard in his thesis with this constraint, he does not explain enough the origin of the constraint, he focuses too much on the benefit).

Here is a plausible general metamodel for resources, I think REST principles can be derived from this metamodel but it has the merit to surface an arbitrary resource lifecycle and an arbitrary inter-action interface.

A uniform interface is a complete myth when the lifecycle of a resource is different than the one of a web page. This is yet another case of a few people (not Roy) applying "theories" well beyond the context for which they were defined. It is already hard enough to come up with a useful theory (ask Newton) but thinking that there are extended areas of application that serendipitously appear here and there is, in general, pure fantaisy.

Sure enough the remoting patrol, jumped on this like a bear on a beehive, they saw yet another uniform communication interface they had not explored. Moreover this communication interface was backed with extremely scalable web servers activation mechanisms. But, yet again, they don't realize that the lack of "lifecycle" and "inter-actions" that made them fail to apply Distributed Computing communication principles to information systems construction, is going to fail them again in REST. As long as we will give people some mechanisms to CRUD the resources, we will face the same wall.

I don't think I am that lonely saying all this, but sure enough, it is not as sexy as telling people: look how successful the Web is, now imagine how successful you are going to be by implementing your information systems with the technologies that made the Web successful. Woa, can't beat that logic !

05/01/08 :: [SOA] WOAmiting  [permalink]

The recent movement of "so called" analysts and experts around WOA has left quite a few people a bit sickened (Jeff, Todd, Miko...)

We have lived in the "transistor age" for the past 40 years, just CRUDing around.

SOA is about building integrated circuits and reusing these ICs when to construct solutions. It is no surprise that some people can't make the switch, so to speak, and will find every possible way to build their solutions from transistors. Yes, the ICs have been hard to come by, because the old mentality is hard to break (Some people will still use an OpAmp as a transistor). It is also true that the IC technologies are barely hitting the market with: SCA and BPEL for instance.  

04/30/08 :: [REST] Who Said Uniform Interface Again?  [permalink]

These days, Tim Bray courageously claims "REST, they say, is the way to go". I guess if "they" say, it must be true, as long as Tim repeats it.

Francois Leygues, a French member of the REST community, sent me a pointer to this blog post from Andrew Townley: "URI Opacity Revisited".

Andrew's findings are in line with mine when it comes to explaining how "uniform" an interface can be. Yes, when a human is in the loop, it works like magic, and...

The Big Problem

While all the above discussion about hypermedia applications is great stuff, the big problem with interesting hypermedia applications is that you need to understand what all the links do. With people, it’s easier, but with automated service consumers, it’s much harder.

"Ben oui", as we say sadly in French. You don't have to be rocket scientist to figure that out. I would not be surprised if many more people figure this out after playing 5 minutes with a REST framework trying to implement some server-to-server communication.

Let's take yet another example. I have a "Tennis Court" resource. This resource can be reserved or not (just to keep it simple). So there are two actions, reserve and free. The resource itself can be in two states: reserved and free. How do you implement a reservation system?

a) the CRUD way: the resource representation has an element called <free>, you change its value and you PUT the Tennis Court resource.

b) you POST something somewhere: that's really like a Web Service operation

c) you PUT a "reservation"/"termination" resource with the Tennis Court ID and your ID (at this point you can either assumes that the Tennis Court resource implementation looks up for the latest reservation or termination and returns the content of the <free> element accordingly, or the implementation invokes a) to avoid figuring out the state of the resource when it's queried again).

Solution a) strongly couples the consumer and the provider, b) is business as usual, c) you mean c) is really simpler than a web service operation? no at end of the day there is something called a "reserve" operation (you could also think of a reservation service), and this "reserve" operation has to fly from the consumer to the provider. When a user is in the loop, all this works automagically, and I explained many times why (Andrew agrees). When two software agents talk to each other, then they both have to agree on what a "reserve" operation is and use a "uniform interface" to exchange the intend.

Now, two words on the topic of "URI Opacity". Again, when humans are in the loop, meaningful URL make perfect sense. I don't think it is a good idea to rely at all on this mechanism in the case of server-to-server communication because a service implementation is usually involving some back-end system and it is likely that this backend system will want to take a look at this metadata too.

At the end of the day, programming is really about not being "sloppy", and REST, as a programming model, is the ultimate sloppy technology. Again, when applied to the architecture of the Web, REST is great, terrific, I can't say enough good things about it. When applied to SOA, its principles are terrible. Except for the concetp of "Resource", REST has absolutely nothing that is required for constructing a SOA: bi-directional interfaces, inter-actions, orchestration, coordination and assemblies. People  like Tim Bray simply don't get it, be sure that they won't be around to help clean up the mess they are creating.

Who said again, that "I was the only one to say what I am saying and therefore I MUST be wrong"? I am simply appalled at the type of things some people in the REST community will do and claim to try to stop any reasonable discussion.

04/26/08 :: [Other] Where's your DVD Drive?  [permalink]

This video caught my attention. So its the Spring of 2008, you are the CEO of 60+ billion dollar ISV and when someone shows you an Air Book, you ask

where's your DVD drive?

I am surprised he did not add, where's your modem?

04/26/08 :: [SOA] What's Cohesive in SOA?  [permalink]

Some people tend to think that I enjoy creating such a controversies, but at the end of the day I don't. Anybody is of course free to say and do what they want, but we seem to have reached a point of evolution that seem to imply "as long as you can say or write something, there is always some truth to it".

SOA could very well be the ideal area to say anything you want. And boy do some people enjoy doing that. Of course, you could very well turn this argument to me, but I pride myself to keep the debate open, no mater what. My only concern is helping IT architects make their day to day decisions, whether they are strategic or tactical.

The short history of SOA is that people once thought 10 years ago to use Web technologies to exchange information between servers (that was a sensitive thing to try). It started with ebXML for B2B applications, then Microsoft and IBM drove it to focus on interoperability while some people threw BPM in the mix. We then realize we could do EAI a bit better with XML and SOAP than the clunky EAI frameworks of the 90s. Now we are reinventing a component model using SOA principles with SCA, while a new community is emerging calling all these effort "crap", and asking us to go back to the basics of the Web (and with it, going back the corniest programming model concepts).

Overall nothing really took shape over the last 10 years, sure there are successful products and technologies, but at the end of the day, you look back and you say Service Orientation did not emerge the way Object Orientation did. I have not lost hope, this one is hard, and there is also a lot more baggage (solid baggage) today than we had in the 70s.

So in this context it is no wonder why so called "experts" and "analysts" can paint an "ubuesque" vision of SOA and say anything they want while FUDing an entire industry one post at a time. I won't say it enough, SOA is unfamiliar and to the people that tell you they were doing SOA 20 years, just ask them what they used for an orchestration engine, how they were "assembling" components together and if their IDL supported "bi-directional" (let me keep the dash to really emphasize the bi) interfaces.

There are many approaches to agent-to-agent "communications"  (I don't want to say "distributed computing" as I see the goal of DC to actually distribute computing resources and make them act as one). DC has been tremendously successful if you ask me, and huge progress is still made every day.

Now, you need a "uniform interface" to communicate for sure. As I said in an earlier post there are plenty to choose from:

• Send, Receive (MEST, WSE)
• Get, Put, Delete, Post (REST)
• Select, Insert, Update, Delete (SQL)
• Request/Response, Notification, Solicit/Response, One-Way (WSDL)
• Publish/Subscribe (JMS and the like)
• Prepare/Commit/Rollback
• OAGIS Verbs
• ...

You can implement agent-to-agent communication with any of them. Each have some merit, I don't really see one better than another, just trade-offs. For instance, Udi does great things with the Publish/Subscribe uniform interface (which is a bit off the beaten paths of SOA). I could even argue that it does not really matter which one you pick, at the end of the day, this collection of interfaces shows that, somehow, at some point, you will need all the other semantics. By picking one you simply agree to layer the semantics of the others on top of the one you picked. Incidentally, you will notice that some of these uniform interface may require some form of coordination, so any "so-called" SOA Reference Model that does not surface "coordinators" are incomplete and require major redesign.

The question, the very question, is do you really have a SOA once you picked one? Is SOA about taking the current programming model and arbitrarily "remote" something using one of these uniform interfaces? Anybody would venture to say Yes? I don't think so when the question is asked in these terms. So what's more to the uniform interface?

Of course, first and foremost, SOA is about creating autonomous software agent that react to a request. By definition an "autonomous" software agent of that sort exhibits a high degree of cohesion. You can't invoke anything else from a Service than what it's interface exposes. So I hope Jim, this is not what you were trying to teach us.

The problem though is this stubborn service interface, that can't be hacked in any way, and if you have to change it, you have to version it, this is yucky. In reality, let's face it, people are used to by-pass cohesive principles all the time. They can never make up their mind about what the best way to fetch or update some data elements. There is always a scenario that requires a new query to be exposed or a new update gram to be built. So over time, low level services interfaces tend to become messy as modifications result in an expanded interface. REST found the solution to this mess, "don't use a service interface", just pretend it is not there, and keep adding URIs (which are really interface point and not resources) to deal with these constant modifications. Ta..da... problem solved? not quite. There are two problems with that: a) in a connected world, there are more than one consumers, nobody owns both ends any more. What happens if you break "the other end"? b) the other problem is precisely that the programming model should not let you CRUD from the consumer side. As long as you will CRUD, you will strongly couple the "consumer" and the "provider". This is not because you chose a uniform interface to happily continue CRUDing remotely that you are building a Service Oriented Architecture. You can tell anyone this is not true, ultimately, this is what people will realize. It does not matter if you apply good cohesion design principle as you are CRUDing, you will fail to build reusable services, not because the service can't be reused at a given point in time, but because you will break existing consumers as you try to accommodate the needs of new consumers. Above the service interface, Cohesion  is Coercion in a SOA.

The goal of a Service Oriented Architecture is precisely about using new technologies a) that support forward compatibility (XML, XSD, WSDL) and b) creating a new factoring of your business logic (SCA, BPEL) that is not CRUD-oriented to avoid this terrible coupling between the consumer and the provider. This is why people were not doing SOA 20 years ago, nobody was doing SOA 20 years ago. Of course ultimately you hit the systems of record, so some CRUD operation is going to be performed. Of course cohesive design principles are going to be applied within a service implementation. But above the service interface I would argue that nothing is cohesive. You actually want to achieve quite the opposite with loose coupling. You want to be able to assemble your service with as many consumers as possible. You may even want the consumer to tell you a few things about how to provide the service by injecting some decision rules or a sub-service provider..., again this is loose coupling. At that level none of the cohesion principles apply. I would even argue that technically, one of the major advances of SOA , (again at the technical architecture level), is that I can now build systems in ways that let me scale, secure, fail-over,... different parts of the systems differently, against all principles of cohesion. Hence, the principles of good cohesion are of no help when it comes to design the service interface.

Now, Jim, have you ever asked how cohesion can apply to an Enterprise Information Model? Information is relational, has always been, will always be. How could you apply any cohesion principles to a data model that is entirely related? There are none that you can apply. And frankly this is also a challenge in SOA. You can't easily do join across service interfaces (REST or WSDL for that matter). It really requires that you factor your data structures in two tiers: an information business entity tier and the parts that constitute a given information entity. If you have a purchase order service and a customer service, how do you pull customer information as you fetch a Purchase Order? (Say, the telephone number of the customer or the contact person, that you would always want to be the latest information available).  Now, let's say you want to use SalesForce.com as you "Customer Service" and "NetSuite" as your Purchase Order Service. You will quickly conclude that pretty much every "Resource Centric" service is "related" to any other service and cohesive boundaries are really hard to define.

If you look at this process (click on the picture to zoom in).

(this figure is coming from Slide 4 of Ross Altman's presentation on composite applications)

There is nothing, absolutely nothing, cohesive about it. You want to be able to create value across cohesive boundaries. Services like "credit score", "blue book value", "appraisal", "payment" can be reused in many different context: home, car, boat, ... loans. Business Information like "Loan Application", "Loan", "Customer", again brought into the process assembly in a lot more opportunistic manner than a cohesive manner. This is where concept where agility, business alignment, reuse... come to play. You are not "programming" in a service oriented architecture, you are "assembling", you are "compositing", you are "orchestrating".

Cohesion offers design concepts antagonist to Service Oriented Architecture and only lives within the service implementation. I understand why, if you CRUD all day with "services", you want (absolutely) to apply cohesion principles like people have been doing it for 40 years, but don't tell me you are doing SOA.

SOA is about the emergence of an Inter-Action Oriented, Asynchronous Peer-to-Peer programming model. Sure the old programming model, the CRUD-oriented synchronous client/server one) is still going to be used "inside" a service implementation, down below, but SOA is new, it is unfamiliar, it is based on technologies that did not exist 20 years ago, all the good software design principles that were discovered back then don't generally apply. Again, I am absolutely not surprised that you or Steve think in these terms, this is precisely the problem. Once you will unlearn some of the great things of the past, and come to SOA with and open mind, you may, understand how to use WSDL, BPEL or SCA.

So yes, I am quite mad at the "cheap" and sometimes incoherent arguments that a small and cohesive community of "experts" and "analysts" are spreading to keep the "old regime" in place. I was just as mad at Assaf Arkin, Howard Smith and Frank Leyman who argued for years that BPEL/BPML were a "Business Process" thing, when most of the semantics to support business processes were missing. This handful of people set the BPM industry 5-7 years back. Can you imagine the magnitude of the loss? The same thing is happening here today, for the exact same reasons: closed minds, cheap arguments over CIRCA 2001 pictures of SOA, a small group of buddies who point at each other to gain credibility... the recipe is well known. At least, they won't be able to tell in the future that they were "unaware" of possible alternatives to their best interpretation of what a Service Oriented Architecture is.

 

04/24/08 :: [SOA] Is SOA ABOUT Building DISTRIBTUTED SYSTEMS?  [permalink]

Stefan commented that "respected members of the distributed computing community for a long time" had to be right about SOA and I had to be wrong because I was a lonely voice.

First, I am not alone, we are simply not as "cohesive" as certain groups. Even saying "we" sounds weird to me. This is not about we, us, them. This is about SOA.

What I am often arguing about is well supported by commercial products and open source frameworks such as Tuscany and Fabric3. There is also this researcher and her group at IBM Zurich, Ksenia Ryndina,, who are actively researching the kind of programming model I advocate for. There is also an entire open source methodology that is based on these principles: http://www.praxeme.org. I may well be the most vocal of all these people, but I am certainly not alone… on this side of the discussion. I am certain most of them have given up a long time ago arguing with the other side.

Amusingly, the large distributed computing power houses (IBM, Sun, Oracle/BEA, and even SAP) have all a composite application strategy, and I don’t think I would be in the position to teach them anything on the topic. At best, I could point out that I have been saying this kind of thing for a long time.

Second, yes, I do have a track record of being a lonely voice and being right. I do remember a time in early 2000s when "Business Process" and pi-calculus was the craze. Even people as famous as Frank Layman from IBM research (whom I met in 2003) were spreading their wisdom about BPEL being a "Business Process Execution Language", when I was the only one, not anyone else, against Frank, against Assaf, against Howard Smith, against John Pike to say that no, BPEL was a programming language and it had nothing to do with "business process". BPEL was and is a Service implementation language that is a key component of an Inter-Action Oriented Asynchronous Peer-to-Peer programming model. Howard Smith and John Spike has been converted. We don't hear Frank on the topic anymore. Jim Webber in a recent presentation clearly positioned BPEL as a programming language. I don't think there is many people to say otherwise. I was also the one, the only one that pi-calculus was not enough, though a solid theoretical foundation, and again these days, there are hardly anybody to argue that leveling this SIL at the level of pi-calculus is not a sensitive thing to do. You can create a pi-calculus VM if you want, but the semantics are incomplete to make it an efficient SIL.

Even David Chappell had told me once (at the 2004 WCF SDR) that I was wasting my career as I was explaining to him the difference between orchestration and choreography and the place of orchestration in SOA (as a SIL). He said, "if Microsoft says this is what it is", why do you fight it? I fight, because of remarks like this, or remarks like yours. If our industry was less full of so-called "experts" we would make progress a lot faster, if you know what I mean. Yes, I have come very close to "experts" and I can safely tell anyone that "experts" are just like everyone else.

So Stefan, if you allow me, if you want to "shut me up" I would request that you use a different argument (this one was really cheap) and maybe, just maybe, we could use technical arguments to reach a conclusion.

Third, and that's the most interesting point, we could ask the question whether SOA is really about Distributed Systems, whether an "expert" in distributed systems is coming to SOA with the right mindset? Do these "experts" really ask themselves why distribution at the programming level has been a failure when applied to information systems (and a big success when applied to other things)? why CORBA failed to deliver a sound architectural foundation? When you read Steve Vinoski's papers you understand what his vision of distributed system is. He wrote recently an excellent one on REST and ERLANG, and what you see is that these respected distributed computing experts are really thinking in terms of activation. He loves ERLANG because it has a fantastic, light weight, very scalable activation model (plus a cool pattern matching capability that fits REST like a glove). This is what he dreamed to build one day. Is this useful, yes, very much so. Is this SOA? not quite.

I wrote many times that SOA is not just about distributed systems, but it is a lot more about "connected" systems. I had great hopes when Don Box started to talk along these lines in 2003, at the LA PDC. He sure delivered a great distributed communication framework, but as a "distributed expert", I think he missed the mark on "connected systems". Connected Systems are about reuse a lot more than they are about activation and "distribution". Reuse is about "connecting" arbitrary pieces of code together, not in a serendipitous fashion, but in a deliberate fashion. Connected systems are about inter-actions, assemblies and forward compatibility, semantically accessible data structures... Don Box himself had said once that Web Services technologies were the first technology that did not require code generation systematically, and that change was profound, this quote is still on the first page of my web site, and of course, the first thing he did was to force code generation in every aspect of WCF, negating the very innovation of Web Service technologies (and XML). Right about the same time, WS-CAF came along. This is truly the best standard work I have ever seen. Jim Webber (and Mark Little) was a co-author of that spec. I always wondered if WS-CAF was consciously designed to participate in this IAOAP2P programming model. I know today that it was a pure coincidence, even the name CAF was right on, but no, this was a coincidence.

At the end of the day, SOA is about replacing a CRUD-oriented Synchronous Client/Server programming model that has been in use for over 40 years as the core foundation of information systems by an Inter-Action oriented Asynchronous Peer-to-Peer programming model. There is no other way around it, COSC/S cannot raise to the challenge of a connected world, COSC/S is responsible for the silos that we have built for the last 40 years. As long as the "experts" will merely XMLize a COSC/S, we will keep asking why we can't get benefits with SOA. The answer is simply because we are NOT doing SOA.

So if you allow me, Stefan, I'll continue patiently hammering these concepts and the difference between distributed and connected (i.e. composite), until this is a well establish concept, even if you think that I am too lonely in this discussion to deserve  consideration by the intelligentsia.

04/20/08 :: [SOA] Cohesive Response  [permalink]

Well I got a lot of people upset: Stefan, Savas, Jim kept the legendary British cool.

Again, I stand by what I am saying, Jim's position on:

• cohesion in SOA (beyond the trivial conclusion that any autonomous asset is by default exhibiting a high degree of cohesion)
• process-to-service relationship
• MVC as a design pattern for SOA/BPM
• Business people becoming IT architects (last but not least by far)

is more than questionable.

Savas, about my communication style, I found that it works -indirectly-: these days, I don't read many more people in the REST community bashing WS-*, Tim Bray's BS on "the end of SOA", or Steve's serendipitous crap. Actually people eventually strengthen their arguments or give them up (they don't want to look stupid).  Everybody saves time: you, the reader, me. Now, yes, it hurts me, you are probably not the first one to think that I am rude and unprofessional, but that's ok, if this is the price to pay, this is a very tiny price compared to the end result.

The reality Savas, Stefan, Jim is that SOA is still raising a lot of questions. We can circle around for years if you want to. I offer an alternative: I just ask anybody to answer just two very simple questions:

Have you ever considered factoring the business logic along the lines of a Resource Lifecycle? How does this fit with whatever your recommendations are?

Answering these two questions alone would end the REST/WOA vs SOA, MEST vs WSDL, BPEL vs BPMN (if you want to) and surface how critical representation-friendly data structures, bi-directional interface definitions, orchestration, assemblies are to designing a Service Oriented Architecture (note that I use concepts not technologies, pick your favorite technology).

I would argue that all of the people that are so prompt to be offended by my posts will never answer these questions, they will never drive the discussion to answer these two questions.

An Inter-Action oriented, Asynchronous, Peer-to-Peer programming is here to replace the CRUD-Oriented, Synchronous Client/Server programming model. This is only a matter of time.

The irony here is that MEST is a lot closer to an IAOAP2P than a COSC/S, but Jim does not see it.

04/20/08 :: [SOA] Loose Coupling and Cohesion  [permalink]

Quite hilarious (and symptomatic), I advise Jim to catch up with modern SOA concepts, technologies and principles and he advises me to read antiquated software engineering books, or articles from Steve Vinoski.

Jim, if all you wanted to tell us is that autonomy requires some degree of cohesion, well you did not need a blog post for that. I was trying to interpret your blog post in the context of an assembly of services performing a unit of work and wondering how cohesive could an assembly of services be?

Again, if you take a modern view of SOA (say post 2005) you really wonder how an assembly (a module for me) can exhibit some degree of cohesion. Even a service within an assembly (a component in SCA terms) is likely to fail "cohesion" litmus tests.

Let's review Steve Vinoski's article.

Functional cohesion occurs when a module does only one thing: take a look at Resource Lifecycle Services, what does "one thing" means. Is your recommendation really that services do only one thing at a time?

Communication Cohesion is when a module carries out multiple operations based on the same input or output data: ok. why not, some services might be designed that way but this is not the case for the vast majority of them. I would argue that this is not even very important in SOA: XML helps you not having the need to establish Communication Cohesion.

Sequential Cohesion is when a module carries out several tasks and the input of one tasks feed to the other. What's the big deal about that one? This is true in a pre-XML civilization. With XML you don't care as much to achieve this level of cohesion.

Logical Cohesion is a condition in which a module's activities are grouped together because they appear to be able to share common implementations: that's a bit closer to SOA but again, have you ever considered that when a Service Interface is implemented by a BPEL definition, all the operations are tied together in the same implementation? unlike OO you don't have necessarily a one-to-one relationship between operation and implementation.

Jim in case you have not noticed, SOA is unfamiliar, it is time to rewrite the books. Concepts like XML extensibility, semantic access (XPath), or forward compatibility precisely alleviate the need to be cohesive. Whatever the principles you learned in school, they most likely don't apply any longer in an Inter-Action Oriented, Asynchronous Peer-to-Peer programming model.

So for me cohesion is not a design principle you can apply at the assembly level and even at the operation level. XML is the technology that allowed us to break from cohesive software engineering (CSE). CSE has been developed for bronze age programming languages. We are in 2008 today, things have evolved a tiny bit. 

I stand by what I have said, a cohesion requirement creates absolutely unnecessary constraints on the design of service interfaces and implementations that actually reduce the degree of reuse of a given service and its capacity to participate in different assemblies. Maybe it is time to become familiar with modern loose coupling concepts that include: bi-directional interfaces, assemblies, orchestration languages, extensible and semantically accessible data structures. Ancient programming techniques have been designed precisely because you did not have these concepts within your programming model.

Every thing you say in your post, from applying cohesion principles to SOA, "a process is a service" or "look at MVC if you don't get it", to "Business people should become IT architects" does not make any sense and are likely to fail anyone trying to build a Service Oriented Architecture.

04/20/08 :: [SOA] Loose Coupling  [permalink]

A couple posts picked my interest today. "Does WOA bring anything new to SOA?" by Mike Meehan. Well, I have already expressed what I thought about WOA and REST so I won't do a repeat here. What I found surprising is that there is actually quite a few people agreeing with some of what I was saying.

If users perceive WOA to be outside the principles of SOA, it could prove an excellent vehicle for building Web-based stovepipes.

The article concludes:

[people] expressed an unequivocal desire to have no more than one something-oriented architecture in their lives

Well someone who is trying single-handedly to topple SOA with a "Something Else Oriented Architecture" is Jim Webber -the MEST guy. Mark Little points out this post by Jim: "Anemic Service Model". I am not sure why. I found his post being completely erroneous. I may be missing something, but Jim is looking at the relationship between Cohesion and Loose Coupling.

He defines Cohesion as follows:

"[Cohesion] measures of the extent to which related aspects of a system are kept together in the same module, and unrelated aspects are kept out."

For me cohesion sounds like a good engineering principle that looks a lot like "Dependency Structure Matrix". In fact Jim concludes:

In fact good software tends to be both loosely coupled and highly cohesive.

This is the part that I find completely bogus (again, unless I am missing something). The goal of loose coupling is precisely to mitigate cohesion as a good engineering principle. You can't be cohesive in a connected system. You can't be both cohesive and loosely coupled. Even in English, associating the two sounds really bad.

The goal of loose coupling is to make to pieces of code work together even though they may have been written at a different time, using different technologies, with a different security model, ...

SOA is about going away from cohesive systems to enable a wider range of reuse scenarios. Nested cohesive systems do provide a "nested" (library-style) reuse model: the upper layers can reuse the lower layers. Unfortunately, it forces us to create systems in a way that is incompatible with the way information systems should be architected. Cohesion is the problem, loose coupling is the solution. Jim, do you really think that people are trying to "keep stuff together in the same module"?

How can you say that when a system is loosely coupled,

even minor changes ripple through multiple components or systems

Have you considered this definition for "Loose coupling"?

The funny thing is that Jim thinks that other people think SOA is best built by "nesting" service invocation in a "Dependency Structure Matrix" style. Jim, I am sorry to say, but this view is what people where talking about in 2001-2002. You need a refresh. The very picture that you think people have of SOA is highly-cohesive.

Then you continue and recommend that people

build out a service that implements [a] process

Jim, what are you talking about a service implements an entire process? for sure that's going to be very cohesive. Now, how can you say that this is an instance of "MVC"? Precisely the fundamental problem with MVC is that is does not have natural boundaries for processes and human tasks. People using MVC keep creating "representations" (i.e. Model), views and controllers that cannot be associated easily to processes and human tasks. Your recommendations and analyses are completely bogus. I would really recommend that you spend more time understanding SOA and less time trying to create specs that are absolutely counter to SOA principles.

You know:

It might sound like a pretty dumb idea that my business stakeholders to become (inadvertent) IT architects.

is one of the dumbest recommendation you can make. You are no George Lucas, there is only one thing anemic about your post, it is your understanding of SOA.

04/20/08 :: [Other] Winter Fatigue? [permalink]

Yes, it's April 20th and yes it has been snowing yesterday, today and it will probably snow tomorrow too. Want some proof? I took this picture of my neighborhood this morning !

04/17/08 :: [SOA] SOA Fatigue? [permalink]

John claims that :

SOA is no longer a mysterious concept that we need conferences, presentations, books and articles to clarify for us

I rarely disagree with John and I can't say that I completely disagree with him on this one. I would argue however that what we need is a lot less of this and a lot more discussions on what we do when "we roll up our sleeves and get to work". My main concern is that when John says:

We get it already - stop debating what it is and go build something already.

I am not sure this is true, I am not sure people get it and know what to do despite the hundreds of conferences that claimed to talk about SOA. I am certain that John gets it, I agree that there are some products and technologies that are mature enough but most people use SOA within the boundaries of the current programming model.

I won't say it enough, over the last 40 years the software industry has developed a "CRUD Oriented Synchronous Client/Server" programming model. Using SOA technologies within this programming model will lead you nowhere.

SOA represents the emergence of an alternative programming model which is "Inter-Action oriented, Asynchronous and Peer-to-Peer".

I don't know many people who say what I am saying and sending people CRUDing around with XML, SOAP and WSDL is likely to create a huge disillusion.

I mean, John, come on, we don't even have a decent SOA Reference Model: Duane Nickull probably delivered one of the worst possible SOA RM , only second to the one developed by the W3C TAG in early 2000.

03/30/08 :: [Other] Interesting contrast on TV this Week [permalink]

I don't watch TV that much except for football. For everything else, I have a little home built digital recorder that occasionally records interesting things.

This week there was an interesting contrast. First there was this show from the University of Washington on "Shared Prosperity in an Age of Global Warming". I can't say enough how proud I am to live in the Northwest. It's almost like living in a different country and nobody captures best the Northwest spirit than this ad from a local insurance company showing all kinds of Northwest characters and claiming "We are a lot like you, ... a little different". I would argue a lot different.

In this presentation Ron Sims, a local county level politician presents his vision about tying together global warming, the global and local economy and poverty. This guy makes you really proud to live here, just like our governor, Christine Gregoire, and so many other people. It looks to me that the US always had forward thinking areas like the greater Chicago area (up to Ohio) in the late 1800s which had incredible innovations, not just in Technology but also economically, architecturally (F.L. Wright) ... or California in the second half of the 20th century. Will the Northwest walk in these footsteps? It is probably too early to tell.

The bottom line is that I feel really privileged to live here. I don't want to comment on Ron's presentation, I think it should be placed in perspective with the current presidential campaign. I'll let you watch it since it is now available on UWTV.org. 

The other show that I watched this week-end was Frontline's "Bush's War", aired on PBS, earlier this week. This show made me sick at two levels (I am constantly sickened by how many people die each day). First it reminded me of 2002-2003 where I could not stand to watch TV and listen to the radio as America was chanting on the path to war. It also made me sick because one of the key turning point the show claims is right after Colin Powell's talk at the UN when the French Prime Minister reacted in Powell's back. They claim that Powell was the only one that could have avoided war and our prime minister killed his credibility with his little anti-war talk. But at the end of the day, it is Powell who delivered that talk to the UN it is he who delivered the case for war (even though we now know that Cheney was the one that wrote that talk, Powell simply delivered it).

Ironically, America is probably the country where there are the most hours of news per day: the networks deliver 3 hours of news in the evening (5 to 7pm) and another 3 hours earlier in the day, yet the quality of what they deliver is abysmal. Worse, they not only make deadly mistakes, but when they venture on the commentary side, they mislead their viewers. Today, the press has completely given up on its role, journalists are no different than a reality show producer, eager to collect ad revenue.

My question is very simple, how can the American press teach anyone a lesson today? The French (and Europe at large) went above and beyond to avoid this war, this is the reality. Where were they then, even Cheney was able to corroborate the false uranium accusation with an article from the NY Times? Where are they today? they want the American people to believe that the French are responsible and sweep everything else under the rug (hundred's of thousands of dead people).

I also found interesting to look back at Tony Blair (I don't want to say the British people or the British Government because they voiced just as hard as the French how terrible a mistake that would be). I'd be really interested to know today what George Bush thinks of himself? Sure his family, and to a certain extend, Britain, have benefited from the strong hand he and Tony put on the Iraki Oil Faucet. But how does he sees himself with respect to his country? What does he think as a Christian and pro-life supporter about all these people dead? What does his wife and daughters think about him? About what he has done to his own country? What Keneth Starr thinks about all this? Why isn't he so prone to impeach Bush and so many other officials for what they have done to their country?

But the saddest thing of all is that the press has learned nothing. It continues to go with the wind, it continues to focus on the wrong things with the wrong attitude. If this country wants to rebuilt itself, it has to rebuilt its press from the ground up. This is no easy task, but the press is one of the most important pillars of democracy, and ultimately freedom.

02/28/08 :: [SOA] WOA: The future of SOA [permalink]

|

There is no shortage of thought leadership when it comes to twisting SOA in every possible way. Dion Hinchcliffe has written for quite some time a series of (generally good) articles on bringing Web 2.0 and SOA together. In his latest article last week, he sketched for us the future of SOA.

He starts by a free for all statement:

Even though Service-Oriented Architecture (SOA) initiatives around the world have the right goals, most efforts have fallen profoundly short of our desired levels of integration and improved business agility.

It hard to know what he means by "profoundly", or "our desired". Being down in the weeds of SOA, working for a large financial institution, I can see first hand how wrong Dion's statement can be. Sure SOA is complex, no doubt about that. People that look at SOA as a lipstick on a pig might very well end up where Dion seems to be, but I would argue that has nothing to do with SOA, precisely, it shows that perpetuating the "old ways" of building information system is wrong, and nothing can fix it, "interoperability" is of no help at all. The truth and the matter is that without a significant architectural change an organization will remain there for ever. Blaming SOA for it, is simply a new way to perpetuate a systemic problem.

Dion sees SOA being reshaped by the Web 2.0 era. He continues:

However, the news isn't all bad, the fascinating story is that there is a place today where the deep integration of our systems and information on a large scale has largely been solved and is a foregone conclusion in most cases. And that place is at the leading-edge of the World-Wide Web, sometimes referred to as Web 2.0.

He sees Mashups leading the charge:

At the leading end of this is the Web mashups story with enterprise mashups being one of the major improvements to the IT landscape that WOA is heralding.

Of course, "WOA puts the focus on Web Resources not Services" (LoL). His assessment could not be clearer:

The basic tenets above paints a picture of a galaxy of nearly infinite granular information resources integrated into a deeply interconnected set of dynamic connections that can be processed individually for a given task, in part (for integrated applications), or as a whole (such as enabling a comprehensive directory or search engine of all data and metadata.) In other words, the Web model provides a single, open, and unified information architecture that is consistent, easily consumed, extremely scalable, securable, very reusable, resilient, and highly federated.

Sometimes I wonder if people read what they write. Why didn't I think of that before? Dion even goes as far as claiming that there is a "timeless" way of building software. I know countless useless ways of building software, but so far, I never found a timeless way.

He concludes:

I should close by emphasizing that I enjoy and use traditional SOA technologies like SOAP, WSDL, and XSD frequently [really?]. But as more and more of the consumer Web moves to a more Web-oriented model, the evidence continues to mount that approaches based on WOA are easier to implement, scale better, and result in much greater uptake and usage scenarios.

And of course:

Traditional SOA is facing a crises of identity at this point [poor SOA, let's give you a big hug], particularly given fairly lackluster results for most, and WOA may just be the prescription we need to make SOA deliver the robust outcomes that we were formerly expecting of it.

I would not hold my breath. The fact and the matter is that real-world composition technologies are emerging, with a real-world composition model. Everyone should understand that pushing composition at the presentation layer simply does not work. You also need process and information composition capabilities. Sure is it a lot less fancy and a lot more technical, but WOA isn't going to cut it, by a large margin. The funny sad thing is that people will never question how they build software.

So REST? WOA? SOA? ROA? COA? How about Duh-OA?

I estimate that 99.999 % of Web Applications are not built in a RESTful/WOA way, but of course it is a terrible sin to build a Web Service with something else than REST/WOA.