Book
05/09/08 :: [REST] Answer to My Open Letter [permalink]
Stefan and Subbu have kindly offered to help me understand how REST can be used to manage the lifecycle of resources. So let's go back to where we were last November and if it is ok with you, here is the example that we could pick (I don't think there is anything particular to this example, and if you'd like you could pick any other one).
This is the beginning of a job application lifecycle. We could focus on implementing the "RejectApplication" operation, i.e. the Job Application Resource transitions from a "submitted" state to an end state (Rejected).
Please note that the red boxes represent some code that would typically invoke the DAL as part of the action implementation. The blue rounded boxes represent states, typically a message event would be sent upon reaching a particular state. But unless you insist, let's forget about events for now. Let's keep it simple.
The trick in this scenario is that once the application has been rejected, the Job Application resource cannot accept any other state changes. Its lifecycle has ended. That would be interesting to show how that can happen based on the implementation that you choose (a, b or c as explained in my previous post).
Ideally, I would like to see both some "client code" and some "service code" (if any). You can pick any programming language -of course. Let's assume that the client is another service -an automated service- that was notified by an ApplicationSubmittedEvent (not represented on the picture) and that service is capable of processing automatically the application to decide to reject the application or not. Upon rejection, the decision service invokes the RejectApplication operation, otherwise, the job application simply waits for interviewers to submit their interviews. In the past, we used to process these applications manually, but James Taylor taught us how we could use a decision service instead, so it's all automated now. SOA really worked for us because we did not have to change a thing on the Job Application Service to make that transition.
Now, I am a lousy analyst and we realized once all this is in production that we can't give a clear status of the application until an interview is submitted (a rejection could still potentially come). What we should have done is to have the decision service tell us also whether the application has passed its criteria. So now, I need to add a new action "AcceptApplication" to my Job Application Service. Could you please explain how this change of the service impacts the "client", i.e. the Job Application Decision Service:
The statelessness constraint isolates the client against changes on the server as it is not dependent on talking to the same server in two consecutive requests.
So to sum up, if you could show me:
a) how to write some client code and potentially some service code that result in transitioning (in a RESTful way) the Job Application resource into a rejected state
b) how once the lifecycle has ended, no further changes can be made to the resource
c) how can I add an operation on the job application service and demonstrate that the decision service client is isolated from this change
I'd be infinitely grateful (and apologetic for all the trouble that I have caused).
Of course, my favorite implementation for the Job Application Service would be to use an SCA composite with a BPEL component to manage the lifecycle, wired to some Java components that implement the resource's DAL. If you allow me, I will publish that Sample code too.
05/08/08 :: [SOA] Open Letter to the (other) Rest Community [permalink]
I know it is far easier to tell me that "You don't understand" than to have an intelligent discussion about the kind of things you dare publishing.
When someone is capable of writing :
The statelessness constraint isolates the client against changes on the server as it is not dependent on talking to the same server in two consecutive requests.
I don't know who does not understand what. Why don't we run a scenario to validate your point? Again, in Roy's world, I see exactly how this works, but when two software agents communicate (say on behalf of two resources inter-acting - a PO and an Invoice), that is pure rubbish. I don't know who convince you of this, but this does not make any sense if unqualified.
I have spent an extensive amount of time reading about your claims
and in particular I read your bible, the RESTful Web Services book, front to
back. All they
explain in this arrogant book is how they created a DAL using REST, and
with that they claim victory. Incidentally, to build a decent DAL, the concept of
"collection" was missing in (Roy's) REST, so APP
was put together to plug that inconvenient hole after years of efforts.
Let me repeat it, your understanding of SOA is pre-2004, what you say about SOA and Web Services might have been true at some point, but things have evolved and it is antiquated today. As Udi pointed out, this is not SOA 2.0, but the picture that we have of SOA today is a lot clearer for a lot more people than it was before 2004.
You know, or course (but that would be too easy) that the best way to convince me (and many other people) is to show me something that does not look like a DAL, so far all the examples I have seen from you or anybody in your community are DAL-oriented:
Yes, you understood that my questions are relevant so you sugar coated your interface with "SubmitOrder" or "Cancel Order" (did I say interface? did you see the interface? the actions? does not look that uniform to me !). You proceed to explain (smartly not in the article, but way down as an answer to a reader's question):
a) one is to PUT a new state to the resource, effectively changing its internal e.g. from booked to shipped.
b) Another way is to do a logical move of the resource from one collection (of booked orders) to another (of shipped orders).
c) A third way is to represent the state change as a resource in itself, e.g. by POSTing it to the order where it becomes a sub-resource. This way, you have a history of changes. The mapping to CRUD is not 1:1 -- a POST can create new resources, or simply process something and return a result. In case a POST is used to create a new resource, the server chooses the URI.
So you would easily admit:
a) is an "UPDATE" and you also understand the coupling it creates between the consumer and provider.
b) is an "INSERT" in a collection, but it is not really practical because you'd have to "search" all collections to figure out in which state the resource is. (I assume that when you say a "logical" move, you are not doing a physical move so other links to this resource would not be broken, right?)
c) is funny, now you "INSERT" another resource to the initial resource, of course that's not CRUDy? How does the consumer gets notified that the order submission failed or succeeded?
And, all the consumer wanted to do is express his or her intent to "submit an order". Do you really think this is a lot simpler than non-CRUD oriented approaches? Who are you fooling?
The reality of everything that you explain is a "build-as-you-go" CRUDing language that can deal with the complex cases supported by SQL or XQuery and which relies on a million different conventions decided in a common document between the developers of the consumer and the provider. This is not coupling, this is called mortar. These conventions will be done and interpreted differently by different people. So what have you gained? absolutely nothing, we just created a bigger mess than before. Sure, some people can now get back to their familiar CRUD-Oriented Synchronous Client/Server architecture and continue CRUDing happily. They don't realize that COSC/S is the core of the problem in IT today. Some people call that "“Full control without much complexity.” Yeah, that works for me.
But of course "I don't understand". How easy! Why don't you show us
the code? Why not? why wait for questions from the readers to even touch
that very central question? (BTW, I was not the one who asked the
question in case you wonder, I always use my true identity - but of
course, I am lonely).
At the end of the day, I have simply never seen any evidence
that the (other) REST community would use (Roy's) REST for anything other than a DAL and CRUD around
with it. When you talk to Steve Vinoski, the action "interface" does not
even exist. Few people are that "extreme", the
non-sectarian REST lovers use POSTs and define an action interface
with WADL. If I understand Roy correctly, that's considered RESTful
not harmful.
You have one way to stop this huge waste of time: why don't you show us a non-trivial scenario implemented with
REST principles? Pick your process, insurance, banking, procurement,
reserve a tennis court… whatever. Maybe, just maybe, you will realize
that you can't build connected systems with just a DAL and a CRUD
mentality. You might also realize that even
this principle will not work: "REST is limited to the
client being told what to do next by the current state of where they are
now". The way Roy
frames REST is counter to SOA principles. It assumes an "application"
boundary. It assumes shackles between the client and the server. SOA is about Peer-to-Peer
software agents performing units of work. There is no client, there is
no server, there is no-single agent telling everybody else what to do.
There are autonomous agents which work Coordination (and not cohesion)
is a central concept of SOA. Go read WS-CAF if you don't understand what I mean.
Yes, I have great hopes for the future, I start seeing that there is a strong movement emerging behind SOA. It is built along the lines of SOA-a-la-WSDL with a bi-directional interface asynchronous mechanism, with orchestration at the core, coordination at the center, with resources too ! SCA and the SCA's Widget capability will kill this silly RESTy CRUDing, have no doubt about it.
I think at this point you guys are just wasting everyone's time. Other "You don't understand" won't cut it much longer... hey, you can use it as much as you want, but, funny enough, there is only one thing that I care, it is precisely to "understand" like the most of us. So why don't you explain? No you won't...
05/03/08 :: [SOA] Roy's Post [permalink]
Subbu pointed to a really interesting post from Roy Fielding.
There seems to be a common thread with most posts here. People have been busy modeling everything as a resource and now they want to know how to do everything in a PUT or DELETE instead of any of the other HTTP methods. That is wrong.
REST is not limited to GET, PUT, and DELETE. Anyone who says so is just making things up as they go along.
REST is limited to the client being told what to do next by the current state of where they are now, aside from the entry point(s) we call a bookmark.
That is feasible because the set of methods is uniform, not because it is limited to CRUD.
If you read the (other) REST community bible, the RESTful Web Services book, all they recommend is to use REST to create a Data Access Layer, so they mostly GET, PUT and DELETE. Unless I am mistaken, I also heard most of the people in the (other) REST community talk about how bad "POST" was, not to mention how a WADLization of the resources would be a tragic design mistake.
I have heard so many times that there is no "actions" in REST that you simply "PUT" resource representations back or new resources in collections at will.
I must admit that your post is refreshing, though I would still need some clarification about the "Client is being told what to do next". As I have explained many times, this makes perfect sense when the client is a browser operated by a human. When the client is a (mundane) piece of code, I would be really curious to understand your opinion on how much "coupling" is needed in that case, and how this coupling would materialize (in relation to the uniform interface).
05/03/08 :: [SOA] SOA Data Services [permalink]
Eric is definitely the kind of people I like (apologies for not tracking your blog earlier).
Yet when I review many SOA enterprise designs, I find that creating specific entity services are most often designed and implemented incorrectly. What many are building is simple CRUD service access to data that provides little business value and causes technical problems with data integrity, performance and tight-coupling with the underlying data structures – see CRUD anti-pattern.
What you want from an entity service is business orientation (e.g. an order or customer entity vs. CRUD like operations), federation (aggregating disparate but related data), high performance (such as caching federated data) and abstraction (data structure and location hidden from the services).
Actually REST (Roy's Theory of the Web) does has something to teach us in the Data Services world. When Data is spread across autonomous systems of record (e.g. services), you need to consider 3 levels of federation:
- navigational (from customer to order)
- attribute level (some customer attributes are in two separate systems)
- record level (customer records are in different systems)
Of course an information system often involves all 3 levels of federation simultaneously. The role of federation is to manage seamlessly the relationships (as in ER) across disparate systems. So for instance, in the first type, a purchase order systems is obviously going to have a bit of customer information embedded in it. Each customer will have an identifier and the goal of a federation system is to make sure that when you navigate from a customer to an order, the correct identifier mapping is applied to get to its orders.
If you use a URI as an identifier (and not as a QBE with this silly URI mapping ritual), REST provides an identifier mechanism that can avoid complex foreign key mappings.
At the attribute level, REST (Roy's Theory of the Web) has also something to teach us. If we go back to the cohesion discussion and the point I was making about "Cohesive Information Models", REST can also help to solve quite elegantly the problem of providing just-in-time information. The purchase order would hold a URI to the customer's contact information (which for a business can change quite often). It is not until you physically need to consume this information (i.e. call the contact, or at least display this information) that you would actually invoke the corresponding GET operation.
I cannot stress enough the importance of Data Federation concepts in a Service Oriented Architecture, precisely because cohesive boundaries are virtually impossible to define.
The only issue I have with the REST community (not Roy) is that they probably see the same benefits as I do, and they go all the way, they say, REST works for everything else, yet what they end up doing is CRUDing because they absolutely refuse to consider how important the concept of Inter-Action is (which is missing from Roy's Theory).
I have nothing against theories, they are obviously the right thing to do, but so many times in my career have I seen theories applied outside the context from which they were developed (no less than the Web here). The first week into my PhD, my advisor asked me to review some of his calculations. I smelled something fishy, but I was not familiar with the theory, and I told him. Remember it was a time where you could not "Google" anything. It was a time where you had to order books from a librarian and wait sometimes several weeks before you could get it. Sure enough during my second PhD (I have technically 1.5 PhD, I dropped out the second), I attended a class where they reviewed the theory that my advisor had applied and he had pretty much violated all assumptions that you needed to validate before you could apply the theory). Similarly, I was working as part of a 50 M DARPA project at HRL, and again a Professor from CU demonstrated that one of the key goal of this project could not be achieved based on the theory of non-linear systems. Yet, our system had only a few discontinuities and was "linear-enough" otherwise to get the job done with linear system controls. So if you allow me, I am quite suspicious when people (PhDs or not) apply "theories" well outside their boundaries without validating (and understanding) the context in which the theories apply.
Roy's developed his theory by looking at the Web as a collection of "resources". IMHO, he has built his theory without considering the concept of "Resource Lifecycles" (for good reasons). He made an approximation, like every scientist does. Here is why, the lifecycle of a Web Page is very basic:
You can question whether "update" is a good thing to do for a Web page, but, whatever the lifecycle is, it is basic and most importantly, it is UNIFORM for all web pages. This uniformity is the reason for the Uniform Interface, there is nothing intrinsic to the "uniform interface" (I think Roy goes a bit overboard in his thesis with this constraint, he does not explain enough the origin of the constraint, he focuses too much on the benefit).
Here is a plausible general metamodel for resources, I think REST principles can be derived from this metamodel but it has the merit to surface an arbitrary resource lifecycle and an arbitrary inter-action interface.
A uniform interface is a complete myth when the lifecycle of a resource is different than the one of a web page. This is yet another case of a few people (not Roy) applying "theories" well beyond the context for which they were defined. It is already hard enough to come up with a useful theory (ask Newton) but thinking that there are extended areas of application that serendipitously appear here and there is, in general, pure fantaisy.
Sure enough the remoting patrol, jumped on this like a bear on a beehive, they saw yet another uniform communication interface they had not explored. Moreover this communication interface was backed with extremely scalable web servers activation mechanisms. But, yet again, they don't realize that the lack of "lifecycle" and "inter-actions" that made them fail to apply Distributed Computing communication principles to information systems construction, is going to fail them again in REST. As long as we will give people some mechanisms to CRUD the resources, we will face the same wall.
I don't think I am that lonely saying all this, but sure enough, it is not as sexy as telling people: look how successful the Web is, now imagine how successful you are going to be by implementing your information systems with the technologies that made the Web successful. Woa, can't beat that logic !
05/01/08 :: [SOA] WOAmiting [permalink]
The recent movement of "so called" analysts and experts around WOA has left quite a few people a bit sickened (Jeff, Todd, Miko...)
We have lived in the "transistor age" for the past 40 years, just CRUDing around.
SOA is about building integrated circuits and reusing these ICs when to construct solutions. It is no surprise that some people can't make the switch, so to speak, and will find every possible way to build their solutions from transistors. Yes, the ICs have been hard to come by, because the old mentality is hard to break (Some people will still use an OpAmp as a transistor). It is also true that the IC technologies are barely hitting the market with: SCA and BPEL for instance.
04/30/08 :: [REST] Who Said Uniform Interface Again? [permalink]
These days, Tim Bray courageously claims "REST, they say, is the way to go". I guess if "they" say, it must be true, as long as Tim repeats it.
Francois Leygues, a French member of the REST community, sent me a pointer to this blog post from Andrew Townley: "URI Opacity Revisited".
Andrew's findings are in line with mine when it comes to explaining how "uniform" an interface can be. Yes, when a human is in the loop, it works like magic, and...
The Big Problem
While all the above discussion about hypermedia applications is great stuff, the big problem with interesting hypermedia applications is that you need to understand what all the links do. With people, it’s easier, but with automated service consumers, it’s much harder.
"Ben oui", as we say sadly in French. You don't have to be rocket scientist to figure that out. I would not be surprised if many more people figure this out after playing 5 minutes with a REST framework trying to implement some server-to-server communication.
Let's take yet another example. I have a "Tennis Court" resource. This resource can be reserved or not (just to keep it simple). So there are two actions, reserve and free. The resource itself can be in two states: reserved and free. How do you implement a reservation system?
a) the CRUD way: the resource representation has an element called <free>, you change its value and you PUT the Tennis Court resource.
b) you POST something somewhere: that's really like a Web Service operation
c) you PUT a "reservation"/"termination" resource with the Tennis Court ID and your ID (at this point you can either assumes that the Tennis Court resource implementation looks up for the latest reservation or termination and returns the content of the <free> element accordingly, or the implementation invokes a) to avoid figuring out the state of the resource when it's queried again).
Solution a) strongly couples the consumer and the provider, b) is business as usual, c) you mean c) is really simpler than a web service operation? no at end of the day there is something called a "reserve" operation (you could also think of a reservation service), and this "reserve" operation has to fly from the consumer to the provider. When a user is in the loop, all this works automagically, and I explained many times why (Andrew agrees). When two software agents talk to each other, then they both have to agree on what a "reserve" operation is and use a "uniform interface" to exchange the intend.
Now, two words on the topic of "URI Opacity". Again, when humans are in the loop, meaningful URL make perfect sense. I don't think it is a good idea to rely at all on this mechanism in the case of server-to-server communication because a service implementation is usually involving some back-end system and it is likely that this backend system will want to take a look at this metadata too.
At the end of the day, programming is really about not being "sloppy", and REST, as a programming model, is the ultimate sloppy technology. Again, when applied to the architecture of the Web, REST is great, terrific, I can't say enough good things about it. When applied to SOA, its principles are terrible. Except for the concetp of "Resource", REST has absolutely nothing that is required for constructing a SOA: bi-directional interfaces, inter-actions, orchestration, coordination and assemblies. People like Tim Bray simply don't get it, be sure that they won't be around to help clean up the mess they are creating.
Who said again, that "I was the only one to say what I am saying and therefore I MUST be wrong"? I am simply appalled at the type of things some people in the REST community will do and claim to try to stop any reasonable discussion.
04/26/08 :: [Other] Where's your DVD Drive? [permalink]
This video caught my attention. So its the Spring of 2008, you are the CEO of 60+ billion dollar ISV and when someone shows you an Air Book, you ask
where's your DVD drive?
I am surprised he did not add, where's your modem?
04/26/08 :: [SOA] What's Cohesive in SOA? [permalink]
Some people tend to think that I enjoy creating such a controversies, but at the end of the day I don't. Anybody is of course free to say and do what they want, but we seem to have reached a point of evolution that seem to imply "as long as you can say or write something, there is always some truth to it".
SOA could very well be the ideal area to say anything you want. And boy do some people enjoy doing that. Of course, you could very well turn this argument to me, but I pride myself to keep the debate open, no mater what. My only concern is helping IT architects make their day to day decisions, whether they are strategic or tactical.
The short history of SOA is that people once thought 10 years ago to use Web technologies to exchange information between servers (that was a sensitive thing to try). It started with ebXML for B2B applications, then Microsoft and IBM drove it to focus on interoperability while some people threw BPM in the mix. We then realize we could do EAI a bit better with XML and SOAP than the clunky EAI frameworks of the 90s. Now we are reinventing a component model using SOA principles with SCA, while a new community is emerging calling all these effort "crap", and asking us to go back to the basics of the Web (and with it, going back the corniest programming model concepts).
Overall nothing really took shape over the last 10 years, sure there are successful products and technologies, but at the end of the day, you look back and you say Service Orientation did not emerge the way Object Orientation did. I have not lost hope, this one is hard, and there is also a lot more baggage (solid baggage) today than we had in the 70s.
So in this context it is no wonder why so called "experts" and "analysts" can paint an "ubuesque" vision of SOA and say anything they want while FUDing an entire industry one post at a time. I won't say it enough, SOA is unfamiliar and to the people that tell you they were doing SOA 20 years, just ask them what they used for an orchestration engine, how they were "assembling" components together and if their IDL supported "bi-directional" (let me keep the dash to really emphasize the bi) interfaces.
There are many approaches to agent-to-agent "communications" (I don't want to say "distributed computing" as I see the goal of DC to actually distribute computing resources and make them act as one). DC has been tremendously successful if you ask me, and huge progress is still made every day.
Now, you need a "uniform interface" to communicate for sure. As I said in an earlier post there are plenty to choose from:
- Send, Receive (MEST, WSE)
- Get, Put, Delete, Post (REST)
- Select, Insert, Update, Delete (SQL)
- Request/Response, Notification, Solicit/Response, One-Way (WSDL)
- Publish/Subscribe (JMS and the like)
- Prepare/Commit/Rollback
- OAGIS Verbs
- ...
You can implement agent-to-agent communication with any of them. Each have some merit, I don't really see one better than another, just trade-offs. For instance, Udi does great things with the Publish/Subscribe uniform interface (which is a bit off the beaten paths of SOA). I could even argue that it does not really matter which one you pick, at the end of the day, this collection of interfaces shows that, somehow, at some point, you will need all the other semantics. By picking one you simply agree to layer the semantics of the others on top of the one you picked. Incidentally, you will notice that some of these uniform interface may require some form of coordination, so any "so-called" SOA Reference Model that does not surface "coordinators" are incomplete and require major redesign.
The question, the very question, is do you really have a SOA once you picked one? Is SOA about taking the current programming model and arbitrarily "remote" something using one of these uniform interfaces? Anybody would venture to say Yes? I don't think so when the question is asked in these terms. So what's more to the uniform interface?
Of course, first and foremost, SOA is about creating autonomous software agent that react to a request. By definition an "autonomous" software agent of that sort exhibits a high degree of cohesion. You can't invoke anything else from a Service than what it's interface exposes. So I hope Jim, this is not what you were trying to teach us.
The problem though is this stubborn service interface, that can't be hacked in any way, and if you have to change it, you have to version it, this is yucky. In reality, let's face it, people are used to by-pass cohesive principles all the time. They can never make up their mind about what the best way to fetch or update some data elements. There is always a scenario that requires a new query to be exposed or a new update gram to be built. So over time, low level services interfaces tend to become messy as modifications result in an expanded interface. REST found the solution to this mess, "don't use a service interface", just pretend it is not there, and keep adding URIs (which are really interface point and not resources) to deal with these constant modifications. Ta..da... problem solved? not quite. There are two problems with that: a) in a connected world, there are more than one consumers, nobody owns both ends any more. What happens if you break "the other end"? b) the other problem is precisely that the programming model should not let you CRUD from the consumer side. As long as you will CRUD, you will strongly couple the "consumer" and the "provider". This is not because you chose a uniform interface to happily continue CRUDing remotely that you are building a Service Oriented Architecture. You can tell anyone this is not true, ultimately, this is what people will realize. It does not matter if you apply good cohesion design principle as you are CRUDing, you will fail to build reusable services, not because the service can't be reused at a given point in time, but because you will break existing consumers as you try to accommodate the needs of new consumers. Above the service interface, Cohesion is Coercion in a SOA.
The goal of a Service Oriented Architecture is precisely about using new technologies a) that support forward compatibility (XML, XSD, WSDL) and b) creating a new factoring of your business logic (SCA, BPEL) that is not CRUD-oriented to avoid this terrible coupling between the consumer and the provider. This is why people were not doing SOA 20 years ago, nobody was doing SOA 20 years ago. Of course ultimately you hit the systems of record, so some CRUD operation is going to be performed. Of course cohesive design principles are going to be applied within a service implementation. But above the service interface I would argue that nothing is cohesive. You actually want to achieve quite the opposite with loose coupling. You want to be able to assemble your service with as many consumers as possible. You may even want the consumer to tell you a few things about how to provide the service by injecting some decision rules or a sub-service provider..., again this is loose coupling. At that level none of the cohesion principles apply. I would even argue that technically, one of the major advances of SOA , (again at the technical architecture level), is that I can now build systems in ways that let me scale, secure, fail-over,... different parts of the systems differently, against all principles of cohesion. Hence, the principles of good cohesion are of no help when it comes to design the service interface.
Now, Jim, have you ever asked how cohesion can apply to an Enterprise Information Model? Information is relational, has always been, will always be. How could you apply any cohesion principles to a data model that is entirely related? There are none that you can apply. And frankly this is also a challenge in SOA. You can't easily do join across service interfaces (REST or WSDL for that matter). It really requires that you factor your data structures in two tiers: an information business entity tier and the parts that constitute a given information entity. If you have a purchase order service and a customer service, how do you pull customer information as you fetch a Purchase Order? (Say, the telephone number of the customer or the contact person, that you would always want to be the latest information available). Now, let's say you want to use SalesForce.com as you "Customer Service" and "NetSuite" as your Purchase Order Service. You will quickly conclude that pretty much every "Resource Centric" service is "related" to any other service and cohesive boundaries are really hard to define.
If you look at this process (click on the picture to zoom in).
(this figure is coming from Slide 4 of Ross Altman's presentation on composite applications)
There is nothing, absolutely nothing, cohesive about it. You want to be able to create value across cohesive boundaries. Services like "credit score", "blue book value", "appraisal", "payment" can be reused in many different context: home, car, boat, ... loans. Business Information like "Loan Application", "Loan", "Customer", again brought into the process assembly in a lot more opportunistic manner than a cohesive manner. This is where concept where agility, business alignment, reuse... come to play. You are not "programming" in a service oriented architecture, you are "assembling", you are "compositing", you are "orchestrating".
Cohesion offers design concepts antagonist to Service Oriented Architecture and only lives within the service implementation. I understand why, if you CRUD all day with "services", you want (absolutely) to apply cohesion principles like people have been doing it for 40 years, but don't tell me you are doing SOA.
SOA is about the emergence of an Inter-Action Oriented, Asynchronous Peer-to-Peer programming model. Sure the old programming model, the CRUD-oriented synchronous client/server one) is still going to be used "inside" a service implementation, down below, but SOA is new, it is unfamiliar, it is based on technologies that did not exist 20 years ago, all the good software design principles that were discovered back then don't generally apply. Again, I am absolutely not surprised that you or Steve think in these terms, this is precisely the problem. Once you will unlearn some of the great things of the past, and come to SOA with and open mind, you may, understand how to use WSDL, BPEL or SCA.
So yes, I am quite mad at the "cheap" and sometimes incoherent
arguments that a small and cohesive community of "experts" and
"analysts" are spreading to keep the "old regime" in place.
I was just as mad at Assaf Arkin, Howard Smith and Frank Leyman who
argued for years that BPEL/BPML were a "Business Process" thing, when
most of the semantics to support business processes were missing. This
handful of people set the BPM industry 5-7 years back. Can you imagine
the magnitude of the loss? The same thing is happening here today, for
the exact same reasons: closed minds, cheap arguments over CIRCA 2001
pictures of SOA, a small group of
buddies who point at each other to gain credibility... the recipe is
well known. At least,
they won't be able to tell in the future that they were "unaware" of
possible alternatives to their best interpretation of what a Service
Oriented Architecture is.
04/24/08 :: [SOA] Is SOA ABOUT Building DISTRIBTUTED SYSTEMS? [permalink]
Stefan commented that "respected members of the distributed computing community for a long time" had to be right about SOA and I had to be wrong because I was a lonely voice.
First, I am not alone, we are simply not as "cohesive" as certain groups. Even saying "we" sounds weird to me. This is not about we, us, them. This is about SOA.
What I am often arguing about is well supported by commercial products and open source frameworks such as Tuscany and Fabric3. There is also this researcher and her group at IBM Zurich, Ksenia Ryndina,, who are actively researching the kind of programming model I advocate for. There is also an entire open source methodology that is based on these principles: http://www.praxeme.org. I may well be the most vocal of all these people, but I am certainly not alone… on this side of the discussion. I am certain most of them have given up a long time ago arguing with the other side.
Amusingly, the large distributed computing power houses (IBM, Sun, Oracle/BEA, and even SAP) have all a composite application strategy, and I don’t think I would be in the position to teach them anything on the topic. At best, I could point out that I have been saying this kind of thing for a long time.
Second, yes, I do have a track record of being a lonely voice and being right. I do remember a time in early 2000s when "Business Process" and pi-calculus was the craze. Even people as famous as Frank Layman from IBM research (whom I met in 2003) were spreading their wisdom about BPEL being a "Business Process Execution Language", when I was the only one, not anyone else, against Frank, against Assaf, against Howard Smith, against John Pike to say that no, BPEL was a programming language and it had nothing to do with "business process". BPEL was and is a Service implementation language that is a key component of an Inter-Action Oriented Asynchronous Peer-to-Peer programming model. Howard Smith and John Spike has been converted. We don't hear Frank on the topic anymore. Jim Webber in a recent presentation clearly positioned BPEL as a programming language. I don't think there is many people to say otherwise. I was also the one, the only one that pi-calculus was not enough, though a solid theoretical foundation, and again these days, there are hardly anybody to argue that leveling this SIL at the level of pi-calculus is not a sensitive thing to do. You can create a pi-calculus VM if you want, but the semantics are incomplete to make it an efficient SIL.
Even David Chappell had told me once (at the 2004 WCF SDR) that I was wasting my career as I was explaining to him the difference between orchestration and choreography and the place of orchestration in SOA (as a SIL). He said, "if Microsoft says this is what it is", why do you fight it? I fight, because of remarks like this, or remarks like yours. If our industry was less full of so-called "experts" we would make progress a lot faster, if you know what I mean. Yes, I have come very close to "experts" and I can safely tell anyone that "experts" are just like everyone else.
So Stefan, if you allow me, if you want to "shut me up" I would request that you use a different argument (this one was really cheap) and maybe, just maybe, we could use technical arguments to reach a conclusion.
Third, and that's the most interesting point, we could ask the question whether SOA is really about Distributed Systems, whether an "expert" in distributed systems is coming to SOA with the right mindset? Do these "experts" really ask themselves why distribution at the programming level has been a failure when applied to information systems (and a big success when applied to other things)? why CORBA failed to deliver a sound architectural foundation? When you read Steve Vinoski's papers you understand what his vision of distributed system is. He wrote recently an excellent one on REST and ERLANG, and what you see is that these respected distributed computing experts are really thinking in terms of activation. He loves ERLANG because it has a fantastic, light weight, very scalable activation model (plus a cool pattern matching capability that fits REST like a glove). This is what he dreamed to build one day. Is this useful, yes, very much so. Is this SOA? not quite.
I wrote many times that SOA is not just about distributed systems, but it is a lot more about "connected" systems. I had great hopes when Don Box started to talk along these lines in 2003, at the LA PDC. He sure delivered a great distributed communication framework, but as a "distributed expert", I think he missed the mark on "connected systems". Connected Systems are about reuse a lot more than they are about activation and "distribution". Reuse is about "connecting" arbitrary pieces of code together, not in a serendipitous fashion, but in a deliberate fashion. Connected systems are about inter-actions, assemblies and forward compatibility, semantically accessible data structures... Don Box himself had said once that Web Services technologies were the first technology that did not require code generation systematically, and that change was profound, this quote is still on the first page of my web site, and of course, the first thing he did was to force code generation in every aspect of WCF, negating the very innovation of Web Service technologies (and XML). Right about the same time, WS-CAF came along. This is truly the best standard work I have ever seen. Jim Webber (and Mark Little) was a co-author of that spec. I always wondered if WS-CAF was consciously designed to participate in this IAOAP2P programming model. I know today that it was a pure coincidence, even the name CAF was right on, but no, this was a coincidence.
At the end of the day, SOA is about replacing a CRUD-oriented Synchronous Client/Server programming model that has been in use for over 40 years as the core foundation of information systems by an Inter-Action oriented Asynchronous Peer-to-Peer programming model. There is no other way around it, COSC/S cannot raise to the challenge of a connected world, COSC/S is responsible for the silos that we have built for the last 40 years. As long as the "experts" will merely XMLize a COSC/S, we will keep asking why we can't get benefits with SOA. The answer is simply because we are NOT doing SOA.
So if you allow me, Stefan, I'll continue patiently hammering these
concepts and the difference between distributed and connected (i.e.
composite), until this is a well establish concept, even if you think
that I am too lonely in this discussion to deserve consideration
by the intelligentsia.
04/20/08 :: [SOA] Cohesive Response [permalink]
Well I got a lot of people upset: Stefan, Savas, Jim kept the legendary British cool.
Again, I stand by what I am saying, Jim's position on:
- cohesion in SOA (beyond the trivial conclusion that any autonomous asset is by default exhibiting a high degree of cohesion)
- process-to-service relationship
- MVC as a design pattern for SOA/BPM
- Business people becoming IT architects (last but not least by far)
is more than questionable.
Savas, about my communication style, I found that it works -indirectly-: these days, I don't read many more people in the REST community bashing WS-*, Tim Bray's BS on "the end of SOA", or Steve's serendipitous crap. Actually people eventually strengthen their arguments or give them up (they don't want to look stupid). Everybody saves time: you, the reader, me. Now, yes, it hurts me, you are probably not the first one to think that I am rude and unprofessional, but that's ok, if this is the price to pay, this is a very tiny price compared to the end result.
The reality Savas, Stefan, Jim is that SOA is still raising a lot of questions. We can circle around for years if you want to. I offer an alternative: I just ask anybody to answer just two very simple questions:
Have you ever considered factoring the business logic along the lines of a Resource Lifecycle? How does this fit with whatever your recommendations are?
Answering these two questions alone would end the REST/WOA vs SOA, MEST vs WSDL, BPEL vs BPMN (if you want to) and surface how critical representation-friendly data structures, bi-directional interface definitions, orchestration, assemblies are to designing a Service Oriented Architecture (note that I use concepts not technologies, pick your favorite technology).
I would argue that all of the people that are so prompt to be offended by my posts will never answer these questions, they will never drive the discussion to answer these two questions.
An Inter-Action oriented, Asynchronous, Peer-to-Peer programming is here to replace the CRUD-Oriented, Synchronous Client/Server programming model. This is only a matter of time.
The irony here is that MEST is a lot closer to an IAOAP2P than a COSC/S, but Jim does not see it.
04/20/08 :: [SOA] Loose Coupling and Cohesion [permalink]
Quite hilarious (and symptomatic), I advise Jim to catch up with modern SOA concepts, technologies and principles and he advises me to read antiquated software engineering books, or articles from Steve Vinoski.
Jim, if all you wanted to tell us is that autonomy requires some degree of cohesion, well you did not need a blog post for that. I was trying to interpret your blog post in the context of an assembly of services performing a unit of work and wondering how cohesive could an assembly of services be?
Again, if you take a modern view of SOA (say post 2005) you really wonder how an assembly (a module for me) can exhibit some degree of cohesion. Even a service within an assembly (a component in SCA terms) is likely to fail "cohesion" litmus tests.
Let's review Steve Vinoski's article.
Functional cohesion occurs when a module does only one thing: take a look at Resource Lifecycle Services, what does "one thing" means. Is your recommendation really that services do only one thing at a time?
Communication Cohesion is when a module carries out multiple operations based on the same input or output data: ok. why not, some services might be designed that way but this is not the case for the vast majority of them. I would argue that this is not even very important in SOA: XML helps you not having the need to establish Communication Cohesion.
Sequential Cohesion is when a module carries out several tasks and the input of one tasks feed to the other. What's the big deal about that one? This is true in a pre-XML civilization. With XML you don't care as much to achieve this level of cohesion.
Logical Cohesion is a condition in which a module's activities are grouped together because they appear to be able to share common implementations: that's a bit closer to SOA but again, have you ever considered that when a Service Interface is implemented by a BPEL definition, all the operations are tied together in the same implementation? unlike OO you don't have necessarily a one-to-one relationship between operation and implementation.
Jim in case you have not noticed, SOA is unfamiliar, it is time to rewrite the books. Concepts like XML extensibility, semantic access (XPath), or forward compatibility precisely alleviate the need to be cohesive. Whatever the principles you learned in school, they most likely don't apply any longer in an Inter-Action Oriented, Asynchronous Peer-to-Peer programming model.
So for me cohesion is not a design principle you can apply at the assembly level and even at the operation level. XML is the technology that allowed us to break from cohesive software engineering (CSE). CSE has been developed for bronze age programming languages. We are in 2008 today, things have evolved a tiny bit.
I stand by what I have said, a cohesion requirement creates absolutely unnecessary constraints on the design of service interfaces and implementations that actually reduce the degree of reuse of a given service and its capacity to participate in different assemblies. Maybe it is time to become familiar with modern loose coupling concepts that include: bi-directional interfaces, assemblies, orchestration languages, extensible and semantically accessible data structures. Ancient programming techniques have been designed precisely because you did not have these concepts within your programming model.
Every thing you say in your post, from applying cohesion principles to SOA, "a process is a service" or "look at MVC if you don't get it", to "Business people should become IT architects" does not make any sense and are likely to fail anyone trying to build a Service Oriented Architecture.
04/20/08 :: [SOA] Loose Coupling [permalink]
A couple posts picked my interest today. "Does WOA bring anything new to SOA?" by Mike Meehan. Well, I have already expressed what I thought about WOA and REST so I won't do a repeat here. What I found surprising is that there is actually quite a few people agreeing with some of what I was saying.
If users perceive WOA to be outside the principles of SOA, it could prove an excellent vehicle for building Web-based stovepipes.
The article concludes:
[people] expressed an unequivocal desire to have no more than one something-oriented architecture in their lives
Well someone who is trying single-handedly to topple SOA with a "Something Else Oriented Architecture" is Jim Webber -the MEST guy. Mark Little points out this post by Jim: "Anemic Service Model". I am not sure why. I found his post being completely erroneous. I may be missing something, but Jim is looking at the relationship between Cohesion and Loose Coupling.
He defines Cohesion as follows:
"[Cohesion] measures of the extent to which related aspects of a system are kept together in the same module, and unrelated aspects are kept out."
For me cohesion sounds like a good engineering principle that looks a lot like "Dependency Structure Matrix". In fact Jim concludes:
In fact good software tends to be both loosely coupled and highly cohesive.
This is the part that I find completely bogus (again, unless I am missing something). The goal of loose coupling is precisely to mitigate cohesion as a good engineering principle. You can't be cohesive in a connected system. You can't be both cohesive and loosely coupled. Even in English, associating the two sounds really bad.
The goal of loose coupling is to make to pieces of code work together even though they may have been written at a different time, using different technologies, with a different security model, ...
SOA is about going away from cohesive systems to enable a wider range of reuse scenarios. Nested cohesive systems do provide a "nested" (library-style) reuse model: the upper layers can reuse the lower layers. Unfortunately, it forces us to create systems in a way that is incompatible with the way information systems should be architected. Cohesion is the problem, loose coupling is the solution. Jim, do you really think that people are trying to "keep stuff together in the same module"?
How can you say that when a system is loosely coupled,
even minor changes ripple through multiple components or systems
Have you considered this definition for "Loose coupling"?
The funny thing is that Jim thinks that other people think SOA is best built by "nesting" service invocation in a "Dependency Structure Matrix" style. Jim, I am sorry to say, but this view is what people where talking about in 2001-2002. You need a refresh. The very picture that you think people have of SOA is highly-cohesive.
Then you continue and recommend that people
build out a service that implements [a] process
Jim, what are you talking about a service implements an entire process? for sure that's going to be very cohesive. Now, how can you say that this is an instance of "MVC"? Precisely the fundamental problem with MVC is that is does not have natural boundaries for processes and human tasks. People using MVC keep creating "representations" (i.e. Model), views and controllers that cannot be associated easily to processes and human tasks. Your recommendations and analyses are completely bogus. I would really recommend that you spend more time understanding SOA and less time trying to create specs that are absolutely counter to SOA principles.
You know:
It might sound like a pretty dumb idea that my business stakeholders to become (inadvertent) IT architects.
is one of the dumbest recommendation you can make. You are no George Lucas, there is only one thing anemic about your post, it is your understanding of SOA.
04/20/08 :: [Other] Winter Fatigue? [permalink]
Yes, it's April 20th and yes it has been snowing yesterday, today and it will probably snow tomorrow too. Want some proof? I took this picture of my neighborhood this morning !
04/17/08 :: [SOA] SOA Fatigue? [permalink]
John claims that :
SOA is no longer a mysterious concept that we need conferences, presentations, books and articles to clarify for us
I rarely disagree with John and I can't say that I completely disagree with him on this one. I would argue however that what we need is a lot less of this and a lot more discussions on what we do when "we roll up our sleeves and get to work". My main concern is that when John says:
We get it already - stop debating what it is and go build something already.
I am not sure this is true, I am not sure people get it and know what to do despite the hundreds of conferences that claimed to talk about SOA. I am certain that John gets it, I agree that there are some products and technologies that are mature enough but most people use SOA within the boundaries of the current programming model.
I won't say it enough, over the last 40 years the software industry has developed a "CRUD Oriented Synchronous Client/Server" programming model. Using SOA technologies within this programming model will lead you nowhere.
SOA represents the emergence of an alternative programming model which is "Inter-Action oriented, Asynchronous and Peer-to-Peer".
I don't know many people who say what I am saying and sending people CRUDing around with XML, SOAP and WSDL is likely to create a huge disillusion.
I mean, John, come on, we don't even have a decent SOA Reference Model: Duane Nickull probably delivered one of the worst possible SOA RM , only second to the one developed by the W3C TAG in early 2000.
03/30/08 :: [Other] Interesting contrast on TV this Week [permalink]
I don't watch TV that much except for football. For everything else, I have a little home built digital recorder that occasionally records interesting things.
This week there was an interesting contrast. First there was this show from the University of Washington on "Shared Prosperity in an Age of Global Warming". I can't say enough how proud I am to live in the Northwest. It's almost like living in a different country and nobody captures best the Northwest spirit than this ad from a local insurance company showing all kinds of Northwest characters and claiming "We are a lot like you, ... a little different". I would argue a lot different.
In this presentation Ron Sims, a local county level politician presents his vision about tying together global warming, the global and local economy and poverty. This guy makes you really proud to live here, just like our governor, Christine Gregoire, and so many other people. It looks to me that the US always had forward thinking areas like the greater Chicago area (up to Ohio) in the late 1800s which had incredible innovations, not just in Technology but also economically, architecturally (F.L. Wright) ... or California in the second half of the 20th century. Will the Northwest walk in these footsteps? It is probably too early to tell.
The bottom line is that I feel really privileged to live here. I don't want to comment on Ron's presentation, I think it should be placed in perspective with the current presidential campaign. I'll let you watch it since it is now available on UWTV.org.
The other show that I watched this week-end was Frontline's "Bush's War", aired on PBS, earlier this week. This show made me sick at two levels (I am constantly sickened by how many people die each day). First it reminded me of 2002-2003 where I could not stand to watch TV and listen to the radio as America was chanting on the path to war. It also made me sick because one of the key turning point the show claims is right after Colin Powell's talk at the UN when the French Prime Minister reacted in Powell's back. They claim that Powell was the only one that could have avoided war and our prime minister killed his credibility with his little anti-war talk. But at the end of the day, it is Powell who delivered that talk to the UN it is he who delivered the case for war (even though we now know that Cheney was the one that wrote that talk, Powell simply delivered it).
Ironically, America is probably the country where there are the most hours of news per day: the networks deliver 3 hours of news in the evening (5 to 7pm) and another 3 hours earlier in the day, yet the quality of what they deliver is abysmal. Worse, they not only make deadly mistakes, but when they venture on the commentary side, they mislead their viewers. Today, the press has completely given up on its role, journalists are no different than a reality show producer, eager to collect ad revenue.
My question is very simple, how can the American press teach anyone a lesson today? The French (and Europe at large) went above and beyond to avoid this war, this is the reality. Where were they then, even Cheney was able to corroborate the false uranium accusation with an article from the NY Times? Where are they today? they want the American people to believe that the French are responsible and sweep everything else under the rug (hundred's of thousands of dead people).
I also found interesting to look back at Tony Blair (I don't want to say the British people or the British Government because they voiced just as hard as the French how terrible a mistake that would be). I'd be really interested to know today what George Bush thinks of himself? Sure his family, and to a certain extend, Britain, have benefited from the strong hand he and Tony put on the Iraki Oil Faucet. But how does he sees himself with respect to his country? What does he think as a Christian and pro-life supporter about all these people dead? What does his wife and daughters think about him? About what he has done to his own country? What Keneth Starr thinks about all this? Why isn't he so prone to impeach Bush and so many other officials for what they have done to their country?
But the saddest thing of all is that the press has learned nothing. It continues to go with the wind, it continues to focus on the wrong things with the wrong attitude. If this country wants to rebuilt itself, it has to rebuilt its press from the ground up. This is no easy task, but the press is one of the most important pillars of democracy, and ultimately freedom.
02/28/08 :: [SOA] WOA: The future of SOA [permalink]
There is no shortage of thought leadership when it comes to twisting SOA in every possible way. Dion Hinchcliffe has written for quite some time a series of (generally good) articles on bringing Web 2.0 and SOA together. In his latest article last week, he sketched for us the future of SOA.
He starts by a free for all statement:
Even though Service-Oriented Architecture (SOA) initiatives around the world have the right goals, most efforts have fallen profoundly short of our desired levels of integration and improved business agility.
It hard to know what he means by "profoundly", or "our desired". Being down in the weeds of SOA, working for a large financial institution, I can see first hand how wrong Dion's statement can be. Sure SOA is complex, no doubt about that. People that look at SOA as a lipstick on a pig might very well end up where Dion seems to be, but I would argue that has nothing to do with SOA, precisely, it shows that perpetuating the "old ways" of building information system is wrong, and nothing can fix it, "interoperability" is of no help at all. The truth and the matter is that without a significant architectural change an organization will remain there for ever. Blaming SOA for it, is simply a new way to perpetuate a systemic problem.
Dion sees SOA being reshaped by the Web 2.0 era. He continues:
However, the news isn't all bad, the fascinating story is that there is a place today where the deep integration of our systems and information on a large scale has largely been solved and is a foregone conclusion in most cases. And that place is at the leading-edge of the World-Wide Web, sometimes referred to as Web 2.0.
He sees Mashups leading the charge:
At the leading end of this is the Web mashups story with enterprise mashups being one of the major improvements to the IT landscape that WOA is heralding.
Of course, "WOA puts the focus on Web Resources not Services" (LoL). His assessment could not be clearer:
The basic tenets above paints a picture of a galaxy of nearly infinite granular information resources integrated into a deeply interconnected set of dynamic connections that can be processed individually for a given task, in part (for integrated applications), or as a whole (such as enabling a comprehensive directory or search engine of all data and metadata.) In other words, the Web model provides a single, open, and unified information architecture that is consistent, easily consumed, extremely scalable, securable, very reusable, resilient, and highly federated.
Sometimes I wonder if people read what they write. Why didn't I think of that before? Dion even goes as far as claiming that there is a "timeless" way of building software. I know countless useless ways of building software, but so far, I never found a timeless way.
He concludes:
I should close by emphasizing that I enjoy and use traditional SOA technologies like SOAP, WSDL, and XSD frequently [really?]. But as more and more of the consumer Web moves to a more Web-oriented model, the evidence continues to mount that approaches based on WOA are easier to implement, scale better, and result in much greater uptake and usage scenarios.
And of course:
Traditional SOA is facing a crises of identity at this point [poor SOA, let's give you a big hug], particularly given fairly lackluster results for most, and WOA may just be the prescription we need to make SOA deliver the robust outcomes that we were formerly expecting of it.
I would not hold my breath. The fact and the matter is that real-world composition technologies are emerging, with a real-world composition model. Everyone should understand that pushing composition at the presentation layer simply does not work. You also need process and information composition capabilities. Sure is it a lot less fancy and a lot more technical, but WOA isn't going to cut it, by a large margin. The funny sad thing is that people will never question how they build software.
So REST? WOA? SOA? ROA? COA? How about Duh-OA?
I estimate that 99.999 % of Web Applications are not built in a RESTful/WOA way, but of course it is a terrible sin to build a Web Service with something else than REST/WOA.