|« How to Run a Chorus.js Orchestration on a Raspberry Pi?||Orchestrating APIs with Chorus.js »|
There is an area in Service and API engineering that has never been really solved: how do you associate the request and response message formats to their underlying information model? We are not talking about a physical data model, the underlying information model is often known as the “Common Information Model” or “Logical Information Model” and provides a coherent structure to all the message types of a service/API or set of services/APIs.
Developers design message formats and interactions by hand without a deliberate traceability to any kind of information model.
People usually try to factor of their XML Schemas into an information model (complex types) and a message model (derived from these types). However these initiatives end up in failure as the information model and the message model need to evolve independently but the structure XML Schemas do not allow for that to happen. When you use XML Schema imports and includes, a change in the information model definition would immediately be reflected in all message definitions that depend on it, making it impossible to deploy that hierarchy at runtime. So you would have to create individual copies of all the versions of your XML Schema based information model. Eventually people realize that this approach amounts to creating self-standing message models. However, in doing so, message types quickly become misaligned with one another.
Due to the same broken design, XML Schema cannot solve another important problem. There is an asymmetry between queries and commands. Queries are generally implemented using a “Query by example (QBE)” mechanism, and require a flexible data structure where most if not all the data elements are optional. On the other hand commands require a rigid format to help validate the input data prior to executing the command. Yes, both query and command types are derived from the Common Information Model. Developers typically result in using the common denominator and making all elements of a type optional rather than having types for queries and other (albeit similar) types for commands.
This is not specific to XML Schema, any Schema language that would be built following the same principles would feature the same problems. For instance, the Swagger’s Pet Shop example has all entities are defined with optional parameter:
Figure 1. Swagger's Order Definition (note everything is optional)
Since when a purchase order is a valid structure when it does not have an id? an item id? a quantity? ... Is that truly “the model”? The reason for that is that this particular structure fits all usage scenarios: create (where the id is not known yet, query by example, or even partial update a particular property (e.g. quantity).
Chorus.js solves that problem directly at the language level by introducing a new concept, projections, which connects with precision message definitions to the Common Information Model. This approach is based on an an article published on InfoQ in 2009: “Message Type Architecture”.
With Cord you can define your Common Information Model (in this case, the Pet Order, Figure 2) and then define message projections for various scenarios (create, query, update...). For instance the CreateOrderMessage expresses that to create an order, you must specify a petId, a quantity, a status, but the shipDate is optional, as it may only be known (and updated) later, once you have made shipping arrangements. Note that the order id is now optional, and that’s all we had to specify, the “deltas”, between the base type and the message type.
Figure 2. Cord's Order definiton (note the Order's projections to create and query orders)
Similarly, the query by example message definition specifies that we can query by petId, by status or both, by id also, but we cannot query by quantity. The quantity property has been “subtracted” from the order base type .
We can now create a service definition. Let’s start with the operation definitions:
Figure 3. Cord's Service definition
The message type definitions use the base entity and the projection, for proper validation. Here is the WSDL file that was generated by Chorus.
As usual, Chorus also generated a PlantUML class diagram.
Figure 4. Class Diagram of the PetShop example, as generated by Chorus.js
Thank you Jouko! appreciate the feedback.
Great work, JJ. I wish I had known about this two weeks ago when I submitted the final draft of “Dependency-Oriented Thinking vol 1″ for publication.
This is an example of how to design an Interface Data Model in line with a Data Dictionary, with some structures to facilitate a DRY way to define each operation.
I will get a reference to it into the next version of my book, when I send them a list of corrections.
A suggestion: How can we support a hierarchy of types with this model? One of the severe problems SOA shops face is the needless pressure to create new versions of operations and services, for entirely extraneous reasons.
If there is a way to simultaneously specify interfaces at a more abstract level (e.g., InvolvedParty in IFW), and implementations to be negotiated to be at more concrete levels (SavingsAccount, CreditCardAccount), then an ESB can not only do a sanity-check validation of a request against the abstract schema, it can also make the decision on which concrete implementation to route to.
It will be an example of data level tools and technology level tools working well together. I have illustrated this as the last example in the book. That used XML schema with subtypes and substitution groups.
Thank you Ganesh.
At the moment I don’t support an “extends” concept for the Entity, but will add it shortly. It was important to get the projection concept working and demonstrated. I also support the concept of “version” of an entity, so I offer traceability of the history of changes to an entity and a particular message formats may reference an older version of the entity until it is updated itself.
I am a bit reluctant go to abstract service definitions. David Webber has done some great work at Oracle on CAM (content assembly mechanism), but I am happy to change my mind. Personally I prefer the other way around, concrete interfaces in the ESB and a general/abstract implementation in the service container.
If you have a more concrete example, I’d be happy to look at it.
You said, ” However these initiatives end up in failure as the information model and the message model need to evolve independently but the structure XML Schemas do not allow for that to happen. When you use XML Schema imports and includes, a change in the information model definition would immediately be reflected in all message definitions that depend on it, making it impossible to deploy that hierarchy at runtime.”
This is true but you can master a bit more smart usage of the XML Schema and its imports, as I did in https://soa.sys-con.com/node/523434, and avoid any impact on the already existing users. Actually, these existing users are offered a to choose whether they care about your change or prefer to go with what they had before. This is the service-oriented approach.
It seems to me that your proposal, in essence, is a replica of the publication I referred above, but published 6 years later.
>> is a replica of the publication I referred above, but published 6 years later
Not quite, I worked on this problem with that solution since 2007, and published this article in 2009 (about 12 months after yours) https://www.infoq.com/articles/message-type-architecture
Now, I am not sure how changes to the store:store type are hidden when you use that type in Listing 8?
There is nothing magic in what I suggest to do, the key is to not rely on import and create self standing artifacts. The only way I know in XML schema to hide a change is to not use import. I must admit I have trouble to parse the way you hide changes. I don’t see what is different when a namespace is introduced: an import is an import, the only way to hide the changes is to copy/paste the definition into a file that will be left alone. Could you elaborate?
The other non starter for me is that you are making all elements optional:>> First, all elements related to the business actions are listed in the construct and all of them are optional.
>> This means a service consumer isn’t obliged to specify any one of them if it’s not needed.
All teams I know that use XML Schema end up using that because XML Schema cannot deal with the variations between queries and commands, therefore everyone is using the common denominator between the two which is all elements optional. By making everything optional, you loose 50% of XML Schema’s value. Required fields are an important part of validation.