Carnets de bord

Robert Scoble thinks so.

The new Path? The one that won a Crunchie last night for great design? It’s not done in HTML 5.

This morning I saw something new coming soon from Storify. Not done in HTML 5. This morning I visited Foodspotting which just shipped hot new apps on iOS, Android, and Blackberry. Not done in HTML 5.

More and more I’m hearing that designers and developers are ignoring HTML 5, especially for the high end of apps.

After building Canappi, I can only wholeheartedly agree with Bob. HTML5 has certainly a place in the mobile landscape, if for nothing else to "mobilize" your web site, but when it comes to user experience, we don't see how it can compete with Native Frameworks at least in the years to come: For every successful mobile web app, someone will build a superior native experience that users will adopt. There is simply no room to compete.

I am currently building Canappi's jQuery Mobile code generator and from what I can tell, it is extremely limited compared to the native iOS and Android SDKs.

The fundamental reason for Web Apps to emerge in the late 90s and 2000s was the "client update" problem and to a lesser degree the number of platforms on which the app should run. In the 90s, assistants used to roam the buildings with CD-ROM (and Floppy disks) to update every single client.

Today, App Stores have completely solved the update client problem (for mobile and desktop apps), and running on multiple platforms really means running on iOS and Android (although the fragmentation of Android is a bit of a concern).

Web Apps have many fundamental flaws, from flowing layouts which simply don't work in small form factors, to cluncky off line experience, not to mention bandwidth and power consumption.

If you combine a horrible developer experience to a horrible user experience, (Mobile) Web Apps have simply no room to grow from here.

AT&T released last week at CES a new API platform that exposes a number of Network APIs:

• SMS/MMS
• Location
• Device Capability
• In-App Payment
• WAP Push
• ...

The APIs implements OAuth 2.0 and capture user consent wherever applicable. I think the in-app payment (that goes directly to thesubscriber's bill) is a game changer for mobile applications, in particular Web  based mobile apps.

The APIs come with an SDK and some code samples.

In order to use these APIs, you need to join the developer program and create a developer account.

You would then create an "application" which basically gives an application id, a shared secret and a short code (yes ! your own short code). These credentials are used to get an access token (See figure below).

If you use the Java SDK (there are others available: PHP, Ruby...), all you have to do is to insert these values in the conf/att-api.properties file and build the project with Eclipse or Ant directly.

Once it is built, you just start the server by executing run.sh and you deploy the Sencha client application in your web server. The API server aquires access tokens automatically.

Here is how I deployed my server:

SDK -> /var/www/att/server -> sh run.sh

Sample app -> /var/www/client -> open http://www.yourdomain.com/client/examples/sms

If you want to change the location of the application, just update the ResourceBase value in the com.sencha.jetty.EmbeddedServer class. In my case, it was set to  webapp.setResourceBase("../../client");

That's it ! And if you wonder, with your very own shortcode, you can also receive SMS too.

I have a few promo codes from AT&T that give you free access to the platform for one year.

I am truly amazed at the degree of innovation of our industry today. In a way, it's quite scary: Are we building tools just in search of a problem? are we really innovating or is it just another "hyped-stand still cycle" ? In the midst of massive scalability improvements and seamless operational environments it is easy to lose track of the semantics view.

I built a "Composite Application Framework" at Attachmate between 2003 and 2005. Even though our team did get to the 1.0 release. The product was shelved before it could get to its first customers when Attachmate was acquired. A composite application is an application which is capable of interacting with systems of record that are beyond its control (typically 3rd party Web APIs, as we call them these days). As we architected the framework, we were faced to a big philosophical question: should we group these Web APIs (they were known as services and operations back then) under a facade that expose a query language interface or should we rather "bind" these APIs to our UI and develop orchestration-based mashup logic in the API consumer (middle-tier or client). I was softly voting for the former, while my team prefered the later. I didn't push back because I knew that implementating a query engine would delay the project and I didn't have a strong rationale for it. It was just cool. At the time, we had done some experiments with a couple virtual database technologies which allowed an in-memory SQL engine to connect to a number of different databases (not APIs) and make them appear as a single database from the client perspective. These product were interesting but they never really got traction and their API bindings were nascent to say the least.

Hence, I have been quite intrigued by Subbu's ql.io project. It is still incomplete since it does not handle updates yet, but it is already quite mature to look like what I had in mind then. I don't want to make too many conclusions until I use it, or other people provide some feedback. When I designed Canappi, I decided to take a binding approach, rather than building a SQL-like facade. So far I have been quite happy with that choice mainly because composite applications rarely need the full power of a query language. The Views map reasonably well to the model (unless it is the other way around ...) and applications follow a "navigational" pattern which is generally well supported by Web APIs. If you need to create some reports, of course, the answer would be quite different.

As REST has forced most of us to CRUD our way to the data, I think that our industry has reached a point where we need to answer objectively: what is different about SQL, NoSQL and (Web) APIs? We have built a "composite world", great, I don't think we will be able to make much progess without establishing a clear articulation between the way we store, relate and access information.

In his introduction to ql.io, Subbu points to a very interesting paper from Erik Meijer and Gavin Bierman, in which they argue: "Contrary to popular belief, SQL and noSQL are really just two sides of the same coin".

I like their analysis, but I am a semantic guy, I really like to see at the semantic level what's new, so I created a simple metamodel:

On the left end side, you have the traditional RDBMSs / SQL model on the right end side the new "No SQL" model (both key-value pairs and document oriented). The color coding is used to (roughly) map concepts from one world to the other. I have adopted a JSON metamodel for the structure of the values of NoSQL databases.

Erik and Gavin pointed out, the key deifference is that in the NoSQL world, an identity is generally a "key":

In the object-graph model, the identity of objects is intensional—that is, object identity is not part of the values themselves but determined by their keys in the store. In the relational model, object identity is extensional—that is, object identity is part of the value itself, in the form of a primary key.

It is of course true, but does it really matter? Did we really create new semantics by making an identity a subclass of a key? (incidentally, it would be very interesting to ask the opposite question, what happens when a "key" IS-An "identity" within a given scope (row, collection, database)?) So, my answer is no, claiming that an identity IS-A key does not change the semantics of what an identity is, regardless of the position of an identity in a result set or in the store. I have argued many times that this was true 8000 years ago when man invented writing and hence data. I have also argued that for instance, in the absence of a reasonable numbering system, ancient writers were left to techniques such as acrostic peotry to enable random access and even sorting.

The authors also argue that another key difference is the "open" data structure of NoSQL databases, but is this openess inherent to NoSQL or, is it inherent to the design of storage engine (and the query language designed to harness it)? First, a "link" is a link semantically speaking, it relates two pieces of information using one or more identity. Nothing would have prevented us to create a "unique id" concept in RDBMSs and allow for querying at the database level (with a statement like SELECT * WHERE ID = '123'). As a mater of fact, if you look at MongoDB which is not a key-value pair store per se but a document store, its data structure includes the concept of collections (~Tables) and the identity of a document is stored separately from the document itself. Second, all these data structure concepts are simply man made, for good reasons of course, but they do not introduce or uncover any new semantics.These "open data structures" simply provide a better opportunity to physically associate data that is often queried together. For instance a purchase order has line items or a prescription has a number of treatments, but that does not remove the need to uniquely identify all individual line items or treatments to associate additional details or independently reference these pieces of information.

Hence, the only semantics that matters in a data store are identity and link, which are common to all data stores, even the ones man used 8000 years ago when writing was invented. Incidentally, we can also notice that would-be data centric technologies like JSON, XML or even ATOM do not reflect correctly the importance and semantics of "links". As a matter of fact, the Web confuses identity and link. A major faux-pas if you ask me.

In the end, the "link" or "relation", or however you want to call it, is the most important concept of a data store and is unlikely to change for another million years, at least. Even the Web or the RESTafarians didn't change that, though its scope world-widened at the expense of bi-directionality. So I don't really view SQL or NoSQL as being semantically different, I don't really see (semantically) an inversion between the association between an identifier and the information being identified (this is merely visual attribute and an implementation decision of the query engine). Of course, I am abstracting all the non functional differences between each type of storage engine, I am just claiming that semantically, we are standing still (and that's good, it would be shocking otherwise).

Web APIs don't change the semantics of a link either. If a given API call were to return any kind of identifier, are long as we know which API(s) can consume it to produce the information associated to it, just like Tables or Collections, we will be in the position to navigate it, even though we might not have an engine that can do it automatically for us. However, the API world introduces a significant asymetry between the way we read and write data. It is clear that you can easily map a Table or a Collection to a set of data returned by a read-only API, but the converse is not true. A set of APIs which all have a side effect will rarely, if ever, map to the ability to write freely to a table or a collection. Again, let's go back to the invention of writing to better understand that point:

Farmers needed to keep records.The Sumerians were very good farmers. They raised animals such as goats and cows. Because they needed to keep records of their livestock, food, and other things, officials began using tokens. Tokens were used for trade.Clay tokens came in different shapes and sizes. These represented different objects. For example, a cone shape could have represented a bag of wheat. These tokens were placed inside clay balls that were sealed. If you were sending five goats to someone, then you would put five tokens in the clay ball. When the goat arrived, the person would open the clay ball and count the tokens to make sure the correct number of goats had arrived. The number of tokens began to be pressed on the outside of the clay balls. Many experts believe that this is how writing on clay tablets began.

Well, obviously, we made some progress on idempotency, but no matter how flexible the data structure is, the semantics of writing information (in that case changing the state of the clay ball to "delivered") at the API level are widely different. Sure, a query language such as SQL and the like in the NoSQL world, would allow full control at the data structure level, just like a scribe would have had on a piece of papyrus thousands of years ago, but who in their right mind would give such privileges to just about anyone?

So I remain unconvinced, today, that a query engine on top of APIs is not the right model to adopt in general. It is a little bit the same argument that RESTafarians used to debunk the WS-Resource Framework specification, how many layers of query language can you stack? How could a "composite" query engine convey business exception as updates fail behind the Web API layer, for instance?

IMHO, in the context of Composite Applications (not analytics for instance), it would be far more efficient to define information entities (with an adequate, shall I risk -modern-, data structure) and associate an access layer composed of entity specific queries on the read side and action model on the write side and bind these queries and actions to specific APIs. Hence it is far more important to understand how Web APIs and modern ("open") data structures can be bound to the programming model (i.e. business logic and views). It seems to me that, problems like tracking transparently the identities of the pieces of information, transparent off-line / on-line operation, ... should be far more interesting to support at the programming level model than offering a general query language.

As a matter of fact, API2MOL is a project lead by Javier Luis Canovas Izquierdo that reverse engineers a set of APIs into a set of classes that describe the information model behind the APIs. I think it would be nice if API2MOL could focus on establishing a clear binding between specific, i.e. concrete, APIs and the views in which these entities are represented and manipulated.

Last week Subbu complained about Web APIs: "they are a pain". Really? Subbu gives four reasons:

1. Interfaces are bound to tradeoff special requirements of their consumers and try to stick to a common denominator

I don't get it, REST was sold to us as "easy". You can't actually give a talk or attend a talk without a guy in the attendance chanting "REST is easy". Why would you stick to a "common denominator" when you can push resource representations at will?

2. Writing client code is slow, repetitive and verbose

Subbu adds: "In one of the Java implementations I looked at, the code was over 300 lines long – that was after excluding the request making and response parsing code which was already factored into into supporting libraries. If the client needs to implement ten such use cases, you got 3000 lines of code just doing data retrieval."

I don't get it. The industry was sold on REST because, we could curl up calls to endpoints, god forbids, even GETting a resource representation directly from a browser's address field, in other words, REST is easy. We were even told that human readable documentation was enough. How in the world could we endup there?

3. API changes that the clients need don’t happen immediately. Sometimes never. The reality is that teams have their own priorities.

I don't get it. REST was sold to us because it had a "uniform interface" and good APIs URLs don't change. HATEOAS was once touted as the end of all evil, links everywhere, autodiscovery, and voila, you could fetch all you need with ease. What happened? why would you need "changes"?

4. Requests are chatty. Three hundred lines of code may not be a big deal, but making so many HTTP requests is.

I don't get it. Resource representations and the uniform interface were touted as what we would ever need and now we can only build "chatty" interactions.

Subbu is by far the most decent RESTafarian I know. Not only does he really know what he is talking about, probably by a factor of 10, when compared to any other RESTafarians, but he is also genuinely looking for a better way to do things. But my questions to you Subbu, are:

a) was not this totally predictable 4 years ago? Who could imagine that by adding some annotations on top of Java (JAX-RS), you would end up anywhere else?

b) Isn't it time that we honestly and objectively look at (the other) "REST" (not Roy's REST)?

c) now that we are in this mess ocean of technical debt, what is the path forward ?

I stepped up to shake hands with Steve Vinoski at QCon last week, as I was getting ready to detail the program of the "Web API" track. That was quite a moment: sorry guys, Pizza is a not vegetable, no matter how many people say it or even when the US Congress says so.

We will showcase Canappi during an AT&T Webinar on "Building Multi-Platform Native Applications" on Wednesday 9/15.

Canappi is a Mobile Application Development Platform that generates iOS or Android code from a DSL. It also generates back-end services (MySQL/PHP and WSO2 Stratos Data Services).

Please register here to view the webinar.