The purchase of Instagram by Facebook is no less than an attempt to redefine the landscape of our industry: today, you are either an app, a service or a platform.
For instance, RIM will become an "app". No disrespect, it can be a formidable app, opening the doors to the Enterprise to an Apple or a Google, but the window of opportunity for RIM to be a platform is gone. In this new world, search is an app which sets the search context with a high degree of precision (yelp, UrbanSpoon...), ads is a service and the Worn out WWW amounts to the Yellow Pages (paper HTML edition).
... and maybe (always smiling) Amazon. The problem with Amazon is that they'll have to choose between being a commerce platform (and compete with WalMart) or an industry platform and play with the big boyz. I don't think these two things work well together and eventually Amazon will return to its commerce roots mainly because of the lack of social capability. Amazon is a formidable company, but they are too late to the game, they have no money in the bank to acquire what they need to compete and they are burdened by a big legacy business, deeply rooted into the Web. As (Microsoft), Google and Apple know, there are only so many social networks that can fuel a healthy platform. Most likely Amazon will become a substrate to support services with AWS, and of course, an "app", the kindle, assuming the platform guys leave these holes open (far from a given).
Yet, Amazon was a decisive factor in establishing the architecture of the platform. It reinforced the power of the vertical integration pioneered by Apple. If you come with a great device or set of devices, and an app store, customers will line up. Something that Google is still struggling to emulate with an open ecosystem, and I am not even talking about Microsoft. In other words, it is not the number of devices you sell that counts, it is the the numbers of end-users actively using the platform to buy digital goods, interact, store their stuff... If you doubt it, just ask Nokia ...
If the social integration is pretty much a given, the game gets really interesting when it comes to the vertical integration. As we have seen with Apple, customers don't see "choice" (or price) as a competitive advantage. People want simple product lines with products they can actually use, with clear and real innovation at each cycle, built by thoughtful product teams: selling incremental junk is no longer an option (as HP painfully learned). And yes, TV or Game consoles will be engulfed in the platform. How could the platform guys leave them on the table? Any device that can run an app will be vertically integrated in the platform (or replaced by one).
My best guess is that:
- MS/FB will partner strongly with or even acquire Nokia and HP.
- Google/Motorola will swallow HTC and/or LG, possibly bring Dell in its ecosystem
- Apple ... huh ... will buy Twitter, and RIM for its Enterprise App
Samsung could team up with Amazon, but most likely it will become the component provider of the platform guys. SONY's, Nintendo's and HP's troubles simply show that there is no more room for Independent Hardware Vendors: you are either a (vertically integrated) platform or you supply the platform with components, no, not even devices (I mean other than speakers and power adapters). The touch points to consumers are going to be completely locked down, not one will be left open.
Even Telco providers will be suppliers to the platform guys, with the Skype/Hangout/Facetime apps being the new face of telephony.
The interesting thing here is that unless social networks are just a fad, MS/FB is the most likely winner of that game. There is a large probability that MS/FB/NOK/HP could take on 80% of a vertically integrated, consumer oriented market. Even Ballmer could pull that off and retire on a real visionary success. Google will lose that game (this is not an engineers game, let alone using corny "Web" paradigms and UX, in any case they will take the decision to move to a vertically integrated model too late without any muscle behind it). Apple will take on a 20% "think different" crowd. Apple is not aggressive enough to win that game, they'll look back to their $100 B in the bank and ask themselves, if they could have used it more "wisely".
Facebook spent a billion, not to buy Instagram but to hold our industry by the balls. With its IPO behind, it will call the shots for years. And yes ... Zuckerberg is next CEO of "The Platform".
Fascinating times ... it's a shame that Steve is no longer with us.
06/18/2012: Microsoft announces the "Surface"
07/04/2012: Google Jelly Bean is a game changer. Google seems to finally get it, UX is key to win this game. I may be wrong after all, Microsoft may never be able to catch up. If Microsoft continues to mess up on the Mobile OS side over the next 6 months, there will be no Microsoft in the Mobile space moving forward.
07/06/2012: Amazon is planning to launch its own smartphone.
07/16/2012: Google (Motorola) is launching the Atrix HD.
07/20/2012: Microsoft reports a big jump in Skype usage, up 50%
07/23/2012: Amazon plans 6 new Kindle variants.
07/26/2012: Facebook works with HTC on its own smartphone
07/28/2012: Microsoft is in trouble, that may well be the statement that signals the decline of the company: Microsoft CEO Steve Ballmer has sought to downplay the notion of Surface competing with partners, recently calling the device “just a design point.
7/31/2012: Details on Microsoft's plans to move to a vertically integrated model
8/20/2012: It looks like Microsoft does not aim at partnering / merging with Facebook, they sold 20% of their stake right after the IPO, that's kind of a shame, without even the shadow of a social network, they will have a hard time getting there. Unless their secret weapon is to make Yammer a general purpose social network.
8/22/2012: "The Platform" is on Amazon's Navigation Bar: App Store, Storage, ... Critically missing is the social network component.
9/06/2012: Amazon unvails the new Kindle, "People don’t want gadgets anymore, says Jeff Bezos, they want services that improve over time. Kindle Fire is a service. It greets you by name. Comes out of box with content preloaded, makes recommendations."
9/06/2012: Mobile gamers outnumber the console core.
9/12/2012: Zuckerberg "Facebook's Mobile opportunity is much bigger than what people think", "Everyone is now "underestimating" Facebook" (I am not ...)
10/02/2012: Windows Mobile market share continues falling
10/02/2012: 1st Apple-approved iOS game controller makes its debut
10/04/2012: Facebook Tops a Billion Users
10/09/2012: Ballmer to Microsoft shareholders: 'a fundamental shift [is] underway in our business' In his annual letter, CEO Steve Ballmer says Microsoft's future is a tight combination of hardware and software
10/12/2012: For the first time this month people searched less than last year. Our interpretation is that people now search more in context, using dedicated apps on their mobile device. This is particularly disruptive because that is the kind of search that has the most impact on commerce and advertising.
10/22/2012 Subliminal message from Zuckerberg: I would have worked at Microsoft if facebook had failed
10/25/2012 Ballmer: Microsoft has more hardware to come.
11/2/2012: Microsoft is testing its own smartphone
11/6/2012: Verizon shuts down its App Store
01/22/2013: Microsoft talks about investing 3B in Dell
02/14/2013: HP to adopt Android
I am truly amazed at the degree of innovation of our industry today. In a way, it's quite scary: Are we building tools just in search of a problem? are we really innovating or is it just another "hyped-stand still cycle" ? In the midst of massive scalability improvements and seamless operational environments it is easy to lose track of the semantics view.
I built a "Composite Application Framework" at Attachmate between 2003 and 2005. Even though our team did get to the 1.0 release. The product was shelved before it could get to its first customers when Attachmate was acquired. A composite application is an application which is capable of interacting with systems of record that are beyond its control (typically 3rd party Web APIs, as we call them these days). As we architected the framework, we were faced to a big philosophical question: should we group these Web APIs (they were known as services and operations back then) under a facade that expose a query language interface or should we rather "bind" these APIs to our UI and develop orchestration-based mashup logic in the API consumer (middle-tier or client). I was softly voting for the former, while my team prefered the later. I didn't push back because I knew that implementating a query engine would delay the project and I didn't have a strong rationale for it. It was just cool. At the time, we had done some experiments with a couple virtual database technologies which allowed an in-memory SQL engine to connect to a number of different databases (not APIs) and make them appear as a single database from the client perspective. These product were interesting but they never really got traction and their API bindings were nascent to say the least.
Hence, I have been quite intrigued by Subbu's ql.io project. It is still incomplete since it does not handle updates yet, but it is already quite mature to look like what I had in mind then. I don't want to make too many conclusions until I use it, or other people provide some feedback. When I designed Canappi, I decided to take a binding approach, rather than building a SQL-like facade. So far I have been quite happy with that choice mainly because composite applications rarely need the full power of a query language. The Views map reasonably well to the model (unless it is the other way around ...) and applications follow a "navigational" pattern which is generally well supported by Web APIs. If you need to create some reports, of course, the answer would be quite different.
As REST has forced most of us to CRUD our way to the data, I think that our industry has reached a point where we need to answer objectively: what is different about SQL, NoSQL and (Web) APIs? We have built a "composite world", great, I don't think we will be able to make much progess without establishing a clear articulation between the way we store, relate and access information.
In his introduction to ql.io, Subbu points to a very interesting paper from Erik Meijer and Gavin Bierman, in which they argue: "Contrary to popular belief, SQL and noSQL are really just two sides of the same coin".
I like their analysis, but I am a semantic guy, I really like to see at the semantic level what's new, so I created a simple metamodel:
On the left end side, you have the traditional RDBMSs / SQL model on the right end side the new "No SQL" model (both key-value pairs and document oriented). The color coding is used to (roughly) map concepts from one world to the other. I have adopted a JSON metamodel for the structure of the values of NoSQL databases.
Erik and Gavin pointed out, the key deifference is that in the NoSQL world, an identity is generally a "key":
In the object-graph model, the identity of objects is intensional—that is, object identity is not part of the values themselves but determined by their keys in the store. In the relational model, object identity is extensional—that is, object identity is part of the value itself, in the form of a primary key.
It is of course true, but does it really matter? Did we really create new semantics by making an identity a subclass of a key? (incidentally, it would be very interesting to ask the opposite question, what happens when a "key" IS-An "identity" within a given scope (row, collection, database)?) So, my answer is no, claiming that an identity IS-A key does not change the semantics of what an identity is, regardless of the position of an identity in a result set or in the store. I have argued many times that this was true 8000 years ago when man invented writing and hence data. I have also argued that for instance, in the absence of a reasonable numbering system, ancient writers were left to techniques such as acrostic peotry to enable random access and even sorting.
The authors also argue that another key difference is the "open" data structure of NoSQL databases, but is this openess inherent to NoSQL or, is it inherent to the design of storage engine (and the query language designed to harness it)? First, a "link" is a link semantically speaking, it relates two pieces of information using one or more identity. Nothing would have prevented us to create a "unique id" concept in RDBMSs and allow for querying at the database level (with a statement like SELECT * WHERE ID = '123'). As a mater of fact, if you look at MongoDB which is not a key-value pair store per se but a document store, its data structure includes the concept of collections (~Tables) and the identity of a document is stored separately from the document itself. Second, all these data structure concepts are simply man made, for good reasons of course, but they do not introduce or uncover any new semantics.These "open data structures" simply provide a better opportunity to physically associate data that is often queried together. For instance a purchase order has line items or a prescription has a number of treatments, but that does not remove the need to uniquely identify all individual line items or treatments to associate additional details or independently reference these pieces of information.
Hence, the only semantics that matters in a data store are identity and link, which are common to all data stores, even the ones man used 8000 years ago when writing was invented. Incidentally, we can also notice that would-be data centric technologies like JSON, XML or even ATOM do not reflect correctly the importance and semantics of "links". As a matter of fact, the Web confuses identity and link. A major faux-pas if you ask me.
In the end, the "link" or "relation", or however you want to call it, is the most important concept of a data store and is unlikely to change for another million years, at least. Even the Web or the RESTafarians didn't change that, though its scope world-widened at the expense of bi-directionality. So I don't really view SQL or NoSQL as being semantically different, I don't really see (semantically) an inversion between the association between an identifier and the information being identified (this is merely visual attribute and an implementation decision of the query engine). Of course, I am abstracting all the non functional differences between each type of storage engine, I am just claiming that semantically, we are standing still (and that's good, it would be shocking otherwise).
Web APIs don't change the semantics of a link either. If a given API call were to return any kind of identifier, are long as we know which API(s) can consume it to produce the information associated to it, just like Tables or Collections, we will be in the position to navigate it, even though we might not have an engine that can do it automatically for us. However, the API world introduces a significant asymetry between the way we read and write data. It is clear that you can easily map a Table or a Collection to a set of data returned by a read-only API, but the converse is not true. A set of APIs which all have a side effect will rarely, if ever, map to the ability to write freely to a table or a collection. Again, let's go back to the invention of writing to better understand that point:
Farmers needed to keep records.The Sumerians were very good farmers. They raised animals such as goats and cows. Because they needed to keep records of their livestock, food, and other things, officials began using tokens. Tokens were used for trade.Clay tokens came in different shapes and sizes. These represented different objects. For example, a cone shape could have represented a bag of wheat. These tokens were placed inside clay balls that were sealed. If you were sending five goats to someone, then you would put five tokens in the clay ball. When the goat arrived, the person would open the clay ball and count the tokens to make sure the correct number of goats had arrived. The number of tokens began to be pressed on the outside of the clay balls. Many experts believe that this is how writing on clay tablets began.
Well, obviously, we made some progress on idempotency, but no matter how flexible the data structure is, the semantics of writing information (in that case changing the state of the clay ball to "delivered") at the API level are widely different. Sure, a query language such as SQL and the like in the NoSQL world, would allow full control at the data structure level, just like a scribe would have had on a piece of papyrus thousands of years ago, but who in their right mind would give such privileges to just about anyone?
So I remain unconvinced, today, that a query engine on top of APIs is not the right model to adopt in general. It is a little bit the same argument that RESTafarians used to debunk the WS-Resource Framework specification, how many layers of query language can you stack? How could a "composite" query engine convey business exception as updates fail behind the Web API layer, for instance?
IMHO, in the context of Composite Applications (not analytics for instance), it would be far more efficient to define information entities (with an adequate, shall I risk -modern-, data structure) and associate an access layer composed of entity specific queries on the read side and action model on the write side and bind these queries and actions to specific APIs. Hence it is far more important to understand how Web APIs and modern ("open") data structures can be bound to the programming model (i.e. business logic and views). It seems to me that, problems like tracking transparently the identities of the pieces of information, transparent off-line / on-line operation, ... should be far more interesting to support at the programming level model than offering a general query language.
As a matter of fact, API2MOL is a project lead by Javier Luis Canovas Izquierdo that reverse engineers a set of APIs into a set of classes that describe the information model behind the APIs. I think it would be nice if API2MOL could focus on establishing a clear binding between specific, i.e. concrete, APIs and the views in which these entities are represented and manipulated.
Last week Subbu complained about Web APIs: "they are a pain". Really? Subbu gives four reasons:
1. Interfaces are bound to tradeoff special requirements of their consumers and try to stick to a common denominator
I don't get it, REST was sold to us as "easy". You can't actually give a talk or attend a talk without a guy in the attendance chanting "REST is easy". Why would you stick to a "common denominator" when you can push resource representations at will?
2. Writing client code is slow, repetitive and verbose
Subbu adds: "In one of the Java implementations I looked at, the code was over 300 lines long – that was after excluding the request making and response parsing code which was already factored into into supporting libraries. If the client needs to implement ten such use cases, you got 3000 lines of code just doing data retrieval."
I don't get it. The industry was sold on REST because, we could curl up calls to endpoints, god forbids, even GETting a resource representation directly from a browser's address field, in other words, REST is easy. We were even told that human readable documentation was enough. How in the world could we endup there?
3. API changes that the clients need don’t happen immediately. Sometimes never. The reality is that teams have their own priorities.
I don't get it. REST was sold to us because it had a "uniform interface" and good APIs URLs don't change. HATEOAS was once touted as the end of all evil, links everywhere, autodiscovery, and voila, you could fetch all you need with ease. What happened? why would you need "changes"?
4. Requests are chatty. Three hundred lines of code may not be a big deal, but making so many HTTP requests is.
I don't get it. Resource representations and the uniform interface were touted as what we would ever need and now we can only build "chatty" interactions.
Subbu is by far the most decent RESTafarian I know. Not only does he really knows what he is talking about, probably by a factor of 10, when compared to any other RESTafarians, but he is also genuinely looking for a better way to do things. But my questions to you Subbu, are:
a) was not this totally predictable 4 years ago? Who could imagine that by adding some annotations on top of Java (JAX-RS), you would end up anywhere else?
b) Isn't it time that we honestly and objectively look at (the other) "REST" (not Roy's REST)?
c) now that we are in this mess ocean of technical debt, what is the path forward ?
I stepped up to shake hands with Steve Vinoski at QCon last week, as I was getting ready to detail the program of the "Web API" track. That was quite a moment: sorry guys, Pizza is a not vegetable, no matter how many people say it or even when the US Congress says so.
We will showcase Canappi during an AT&T Webinar on "Building Multi-Platform Native Applications" on Wednesday 9/15.
Canappi is a Mobile Application Development Platform that generates iOS or Android code from a DSL. It also generates back-end services (MySQL/PHP and WSO2 Stratos Data Services).
Please register here to view the webinar.
Since 2007, the industry was told by a handful of people that the "Web" could teach us everything about distributed computing, and that everything we had done before was simply wrong. Four years later, we can only see that the emperor is completely naked (what else could we expect?).
REST was designed with an end-user in mind, operating a user agent: the browser (I know Roy is going to get mad if he reads this). Everything that has been written to apply REST principles to a software agent-to-server scenario is simply wrong. To quote Roy himself:
A REST API should be entered with no prior knowledge beyond the initial URI (bookmark) and set of standardized media types that are appropriate for the intended audience (i.e., expected to be understood by any client that might use the API).
A REST API should never have “typed” resources that are significant to the client. ... The only types that are significant to a client are the current representation’s media type and standardized relation names.
A REST API must not define fixed resource names or hierarchies (an obvious coupling of client and server). Servers must have the freedom to control their own namespace.
How many Web APIs are actually designed that way? It is time to move on from all this mind bending serendipitous complexity that brings absolutely nothing, except more lost productivity by forcing to everyone to hand code everything we got for free in previous distributed computing paradigms.
So I encourage everyone to unREST with pride. No offense to Roy, REST is brilliant. It is just not applicable to the problem at hand and we should give him some REST in return, for all that he has given to us. It's cool to be RESTless. Let's not waste any more time.
So what does it mean to unREST?
a) Define a uniform interface that suits your domain
There are many uniform interfaces to choose from: Send/Receive is one, GET/PUT/POST/DELETE is the "REST" one, CRUD has been used for 30 years world widely, ...
Don't be shy, forget the 4 little HTTP verbs, you can even define your own: Query/Do/Notify/Synchronize is a perfectly good one. It means that all methods in your Web API will be of type query, action, notification or sync request.
One could even imagine a 3-legged uniform interface, well suited to support modern converged / composite applications.
In fact, REST's so-called "uniform interface" is quite broken outside of the Web as some queries must be implemented with a POST and lots of people struggle to use PUT, POST and DELETE meaningfully. I have seen a standard using DELETE to implement the "cancel" payment operation. (I bet that's what some would like to do with their CDS transactions... )
A uniform interface is essential to a healthy API, pick one that fits your domain and stick to it. Let your infrastructure know about it from a security , scalability ... perspective.
b) Specify messages
when two agents communicate, they exchange messages. Duh? These messages rely on a shared understanding of some sort be it a standard type or semantic. Define your API messages loud an clear. This is a major step in becoming unREST. The syntax of your message doesn't matter, who cares about XML, JSON, Protocol Buffers? What is essential is that your syntax must be extensible. This is what allows forward versioning.
Explicit message definitions which can be explicitely versioned is essential to a healthy API definition. Message Extensibility is the property that makes distributed computing work.
c) Specify contracts with a clear versioning strategy
There has been many "IDLs" defined over the years, few were truly message based. Think of a relationship between software agents as forming a machine readable "Contract" that defines the rules upon which messages are exchanged between them. A contract is something you break only when you have no other choice. A Contract is most often bi-directional and asynchronous (anyone is receiving any notification on their phone?). Thinking of every relationship as being synchronous and of type client/server, is pure nonsense. Convincing people that they could do without a contract, just with some "human readable documentation" (HRD) is also pure nonsense. How many millions of hours have been wasted worldwide the day we decided to drop WSDL for "HRDs". A uniform interface is not enough to form a contract, it is only the substrate on which a contract is defined. That contract is not uniform, by any stretch of imagination.
Machine readable contracts are the foundation on which to build a healthy ecosystem of Web API consumers. Versioning is essential to evolution and reuse, and hence a healthy API.
Please note that, reuse, in distributed computing, happens the other way around when compared to traditional software engineering. It is the old "clients" of a Web API that "reuse" the new versions of a service (without breaking). Who would expect that anyone could design an API that 3 years down the line would be "reusable" by any 3rd party consumer? It is far more valuable to enable existing client to "reuse" the new features of your API, at their pace, without breaking them, for every tweak you make into your message formats.
That's it, that's all you need to create a successful API nothing else, nothing more. Sure, it would be great if we converge on a handful of uniform interfaces and versionable message /contract specifications. I am not sure we can find "the one" that works for everyone.
I have participated in crafting web-based protocols for most of the last 12 years, starting with ebXML in 1999. The big difference between today and then, is that we have thousands of openly available data points teaching us, we, the "standard" guys, how people actually create APIs, for what purpose. Maybe for once, they could teach us a thing or two. That would avoid picking just a couple of use cases convenient to a vendor's product and forcing the REST of the world fit under that petty umbrella.
You are now free to unREST and laugh quietly at anyone who asks you to marvel at an HTTP header, a "link", use a 7 letter acronym, or tells you with the authority of a story teller that you don't get it. REST was just a (bad) dream, unREST is what you want to do.
The core of the presentation is on two slides (10 & 11). They focus on two important dimensions of mobile apps: the information dimension and the actions dimension (there are more such as the Converged dimension or the UX dimension).
Mobility is a major paradigm shift because mobile apps are based on a 3-legged information model: Product-Place-People, unlike the tethered Web that has no actionable location information. Amazon's business model, for instance, relies solely on linking products to people and I would not be surprised if a startup coming out of nowhere disrupts Amazon's eCommerce franchise.
Why? because location is not just about finding disruptive way to link products and people, it is also weaving time (i.e. events) into the business model and, most importantly, it makes the world "actionable" (taking actions at a distance doesn't always work well...).
This new relationship between people and products (via places, events and action) gives the opportunity to create a virtuous circle (slide 11): people, places and events enable advanced advertising/offers models and as people take actions on goods, services, media, knowledge ... (such as purchase, consume, invoke, favor, link, collect ...) it gives the ability to capture favorites and historical data (which would be impossible to tie to a location or event without a mobile device) thereby providing a formidable feedback loop to advertising models and offers in very innovative ways.
Hope you like the presentation and join us in Redmond if you can !