REST is for sleeping. MQTT is for Mobile

REST is designed around a simple request/response model.  So you ask “did my account balance change” and the response is returned “no it did not“.  So you check again a few minutes later, and get the same response.  Sound like a silly example?  Actually we’ve learned that it is a very real issue that many customers obsessively check throughout the day, as many as 60 times inflicting a load on back-ends that they weren’t designed for.

Compounding the issue for mobile apps is REST via HTTP which wasn’t designed to work on mobile networks.  HTTP on mobiles is a bit heavy, fragile and slow and drains batteries quickly.

So notifications, isn’t that what Google and Apple Push are for?  Well sure, up to a point.  But there are some serious issues there.  They offer no quality of service, really don’t have much in the way of guaranteed messaging. The practical result customers see is that notifications arrive quickly, late or not at all.  And it is the not at all that is particularly troubling because there is  no way for the sending party to know whether it was delivered.  I hear a lot of frustrations from customers over this with one telling me “Sure, it is great to find out there is a new level of Angry Birds but it isn’t anything I can run my business on“.

So isn’t there smartphone technology that solves this nicely?  Not really.  But there is an obscure machine-to-machine protocol that does.  Andy Stanford-Clark and Arlen Nipper invented MQTT to solve a problem they had: how to do reliable messaging over unreliable networks?  In an industrial environment with computationally”challenged” devices, with restricted power due to solar or battery power, on extremely low bandwidth and often brittle RF communications including satellite.  It had to work reliably, it had to use very few computational cycles, consume trivial power and could not hog what little bandwidth there was.  And it had to be pub/sub so to break the cycle of violence being inflicted by heavy unnecessary workloads and bandwidth consumption due to all the request/response polling-based monitoring & control systems that were in place at that time.

So Arlen & Andy developed a very simple, extremely efficient publish/subscribe reliable messaging protocol and named it MQ Telemetry Transport (MQTT).   A protocol that enabled devices to open a connection, keep it open using very little power and receive events or commands with as little as 2 bytes of overhead.  A protocol that has the things built in that you need for reliable behavior in an unreliable or intermittently connected wireless environments.  Things such as “last will & testament” so all apps know immediately if a client disconnects ungracefully, “retained message” so any user re-connecting immediately gets the very latest business information, etc.

Fast forward a few years to Facebook’s ambition to create a communications platform for hundreds of millions of people.  A platform that would have to have a dramatically better, more responsive user experience than what others were providing.  Lucy Zhang, the engineer in charge was experienced enough to know that the 3 key issues were going to be:

• latency – how to get faster phone-to-phone communications
• battery – and do that without killing batteries
• bandwidth – or sucking up the user’s available bandwidth

While chatting over drinks in Barcelona, a former Facebook  employee told me that the brilliant bit of lateral thinking that Lucy did was instead of trying to brute force the problem with HTTP or HTTP-based protocols like XMPP (which inherit all of HTTP’s issues on mobile), she adopted an obscure m2m protocol “MQTT”

With the result being that Facebook gained a communications platform with a compelling user experience (well, good design had something to do with it too) that garned great reviews, tremendous stickiness, and  today has 680M mobile users, projected to hit 1B within a year.

Verizon Wireless this spring had their engineers do an engineering assessment of the top applications on their network.  Their assessment was that Facebook Messenger using MQTT is 5 stars for security, 5 stars for battery, and 4 stars for bandwidth.  Which is impressive considering the bandwidth measure is compared to things like Angry Birds which use zero bandwidth.

Stephen Nicholas did a fascinating apples to apples comparison of MQTT vs HTTPS on Android, 3G and WiFi which you can read here.  The 3G results are quite interesting:

• 93x faster throughput
• 11.89x less battery to send
• 170.9x less battery to receive
• 1/2 as much power to keep connection open
• 8x less network overhead

So if you follow the NY Times or any of the IT trade magazines you may have seen a lot of news about MQTT in the past 2 weeks.  It is now in progress to become an OASIS M2M standard.  And IBM has open sourced all of its MQTT source code via Eclipse.org including the new HTML5 MQTT over WebSocket JavaScript which enables MQTT in any HTML5 container including mobile browsers, desktop browsers, vehicle infotainment, consumer electronics.  I’m doing 13,000 messages a second on my iPad with a pure HTML5 app developed in Worklight.  I don’t think you could do that with HTTP.  You can also google for “MA9B” to download zip from IBM of source code for a 1/2 dozen different mobile clients.

MQTT is small footprint, efficient, low power on the device.  So what happens when you flip that around to the data center or mobile operator side?  And drive it with new acceleration technology from IBM Labs?  You get something very dense, very green, and very fast.  As in concurrently connect as many mobiles, cars, or devices as 1,000 web servers with just 1 rack.  Product Management is OK when I say 21M concurrently connected things per rack.  They frown when I say the actual number.  And messaging throughput?  With the Beta firmware we were getting 273M mobile messages/sec per rack.  The performance  results we see in the GA firmware are “higher” and will be published soon.  End-to-end, app-to-app on fast network with this appliance using MQTT is in µs not milliseconds.  That’s why we’re now using this appliance and MQTT for high-speed Big Data analytics, driving millions of low-latency analytics decisions per second.

More on that at ibm.com/MessageSight with videos here and here.

I’m quite fanatical about this notion that response time = revenue, response time = business performance, and most importantly response time saves lives.  It isn’t about machines anymore folks, “Connected Life” is you, me and everything around us and how it all interacts.  And anytime there is a human in that interaction, then latency matters very much.

Well, that’s probably enough of a diatribe for one day.  I have an IBM internal blog which is “mobile musings”, my intention here was “mobile bit” i.e. bitten, smitten but also a small piece, short.  I’ll hew closer to that in the future.

Best,
– Joe