REST is for sleeping. MQTT is for mobile

REST is designed around a simple request/response model.  So you ask “did my account balance change” and the response is returned “no it did not“.  So you check again a few minutes later, and get the same response.  Sound like a silly example?  Actually we’ve learned that it is a very real issue that many customers obsessively check throughout the day, as many as 60 times inflicting a load on back-ends that they weren’t designed for.

Compounding the issue for mobile apps is REST via HTTP which wasn’t designed to work on mobile networks.  HTTP on mobiles is a bit heavy, fragile and slow and drains batteries quickly.

So notifications, isn’t that what Google and Apple Push are for?  Well sure, up to a point.  But there are some serious issues there.  They offer no quality of service, really don’t have much in the way of guaranteed messaging. The practical result customers see is that notifications arrive quickly, late or not at all.  And it is the not at all that is particularly troubling because there is  no way for the sending party to know whether it was delivered.  I hear a lot of frustrations from customers over this with one telling me “Sure, it is great to find out there is a new level of Angry Birds but it isn’t anything I can run my business on“.

So isn’t there smartphone technology that solves this nicely?  Not really.  But there is an obscure machine-to-machine protocol that does.  Andy Stanford-Clark and Arlen Nipper invented MQTT to solve a problem they had: how to do reliable messaging over unreliable networks?  In an industrial environment with computationally”challenged” devices, with restricted power due to solar or battery power, on extremely low bandwidth and often brittle RF communications including satellite.  It had to work reliably, it had to use very few computational cycles, consume trivial power and could not hog what little bandwidth there was.  And it had to be pub/sub so to break the cycle of violence being inflicted by heavy unnecessary workloads and bandwidth consumption due to all the request/response polling-based monitoring & control systems that were in place at that time.

So Arlen & Andy developed a very simple, extremely efficient publish/subscribe reliable messaging protocol and named it MQ Telemetry Transport (MQTT).   A protocol that enabled devices to open a connection, keep it open using very little power and receive events or commands with as little as 2 bytes of overhead.  A protocol that has the things built in that you need for reliable behavior in an unreliable or intermittently connected wireless environments.  Things such as “last will & testament” so all apps know immediately if a client disconnects ungracefully, “retained message” so any user re-connecting immediately gets the very latest business information, etc.

Fast forward a few years to Facebook’s ambition to create a communications platform for hundreds of millions of people.  A platform that would have to have a dramatically better, more responsive user experience than what others were providing.  Lucy Zhang, the engineer in charge was experienced enough to know that the 3 key issues were going to be:

  • latency – how to get faster phone-to-phone communications
  • battery – and do that without killing batteries
  • bandwidth – or sucking up the user’s available bandwidth

While chatting over drinks in Barcelona, a former Facebook  employee told me that the brilliant bit of lateral thinking that Lucy did was instead of trying to brute force the problem with HTTP or HTTP-based protocols like XMPP (which inherit all of HTTP’s issues on mobile), she adopted an obscure m2m protocol “MQTT”

With the result being that Facebook gained a communications platform with a compelling user experience (well, good design had something to do with it too) that garned great reviews, tremendous stickiness, and  today has 680M mobile users, projected to hit 1B within a year.

Verizon Wireless this spring had their engineers do an engineering assessment of the top applications on their network.  Their assessment was that Facebook Messenger using MQTT is 5 stars for security, 5 stars for battery, and 4 stars for bandwidth.  Which is impressive considering the bandwidth measure is compared to things like Angry Birds which use zero bandwidth.

Stephen Nicholas did a fascinating apples to apples comparison of MQTT vs HTTPS on Android, 3G and WiFi which you can read here.  The 3G results are quite interesting:

  • 93x faster throughput
  • 11.89x less battery to send
  • 170.9x less battery to receive
  • 1/2 as much power to keep connection open
  • 8x less network overhead

So if you follow the NY Times or any of the IT trade magazines you may have seen a lot of news about MQTT in the past 2 weeks.  It is now in progress to become an OASIS M2M standard.  And IBM has open sourced all of its MQTT source code via Eclipse.org including the new HTML5 MQTT over WebSocket JavaScript which enables MQTT in any HTML5 container including mobile browsers, desktop browsers, vehicle infotainment, consumer electronics.  I’m doing 13,000 messages a second on my iPad with a pure HTML5 app developed in Worklight.  I don’t think you could do that with HTTP.  You can also google for “MA9B” to download zip from IBM of source code for a 1/2 dozen different mobile clients.

MQTT is small footprint, efficient, low power on the device.  So what happens when you flip that around to the data center or mobile operator side?  And drive it with new acceleration technology from IBM Labs?  You get something very dense, very green, and very fast.  As in concurrently connect as many mobiles, cars, or devices as 1,000 web servers with just 1 rack.  Product Management is OK when I say 21M concurrently connected things per rack.  They frown when I say the actual number.  And messaging throughput?  With the Beta firmware we were getting 273M mobile messages/sec per rack.  The performance  results we see in the GA firmware are “higher” and will be published soon.  End-to-end, app-to-app on fast network with this appliance using MQTT is in µs not milliseconds.  That’s why we’re now using this appliance and MQTT for high-speed Big Data analytics, driving millions of low-latency analytics decisions per second.

More on that at ibm.com/MessageSight with videos here and here.

I’m quite fanatical about this notion that response time = revenue, response time = business performance, and most importantly response time saves lives.  It isn’t about machines anymore folks, “Connected Life” is you, me and everything around us and how it all interacts.  And anytime there is a human in that interaction, then latency matters very much.

Well, that’s probably enough of a diatribe for one day.  I have an IBM internal blog which is “mobile musings”, my intention here was “mobile bit” i.e. bitten, smitten but also a small piece, short.  I’ll hew closer to that in the future.

Best,
- Joe

About these ads

18 thoughts on “REST is for sleeping. MQTT is for mobile

  1. Brilliant article, Joe – well written and seriously emotional. After I saw MQTT in action at IMPACT, I am planning to conduct a POC and analysis.
    Gene

    • That’s great Gene. Thanks for the complement. I really am quite unreasonably invested in this. It isn’t that I care because it is my job. It is rather the other way around. I took the job because I cared, was already so deeply invested in this. I really see potential in it making a difference in how people and things interact. I know it sounds corny but that’s me :-) I enjoyed spending time with you and Sridhar last week, very excited to work with you!

  2. Nice write-up. But there are also other contenders in this space. Mainly CoAP which tries to combine the advantages of the REST approach with the constraints of very low-resource devices.

    If REST is for sleeping, MQTT for mobile then CoAP is for the IoT ;-)

    • Hi Leo, yeah CoAP is lighter than HTTP solution for REST. Isn’t it still request/response? what we’re learning is that IoT and sensor networks do best with event driven pub/sub. One of my customers is a very large electronics conglomerate that is in the process of flash updating every CoAP sensor (power, water, traffic, buildings, et al.) in their national smarter city infrastructure to switch them to MQTT. They tell me after using it for some time, they determined that CoAP wasn’t the best fit for their national sensor network. And if you do need request/response you can still do that via MQTT pub/sub, just look at what the automakers are doing with MQTT remote controls of cars for “key fob” response time via cloud for car remote functions using a smartphone.

  3. Nice write-up ! However MQTT is not the only alternative to REST over HTTP. There is also CoAP which combines the advantages and simplicity of REST and ultra-low-power & bandwidth requirement over lossy networks.

    So if REST is for sleeping and MQTT is for mobile, then CoAP is for the Internet of Things (IoT). :)

  4. Please let be aware that MQTT is also availabel on Arduino uc and there are also lite clones as Mosquitto quite nice and small.

    • Hi Roberto, Definitely love the combo of MQTT with Arduino, Rasberry Pi, mbed. Works great on those because small footprint, tiny overhead, low power. We have MQTT-powered Arduino RC Ford truck and police car running around the office. They were easier to fly with than an actual test bed vehicle :-) If you drive the Arduino RC Ford truck too far away from the meshed police car, then the police give chase. Mosquitto and RSMB (really small message broker) are popular with developers and hobbyists. I’ve spent more time using RSMB which is a 75KB MQTT broker for desktop and embedded systems.

  5. I also saw MQTT at Impact this year and we’ve done a heap of little PoCs since then integrating it with Worklight Mobile Apps and WebSphere Integration Bus on the backend. Fantastic technology whose time has come. Lots of customers are very excited about it this year. Expect to see many pick up where Facebook left off.

  6. Pingback: MQTT with Vert.x | Tales from a Trading Desk

  7. Pingback: Pushy Node.js II: The Mullet Architecture | Dejan Glozic

  8. Pingback: The Direction is toward Open Standards for M2M and IoT | Building Context

  9. Pingback: Pub/Sub Communications — Social, Simple, Tried-and-True | Building Context

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s