thoughts about using IoT MQTT for V2V and Connected Car from CES 2014

Had a great time at CES 2014 showing my IoT tech with Continental AG in the Renaissance Hotel, Jaguar Land Rover + Intel at GENIVI Show Trump Tower, QNX + Qualcomm in North Hall LVCC, and IBM in the Venetian.  Really enjoyed CES, GENIVI Show, Tao, and especially Firefly on Paradise with my guys.  Thanks!

IMG_8029 IMG_8080 IMG_8025 IMG_8075

Spent 4 days with dozens of automakers and car system providers, fielding questions and showing live demos.  Based on those discussions I’ve pulled together information I hope is useful.  Or if not useful, at least interesting:

  • thoughts
  • rants
  • slides
  • monologue
  • MQTT vs HTTPS on mobiles – 93x faster, 11x less power to send, 170x less power to receive, 8x less network overhead.  QoS and LWT for reliable comms on unreliable networks.
  • IBM MessageSight – secure IoT appliance and core component of global IBM IoT Cloud on Softlayer.   Concurrently connects 21M vehicles & mobiles at up to 340.2M telematics messages/second per rack.  Low latency, 60μs device-to-device MQTT messaging on fast network (i.e. 10GigE) under load.
  • video of AT&T iPhone & Verizon iPad remote control of GENIVI Tizen HVAC, note the low latency  Scottsdale > Dallas cloud > car in MA changes temp > Dallas > Scottsdale 
  • demos, how-to examples and source code 
  • demo video using QNX TI OMAP5 car
  • another demo or even better is this one
    note the data path for what you’re seeing in the 2 side-by-side HTML5 apps is Verizon iPad Scottsdale AZ > Dallas Softlayer cloud > GENIVI Tizen car unit in Dartmouth MA > Dallas Softlayer cloud > AT&T iPhone Scottsdale AZ
  • try this MQTT HTML5 whiteboard demo on several mobiles side-by-side, and observe the latency.  This is running against  Softlayer’s Dallas cloud in North America.  Let me know if you need the URLs for this in Asia, Europe or elsewhere.
  • MQTT + MQTT-SN simplifies V2V / V2X. MQTT for V2V via cloud 10s-100s ms, MQTT-SN for V2V via DSRC, ZigBee, 6LoWPAN, serial, UDP, et al.  And Eclipse Mosquitto RSMB 1.3 is 75KB MQTT + MQTT-SN broker for embedded that bridges V2V mesh to cloud.  I think it wouldn’t be an exaggeration to say that RSMB is battle hardened.
  • MQTT/MQTT-SN for transport independent V2V/V2X via dynamic mesh of DSRC/WAVE, Wi-Fi, LTE-A, cloud a’la ITA’s MQTT Sensor Fabric
  • Yes, the cloud is  too slow to tell me the car ahead hit their brakes.  For that you need 20ms latency.  But you know what else besides DSRC can do that?  Cameras, radars, lasers, many things.  And DSRC doesn’t help  with what’s around the next bend, what’s over the next hill.  It is very localized.  Which is an issue because there is a whole class of V2V problems that require macro-level awareness and real-time decisioning for which higher latency (100ms+) is acceptable.  Continental is public example of someone that tackling those using IoT + real-time Big Data geospatial decisioning in the cloud.
  • Yes, not every car is connected nor will they be in the near term, so what about the blind spots?  There are things that help mitigate that.  For example if a car senses other cars and publishes that data to the cloud, then the cloud knows about those cars even though they’re not connected.   IoT MQTT cloud connectivity is software, just requires is a programmable TCU.  Ford already made public that they’re pushing an OTA update to 3.4M existing cars adding more connected car capabilities, so many existing models can enabled for V2V cloud, not just future ones.  Automakers are launching aftermarket OBD2 dongles such as this one w 3G/GPS/accelerometers/etc. are $100 and less at retail ($20 in volume  for an OEM?) that cloud connect via MQTT any car built after 1995.
  • Yes, it is an issue there that each OEM is having their own cloud.  Solutions include having them use 1 cloud, for example what Continental AG is doing with having IBM give them the global connected car cloud which will be used by multiple OEMs that they supply.  Can also tackle it via federated car clouds, interconnecting them and exchanging data in real-time.  While not trivial, federated clouds and federated ESBs is fairly well traveled ground.  The technology is there.  MQTT even has a bridging convention whereby an MQTT broker (server) can also be a client.  The same approach you use to have an MQTT broker like RSMB in the car bridging a local V2V mesh to the cloud can also be used to bridge one cloud to another, effectively interconnecting them.  Yes that requires agreement re data format (preferably protobuf or JSON) and topic space, but those are solvable.
  • I’ve nothing against DSRC, in fact I’m very excited to have MQTT-SN running atop DSRC/WAVE, I think the pub/sub topics and QoS adds to DSRC’s usefulness.  That said, DSRC is early 90’s RF tech that is beginning to show its age.  And many of the use cases DSRC was intended for in V2V are becoming obviated by other sensor and wireless tech.  I don’t need DSRC to tell me when the car ahead hits their brakes, the car is already getting several other sensors that can do that.  And nobody has figured out where the $3 trillion dollars is going to come from to get DSRC radios into all the cars and infrastructure.  And we’re talking about adding that DSRC cost to cars that are already getting LTE Advanced and/or WiFi.  You know what else besides DSRC can do peer 2 peer?  LTE Advanced and WiFi.  Credit for this  to Roger Lanctot @rogermud who clued me into this.  He’s also the one that pointed out to me that latency is a driver distraction issue.
  • I already knew latency is an owner satisfaction and brand perception issue.  She’s not going to buy your car if her buying experience includes making her standing in the rain for 30 to 90 seconds while pondering her request to unlock the doors.
  • MQTT client source for C, C++, Java, JavaScript, Python, Lua
  • MQTT + MQTT-SN bridging broker for embedded systems. 
  • MQTT-SN “MQTT for Sensor Networks” is designed for WSNs and mesh networks using datagrams instead of socket.  It works over DSRC, ZigBee, 6LoWPAN, LTE Advanced p2p, UDP, et al.  Like MQTT it is open standard, open source and royalty free.  Some think it can greatly improve V2V / V2X by making it publish/subscribe based and transport independent (like the military’s).  The meshing capabilities and ability to propogate topics and messages across the mesh w QoS is really quite amazing.   I’m trying to get permission to youtube the videos form the field trials.
  • MQTT Sensor Fabric secure WSN mesh used by US and UK military uses RSMB which is now open source per above.
  • MQTT 3.1 spec
  • MQTT 3.1.1 spec draft
  • MQTT-SN 1.2 spec 
  • Q:  Is MQTT security better or worse than HTTPS?  A:  Neither.  It is the same.  They are both socket protocols that rely on TLS/SSL connection level encryption.  For example to the cloud I often suggest TLS 1.2 w mutual authentication using “signed at the foundry” keys built into the chipset.

Latency is a Driver Distraction issue

Latency is a Driver Distraction issue. Faster communications, faster decisioning saves lives.  There are many automakers and tier-1 suppliers working with me on that. Obviously there are also the other values it brings such as having a better driver experience and a better owner experience. For example being able from your smartphone to find my car, unlock my car, etc. with “key fob response time”, i.e. sub-second, faster than you lift your finger off the button.  Which is more impressive after you learn that the connected car systems out there today are typically yielding 15 to 90 second response times.  One luxury German brand for example, when you click the smartphone app to “find car” brings up a dialogue saying “Finding your car.  This may take a few minutes.”  And it does.

Somewhere you can see the alternative is the faster connected car driver experience that Sprint Velocity is showing at the upcoming LA Auto Show.


MQTT Sensor Fabric – Everything you were afraid to ask DARPA, UK MoD, Army Research Lab

Everything you ever wanted to know about creating a military-grade sensor mesh.  And a few things you’d rather not think about..

I understood (or was told) that I can not discuss this.  But info is now declassified and readily available if you know what you’re looking for.

There are lots of smart applications to exploit this technology to improve quality of life.  Think first responders, disaster relief, agriculture, healthcare and yes even connected car.  And just like in V2V, V2X safety scenarios, low latency & reliability matters.  MQTT is engineered for low bandwidth and unreliable networks providing fast reliable communications under the worst possible conditions and is RF agnostic.

And MQTT is very low power.  Which is critical in the battlefield.  Every pound of battery a soldier carries is a pound of ammunition and medical supplies that they must leave behind.  Brig. General Kevin Nally (CIO US Marine Corp) says battery requirements have increased 1,290% and reducing power consumption is a priority.  Clearly this is an important matter since dead batteries can mean dead soldiers.

Videos of the field trials are fascinating.  Now if I can just get permission to post those ..

MQTT Sensor Fabric is an inherently resilient architecture for the field with no single point of failure.  And interoperable across coalition forces.  That’s what open standards can do for you.  You can download the MQTT Sensor Fabric source code here.   Everything that follows is now public so I am disclosing nothing:

Coalition Sensor Interoperability and Data/ Information Sharing via the ITA Sensor Fabric and Policy Management Toolkit

This unprecedented technology enables coalition sensor assets to be integrated and networked, and data/information to be shared and disseminated under distributed policy‐control to support dynamic coalition operations . ARL is collaborating with the United Kingdom Ministry of Defence to develop this unique coalition capability for Network Enabled Operations with distributed intelligence, surveillance and reconnaissance (ISR) assets and users at the edge of the network . The collaborative effort leverages two key technology components developed within the U .S .‐U .K . International Technology Alliance program, Sensor Fabric and Policy Management Toolkit. The fabric is a flexible middleware infrastructure for sensor discovery, access and control, and data consumability . The policy tools perform a variety of distributed management functions within the fabric infrastructure such as sensor/platform command and control, data access control, and filtering of data/information. In June 2011, this coalition technology was successfully demonstrated with disparate ground sensors and aerial platforms in an operational environment at Camp Roberts, Calif . In 2012, the ITA team will integrate and demonstrate the technology as part of a U .K . Persistent Wide Area Surveillance (PWAS) capability concept demonstration . The technology development effort is funded by the Office of the Secretary of Defense Coalition Warfare Program .  page 28

The Information Fabric is a two-way message bus and set of middleware services connecting all of the network’s assets to each other and to users, facilitating universal access to intelligence data from any point, and maximizing its availability and utility to planning services, analysis applications (including fusion algorithms and agents), and human analysts. The Fabric leverages a publish/subscribe messaging model with multi-hop capabilities, ensuring that messages are propagated efficiently, without duplication, and with the minimum use of valuable network bandwidth.

Screen Shot 2013-05-14 at 9.39.02 AM

Click to access 09_Wagget.pdf

Screen Shot 2013-05-14 at 9.30.56 AM

Screen Shot 2013-05-14 at 9.14.30 AM

Screen Shot 2013-05-14 at 9.14.01 AM Screen Shot 2013-05-14 at 9.13.38 AM Screen Shot 2013-05-14 at 9.13.24 AM

Click to access acita2008a.pdf

Screen Shot 2013-05-14 at 9.15.16 AMScreen Shot 2013-05-14 at 9.15.32 AMScreen Shot 2013-05-14 at 9.15.58 AM

Click to access milcom2008.pdf

Screen Shot 2013-05-14 at 9.44.42 AM

Screen Shot 2013-05-14 at 9.44.13 AMScreen Shot 2013-05-14 at 9.45.04 AM Screen Shot 2013-05-14 at 9.44.53 AM  Screen Shot 2013-05-14 at 9.44.31 AM

Click to access KSCO-2012-Pham-Sensor%20-Data-Info-Sharing.pdf

Screen Shot 2013-05-14 at 9.37.24 AMScreen Shot 2013-05-14 at 9.35.28 AM

Screen Shot 2013-05-14 at 9.37.24 AM

Click to access ie.10.pdf

Screen Shot 2013-05-14 at 9.18.24 AMScreen Shot 2013-05-14 at 9.19.06 AMScreen Shot 2013-05-14 at 9.19.25 AMScreen Shot 2013-05-14 at 9.19.43 AMScreen Shot 2013-05-14 at 9.20.36 AMScreen Shot 2013-05-14 at 9.21.48 AMScreen Shot 2013-05-14 at 9.23.14 AM

Click to access phd.pdf

Click to access The_ITA_Sensor_Fabric_V4.pdf

Click to access Policy%20Enabled%20ITA%20Sensor%20Fabric.pdf

Click to access ITA%20Sensor%20Fabric.pdf

Click to access service-composition-acita2010.pdf

Click to access 12%20-%20Sensor%20Assignment.pdf

Click to access gdemel-8389-29.pdf

Click to access ITA_Newsletter-Vol_01-Issue_01-March_2008.pdf

Click to access Norman-ITA.pdf


tangentially related but fascinating:

Screen Shot 2013-05-14 at 11.21.10 AM

Screen Shot 2013-05-14 at 11.18.37 AM

Click to access 2012_TOSN_CB.pdf

REST is for sleeping. MQTT is for Mobile

REST is designed around a simple request/response model.  So you ask “did my account balance change” and the response is returned “no it did not“.  So you check again a few minutes later, and get the same response.  Sound like a silly example?  Actually we’ve learned that it is a very real issue that many customers obsessively check throughout the day, as many as 60 times inflicting a load on back-ends that they weren’t designed for.

Compounding the issue for mobile apps is REST via HTTP which wasn’t designed to work on mobile networks.  HTTP on mobiles is a bit heavy, fragile and slow and drains batteries quickly.

So notifications, isn’t that what Google and Apple Push are for?  Well sure, up to a point.  But there are some serious issues there.  They offer no quality of service, really don’t have much in the way of guaranteed messaging. The practical result customers see is that notifications arrive quickly, late or not at all.  And it is the not at all that is particularly troubling because there is  no way for the sending party to know whether it was delivered.  I hear a lot of frustrations from customers over this with one telling me “Sure, it is great to find out there is a new level of Angry Birds but it isn’t anything I can run my business on“.

So isn’t there smartphone technology that solves this nicely?  Not really.  But there is an obscure machine-to-machine protocol that does.  Andy Stanford-Clark and Arlen Nipper invented MQTT to solve a problem they had: how to do reliable messaging over unreliable networks?  In an industrial environment with computationally”challenged” devices, with restricted power due to solar or battery power, on extremely low bandwidth and often brittle RF communications including satellite.  It had to work reliably, it had to use very few computational cycles, consume trivial power and could not hog what little bandwidth there was.  And it had to be pub/sub so to break the cycle of violence being inflicted by heavy unnecessary workloads and bandwidth consumption due to all the request/response polling-based monitoring & control systems that were in place at that time.

So Arlen & Andy developed a very simple, extremely efficient publish/subscribe reliable messaging protocol and named it MQ Telemetry Transport (MQTT).   A protocol that enabled devices to open a connection, keep it open using very little power and receive events or commands with as little as 2 bytes of overhead.  A protocol that has the things built in that you need for reliable behavior in an unreliable or intermittently connected wireless environments.  Things such as “last will & testament” so all apps know immediately if a client disconnects ungracefully, “retained message” so any user re-connecting immediately gets the very latest business information, etc.

Fast forward a few years to Facebook’s ambition to create a communications platform for hundreds of millions of people.  A platform that would have to have a dramatically better, more responsive user experience than what others were providing.  Lucy Zhang, the engineer in charge was experienced enough to know that the 3 key issues were going to be:

  • latency – how to get faster phone-to-phone communications
  • battery – and do that without killing batteries
  • bandwidth – or sucking up the user’s available bandwidth

While chatting over drinks in Barcelona, a former Facebook  employee told me that the brilliant bit of lateral thinking that Lucy did was instead of trying to brute force the problem with HTTP or HTTP-based protocols like XMPP (which inherit all of HTTP’s issues on mobile), she adopted an obscure m2m protocol “MQTT”

With the result being that Facebook gained a communications platform with a compelling user experience (well, good design had something to do with it too) that garned great reviews, tremendous stickiness, and  today has 680M mobile users, projected to hit 1B within a year.

Verizon Wireless this spring had their engineers do an engineering assessment of the top applications on their network.  Their assessment was that Facebook Messenger using MQTT is 5 stars for security, 5 stars for battery, and 4 stars for bandwidth.  Which is impressive considering the bandwidth measure is compared to things like Angry Birds which use zero bandwidth.

Stephen Nicholas did a fascinating apples to apples comparison of MQTT vs HTTPS on Android, 3G and WiFi which you can read here.  The 3G results are quite interesting:

  • 93x faster throughput
  • 11.89x less battery to send
  • 170.9x less battery to receive
  • 1/2 as much power to keep connection open
  • 8x less network overhead

So if you follow the NY Times or any of the IT trade magazines you may have seen a lot of news about MQTT in the past 2 weeks.  It is now in progress to become an OASIS M2M standard.  And IBM has open sourced all of its MQTT source code via including the new HTML5 MQTT over WebSocket JavaScript which enables MQTT in any HTML5 container including mobile browsers, desktop browsers, vehicle infotainment, consumer electronics.  I’m doing 13,000 messages a second on my iPad with a pure HTML5 app developed in Worklight.  I don’t think you could do that with HTTP.  You can also google for “MA9B” to download zip from IBM of source code for a 1/2 dozen different mobile clients.

MQTT is small footprint, efficient, low power on the device.  So what happens when you flip that around to the data center or mobile operator side?  And drive it with new acceleration technology from IBM Labs?  You get something very dense, very green, and very fast.  As in concurrently connect as many mobiles, cars, or devices as 1,000 web servers with just 1 rack.  Product Management is OK when I say 21M concurrently connected things per rack.  They frown when I say the actual number.  And messaging throughput?  With the Beta firmware we were getting 273M mobile messages/sec per rack.  The performance  results we see in the GA firmware are “higher” and will be published soon.  End-to-end, app-to-app on fast network with this appliance using MQTT is in µs not milliseconds.  That’s why we’re now using this appliance and MQTT for high-speed Big Data analytics, driving millions of low-latency analytics decisions per second.

More on that at with videos here and here.

I’m quite fanatical about this notion that response time = revenue, response time = business performance, and most importantly response time saves lives.  It isn’t about machines anymore folks, “Connected Life” is you, me and everything around us and how it all interacts.  And anytime there is a human in that interaction, then latency matters very much.

Well, that’s probably enough of a diatribe for one day.  I have an IBM internal blog which is “mobile musings”, my intention here was “mobile bit” i.e. bitten, smitten but also a small piece, short.  I’ll hew closer to that in the future.

– Joe