MQTT Sensor Fabric – Everything you were afraid to ask DARPA, UK MoD, Army Research Lab

Everything you ever wanted to know about creating a military-grade sensor mesh.  And a few things you’d rather not think about..

I understood (or was told) that I can not discuss this.  But info is now declassified and readily available if you know what you’re looking for.

There are lots of smart applications to exploit this technology to improve quality of life.  Think first responders, disaster relief, agriculture, healthcare and yes even connected car.  And just like in V2V, V2X safety scenarios, low latency & reliability matters.  MQTT is engineered for low bandwidth and unreliable networks providing fast reliable communications under the worst possible conditions and is RF agnostic.

And MQTT is very low power.  Which is critical in the battlefield.  Every pound of battery a soldier carries is a pound of ammunition and medical supplies that they must leave behind.  Brig. General Kevin Nally (CIO US Marine Corp) says battery requirements have increased 1,290% and reducing power consumption is a priority.  Clearly this is an important matter since dead batteries can mean dead soldiers.

Videos of the field trials are fascinating.  Now if I can just get permission to post those ..

MQTT Sensor Fabric is an inherently resilient architecture for the field with no single point of failure.  And interoperable across coalition forces.  That’s what open standards can do for you.  You can download the MQTT Sensor Fabric source code here.   Everything that follows is now public so I am disclosing nothing:

Coalition Sensor Interoperability and Data/ Information Sharing via the ITA Sensor Fabric and Policy Management Toolkit

This unprecedented technology enables coalition sensor assets to be integrated and networked, and data/information to be shared and disseminated under distributed policy‐control to support dynamic coalition operations . ARL is collaborating with the United Kingdom Ministry of Defence to develop this unique coalition capability for Network Enabled Operations with distributed intelligence, surveillance and reconnaissance (ISR) assets and users at the edge of the network . The collaborative effort leverages two key technology components developed within the U .S .‐U .K . International Technology Alliance program, Sensor Fabric and Policy Management Toolkit. The fabric is a flexible middleware infrastructure for sensor discovery, access and control, and data consumability . The policy tools perform a variety of distributed management functions within the fabric infrastructure such as sensor/platform command and control, data access control, and filtering of data/information. In June 2011, this coalition technology was successfully demonstrated with disparate ground sensors and aerial platforms in an operational environment at Camp Roberts, Calif . In 2012, the ITA team will integrate and demonstrate the technology as part of a U .K . Persistent Wide Area Surveillance (PWAS) capability concept demonstration . The technology development effort is funded by the Office of the Secretary of Defense Coalition Warfare Program .

http://www.arl.army.mil/www/pages/172/docs/ARL_Annual_Review_2011.pdf  page 28

The Information Fabric is a two-way message bus and set of middleware services connecting all of the network’s assets to each other and to users, facilitating universal access to intelligence data from any point, and maximizing its availability and utility to planning services, analysis applications (including fusion algorithms and agents), and human analysts. The Fabric leverages a publish/subscribe messaging model with multi-hop capabilities, ensuring that messages are propagated efficiently, without duplication, and with the minimum use of valuable network bandwidth.

Screen Shot 2013-05-14 at 9.39.02 AM

http://www.us-czic.org/000/Day-2-afternoon/09_Wagget.pdf

Screen Shot 2013-05-14 at 9.30.56 AM

Screen Shot 2013-05-14 at 9.14.30 AM

Screen Shot 2013-05-14 at 9.14.01 AM Screen Shot 2013-05-14 at 9.13.38 AM Screen Shot 2013-05-14 at 9.13.24 AM

http://users.cs.cf.ac.uk/A.D.Preece/publications/download/acita2008a.pdf

Screen Shot 2013-05-14 at 9.15.16 AMScreen Shot 2013-05-14 at 9.15.32 AMScreen Shot 2013-05-14 at 9.15.58 AM

http://users.cs.cf.ac.uk/A.D.Preece/publications/download/milcom2008.pdf

Screen Shot 2013-05-14 at 9.44.42 AM

Screen Shot 2013-05-14 at 9.44.13 AMScreen Shot 2013-05-14 at 9.45.04 AM Screen Shot 2013-05-14 at 9.44.53 AM  Screen Shot 2013-05-14 at 9.44.31 AM

http://ksco.info/ksco/ksco-2012/papers/KSCO-2012-Pham-Sensor%20-Data-Info-Sharing.pdf

Screen Shot 2013-05-14 at 9.37.24 AMScreen Shot 2013-05-14 at 9.35.28 AM

Screen Shot 2013-05-14 at 9.37.24 AM

http://www.cs.rpi.edu/~szymansk/papers/ie.10.pdf

Screen Shot 2013-05-14 at 9.18.24 AMScreen Shot 2013-05-14 at 9.19.06 AMScreen Shot 2013-05-14 at 9.19.25 AMScreen Shot 2013-05-14 at 9.19.43 AMScreen Shot 2013-05-14 at 9.20.36 AMScreen Shot 2013-05-14 at 9.21.48 AMScreen Shot 2013-05-14 at 9.23.14 AM

http://bourdenas.appspot.com/files/phd.pdf

https://www.usukitacs.com/papers/4835/The_ITA_Sensor_Fabric_V4.pdf

https://www.usukita.org/papers/3665/Policy%20Enabled%20ITA%20Sensor%20Fabric.pdf

https://www.usukitacs.com/papers/2930/ITA%20Sensor%20Fabric.pdf

http://www.cs.rpi.edu/~szymansk/papers/service-composition-acita2010.pdf

http://www.ieee-infocom.org/2009/demos/12%20-%20Sensor%20Assignment.pdf

http://homepages.abdn.ac.uk/g.demel/pages/myHome/publications/gdemel-8389-29.pdf

https://www.usukita.org/papers/3406/ITA_Newsletter-Vol_01-Issue_01-March_2008.pdf

http://scone.cs.st-andrews.ac.uk/cybersecurity/slides/Norman-ITA.pdf

itasensorfabricwmqtt

http://www.academia.edu/1502079/A_Dynamic_Infrastructure_for_Interconnecting_Disparate_ISR_ISTAR_Assets_the_ITA_Sensor_Fabric_CX

tangentially related but fascinating:

Screen Shot 2013-05-14 at 11.21.10 AM

Screen Shot 2013-05-14 at 11.18.37 AM

http://nesl.ee.ucla.edu/fw/documents/journal/2012/2012_TOSN_CB.pdf

REST is for sleeping. MQTT is for Mobile

REST is designed around a simple request/response model.  So you ask “did my account balance change” and the response is returned “no it did not“.  So you check again a few minutes later, and get the same response.  Sound like a silly example?  Actually we’ve learned that it is a very real issue that many customers obsessively check throughout the day, as many as 60 times inflicting a load on back-ends that they weren’t designed for.

Compounding the issue for mobile apps is REST via HTTP which wasn’t designed to work on mobile networks.  HTTP on mobiles is a bit heavy, fragile and slow and drains batteries quickly.

So notifications, isn’t that what Google and Apple Push are for?  Well sure, up to a point.  But there are some serious issues there.  They offer no quality of service, really don’t have much in the way of guaranteed messaging. The practical result customers see is that notifications arrive quickly, late or not at all.  And it is the not at all that is particularly troubling because there is  no way for the sending party to know whether it was delivered.  I hear a lot of frustrations from customers over this with one telling me “Sure, it is great to find out there is a new level of Angry Birds but it isn’t anything I can run my business on“.

So isn’t there smartphone technology that solves this nicely?  Not really.  But there is an obscure machine-to-machine protocol that does.  Andy Stanford-Clark and Arlen Nipper invented MQTT to solve a problem they had: how to do reliable messaging over unreliable networks?  In an industrial environment with computationally”challenged” devices, with restricted power due to solar or battery power, on extremely low bandwidth and often brittle RF communications including satellite.  It had to work reliably, it had to use very few computational cycles, consume trivial power and could not hog what little bandwidth there was.  And it had to be pub/sub so to break the cycle of violence being inflicted by heavy unnecessary workloads and bandwidth consumption due to all the request/response polling-based monitoring & control systems that were in place at that time.

So Arlen & Andy developed a very simple, extremely efficient publish/subscribe reliable messaging protocol and named it MQ Telemetry Transport (MQTT).   A protocol that enabled devices to open a connection, keep it open using very little power and receive events or commands with as little as 2 bytes of overhead.  A protocol that has the things built in that you need for reliable behavior in an unreliable or intermittently connected wireless environments.  Things such as “last will & testament” so all apps know immediately if a client disconnects ungracefully, “retained message” so any user re-connecting immediately gets the very latest business information, etc.

Fast forward a few years to Facebook’s ambition to create a communications platform for hundreds of millions of people.  A platform that would have to have a dramatically better, more responsive user experience than what others were providing.  Lucy Zhang, the engineer in charge was experienced enough to know that the 3 key issues were going to be:

  • latency – how to get faster phone-to-phone communications
  • battery – and do that without killing batteries
  • bandwidth – or sucking up the user’s available bandwidth

While chatting over drinks in Barcelona, a former Facebook  employee told me that the brilliant bit of lateral thinking that Lucy did was instead of trying to brute force the problem with HTTP or HTTP-based protocols like XMPP (which inherit all of HTTP’s issues on mobile), she adopted an obscure m2m protocol “MQTT”

With the result being that Facebook gained a communications platform with a compelling user experience (well, good design had something to do with it too) that garned great reviews, tremendous stickiness, and  today has 680M mobile users, projected to hit 1B within a year.

Verizon Wireless this spring had their engineers do an engineering assessment of the top applications on their network.  Their assessment was that Facebook Messenger using MQTT is 5 stars for security, 5 stars for battery, and 4 stars for bandwidth.  Which is impressive considering the bandwidth measure is compared to things like Angry Birds which use zero bandwidth.

Stephen Nicholas did a fascinating apples to apples comparison of MQTT vs HTTPS on Android, 3G and WiFi which you can read here.  The 3G results are quite interesting:

  • 93x faster throughput
  • 11.89x less battery to send
  • 170.9x less battery to receive
  • 1/2 as much power to keep connection open
  • 8x less network overhead

So if you follow the NY Times or any of the IT trade magazines you may have seen a lot of news about MQTT in the past 2 weeks.  It is now in progress to become an OASIS M2M standard.  And IBM has open sourced all of its MQTT source code via Eclipse.org including the new HTML5 MQTT over WebSocket JavaScript which enables MQTT in any HTML5 container including mobile browsers, desktop browsers, vehicle infotainment, consumer electronics.  I’m doing 13,000 messages a second on my iPad with a pure HTML5 app developed in Worklight.  I don’t think you could do that with HTTP.  You can also google for “MA9B” to download zip from IBM of source code for a 1/2 dozen different mobile clients.

MQTT is small footprint, efficient, low power on the device.  So what happens when you flip that around to the data center or mobile operator side?  And drive it with new acceleration technology from IBM Labs?  You get something very dense, very green, and very fast.  As in concurrently connect as many mobiles, cars, or devices as 1,000 web servers with just 1 rack.  Product Management is OK when I say 21M concurrently connected things per rack.  They frown when I say the actual number.  And messaging throughput?  With the Beta firmware we were getting 273M mobile messages/sec per rack.  The performance  results we see in the GA firmware are “higher” and will be published soon.  End-to-end, app-to-app on fast network with this appliance using MQTT is in µs not milliseconds.  That’s why we’re now using this appliance and MQTT for high-speed Big Data analytics, driving millions of low-latency analytics decisions per second.

More on that at ibm.com/MessageSight with videos here and here.

I’m quite fanatical about this notion that response time = revenue, response time = business performance, and most importantly response time saves lives.  It isn’t about machines anymore folks, “Connected Life” is you, me and everything around us and how it all interacts.  And anytime there is a human in that interaction, then latency matters very much.

Well, that’s probably enough of a diatribe for one day.  I have an IBM internal blog which is “mobile musings”, my intention here was “mobile bit” i.e. bitten, smitten but also a small piece, short.  I’ll hew closer to that in the future.

Best,
– Joe