Deterministic, Time-Sensitive Networking for IoT
Having read my post on the Path Computation Element, “Pixie Dust and Unicorn Stuff,” and finding only one small error [see the footnotes for details], our resident PCE expert graciously informed me that “you got off pretty lightly there.” Which, of course, I had. He also asked if I was aware of the deterministic networking application for the PCE, which I hadn’t covered. Without even having to lie, I responded that I was, having joined the IETF DetNet Working Group mailing list back in May 2015 -- even prior to the BoF attaining working group status in early October 2015. Though not a liar, I am a complete idiot, in that I had totally forgotten about it when it came to writing that post. While I thought this was a perfect opportunity for him to write a guest blog on the topic, he reminded me that he has real work to do and couldn’t spare the time. A great pun, considering the topic in question, and a statement I couldn’t argue with. I hate people with real jobs.
Now, I know it’s not unusual to be to be 175 words deep into one of my posts and still not have a clue what I’m talking about… and serving up the Layer 2-focused IEEE Task Group designation -- 802.1 Time Sensitive Networks (or simply TSN) -- probably won’t help much. Call it what you will, it’s probably one of the most important networking initiatives you’ve never heard of… or remembered, in my case. DetNet and TSN represent another collaborative effort to solve one of the most elusive networking issues, and one that is becoming increasingly critical in the age of IoT, the smartgrid and infrastructure optimization or bin packing. (Yay! Bin packing!)
Built from and enhancing IEEE 1588, the Precision Time Protocol (PTP), a standard very familiar to anyone who has set foot in a central office or been involved in high-frequency trading system, TSN is simplified for Layer 2 Ethernet networks, but extended to support other “coordinated shared networks” like Wi-Fi. Timing and Synchronization for Time-Sensitive Applications in Bridged Local Area Networks is detailed within 802.1AS, though the working group has also defined VLAN extensions under the auspices of the 802.1Q Working Group. From its inception, early in the decade, TSN set out to solve local area networking issues foreign to those of us more familiar with classic (non-IoT) network operator infrastructure. Most notably, this includes Audio / Video Bridging (AVB) systems, from which the TSN task group was born (in November 2012) once it was recognized that the standards being developed were applicable to applications beyond that of the live content production studios, to more generic industrial automation and control systems.
Why is TSN critical in AVB and industrial applications? Audio technologies like Auto-Tune induce compute delays, but musicians must hear themselves in their earpiece monitors with less than 10ms delays, or else they become confused. With the inline DSP and mixer delays averaging 8ms, there’s just 2ms left for network latency. While any media production house -- even if the content is for future playback -- will have low tolerances to delay and packet loss, the proliferation of distributed residential entertainment systems (wireless speaker arrays and the like) literally brings the problem home. With our ability to easily detect lip-sync issues exceeding -125ms and +45ms, even the smallest disparities in the timing between these decoupled systems can be detrimental to the viewing experience. Generally speaking (excuse the pun), the delay variation between speakers in an array should be less than 10μs, but manufacturers and audiophiles (aka weenies1) would prefer a number closer to 1μs.
ITU-R BT.1359: Detectability and acceptability thresholds2
In the manufacturing world, things escalate quickly from simple annoyances affecting individuals to critical safety issues that can put production lines and lives at risk. While high-end machines are designed to fail-safe on as few as two consecutive packet delays or losses, doing so on a regular basis will have a detrimental effect on the manufacturing business. Architecturally speaking, therefore, it’s better to plan for success in these machine-to-machine applications than to apologize for such failures.
For those of you thinking that these examples are pretty lame, my goal was simply not to be overly dramatic… well, outside directly correlating packet losses to lost lives, at least. The applications for time-sensitive networking are much broader, as you might image, including everything from energy production and distribution to transportation, ranging from automotive to space exploration. Yes – it took me this long to get to connected cars, but the problem is real: Lane assists and proximity detectors for breaking and blind spot detection all decrease in value if, after combining the input from multiple sensors, alerts and autonomous system controls are delayed.
The goal of TSN is therefore to mitigate or eliminate buffering delays and packet loss or retransmissions of such mission-critical packets within localized environments. There are three standardization areas associated with TSN, ranging from the base timing and synchronization specification (802.1AS-Rev(ision)) to forwarding and queuing (802.1Qav/802.1Q Clause 34) and stream reservation/SRP (802.1Qat/802.1Q Clause 35). Bringing IEEE 1588 in the Ethernet (802.3) and Wi-Fi (802.11) bridged LAN domains, 802.1AS employs the same (grand) master/slave clock arrangement, but a generalized variant of the precision time protocol (aka gPTP). A simplified Best Master Clock Algorithm (BMCA) selects… well… I think it’s obvious what it selects. The chosen grandmaster then propagates time sync messages to all other bridges, with a goal of achieving syntonization (operating at the same frequency) to within +/-500ns between any two nodes, up to maximum of seven network hops or six switches.
Applying quality of service to a stream first requires us to identify a stream and then reserve the appropriate resources to support it. This role falls to the stream reservation protocol (SRP), which, riding on top of the existing Multiple Registration Protocol (MRP), defines a stream ID as the transmitter's MAC address with a 16-bit appendix. SRP also includes the all-important characteristics and prioritization information, such as the maximum frame size and rate (Tspec) used for traffic shaping; one of three priority classes (control traffic, emergency/Class A and non-emergency/Class B); and finally data regarding the maximum amount of end-to-end latency the stream can incur. This is fixed on the high end at 2ms for Class A traffic and 50ms for Class B -- again, over seven hops or six switches. Meanwhile, 802.1av set out to define how packets are handled by the intermediate switches, outlining the implementation of a (pretty) unique credit-based fair queuing (shaping) algorithm,3 which is generally thought to be more computationally efficient than classic fair queuing techniques. Worst case, a Class A or B packet might be delayed at every bridge by the ongoing transmission of a maximum-sized frame belonging to a best-effort traffic class.
With their Working Group Charter approved by the The Internet Engineering Steering Group (IESG) on Oct. 1, 2015, DetNet set out to formally expand the scope of TSN into the Layer 3 domain. This was primarily because of the growing need to extend the data plane and control plane of elements, demanding very strict quality-of-service guarantees, beyond the local area network. The most obvious application here is telemetry systems for autonomous vehicles, but there are numerous IoT uses, such as industrial M2M, where both data and signaling protocols must be transported with minimal delay and delay variations, with consequences far more dire than a short audio disruption or a few bad video frames.
Whether rotating, static, fixed or fast-moving, the potentially massive number of network-connected sensors, actuators and control loops that make up IoT applications are often perfect candidates for DetNet, given their need for reliability and accuracy and their low-cost, constrained nature, which limits on-board buffering for reassembling unsequenced packet flows. There are, however, many issues to resolve when it comes to implementing DetNet, not least of which are the data and control plane protocols. Like TSN, here is a need for resource reservation and strict priority queuing, but there’s also the need for explicit routing and packet replication/deletion. The DetNet working group divides the data plane into two distinct parts: The service layer and the transport layer. There are a broad range of protocols and encapsulation options under consideration for the service layer but MPLS derivatives, like EVPN, appear to have the edge. Similarly, native MPLS IP encapsulation seems to be in pole position in the transport layer, There are, however, some protocols in both layers that might make the grade owing to some unique properties the possess.
DetNet Data Plane: Candidate Protocols4
Take the IETF BIER working group initiative, for example… [pause while others insert a well-oiled pun]. Bit Indexed Explicit Replication was originally conceived as a highly efficient alternative to existing multicast solutions, which have long been regarded as absolutely terrible. Employing existing unicast discovery mechanisms, rather than dedicated multicast ones [did I mention that they are awful?], both Base BIER and new traffic engineering extensions (BIER-TE) can meet the need for flow replication without a large amount of overhead.
From a control plane perspective there are, as usual, two primary high-level candidates once flows have been appropriately identified. DetNet could employ a distributed hop-by-hop (aka IntServ/RSVP) technique or a simpler centralized model akin to today’s popular “software defined” networks.5 OK -- well, there are three, actually, as a hybrid approach could also be used, alleviating the wait time as downstream nodes are configured. Either way, this meets the explicit routing requirement, but to do it well -- fully utilizing infrastructure capacity [i.e. Bin Packing. Yay! Bin packing!] while eliminating the need to oversubscribe links and network functions -- you need a Path Computation Element, which is why the DetNet working group is initially focusing on that.
Indeed, PCE gets a lot of attention in “Deterministic Networking Use Cases,”6 one of the first draft IETF documents being developed by the working group. This is primarily due to the IoT connection, where the PCE is already a critical component of IETF IPv6 over the TSCH mode (6TiSH) infrastructures, which built on IEEE 802.15.4e Time-Slotted Channel Hopping (TSCH), of course. The deterministic demands of 6TiSH could be fulfilled by work under way in the DetNet working group, and a hierarchical PCE (H-PCE) implementation can ensure that a complete end-to-end deterministic forwarding path across both the TSCH and TSN domains can be computed.
If you want to know more about the PCE operating in Low Power Lossy Networks (LLN), look no further than… errr… here: Pixie Dust and Unicorn Stuff. If it feels like this post has now come full circle, you should have been the one writing it. My head is spinning. Once again, though, this does prove how incestuous this business is and how interrelated technologies may be, even though it’s not always apparent on the surface to laymen like myself. This, in all fairness, is probably why I landed in marketing, rather than getting a real job.
To learn more about our Path Computation Element, please contact us.
1) I apologize for such a general and quite frankly unnecessary characterization of those enamoured with the finer details or sound reproduction. Feel free to tweet your disdain to @dredgie #ahole.
2) ITU-R BT.1359 | Relative Timing of Sound and Vision for Broadcasting | https://www.itu.int/dms_pubrec/itu-r/rec/bt/R-REC-BT.1359-0-199802-S!!PDF-E.pdf
3) 802.17 Resilient Packet Ring (RPR) networks also employ credit-based fair queuing / traffic shaping.
4) https://tools.ietf.org/html/draft-dt-detnet-dp-alt-00 as of this post.
5) The cool kids call this ‘SDN’. I’m sure it’s just a fad.
6) https://tools.ietf.org/html/draft-ietf-detnet-use-cases-09 as of this post.
Simon is the Director of Technical Marketing and a man of few words.