![[INET 2000]](/web/20160103040155im_/http://www.isoc.org/inet2000/cdproceedings/inet00sm.gif)
![[ Up ]](/web/20160103040155im_/http://www.isoc.org/inet2000/cdproceedings/up.gif)
![[Prev]](/web/20160103040155im_/http://www.isoc.org/inet2000/cdproceedings/prev.gif)
![[Next]](/web/20160103040155im_/http://www.isoc.org/inet2000/cdproceedings/next.gif)
With growing volume in data traffic, SONET/SDH networks must now carry a significantly large number of data packets in addition to traditional T1/T3 channels. Current protocols on these networks, however, do not allow transmission of different data types on a single fiber (or wavelength) without seriously compromising timing relationships and/or wasting available bandwidth. This paper describes a Hybrid Data Transport (HDT) protocol that allows transmission of Fractional T1 (in increments of DS0, starting at DS0 bandwidth), T1/T3, ATM, IP and any other protocol data in a single SONET frame, allowing maximum bandwidth usage on a fiber. With a unified packet framing across a mix of SONET and non-SONET networks, dynamic bandwidth provisioning on a packet-by-packet basis, and hybrid data mixing capability, this protocol maximizes bandwidth usage and yields major cost savings in fiber infrastructure, equipment, and operation.
SONET was designed to efficiently carry Plesiochronous Digital Hierarchy (PDH) telephony channels such as T1/T3. This was easily achieved by dividing its payload area in fixed slots called virtual tributaries (VT). Nowadays, wide-area and metropolitan-area network (MAN) communications rely on short- and long-haul fiber optic networks to transport data and telephony traffic. As the volume of packet data on optical networks has gone up, many methods have been developed to force-fit packet data onto the SONET framework, which is better suited for PDH transport. Using current protocols for simultaneous support of packet network and PDH traffic on SONET results in inefficient use of precious optical fiber bandwidth.
This paper refers to SONET for all of its discussions, with similar implications for SDH (Synchronous Digital Hierarchy).
Different methods of supporting data traffic (such as IP and ATM) over SONET networks rely on one of the following two core technologies:
In each of these methods, a unique Path Signal Label (PSL) value in the POH (Path Over Head) field of the SONET frame identifies the type of data within the payload area. Because a PSL value identifies the contents of the entire SONET payload envelope, only one type of transmission can be supported at a time within a SONET payload. To understand ways of bandwidth maximization, it is important to discuss the architecture and limitations of current approaches for packet transport on SONET. A brief overview of these technologies is presented here.
The SONET payload area is filled with IP (or other packet-oriented protocol) by using POS packet protocol (RFC2615 [3]) with packets delimited by 0x7E at both ends of a packet. Many packets can be put inside a single SONET Synchronous Payload Envelope (SPE). This method, however, can only support variable-length packet protocol such as IP and cannot transport T1/T3 channels or ATM cells. If a SONET fiber (or wavelength) carries POS frames it cannot support other SONET frames containing T1/T3 or ATM cells. This is because each frame containing POS would introduce a 125-microsecond delay (interval for one SONET frame), and this delay is not acceptable for T1/T3 circuits and most real-time services using ATM cells.
In traditional SONET networks, a frame originating from a node must travel all around the ring until it is removed by the sending node, resulting in a waste of bandwidth. One such method, Spatial Reuse Protocol (SRP) [8], improves bandwidth through bandwidth reuse by removing a SONET payload after the destination node has received it.
SRP and other similar protocols treat a SONET ring as a big, shared medium (as opposed to the usual point-to-point approach) and use Ethernet addressing to address nodes on the ring. The destination node terminates the packet, and other nodes down the network can use the released bandwidth. These protocols, however, only work with packet-oriented protocols, and leave issues such as guaranteed bandwidth, QoS, to the network layer protocols such as IP. Also, as with POS, a ring carrying these protocols cannot support T1/T3 or real-time ATM traffic.
Another way to fully utilize SONET bandwidth is to fill the payload area with ATM cells using a technique known as ATM VP (Virtual Path) multiplexing [10], where packets are encoded in ATM cells and then put inside a SONET SPE. In this method, ATM Circuit Emulation Service (CES) is normally used for carrying PDH traffic, and data packets are transmitted using multiprotocol encapsulation over ATM encapsulation [4] or interworking (such as Frame Relay-ATM) protocols. Small, fixed-size ATM cells are also easy to work with cross-connect or add/drop multiplexer devices to switch traffic to different destinations.
However, Operations, Administration, and Maintenance (OAM) activities of the ATM network are different from that of SONET, and management of the two becomes difficult. Similarly, ATM network routing and switch-to-switch signaling data paths are different from IP network routes, resulting in additional network operational complexity.
It has been found in a study [2] that a good percentage of network traffic consists of IP packets with small packet sizes of around 40 bytes. With IP over ATM (RFC 2684 - Multiprotocol Encapsulation over ATM AAL5), payload size slightly exceeds what can fit inside a single ATM cell. This results in transmission of two ATM cells, with the second cell mostly containing ATM overhead and stuffing bytes. This, with other ATM overhead, means having to allocate extra SONET bandwidth for data transport.
In addition, using ATM for sending different services requires implementation of ATM transport protocols and interworking for all related protocols (such as IP-over-ATM, frame relay-ATM internetworking, circuit emulation, etc.) in networking devices.
In this method, some VT channels are used to transport data such as ATM and IP while others continue to carry the usual PDH traffic. Because an individual virtual tributary has a limited bandwidth, extra mechanisms have to be used to send data packets of higher bandwidth using multiple tributaries.
Virtual concatenation [9] combines several VT channels into bigger virtual pipes to carry higher bandwidth traffic. With this protocol, while some virtual tributaries carry T1/T3 data as usual, others are concatenated for transport of higher bandwidth data traffic such as 10/100Mbps links.
However, virtual concatenation allocates a fixed bandwidth for packet traffic, and it cannot dynamically adjust bandwidth usage on a packet-by-packet basis. While it is possible to change the concatenated bandwidth through software in a reasonably short time, the bandwidth utilization is poor because bandwidth usage on packet data channels changes from packet -to packet.
Another problem with virtual concatenation is that while one virtual pipe may be overloaded with traffic, others may be underutilized; and virtual concatenation cannot dynamically adjust network loads on different channels.
Figure 1: Inverse Multiplexing
In inverse multiplexing, data from a high bandwidth source such as a 10Mbps Ethernet is sent over multiple VT1.5 tributaries (seven VT1.5 are required for transporting a 10Mbps LAN traffic) using inverse multiplexing protocol encapsulation. The data is later recovered by combining individual channels.
Virtual tributaries are of a fixed bandwidth, and their use as a single entity or a group creates another (higher) fixed bandwidth channel.
Figure 2: Lack of data mix in a SONET payload results in poor bandwidth
use
In a SONET payload, only one type of data (such as ATM cells, POS, or PDH traffic), identified by a unique PSL (Path Signal Label) C2 byte value in the POH, can be carried at one time.
This lowers bandwidth usage even for networks carrying only delay-tolerant data traffic. In the figure [xx], for instance, nodes at different points in the SONET ring have different types of data to send on the network. Assume, for instance, that node A has ATM cells to transmit on the SONET ring. It creates a SONET SPE, sets PSL value (in the POH) for ATM cells and sends the SONET frame down the link. Even if the SPE is only partially full, an entire SPE frame must be transmitted. If node B has IP packets to send, it cannot use the partially filled SONET frame received from A to add POS packets because the PSL value identifies only ATM cells. It must wait out an entire 125 microseconds for the partially filled packet to go by [Figure 4].
Statistically, a significant percentage of network traffic consists of IP packets that are quite small in size [2]. Because higher-speed concatenated SONET frames are large in size, in many cases a SONET SPE may not be completely filled with packets or cells at the origin. A SONET SPE can only transport one type of data, preventing other downstream nodes from entering any other type of data to the SPE, and the SONET link may be heavily underutilized
With current SONET protocols, separate fiber links (or wavelengths) are needed for transporting ATM, PDH traffic, and variable-length packets such as IP (POS).
Figure 3: Multiple fiber networks needed for different data types
Figure 4: Inserting different data types in SONET
PDH channels are created by strict timing relationships between consecutive STS-1 frames (to form a superframe). Any intervening STS-1 frames containing ATM or POS packets will violate this timing significantly.
In addition, SONET fiber links containing ATM cells cannot carry POS frames because ATM cells frequently carry QoS-sensitive data such as CES or multimedia traffic, and introduction of SONET frames containing POS will cause significant delays (125 microseconds for each POS frame inserted in the link).
To understand bandwidth limitations of SONET/SDH and data-over-fiber network, consider a typical access network applications as shown in Figure 5.
Figure 5: Access network using different types of data
In this legacy network such as shown above, SONET rings must employ an architecture that takes a compromise path to support data and PDH traffic. Bandwidth on different SONET rings is either overcommitted but never used or is underprovisioned and causes traffic congestion. All PDH traffic goes over slotted blocks using virtual tributaries, and any data traffic must be sent on remaining virtual tributaries. One or more of these rings may be underutilized while another could be saturated, and there is no way of adjusting traffic using current protocols.
In the other approach where the entire payload is used for sending data traffic (such as for network F, or for A and G without DLC support), only one type of packet is sent because of a single PSL value restriction as described earlier. This results in a major waste of bandwidth because an entire fiber (or wavelength in the case of WDM) must be allocated for just one type of traffic.
With the use of the proposed HDT protocol it is possible to provision a single fiber to use its full capacity for transport of different types of traffic and to manage its bandwidth usage dynamically on a packet-by-packet basis. This architecture features hybrid transport of different types of data, spatial reuse of bandwidth, allocation of PDH bandwidth in 64kbps increments, and seamless operation over point-to-point and ring networks with SONET/SDH and/or direct data-over-fiber configurations.
In addition to powerful bandwidth management, MPLS support in HDT allows packets to be sent across dissimilar networks without internetworking translations or protocol awareness at intermediate nodes. For instance, ATM cells can be transported across non-ATM networks (such as frame relay, PPP, or data-over-fiber configurations) without performing any protocol translations (frame relay-ATM interworking is still required for accessing an ATM node from within a frame relay cloud, and vice versa). This results in savings both in data routing overheads and equipment costs.
With this protocol, IP (or other protocol) packets, POS, ATM cells, G.702-based PDH (T1/T3), SRP, frame relay, and other types of data can be mixed inside a SONET payload dynamically and sent on a single fiber (see figure 6).
Because of its robust scrambling and unified packet transport over ring and point-to-point networks it is ideally suited for any mix of SONET and non-SONET configurations such as point-to-point WDM networks or a hybrid mix of point-to-point and ring topologies.
Figure 6: Single fiber needed for all data types
Figure 7: HDT frames inside a SONET SPE
Figure 8: HDT frame structure
Traditional SONET framing carries only one type of payload identified by a unique PSL value for each type. HDT extends this identification to a per-packet level by marking every packet individually.
In HDT, a user data packet is encapsulated with 32-bit Payload Header (PH) that identifies the type of the packet and provides other information about the packet. This whole packet is then framed using a four-byte Simple Data Link (SDL) [1,6] framing header. This and other HDT packets are then put inside a SONET SPE to carry a wide mix of fixed and variable bandwidth data (see figure 7).
The SDL framing protocol prefixes a payload with a 32-bit word, 16 bits of which hold the length of the packet. The other 16 bits contain CRC (Cyclic Redundancy Check) for the length field. SDL provides a robust CRC-16 based framed boundary delineation mechanism (compared with the traditional 0x7E delimiter) that solves all current POS issues like robustness in bad BER conditions, variable packet size expansion, and malicious long-packet scrambler manipulation.
A null packet inside a frame is a 32-bit length/CRC construct with the length field equal to zero. Packets are located by hunting for a length/CRC match, the same way that ATM cells are located by HEC synchronization. The next packet within a SONET payload is located by jumping length bytes in the frame and again looking for a length/CRC match. In case of data corruption at the location of a length CRC field, the hardware begins a byte-by-byte hunt for the length/CRC construct until a match is found.
Figure 9: Payload header bit definitions
It is not necessary to use HDT with SDL framing; it can easily be transported with standard HDLC-type framing (with 0x7E frame delimiter). However, using standard HDLC framing has following disadvantages:
A payload header precedes every packet and carries all information required for supporting HDT. A uniform structure of the header across all packet types simplifies design of optical nodes for packet processing for both SONET and direct data-over-fiber networks.
The packet can contain data of any kind: IP, PPP, frame relay, ATM cells, raw bit stream, T1/T3, NxDS0, etc.
The header contains a bandwidth allocation/reusability bit that is set by the sending node. If this bit is cleared, then a destination node resets the data identification bits to free up the packet area for reuse by new data packet, either at the destination node or other downstream nodes.
By using the payload header to identify the packet type, HDT is able to extend data identification beyond PSL-based SPE-level data typing and put multiple data types inside a SONET SPE (see inset for PHbit details).
A value of 0000 indicates the packet area (length of the packet area is always given by the length value in the outside SDL framing) does not contain any useful data and can be reused to insert new data.
With HDT it is easy to support traditional PDH and other guaranteed bandwidth channels. In SONET networks a frame repeats every 125 microseconds, resulting in a bandwidth of 64KHz for every byte in the payload. By fixing the starting location of some packets inside the SONET frame, one can create slots for sending TDM-style traffic. Because the packet length is changeable in one-byte increments, such slots can be created in increments of 64kbps (DS0). Because packets are dynamically created, fixed-bandwidth channels can be created on the fly by clearing the bandwidth allocation/reusability bit in the payload header.
Figure 10: Fixed bandwidth allocation for packets
As SONET frames containing these fixed bandwidth channels go around the ring, intermediate nodes detect these packets (bandwidth allocation bit reset), note the offsets of these packets, and preserve their offsets when recreating the frame (maybe after adding packets from local input ports) for outbound traffic (see figure 10).
A device supporting HDT protocol works much the same as a normal SONET transport. Operations for processing ATM cells, POS, and PDH (T1/T3) are the same; HDT only adds a header to packets to allow their mixing within the same SPE. Much of the HDT operation is related to processing of the header to identify the type of packet and then passing the starting address of data bytes to standard logic for handling the individual packet types.
Recall that support of PDH-type channels requires a fixed starting location for the channel in every frame. If PDH support is not needed, packets of any mix can be put anywhere inside the SPE and we can achieve excellent bandwidth utilization without much operational complexity. When fixed bandwidth channels are carried, some data packets may need to be fragmented when they hit a static location. Fragmentation of a packet, however, is quite easy in SONET networking because all bytes in an SPE are transmitted sequentially, and there isn't any problem in recovering fragments and putting them together.
The PSL value proposed for use with HDT is currently the same as that for SDL frames [6,7].
When a node receives a SONET frame, it uses its PSL value in the POH to determine the type of protocol carried inside its SPE. If the PSL shows POS, ATM, or PDH traffic, the device works normally as it does in current systems for handling POS/ATM/PDH.
If the PSL shows the SPE contains HDT frames, the node uses additional logic for HDT processing to detect and route different types of packets embedded in the SONET SPE. Basically, once the payload header has been processed and different packet types are identified, the hardware can use header fields to retrieve the payload and use usual hardware blocks to handle it.
Figure 11: Receiving packets using SDL
Figure 12: Processing packet using HDT
In traditional ATM transport (such as ATM VP multiplexing), ATM cells are retrieved by first looking at the PSL value to determine their presence and then parsing the SPE to get fixed-length ATM cells, either with or without HEC-based cell delineation.
With HDT, if one of the payload headers shows the payload contains ATM cells, the device retrieves payload bytes (up to the number of bytes specified in the length field) and sends the byte stream to an existing ATM cell processing logic. The ATM cell processing logic would then work on the byte stream using HEC hunting just as it would on an entire SPE containing only ATM cells in its payload.
In a transmit operation, a node takes inputs from different sources, adds an HDT header to each of the packets, encapsulates with SDL length/CRC fields, and then puts these inside an SPE. The node does not have to send a fresh frame on the network. It can reuse available space in an incoming SPE (containing HDT frames).
A device supporting HDT would receive a packet from its system side that it needs to transmit. It then looks in its output packet buffer to see if there is any space available where the packet can be inserted. If there is enough space, the entire packet is stored (with proper SDL framing and HDT header bytes). Any remaining bytes, depending on the size are either filled with a null HDT packet (the payload header identification bits are 0000), filled with SDL null packets (pairs of length/CRC with a null length field), or (if the size is less than four bytes) accounted for as tail-end padding.
Figure 13: Transmit operation using HDT
In case the device runs into a fixed-bandwidth channel allocation midway through the packet allocation, it must fragment the packets. In this case, a portion of the packet is stored at one place and other fragments are stored at another free location, with the first fragment's offset pointer containing the starting location of the second fragment. Because bytes are transmitted sequentially in an SPE, reassembling fragments isn't very difficult.
If the node detects an incoming SONET frame on its receive port, or if there is a frame in its transmit/receive queue, it checks the frame to see if there are unused/reusable areas in the incoming/queued frame that can be used for sending its own data. If there is enough space available in the frame, it fills the space with its own data before sending the frame out.
In HDT, PDH channels of any bandwidth (up to allowable SONET bandwidth limits) can be provisioned anywhere inside an SPE. To achieve precise timing, PDH bytes must begin at the same offset inside the SPE. However, allocation of PDH channels at different locations inside a SONET SPE is likely to result in sections that are unusable for data packets as they are used up for allocation of PDH channels.
When a data packet is sent inside an SPE that is already carrying some PDH channels, it may happen that the data bytes hit the starting location of a PDH channel. In this case, the packet is continued at the next available empty space in the SPE. All fragments look like complete packets with proper SDL framing and fragment length information. Each points to the next by storing the starting address of the next fragment in its Next Fragment Offset field. Fragmentation of HDT frames allows filling the entire SPE with different types of data to its full capacity.
Through the use of a separate CRC for PH bytes, HDT provides an easy way of detecting and discarding bad packets. Without this method, it becomes quite difficult to isolate optical nodes in long-haul networks that introduce errors in frames on their outgoing links. Bad frames coming from faulty nodes result in massive retransmissions and cause a bandwidth bottleneck.
For instance, in figure 14, if some bit errors occur at an upstream node that receives a packet with a correct CRC, the downstream node will never learn about the bit errors if the upstream node recomputed CRC for the packet before transmission.
With modern protocols such as MPLS, a packet-processing node would almost always have to recompute CRC on its output port. With MPLS, a node that receives the packet usually swaps the label with a different value, pops the label, or adds a new label to the stack. Because MPLS labels are embedded inside a packet, payload CRC will change at each node.
Figure 14: Problem in error detection due to CRC regeneration
An ideal solution is to check the CRC at input, but not to recompute the CRC on packet output. An efficient way to do this is to separate header CRC from payload CRC. This way, header CRC is recomputed easily and quickly at intermediate nodes while the payload CRC is preserved end-to-end.
With HDT, all header labels and other temporary information for the packet are carried outside of the payload. The payload data/CRC is not modified at any of the intermediate nodes.
MPLS specification states that MPLS labels are carried inside a packet, their existence known either by a unique protocol identifier or prearrangement among nodes. For optical nodes, however, having to go deeper into the frame at intermediate nodes to get MPLS tags requires that all nodes be protocol-aware. For instance, to learn of shim headers, PPP MPLS, etc., the node must know the packet protocol to find out whether MPLS labels are present and how they are encoded inside.
While physical medium-dependent link layer protocols such as Ethernet must have an Ethernet-style framing with embedded MPLS labels, optical transport and switching nodes need not do so. Because MPLS labels are used for switching packets at nodes and are not part of the payload, having pure MPLS fields outside the payload simplifies node design significantly. Using this technique it is possible to design high-speed switching logic at nodes without having to incorporate protocol-specific knowledge for packets. Notice that HDT is part of an optical data framing, just as POS is for data packets. MPLS labels can be processed by optical nodes outside of the actual payload independent of the type of packet data. If optical nodes are MPLS capable, for instance, packets can be carried from network to network without performing an inter-working function (IWF) at the network boundaries.
With a powerful support for MPLS that is transported independent of payload, one can design networks that can have alternative LSP (Label Switched Paths) links for a highly robust redundancy. For example, nodes on a SONET ring could be connected through another network supporting a different network topology [5]. The backup path could be a high-speed point-to-point link or a ring network that is geographically quite diverse. By providing a common network protocol engine, HDT permits configuration of these networks quite easily without requiring complex protocol translation logic for different network configurations.
Figure 15: Spatial reuse of bandwidth
Figure 16: Variable-length hybrid data ADM
With HDT variable-length hybrid-protocol add/drop multiplexers and spatial reuse of bandwidth can be implemented easily. This is unlike other protocols where either fixed-slot bandwidth channels or only one type of data can be added/dropped, limiting efficient use of SONET bandwidth. Spatial reuse of bandwidth across network nodes is achieved by permitting full or partial termination of individual packets at any node and reuse of bandwidth reclaimed from the terminated packet.
As the frame comes in its initial bytes are placed in a small transit buffer. Through MPLS labels contained in the header section or through internal packet fields a node would be able to determine whether the packet belongs to it. If the packet does not belong to the node, the bytes are streamed out of transit buffer to the output port. However, the packet belongx to the node if
Packets are added either using a fresh SONET SPE or by reusing bytes inside an incoming or a previously created/queued frame.
The decision of which packet to add to a void or reusable packet area inside an SPE can be made using one or more of the following rules:
The SDL framing mechanism uses length/CRC pair as a header and a frame delimiter. Its robust scrambling and frame locator technique makes it a good candidate for direct data-over-fiber networks without SONET framing. It is possible that over time, use of OAM packets will obviate the need for complex SONET framing and link management overheads. In such a case of direct data-over-fiber networks, the HDT protocol structure will run unmodified over fiber.
Figure 17: Operation of HDT in a mixed optical network (non-SONET and
SONET)
A unified framing structure of HDT for both SONET and non-SONET networks simplifies design of optical nodes in a mix of point-to-point and ring networks with/without SONET framing.
In the network shown in figure 17, data travels in HDT packets on a non-SONET WDM network (data-over-fiber) and the packets go inside a SONET frame unmodified. The same SONET SPE can also house a PDH channel to maximize use of available SONET bandwidth. Because bandwidth can be provisioned on a packet-by-packet basis, any of the data packets can be given a guaranteed bandwidth over the SONET network without any difficulty.
Cypress Semiconductor Corporation owns intellectual property rights to some of the key technologies disclosed in this document. Cypress intends to license them on reasonable and nondiscriminatory terms once the protocols become a standard.
[1] Doshi, B., Dravida, S., Hernandez-Valencia, E., Matragi, W., Qureshi, M., Anderson, J., Manchester, J., A Simple Data Link Protocol for High Speed Packet Networks, Bell Labs Technical Journal, pp. 85-104, Vol.4 No.1, January-March 1999.
[2] Anderson, J., Manchester, J., Rodriguez-Moral, A., Veeraraghavan, M., Protocols and Architectures for IP Optical Networking, Bell Labs Technical Journal, pp. 105-124, Vol.4 No.1, January-March 1999.
[3] Malis, A., Simpson, W., RFC 2615 PPP over SONET/SDH (http://www.ietf.org, June 1999.
[4] Grossman, D., Heinanen, J., 84 Multiprotocol Encapsulation over ATM Adaptation Layer 5, September 1999.
[5] Haskin, D., Krishnan, R., A Method for Setting an Alternative Label Switched Paths to Handle Fast Reroute, draft-haskin-mpls-fast-reroute-01.txt (http://www.ietf.org, June 1999.
[6] Carlson, J., Langner, P., Hernandez-Valencia, E.J., Manchester, J., PPP over Simple Data Link (SDL) using SONET/SDH with ATM-like framing, draft-ietf-pppext-sdl-04.txt (http://www.ietf.org, September 1999.
[7] Carlson, J., Langner, P., Hernandez-Valencia, E.J., PPP over Simple Data Link (SDL) using raw lightwave channels with ATM-like framing, draft-ietf-pppext-sdl-pol-00.txt (http://www.ietf.org, June 1999.
[8] Tsiang, D., Suwala, G., The SRP MAC Layer Protocol, draft-tsiang-srp-00.txt (http://www.ietf.org, June 1999.
[9] Jones, N., Murton, C., Extending PPP over SONET/SDH with virtual concatenation, high order and low order payloads, draft-ietf-pppext-posvcholo-00.txt (http://www.ietf.org, June 1999.
[10] Wu, Tsong-Ho, Cost-effective Network Solution, IEEE Communications Magazine, September 1993, pp. 64-73.