MICAN has decades of experience with optical fibre networks. We can help you with your designs too. We understand all of the fibre options and can help you to sort through these confusing choices.

Our expertise is in the blending scenarios whereby both Information Technology (IT) and Operational Technology (OT) are jointly deployed. Whether it is for the factory floor, the smart grid, or for a mining operation, we have been working in these industries since the days of analog fibre as far back as the 1980 Lake Placid Olympics where analog fibre was first deployed commercially for broadcast purposes, and to the mid-1980s when SaskTel build what was then the world’s longest commercial fibre optic network of over 3,200 km. Since then, our team members have deployed optical fibre solutions to telecommunication towers, 10,000+ feet below the earth underground in an operational nickle mine, in Chile from the coast to the Andres Mountains into the 11,500 foot above sea level in the Atacama Desert, and even at the several other sporting venues and events.

Giant Slalom and Ski Jumping Broadcast Signals Carried by Optical Fibre Links

We have worked with Utilities around the globe on smart grid project with IP/MPLS, Carrier Ethernet, SONET, GigE, OTN, and more. We can help you with your optical fibre project too.

Many customers, such as Utilities building smart grids demand and generate lots of data, so optical fibre networks are a big part of the make-up of every large scale installation.

Whether it is optical fibre that is buried, aerial fibre on poles, or optical fibre nested into OPGW (optical ground wire) does not matter much. What matters is that the connections are made and the network meets the performance criteria for the applications.

These days, with the NERC CIP rules in flux, the pressing question is the level of integration between IT (information technology) and OT (operational technology). Some say they must be two discrete networks, while others argue that they can be fully integrated. Some think that reality is a place in the middle between two disparate networks and one holistic solution. Whatever your point of view, suffice it to say that optical networks are a big part of any smart grid strategy, no matter how it is designed.

Cisco Smart Grid
Cisco smart grid network designed with RAD statistical multiplexers 2011

Another burning question is the way that the signals are transmitted. Do we continue to use SONET, or do we look to IP/MPLS or Carrier Ethernet? What about the next generation software defined networks (SDN)?

Early Optical fibre Solutions from Grass Valley Group

All have merit. The manner in which the customer is organized plays a big part in technology selection. Most agree that SONET is well beyond its useful life so the debate turns to IP/MPLS, Carrier Ethernet, or SDN. When the customer is driven by the OT department, then Carrier Ethernet is preferred, however, if the IT team is leading, then IP/MPLS or SDN is desired. It is a matter of perspective and priorities based upon the leadership’s background. Of course, this is not always the case, but it is usually. So, understanding who is driving the adoption helps to predict the desired solution. Is this the best way to approach technology selection, not really, but it is a reality of today’s industry approaches. The best solution should be selected based upon a number of technology and business parameters built upon qualitative and quantitative metrics. Personal bias should not be totally ignored, but it should not be the dominate rating and ranking judgement factor either. So, let’s consider the options.

SONET

SONET (synchronous optical networking) has been the mainstay of telecom networks for decades and has served their needs well. Often, it was just used by the OT team and is mostly found on OPGW. However, as new applications evolved, especially from the IT side of the house, SONET was not able to support them or was too expensive to scale as necessary. It is a clocked network, so that makes it ideal for the OT folks who demand deterministic delivery of datagrams. But, it is challenged to meet all of the needs for all applications. As the global demand for SONET products shrinks in favour of SDN, IP/MPLS, and Carrier Ethernet, availability of supply and high cost make it hard to justify further investment in this technology.

IP/MPLS Links carried as OPGW, on Aerial Links and Buried with Pipelines in Chile

IP/MPLS

IP/MPLS (internet protocol / multi-protocol label switching) is a leading technology with great ability to scale and impressive data handling capacity. There are many quality vendors and several equally impressive bolt-on standards to help mitigate some integration challenges for the OT side of the house. The IT folks can embrace it fully. But, depending upon the current and future approaches to applications, it may not be the best choice for OT. There are a few issues that make IP/MPLS difficult for OT. For example, it is a layer 3 technology, so it has the overheads and delays of TCP/IP. It can use different paths in the forward direction versus the reverse direction meaning that the latency in both directions can be different. The clocking needs to be added with technologies like ITU’s SyncE (synchronous Ethernet), IETF’s NTP (network time protocol), or the ever popular IEEE 1588 version 2, however it is still non-deterministic even with clocking added. Numerous protocols need to be set and maintained within the IP/MPLS networks, so it can be complex to operate.

A State-wide Carrier Ethernet Network with DWDM to Connect 20 Utilities

Carrier Ethernet

Carrier Ethernet has advantages over IP/MPLS and is often preferred by the OT side of the house since it is the natural outgrowth of SONET in the packetized world. Carrier Ethernet is generally considered to be lower cost to implement compared to IP/MPLS. While IP/MPLS can scale to thousands of sites, Carrier Ethernet can scale to hundreds of sites, still very respectable and normally not a barrier for most customers (except for telecom carriers that service more nodes). Carrier Ethernet is the preferred technology by telecom carriers to connect data centres and other mission critical network nodes. Carrier Ethernet offers lower latency compared to IP/MPLS. It is a Layer 2 solution. Carrier Ethernet is an all Ethernet structure so it can connect end to end with Ethernet, it is better than IP/MPLS in this regard which needs to convert between MPLS and Ethernet at the network edges.

Wave Division Multiplexing

Wave Division Multiplexing (WDM) is a means to carry multiple wavelengths or frequencies of light simultaneously over a single strand of optical fibre. WDM is able to provision bidirectional communications over the same strand of glass. The first systems used just two wavelengths of light but it is possible to multiplex up to 160 frequencies today, although lower number of frequencies are more common today such as 16 or 24 wavelengths. Two flavours of WDM are used, Course Wave Division Multiplexing (CWDM) and Dense Wave Division Multiplexing (DWDM). Other variants of these two types are seen, but these two are the most popular. In the many industries, we normally just use DWDM. This approach uses the 1550 nanometre (nm) band. So, the glass used must have this wavelength window available, Normally, glass is purchased with three windows for the greatest flexibility, 850 nm, 1310 nm, and 1550 nm. Reconfigurable Optical Add-Drop Multiplexer (ROADM) is the main termination device used at fibre nodes to deliver or add traffic to the network. ROADM devices operate on WDM networks and can selectively manage individual wavelengths. When the optical fibre network is configured as a mesh architecture, then Optical Cross Connects (OXC) matrices are used to map the fibres to each other. For long distance transports of signals, IP/MPLS and Carrier Ethernet ride over these DWDM networks and can use ROADM and OXC at termination points or network nodes.

Other considerations include legacy solutions like OTN and propriety GigE links.

Optical Transport Network

OTN (optical transport network) is an ITU standard for wrapping a variety of signal formats like SONET, Gigabit Ethernet, and Fibre Channel. It is used to wrap these different signalling formats into a common Layer 1 optical foundation to improve the functionality of transport, multiplexing, switching, management, supervision and survivability of optical channels carrying client signals. It is used with WDM (wave division multiplexing) and applies each signal to its own wavelength. Rather than using a form of encapsulation into a common carrier format like Carrier Ethernet or IP/MPLS, OTN permits the discrete signal formats to be individually mapped to the OTN structure and are carried in parallel.

OTN
Image sourced from Microsemi on December 12, 2014 from http://www.microsemi.com/products/timing-and-synchronization/otn-timing

While OTN is cost effective, it is limited in flexibility and usability for applications. OTN varies its latency depending upon the source signal being wrapped and therefore different signal types will flow over the same optical pathway in different time duration.

Gigabit Ethernet

Gigabit Ethernet is a common access connection solution in baseband formats over copper wire. But, it can be delivered in five physical layer standards for Gigabit Ethernet using optical fibre (1000BASE-X),using copper on twisted pair cable (1000BASE-T), or shielded balanced copper cable (1000BASE-CX). Sometimes it is used in intra-city links as point to point connections to connect sites. It is not scalable and should be replaced when data rate demand a better solution.

Software Defined Networks

The IT and OT technology worlds are going through a new phase of internetworking transition.  They are moving from a static setting towards a highly dynamic model.

Historically, networks devices and appliances connected to the network were all hardware based solutions and were installed in a fixed manner to deliver connectivity and protection of data.  They were set up and placed into service – all very static.  But, the set up took a long time to implement normally measured in many weeks or several months.  Activation to bring these devices and appliances online was also slow because it had to be done precisely, so it was highly prone to errors, and demanded lots of testing and evaluation before going live.

sdn-nfv-network-transformation-16x9.png.rendition.intel.web.416.234
Network Transformation

Something must change to speed up the overall implementation process, make it all more agile and responsive to changes in user needs, and most importantly, to make it more trustworthy and secure.

In data centres, the virtualization of servers is having a profound effect on both capital and operating costs and scalability of resources to react to the needs of the traffic flows.  As servers were virtualized, this proved to be a smart and valuable strategy to squeeze out every compute cycle and to maximize the utilization of the server.  Storage has been virtualized too added even more value to the equation.  Electrical consumption and air handling costs were seriously reduced to deliver big saving compared to the 1:1 model for applications versus servers.

Therefore, it was a natural next step to virtualize the networks and the connected devices that protect these networks.

A Software Defined Network (SDN) is an open approach to managing the network.  A centralized controller remotely controls the routers and switches within the network fabric, which are typically located far away at the network’s edge.  SDN uses automation and centralized control to provide speed to network configuration and permit dynamic, on-demand changes to react to fluctuations in the traffic flows.  The SDN solution operates hand-in-hand with server virtualization within data centres.  As servers are brought online or removed from service to react to varying demands for traffic requests, the network can be scaled to map these same traffic flows to the server variations.  The provisioning is automated.  SDN separates the data flow plane from the network control plane to permit this virtualization of the network fabric.

Network Function Virtualization (NFV) is similar and interrelated to SDN, but it does not need to be necessarily coupled with SDN.  Although they are often seen together.  NFV is the application of software defined appliances to the network.  These appliances include devices such as edge authentication, firewalls, load balancers, and routing functions.  Traditionally, these network function appliances were hardware based and now under the NFV approach they are software applications running on network edge servers.  Like SDN, these appliances are centrally controlled and managed, which delivers speedy agility, permitting rapid change to settings and configurations.  Policies and procedural settings can be downloaded and updated dynamically too.  These network appliances are virtualized, exactly like the way the networks are managed with SDN.

Many believe that the SDN strategy built with NFV is the future and all hardware based network technologies will disappear in time. However, like all network solutions, there are always both ‘PROs’ and ‘CONs’ to be considered. Nothing is perfect. So, this transition to transform all networks has many hurtles to overcome before it is deemed to be universal.

Passive Optical Networks

The advent of Passive Optical Networks (PON) is now well upon us.  PON is the primary access tier technology used to create links for FTTH (Fiber To The Home).

FOTS 5
Installer Connecting a Home with a PON Drop

Fiber optics uses light signals to transmit data. As this data moves across a fiber, there needs to be a way to separate it so that it gets to the proper destination.

There are two important types of systems that make fiber-to-the-home broadband connections possible. These are active optical networks and passive optical networks. Each offers ways to separate data and route it to the proper place, and each has advantages and disadvantages as compared to the other [source: FTTH Council].

Passive optical networks, or PONs, have some distinct advantages. They are efficient, in that each fiber optic strand can serve up to 32 users.  PONs have a low building cost relative to active optical networks along with lower maintenance costs.  Because there are few moving or electrical parts, there is simply less that can go wrong in a PON.

Passive optical networks also have some disadvantages. They have less range than an active optical network, meaning subscribers must be geographically closer to the central source of the data.  PONs also make it difficult to isolate a failure when they occur. Also, because the bandwidth in a PON is not dedicated to individual subscribers, data transmission speed may slow down during peak usage times in an effect known as latency.  Latency quickly degrades services such as audio and video, which need a smooth rate to maintain quality.

FOTS 3
PON Ecosystems from Headend to the Residence.

Active optical networks offer certain advantages, as well.  Their reliance on Ethernet technology makes interoperability among vendors easy.  Subscribers can select hardware that delivers an appropriate data transmission rate and scale up as their needs increase without having to restructure the network.

Active optical networks, however, also have their weaknesses.  They require at least one switch aggregator for every 48 subscribers. Because it requires power, an active optical network inherently is less reliable than a passive optical network.

FOTS 4
A Typical PON In-home Adapter

In some cases, FTTH systems may combine elements of both passive and active architectures to form a hybrid system.

Ring versus Mesh Architecture

One of the more interesting aspects of optical network design, is the approach to the network architecture pertaining to redundancy and network robustness. Specifically, should the network be designed as a ring or a mesh? Historically, rings have been the most popular approach but that was driven by the legacy SONET networks that used counter-rotating rings to add robustness to the design. However, now with SDN, IP/MPLS and Carrier Ethernet, we can continue to employ the ring approach or consider the mesh model. Both have merit and both work well for IT applications. But, for OT applications, it is important to know the latency of the pathway and to appreciate the convergence time delays when the primary path fails and the secondary pathway is switched into circuit. If the existing fibre strands are already deployed in a ring model, then it is likely that a ring model architecture will continue in the future designs. However, some aspects of the mesh model may be incorporated into the new design either in part or whole. When using OPGW, the ring model may be the only option as well, since the nature of the transmission corridor is in long trunks that facilitate rings instead of mesh architecture.

Network Management System

A surprising majority of private optical network projects neglect the NMS (network management system) requirement. Sometimes, element managers are purchased to support the optical network, but that is not always the case and all to often they do not include integration with other network aspects like point-to-point microwave links, point-to-multipoint wireless networks, and baseband routing and switching to name a few. It is critical to view all of the network elements as one seamless network fabric. Therefore, a manager of managers (MOM) is needed to orchestrate the many element managers into one harmonious solution that interacts well together and “makes and breaks” connections upon demand. This MOM should also include an out-of-band diagnostic system that feeds troubleshooting back to the NOC (network operation centre). The MOM often supports network upgrades and pushing new images to the end devices, such as firewalls.

NMS
A Classic Stacked Manager of Managers Architecture

Multiplexers and Legacy Interfaces

How the traffic connects to the network is always challenging for smart grid networks since there are so many legacy interfaces that need to be respected in the design. Therefore, encapsulation / de-encapsulation is used for IP/MPLS and Carrier Ethernet. This process will add latency to the link. So, great care is needed to respect the latency issue for some applications applications, such as the Utility Industry need for teleprotection. There are several ways to interface. Some baseband routers and switches can accommodate a direct connection and other need to be connected via an intermediary device, most often a statistical multiplexer. There are many good statistical multiplexers available today that are well suited for most applications and are loaded with features to make traffic flow management and grooming easier. These devices need their own element manager in the NMS.

Multiplexer
Legacy Connections are Embedded into the Network with a Statistical Multiplexer

So, the world of optical networks within private network environments does not need to be overly complex. Once the traffic types are known, and properly quantified, then the best solution can be evaluated and identified. Deep knowledge of the applications and their expected traffic flows, regardless if they are IT or OT, will make the task of selecting a technology solution much easier. So, begin with the needs first, then look to the architecture, and finally consider the technology to make it all work. Too often, we see clients start with the technology first – and this is wrong. A perfect network is one that is invisible to the applications. Business is run on the applications, then networks are built to serve them. Do not get caught in the techolust arguments that we see far to often. Engineers love their technology, but it is the business of running a company that is paramount. Technology is the foundation upon which the business is operated. So, while it is important, it should not dictate how the business is run, the best solutions underpin the business.