WAN Design

For this section of the course I decided to sort of start from the outer edge of the enterprise, with WAN connections and Branch design and moving our way in towards the campus and datacenter.

Now over the years the WAN has changed a lot, mostly that now we have a lot of different options available. Each with heir own benefits and considerations for us as the network designers to take into account to ensure the most appropriate options are presented for the design.

Traditional WAN technologies fall into 3 categories. Circuit switched, packet switched, and leased lines. Circuit switched would be like the PSTN, which is the ‘public switched telephone network’, or ISDN, which is actually a communications standard for use over the PSTN. These are circuit switched because the whole circuit needed for the call to connect is determined by the system and is reserved for the duration of the call. This differs from a packet switched network where you have what’s called a virtual circuit. This is sometimes referenced as a Permanent virtual circuit, or a switched virtual circuit.

Each packet is routed or switched independently, which allows for redundant paths to the destination to exist and be used in the event of a failure. With PSTN if there’s a line failure your call will be disconnected and will need to be initiated again; in a packet switched network, when the next packet arrives, it can just be routed around the failure. 

It’s a virtual circuit because the actual physical path the data takes is generally not defined. The service provider just owns a lot of routers and lines connecting them and your data will take whatever the least cost path is based on whatever protocol is being used. In the case of MPLS this is typically multiprotocol BGP with label distribution protocol.

Finally we have leased lines. These are the ultimate in control. This is when the service provider has a line that it has laid, like fiber, that they aren’t using at the moment, so you have the ability to lease that line for your enterprise to exclusively use. This is seen a lot in connections from the campus to the datacenter or connecting 2 datacenters, where you need your 10, 20, 40Gb connections. Leased lines typically are very expensive, but may turn out to be worthwhile for the speed, or you also get this physical line, so you can get layer 2 connectivity across the line, often needed for some of your server high availability clustering features. Also, since your traffic is the only traffic going over these lines for the duration of the lease, this is about as secure as you can get when it comes to service provider offerings.

So those are the categories of connectivity technologies you’ll likely encounter not lets talk about the topologies available and some considerations for them.

First up is the known and trusted hub and spoke topology. This is where you have a star formation; you know some main head office that’s your hub here and the branches or remote offices all connect in to the hub. This has some big benefits that rightfully makes it one of the most popular topologies I’ve seen. Number 2 here is almost entirely due to number 1, and that’s this topology requires the fewest number of links to allow for full connectivity. Fewer WAN links means lest recurring cost. This also means more simplified configuration when a new office is brought online, and simpler troubleshooting when there’s issues. Something some business go for as well is to have all of their branch’s internet traffic go through the hub firewall as well, allowing for centralized filtering and scanning of that traffic. You can imagine that might make your life easier, right? To have 1 spot to manage for these functions and features. 

In that same fashion you can also imagine the drawbacks I’m sure. You’ll need a pretty beefy router or firewall here at the hub to make sure it doesn’t get bogged down since all spoke to spoke traffic also needs to pass through the hub. Also this potentially provides a single point of failure at the hub, I’d surely hope that you put redundant devices at the hub to protect against device failure, but this still provides a single location where if the poorly placed backhoe comes and *schwooop* cuts that line then your spokes would have no connectivity to at least corporate resources and potentially internet access.

So lets move over to the other extreme, the full mesh topology. This is of course the ultimate in redundancy. Assuming you have dynamic routing happening with OSPF or EIGRP or something, then if a path goes down, you can simply route around it. Say any one guy gets cut, *schwooop*, then you can just come over here through this guy to get to the destination. Similarly you have connectivity directly between all locations so there’s no one router that’s getting bogged down here trying to handle all of the traffic.

Note for test takers, you can probably feel it as a question, right? Like ‘How many links are required to connect 8 branch offices in a full mesh topology?’ This here is how you calculate the number of links required for a full mesh.

# links = (N – 1) * N / 2

Immediately the thing that probably pops into anyone’s head is ‘that’s a lot of links’, and you’re absolutely right. Depending on the technology used to deploy this topology, this could be very administratively intensive or even infeasible. If these are the private WAN links, using frame relay, MPLS, or leased lines then the cost of this topology can become rather great. You know these service providers not only charge you per link usually but also based on distance, these guys could be thousands of miles apart and simply not be worthwhile due to the cost.

The last topology provides a happy median between the 2 extremes. The partial mesh topology provides more connectivity than a simple hub and spoke but less than a full mesh configuration. This is really great if perhaps you have 2 larger branch offices that have a lot of traffic going between them and you want to offload that from your hub router in the hub and spoke and you add a direct link between them. Also this can be used if you wanted to add a redundant hub to your hub and spoke, with all of the branches connecting to 2 hubs and possibly distribute the load between them.

Just like with full mesh, management can get a little tricky, but there can be many benefits to a partial mesh than a pure hub and spoke and a savings of both money and administrative overhead in complexity against a full mesh.

So, with the exception of leased lines which are actual private networks, most WAN connectivity options are virtual private networks. This connectivity is either due to the service provider separating your traffic from other customers traffic by various means and thereby creating a virtual private network for you, or this can be accomplished by the enterprise itself over the public internet by creating a tunnel between your offices so you can address the private networks in the various offices from across the public internet connection.

Of course over the internet you’d certainly encrypt your data to ensure that bad actors on the internet can’t snoop in on your traffic, but there’s also several technology options we have available for accomplishing this, each with their own benefits.

So in the service provider managed VPN world, we typically have the option between a layer 2 VPN or a layer 3. Now, layer 2 works just like how it sounds, the service provider looks like from your perspective a network switch in the cloud. You’ll form neighbor relationships with your branch routers and even things like CDP and LLDP would work over this. This can be really great, you can even tag vlans across this connection. Typically your layer 2 VPN offering will be advertised as a virtual private LAN service, or virtual private wire service. Where VPWS would be a point-to-point version of VPLS.

The network switch in the sky feels really cool, but it also presents some problems. For 1, it doesn’t scale well. At a certain point you start having trouble with broadcasts and large numbers of neighbor relationships forming between all the routers at your branch sites. Also, any QoS prioritization and queueing needs to be done on your end before the traffic enters the service providers network.

Layer 3 VPN is usually provided with MPLS. In this configuration the service provider participates in routing protocols with you, their router will exchange routes with you with BGP or OSPF or something like that to learn and advertise your routes for your offices connected to the VPN. You aren’t forming a neighborship with your own equipment on the other end, but with the service providers’. This can be much more manageable and also allows for the service provider to perform traffic engineering with QoS on your tagged traffic in their network. You’ll want to speak with your service provider to understand the level of service and features available in this regard as not all MPLS providers will provide traffic engineering and may drop your mission critical traffic in times of high congestion.

A big practical lesson to learn is to always make sure you deeply investigate your SLAs. Make sure you know that if you go over your prescribed bandwidth that they’ll drop and police your traffic instead of shape it. This of course applies to outages and the SLA for both uptime and time to resolution during an outage. This is where a lot of the money is; you might look at 2 MPLS providers, with the same specs, but one is twice the cost of the other and it’s because it has a 4 hour SLA for repair of a cut line and the other is 24 hrs.

Cost is a huge factor in design. And trust me, if you’ve never seen what an MPLS connection costs, you’ll be in for a surprise. For a long time MPLS connections have been related to public internet connections by how many hundreds of times the cost it is per megabit of bandwidth. Now mind MPLS connections have come down in cost pretty greatly over the years, but you’re still looking at like $700 us dollars per month for maybe a 20 megabit connection.

When you consider that the same connection to the public internet can be had for a third of the price or less, it starts looking very attractive to use enterprise managed VPN technologies over the public internet. You can even get two of these for less than the cost of the MPLS VPN. This is where I’ve seen a ton of businesses go, but there’s also a few considerations.

First of which is that there’s no QoS over the internet, so all traffic is handled as Best Effort. This could cause problems if you have mission critical applications that are not performing due to congestion with your ISP. Second, public internet connections are generally less reliable. Finally, you’ll want to consider what VPN technology you’ll use.

Here’s a quick table I mostly got from cisco press. Standard IPsec doesn’t encapsulate multicast traffic. And what do all of our routing protocols use to form neighbor relationships? Multicast. However we can get around this limitation by creating a GRE tunnel and encapsulating that with IPsec. This is a great option, but if you want 2 locations to communicate directly with each other, you’ll need to go in there and configure a tunnel between them. This is where DMVPN comes in.

Dynamic Multipoint VPN, this uses GRE over IPsec combined with Next Hop Resolution Protocol, NHRP. Say you have a hub and spoke topology, the spokes are just pointed to the hub, and when spoke to spoke communication is needed the hub tells the spokes where to find each other so they can bypass the hub.

Now GETVPN, which is group encrypted transport, doesn’t encrypt the source and destination IPs, only the data. You see this used sometimes in an MPLS connection; you’ll talk to the network architect and he’ll be like ‘I trust no-one…’.

Now if your company is big enough or requires this much bandwidth, or really is in 2 datacenters that the same provider is in, you may have the option to lease a line. Something we’re seeing more often is that with all this fiber that ISP are laying a fair bit of it is just unused, this is called dark fiber. So what happens is you pay the service provider lots of money per month and they let you use that fiber line. In a leased line scenario you typically are the only customer using that physical connection so you have maximum security. Being a point to point link, you are responsible for doing any queueing or traffic prioritization.

The last bullet point here I want to mention because I’ve seen it on the exam. DWDM is ‘dense-wavelength division multiplexing’. See, normally when you are sending a single stream of data across a fiber line, it uses a single wavelength of light. What DWDM does is it splits up your data into multiple streams and sends each simultaneously using a different wavelength. This can allow you to get higher throughput across an existing fiber line.  Say you have a 10Gb line already, and you need 40Gb, you may be able to utilize DWDM instead of getting additional fiber lines from your ISP.

The last slide I want to do here is a very quick one about backup WAN links. You, of course being an experienced network engineer know that a single WAN link is unacceptable when your branch office operation completely relies on that connection being functional. So you have 2 internet lines, but please please do your homework. A backhoe in the wrong spot over here, can *schwooop* and cut both your primary and secondary internet connections because the carriers run their cables on the same utility poles. 2 connections can also be good for doing load balancing and perhaps increasing your available bandwidth.

Course Video