Multihoming Your Network Using the Border Gateway Protocol (BGP)

By Chris Kilbourn
Editied by Justin W. Newton, Senior Director, Networking and Telecommunications, NetZero, Inc.

BGP stands for Border Gateway Protocol and is generally used as an exterior gateway network routing protocol. When it is used as an exterior protocol, it only passes information about external network information it receives, and does not transmit any information about internal network routes or structures to the outside world.

BGP is only useful if you are multihomed (have more than one Internet connection.) If you only have one Internet connection, you only have one path to the Internet, and BGP would only ever announce that one path to your network. If that one link goes down, there is no failover that can be done via BGP. A network topology with only one path would be much better served by using static IP routes.

BGP allows for the announcement to the rest of the Internet that you have more than one path into your network. This means that any traffic destined for your network has a redundant path and having redundant paths into your network unshackles you from a single point of network transit failure.

This is generally accepted to be a Very Good Thing due to the increased uptime and accessibility of your network when running BGP.

BGP had its roots in EGP (Exterior Gateway Protocol) as put forth in October of 1982[1] which introduced the key concepts of autonomous systems, network neighbors, the routing core and routing updates.

EGP described a system of autonomous systems of networks which exchanged network reachability information to network neighbors. This was done via routing updates when the network status changed state (up/down) in the core as reported by neighboring networks.

Since 1982, EGP and then BGP protocols have gone through several changes to support new technologies and the challenges of scaling the Internet routing system. Currently, BGP-4 is actively deployed in the 'core' of the Internet.

Today, the core of the Internet is considered to be network mesh of boundary routers between autonomous networks.

BGP is used to pass network route information between autonomous networks on the Internet. Unlike active routing protocols like AppleTalk that periodically pass the entire route table, BGP only passes network change information when it occurs.

BGP messages passed between network neighbors fall into one of four categories: open messages, update messages, notification messages, and keep-alive messages.[2]

Open messages are used to establish a routing session between network neighbors, and include the BGP version number being used, the ASN (Autonomous System Number) of the originator, a hold time used to set session timing to prevent message flooding, a BGP identifier which is set to an IP address of the sender and optional parameter information which is generally used for security.

Update messages serve double duty by passing new route announcements with network path information and by informing neighbors of withdrawn routes and networks.

Keep-alive messages are exchanged between neighbors to let neighbors know that they are still there and routing. If a neighbor does not receive a keep-alive message, it will back off sending data to that neighbor until a new open message is received and withdraw those routes from the local routing table.

Notification messages are used to report errors to neighbors, and the BGP session is then closed to prevent invalid data to be injected into the routing table.

BGP neighbors exchange network route information that are passed as belonging to a particular ASN, and routing decisions are made on ASN reachability information. The shortest ASN path is generally chosen when presented with multiple paths to a given network.[3]

Migrating to BGP

Migrating your network to BGP routing can be a challenging process, and it is not for everyone. In fact, the Internet community has stringent requirements before you can send your first open message to your first BGP neighbor.

These requirements include being multi-homed, having a powerful enough router to do BGP routing, being assigned your own block of IP address space, and having an ASN number assigned to your organization.

Usually, the first BGP routing you will do will be to your upstream Internet transit providers. Since each network that you connect to may have different requirements, you should inquire with your future BGP peers to determine what requirements they have.

The first step, becoming multi-homed, is the easiest. All you need to have is more than one Internet connection from different Internet access providers who will exchange BGP routes with you.

RAM - Routers Need It Too!

As you saw in the main part of the article, this and many other types of router uses requires lots of RAM. And, this isn't just any RAM - it's the kind that can get expensive quickly for several reasons.

First, the RAM needs to be high quality and relatively fast. Second, is the configuration of the RAM module itself - it's not the same as what you'd buy for your Mac or PC.

After checking around with router people "in the know", we came up with two places to get router RAM. First, is to get it from a Cisco reseller. This is the most expensive way to go. Second, we found Rocky Mountain RAM, in Boulder, Colorado - and saved a considerable amount of money compared to the local Cisco reseller. The rep we spoke to was Eric Thomas,

ethomas@ram-it.com, 800-543-0932 - and he knew quite a bit about Cisco's.

Got the RAM, popped it in, and never thought about it again. Just as life should be.

Acquiring a powerful enough router is a little harder due to cost issues. Currently, you need at least 128MB of RAM in your router (if you use Cisco gear,) to accept the full Internet BGP route table. Router RAM is expensive, and the more connections you have to the Internet, the more processing power you will need. There is a fairly active used market right now, so be sure to shop around, and compare vendors!

IP address space allocation is the hardest part of the process as networks must meet stringent requirements to demonstrate need.[4],[5] There are two methods of obtaining IP addresses: from a regional IP registry, or from one of your upstream ISP's.

Unless your network is already utilizing 2,048 IP addresses internally or with downstream clients, you will not qualify for your own address space allocation from a registry and will need to request space from one of your ISP's.

Due to IP address depletion, you should only ask for the amount of IP space you really need. While it used to be the case that only large IP blocks were routeable in BGP, small blocks are now commonly routed so there is no need to ask for extra IP space any more.

Once you have been allocated IP address space from a registry or your ISP and are currently multihomed, you can begin the process of applying for an ASN.[6]

ASN's are defined by a 16 bit identification number assigned by ARIN[7] for North and South American networks, RIPE[8] for European and African networks, and APNIC[9] for Asian and Pacific networks. These organizations are also responsible for IP address allocations for their respective regions.

It is possible to obtain an ASN without an IP network address allocation, but its utility is somewhat limited. This is because ASN's obtained this way are limited to running BGP in a closed system and are prevented from advertising their networks to the rest of the Internet via BGP.

Since there are only 65,536 possible ASN's, it is important that you only request an ASN if you are immediately about to multi-home. No one will think that you are 'cool' if you have your own ASN and are not using it. Quite the opposite, in fact.

Once allocated your ASN[10], you are ready to begin BGP routing.

Getting Started With BGP Routing Commands

All of the following examples are based on Cisco's IOS command set, and show reserved address space for route announcements and ASN's. Additionally, the examples shown reflect a bare-bones configuration for simplicity's sake.

It is also assumed that you have already created your internal network routes and defined interfaces.

You should perform your own research before copying these examples and deploying them in your network.[11]

The author would also like to point out that there are many different ways to configure BGP, and that best current practices are constantly evolving. Study, evaluate and decide what configuration parameters, options and methods will be best for your network.

Additionally, you will need to coordinate with your ISP's to begin BRP routing. Plan ahead to make sure that they are ready to accept your BGP sessions, and that these changes are done duing your normal maintenance window in case something goes wrong.

Enter command mode and tell your router what your ASN is:

autonomous-system 64512

Next, you need to tell the router that you want to enter some BGP commands, prevent an arbitrary router from trying to synchronize with ours, and also tell our router what networks are local to our ASN:

router bgp 64512
no synchronization
network 192.168.0.0 mask 255.255.224.0
network 192.168.145.0
network 192.168.225.0
network 10.4.0.0 mask 255.255.0.0

In the above example, we will be announcing that networks 192.168.0/19, 192.168.145/24, 192.168.225/24 and 10.4/16 belong to ASN 64512.

Now we need to define our BGP neighbors:

neighbor 172.16.45.3 remote-as 64828
neighbor 10.128.47.16 remote-as 65123

Note that the neighbor addresses are remote port address that should be provided by your ISP's.

Now, we want to tell the router to aggregate our IP address blocks for supernetting and to make sure it will only distribute the supernet route and not a more specific network route:

aggregate-address 192.168.0.0 255.255.224.0 summary-only
aggregate-address 192.168.145.0 255.255.255.0 summary-only
aggregate-address 192.168.225.0 255.255.248.0 summary-only
aggregate-address 10.4.0.0 255.255.0.0 summary-only

Lastly, we want to prevent the router from auto-summarizing BGP-3 routes that are injected into the routing tables:

no auto-summary

Now escape out of command mode and save your changes.

Here is what it would look like all printed out together:

autonomous-system 64512
      !
router bgp 64512
no synchronization
network 192.168.0.0 mask 255.255.224.0
network 192.168.145.0
network 192.168.225.0
network 10.4.0.0 mask 255.255.0.0
neighbor 172.16.45.3 remote-as 64828
neighbor 10.128.47.16 remote-as 65123
aggregate-address 192.168.0.0 255.255.224.0 summary-only
aggregate-address 192.168.145.0 255.255.255.0 summary-only
aggregate-address 192.168.225.0 255.255.248.0 summary-only
aggregate-address 10.4.0.0 255.255.0.0 summary-only
no auto-summary

At this point, you should be ready to go. Now it is time to call your ISP's and have them accept your BGP routing sessions. Once you have a network engineer on the phone, you will need to reset your port to force an open message exchange:

clear ip bgp 172.16.45.3

Notice that this is the remote port of your ISP. You will need to do this for each network connection you have when you first come online with BGP with that provider.

You should also confirm with your ISP that they are announcing your routes from their BGP sessions to the rest of the world (this is what you are paying them for after all!) This may require them to update their route filters which can take some time, depending upon the ISP.

Checking Your Work

Now you should check to see that you have everything set up and running correctly for traffic flowing out of your network and to make sure that people can get into your network.

From your router's prompt, check to see that you have BGP routes to a site outside of your network:

show ip bgp 17.254.0.91

This should show an output like this:

BGP routing table entry for 17.128.0.0/9, version 17940452 Paths: (2 available, best #2)
Not Advertised to Any Peer
64828 702 701 10911 714
172.16.45.3 from 172.16.45.3 (172.16.45.3)
Origin IGP, localpref 100, valid, external 65123 1239 10911 714 10.128.47.16 from 10.128.47.16 (10.128.47.16)
Origin IGP, localpref 100, valid, external

This shows two routes out of your network via different networks and that the second route is the preferred route due to the fact that it has the shortest number of hops through other networks to get to the final destination.

Next, connect to a public route server[12] and perform the same command, but with a destination address inside of your network.

The output should be similar to this:

show ip bgp 192.168.14.12
BGP routing table entry for 192.168.0.0/19, version 5055628 Paths: (4 available, best #3)
Not advertised to any peer
64802 64739 64565 64917 65034 64828 64512 172.16.62.94 from 172.16.62.94 (172.16.62.94)
Origin IGP, localpref 100, valid, external 64721 65022 64631 65123 64512 10.8.3.19 from 10.8.3.19 
   (10.8.3.19)
Origin IGP, localpref 100, valid, external 6294 64828 64512 172.16.12.9 from 172.16.12.9 (172.16.12.9)
Origin IGP, localpref 100, valid, external 64631 65022 64802 65123 64512 10.17.224.45 from 10.17.224.45 
   (10.17.224.45)
Origin IGP, localpref 100, valid, external

This view shows us that there are four routes to our network with route number three being the best route. Looking at the next to last hop ASN, we see that both of our upstream ISP's are in the ASN path list, so both are announcing our routes to the rest of the world.

If the next to last hop was always the same, we would need to call the ISP that was not shown, and ask them to make a route announcement for us.

Do be aware that routing announcement changes can take up to an hour before routes converge, or are fully propagated through the global BGP system. This means that troubleshooting can sometimes be delayed as you wait for route convergence.

What Can Go Wrong

The careful reader will note in the above examples that there is absolutely nothing to prevent you from announcing 0.0.0.0 or any other network to your BGP neighbors. When you announce routes that you do not own, you 'blackhole' those routes.

Remember that BGP only propagates routing changes and if you announce a network that you do not administer or have a route to, the rest of the world will now think that you are the best path to that network and start sending you traffic. The true network administrator's traffic begins to drop off to zero as the routes converge, thus a 'Black Hole' network; one that does not have a valid route on the Internet.

These sorts of situations happen in small and large scales on the Internet every now and then. In these cases, the false routes must be withdrawn by removing the incorrect network statements and resetting the BGP session. The true administrator of the falsely announced network then must reset their BGP sessions in order to inject the routes back into the global route table.

This process obviously needs to happen in a coordinated fashion and requires resetting the BGP session. Every time you reset a BGP session, you 'flap' a route. This means you send a new update message that is passed around the world.

Many ISP's do flap dampening[13] to prevent excessive routing table churn, which can slow down route processing. Flap dampening works by ignoring BGP update messages from a neighbor if the BGP session is reset too many times in a given time period.

Flap dampening prevents minor typos from causing excessive route churn. Route flaps are generally caused by internal network information leaking into BGP and inexperienced BGP network administrators resetting their BGP sessions to fix their mistakes. Route filtering is where the real protection from ignorant mistakes comes into play though.

Cisco 3600 Family of Routers

You've been reading about BGP and are probably wondering about the type of hardware that you would need. Aside from a good deal of RAM in the router, you need to pick a capable router that is rock solid, and has the features to make BGP possible.

For our testing, we went with a Cisco 3640 router. We chose the 3600 series because it had the ability to run current versions of the Cisco's IOS, as well as have enough RAM and ports in the box to do the job we needed.

Furthermore, in our minds, Cisco routers feel like that old TV commercial for Master padlocks. You know, the one where they shoot a bullet through it, and the lock keeps working? Cisco's are the same way. They just keep going and going and going ... a good feature to have in a router.

Specifics on the 3640

The Cisco Systems 3600 series is the multiservice solution that Cisco says it has designed for "branch offices". But, because it is flexible, modular, high performance, and cost-effective, it may be good for your main office. :) It all depends on the amount of traffic you are flowing.

The 3600 Family comes in three main flavors (3620, 3640, and the 3660) which differ in their expandability and throghput capabilities. The 3640 has a 100-MHz IDT R4700 RISC processor; 8 MB Flash, ugradable to 32 MB; 16 MB DRAM, upgradable to 128 MB.

Key Benefits

What makes the 3640 special is that you can do so much with it. In one box, you can combine dial-up access, advanced LAN-to-LAN routing services, and multiservicce integration of voice, video, data into a single box. In typical Cisco fashion, the design is both modular and flexible supporting a wide array of network modules. As you would expect, everything is highly configurable and scalable.

If you are into such things, you can use the 3640 for standards based support for Voice over IP and Voice over Frame Relay. If you aren't already familiar with Cisco's fully integrated IOS software, it comes with extensive security features, and multimedia support with robust QoS, and guaranteed interoperability across all Cisco routers. Since Cisco routers make up so much of the Internet, this gives you a great deal of interoperability.

When it comes to management, you'll be able to use a console port, Simple Network Management Protocol (SNMP), or Telnet for remote management and monitoring. Having a simple, clean Telnet interface was a big plus for us and made it easy to access the router from anywhere on our network.

When to Deploy a 3620 or a 3640/3660, and what can you do with them

Of the three, which do you go with? With 2 slots in a Cisco 3620 with performance of 20-40 kpps, 4 slots in a Cisco 3640 with performance of 50-70 kpps, and 6 slots within a 3660 with performance of 120 kpps. It all comes down to the number of slots you need and the throughput you want.

The 3600 series allows you basic hardware integration including integrated CSU/DSU and analog and digital modems to the applications integration of Voice/Fax/Video/Data. The Quality of Service (QoS) features of the Cisco IOS and the power to support them at bandwidths in excess of multiple T1/E1. Features such as Weighted Fair Queuing (WFQ), IP Precedence, Resource Reservation Protocol (RSVP) and Committed Access Rate (CAR) provide both the traffic shaping and prioritization necessary for a robust multiservice platform that can handle mission critical networking.

The 3600's can also handle Virtual Private Networking (VPN) through advanced tunneling features including L2F and L2TP, standards based IPSEC encryption, IOS Firewall Feature Set, and diverse WAN and dial interfaces to yield a choice great for both VPN entry points and home gateways.

LAN media support is flexible in that you can support Ethernet, Fast Ethernet, and Token Ring as well as high density ISDN, async, and analog and digital modem support. In a 3640, you can have up to 8 PRI, up to 96 ports for supporting external modem banks, up to 48 analog (POTS) modems, or up to 60 digital modems (at 56 kbps speeds).

How the Cisco 3600 Series Stacks Up

The Cisco 3600 series offers support for the most widely used network protocols, including IP, AppleTalk, Novell IPX, DECnet, and a wide range of routing protocols. For bandwidth optimization there are a series of features including data compression and multiple traffic prioritization techniques which ensure that mission-critical data is accommodated, while features such as protocol spoofing, snapshot routing, bandwidth on demand and dial on demand guarantee that the cost of usage-based services such as ISDN is minimized.

There's enhanced multimedia and virtual LAN (VLAN) support: Internal Group Management Protocol, RSVP, Protocol Independent Multicast, WFQ, Simple Multicast Routing Protocol, and Inter-Switch Link enable the Cisco 3600 series to support audio and video service applications as well as virtual LANs.

For security, there's user authentication and the IOS Firewall Feature Set allow only approved traffic onto the network. Event logging and audit trails, encryption, and VPN tunneling provide increased network security. In addition, TACACS+ and RADIUS are also supported.

Our Conclusion

The only real issues that we faced were that, like many Ethernet interfaces from other vendors, auto-sensing of duplex doesn't necessarily work. It's just safer to lock them down manually ... and that's what the experts do.

If it sounds like we like the 3640, we do ... a lot ... and highly recommend them for both the heart of your network and the satellite office communications that you may want to facilitate.

The nice thing about Cisco's line is that if this is too much or too little of a box for your needs, there are additional models above and below it ... and they all work in a consistent way.

You can find more information out about Cisco and their products at: Cisco Systems Inc., 170 West Tasman Drive, San Jose, CA 95134, http://www.cisco.com, 800-553-NETS (6387), 408-526-4000, Fax: 408-526-4100.

Protecting Your Networks

Route filtering is more granular because you can pick and choose which networks you want to accept into your BGP routing table. Route filtering for your network should reflect your network's policy of what routes you want to accept and reject, and to protect you from other people's mistakes.

A solid route filtering policy[14] will prevent the acceptance of nonsensical routes that could cause all sorts of havoc if you accepted them and then passed them on to your neighbors.

Examples of routes you want to avoid are the default route of 0.0.0.0, any RFC 1918[15] address space, loopback, etc.

Do bear in mind that if you place more than one BGP network engineer in a room, you can instantly start a debate that could rise to fisticuffs by casually suggesting what a 'perfect' BGP route filter policy is. Everyone's network is different, and as such, your route filtering policy should reflect your goals.

Public route servers,[16] where network engineers store network route information, is one source of information that can be used in building route policies. These IRR's (Internet Routing Registries) provide an automated way to build your route filtering policies by pulling down network route information and programmatically building route filters based on the data retrieved.

Conclusion

Multihoming your network and running BGP is a boon to network uptime as it provides multiple routes in and out of your network for traffic to flow on. The downside is that the requirements to do BGP routing can be hard to attain without demonstrated need, sufficient capital and experienced staff.

If you feel that you need the benefits of BGP, but feel that migrating to BGP routing is beyond your organization's ability, you should investigate other methods of network redundancy which are easier to implement. As part of those methods, be sure to utilize ISP's that are using BGP and are willing to assist you with implementing your redundant configurations.

• [1] http://www.freesoft.org/CIE/RFC/Orig/rfc827.txt
• [2] http://www.cis.ohio-state.edu/htbin/rfc/rfc1771.html
• [3] http://www.cisco.com/warp/public/459/25.html
• [4] http://www.arin.net/regserv.html
• [5] http://www.arin.net/regserv/initial-isp.html
• [6] http://www.arin.net/regserv/asnguide.htm
• [7] http://www.arin.net
• [8] http://www.ripe.net
• [9] http://www.apinc.net
• [10] ftp://rs.arin.net/netinfo/asn.txt
• [11] http://www.cisco.com/univercd/cc/td/doc/product/software/ios120/12cgcr/np1_c/1cprt1/1cbgp.htm
• [12] http://www.merit.edu/~ipma/tools/lookingglass.html
• [13] http://www.merit.edu/internet/documents/rfc/rfc2439.txt
• [14] http://www.merit.edu/ipma/docs/help.html
• [15] http://www.merit.edu/internet/documents/rfc/rfc1918.txt
• [16] http://www.radb.net/

Chris Kilbourn <chrisk@forest.net> is the Founder and Chief Technical Officer for digital.forest, a server colocation, database- and application-hosting company serving clients worldwide.