Excerpts from The Internet Peering Playbook: Connecting to the Core of the Internet
In this chapter we introduce the other dominant form of interconnection, called Internet Peering. Internet Peering is a local routing optimization, a way to exchange some of your traffic with neither party incurring Internet Transit fees. We describe the process of peering, introduce (a lot of) terminology, and enumerate the top five motivations ISPs give for peering. The chapter closes with a brief discussion of Paid Peering.
William B. Norton presenting at the African Peering and Interconnection Forum in Accra, Ghana.
One of the most common discussions within the peering coordinator community is
“Internet Transit is so inexpensive, why do we need anything else?”
The answer is related to transit traffic volume.
As stated earlier, as the price of Internet Transit has dropped an average of 30% per year, the average volume of Internet traffic has historically grown by 40–50% per year. This growth leads to the growth of the transit fees paid by and paid to the upstream ISPs.
For example, while the price for Internet Transit drops from $3/Mbps down to $2/Mbps, the volume of traffic during the same period might grow from 6Gbps to 10Gbps. The net result of these simultaneous forces is a 25% increase in monthly transit fees, from $16,000 per month to $20,000 per month. The percentages vary, but every year the Internet packet transmission market could grow as shown in Figure 4-1.
Figure 4-1. The unit cost of Internet bandwidth drops, but the volume increases.
The result is a monthly Internet Transit bill that continues to rise.
So even though the unit cost of transit drops, the monthly transit bill increases. Decreasing the absolute cost of Internet Transit is one of the principal drivers for pursuing an Internet Peering strategy.
Definition: Internet Peering is the business relationship whereby two companies reciprocally provide access to each other’s customers.
Internet Peering is typically settlement-free, meaning that neither party pays the other for access to each other’s customers, reflective of the underlying notion that peering is a relationship of approximately equal value to each party. Since both parties benefit about the same from the relationship, there is no need to bother with the overhead of measurement and settlement.
There is also no standard way to calculate and monitor the absolute value derived from a peering relationship. Is the value of the peering relationship proportional to the volume of traffic freely peered bi-directionally? Or is it proportional to the desirability or uniqueness of the routes? Or is the value the number of people reached?
For these and other reasons, the dominant form of peering is settlement-free. When you see the term “peering” from this point on, it means settlement-free peering.
To illustrate Internet Peering, consider the mini-Internet Peering Ecosystem shown in Figure 4-2 with only three ISPs: WestNet, MidNet, and EastNet.
Figure 4-2. Internet Peering provides reciprocal access to each other’s customers.
Using graphical peering notation, we see that:
After these two peering sessions are established, the routing tables are in place (as graphically shown as colored circles in the “routing table” beneath the ISP clouds). This diagram shows that MidNet peers with both EastNet and WestNet, and therefore MidNet customers can reach both EastNet and WestNet customers.
It is important to observe that the routing announcements go in the opposite direction as the traffic to that destination. The Internet routing scheme separates the control plane from the data plane.
For convenience, we will use the simplified and equivalent notation shown in Figure 4-3 to indicate a settlement-free peering relationship between two parties.
Figure 4-3. Equivalent graphical notations for an Internet Peering relationship between ISP A and ISP B.
These notations concisely convey that the two parties (ISP A and ISP B) are in a peering relationship.
Three key points often get lost when one is first introduced to Internet Peering. It is worth reading the following points a few times:
Discussions with the peering coordinators highlighted several dominant motivations for Internet Peering:
Figure 4-4. Internet Peering bypasses metered Internet Transit.
We will work through a detailed mathematical calculation of this reduction of transit costs in “The Business Case for Peering” chapter.
Notes from the field.
Without Peering, Packets travel the Ocean. Twice.
In the late 1990’s, traffic between the United Arab Emirates and Saudi Arabia traversed two international ISPs and across an overloaded exchange point in Washington, D.C. By peering this traffic in the Middle East instead, these ISPs reduced their transit costs and improved the performance for both customer bases.
Notes from the field.
End-User Experience Key Driver for Content Provider Peering
It is interesting to note that the content providers that started peering all said that the end-user experience was the primary driver for peering, and that control over routing was the means to accomplish a good end-user experience.
Yahoo! for example said that when it sees packet loss on a flow it adjusts its routing to route around the troubled path. The company could not do this if it simply allowed its upstream ISP to handle its traffic. For Yahoo! and most other content providers that are peering, the end-user experience—and therefore control over their traffic—is strategic.
Notes from the field.
ISPs Make More Money by Peering
Dave Rand first highlighted this effect, and conversations with European ISPs showed increasing adoption of the usage-based billing model.
Notes from the field.
Hurricane Electric Markets Broad Peering
A large-scale ISP and hosting company called Hurricane Electric highlights all of the interconnect points across the globe in its marketing literature. The marketing message is that the company takes responsibility for traffic all the way to the access networks, and that it leverages peering to provide better performance for customers.
Now let’s talk about the process of peering.
Internet Peering has essentially three phases:
We will examine each peering negotiation phase in turn. This section speaks from the perspective of an ISP exploring whether peering makes sense. Most ISPs go through these processes, and later on we will see that they apply to content providers as well.
The first process is identifying the set of potential peering targets. ISPs determine where their traffic is going to and coming from to identify. They are looking for peering partners that should be similarly incentivized to directly interconnect their network, to save money, to improve performance, etc., as shown in Figure 4-5.
Figure 4-5. Traffic analysis helps identify potential peers.
For example, in the figure, ISP A would look at its traffic and determine that a large volume of traffic goes to or comes from ISP B. Since both ISP A and ISP B are paying transit fees for that traffic exchange, it might be in both parties’ interest to “peer” with each other.
This traffic analysis is done with NetFlow or other traffic management software such as PeakFlow provided by Arbor Networks. The result of this analysis is a top 50 list of networks sorted by traffic volume. When sorted, a typical transit traffic histogram looks something like the graph shown in Figure 4-6.
Figure 4-6. Traffic distribution follows a power law.
The bad news is that the top traffic sources and sinks are typically the larger ISPs that do not peer openly. These top destinations represent a large chunk of traffic that is ultimately not peerable; these ISPs simply will not peer with you (we will discuss why later). The remainder of the traffic, however, is potentially “peerable”.
Once the top traffic destinations are identified and associated with specific ISPs, these ISPs are targeted for potential peering relationship discussions. Figure 4-7 shows a sample “Peering Top 50 list” template—peering coordinators use some form of this template to help them systematically track their interactions and progress with new potential peers.
Figure 4-7. Worksheet for prioritizing and tracking peering discussions.
These target peers are ordered by traffic volume and sometimes color-coded to reflect the likelihood of obtaining peering.
ISPs typically have a person or group specifically tasked with peering and traffic engineering issues. For example, Verizon has a “Peering Steering Committee” to evaluate peering requests, a structure now common in many of the larger ISPs.
Interviews I have had with ISPs have highlighted a key challenge: Finding the right person to speak with at the target ISP is more difficult than you might expect; in fact, it is a time-intensive process.
Peering coordinators change jobs, and there is no single standard way to find out who handles this task. Even if the name is known, peering coordinators are often traveling, way behind in e-mail, and prioritizing e-mail based on the subject or the sender. In this area, “people networking” helps a great deal, and hiring experts for their contacts speeds this initial contact process up significantly.
Next we will enumerate the most effective ways to put target ISPs in context.
Notes from the field.
Tier 1 ISP Operations Meetings
There are also “restrictive” operations meetings, such as the Tier 1 ISP operations and interconnection meetings. These meetings are not advertised anywhere since the Tier 1 ISPs all know each other, and the sole focus is on planning for interconnection growth among those Tier 1 ISPs who are already peering. I was invited to one of these meetings to present my first white paper, “Interconnection Strategies for ISPs”, which documented the math that justified building into an IXP with dark fiber to decrease the cost of interconnection. I don’t list these meetings since they would not be an effective avenue for acquiring peering.
Peering request e-mails are among the easiest e-mails to ignore. Many ISPs pass peering e-mail duties around on a weekly basis, with many engineers viewing the peering activity as “not real engineering”. Some sneeringly view peering work as secretarial in nature. Therefore, e-mail messages that are poorly written or missing information are much less likely to receive a reply. It is worth your time to write the peering request carefully and completely.
Peering requests need to include the relevant information. Peering request e-mail messages that fail to include the necessary information will delay or prevent peering from occurring. Including too much information is also a problem; the request may be perceived as too burdensome. Figure 4-8 illustrates the point with a real-world example of a poorly worded peering request.
Figure 4-8. Example of a poorly worded peering request.
The template shown in Figure 4-9 is based on a sample peering template from Serge Radovcic (Euro-IX):
“I put this together a few years ago. I did this by first looking through several peering coordinators’ e-mails that they had passed on to me, and after putting together a first draft I sent it back to three or four coordinators and I ended up with the draft I sent you.”
Figure 4-9. A template for a better peering request.
Serge goes on to say, “My original idea was based on a conversation that we had a few years ago where we would ask some peering coordinators from different countries to translate this letter and we would then automate the languages to what the Requestor and Requestee needed.”
The full template is available at: http://scripts.drpeering.net/pr3.html
Next, if a discussion is scheduled, mutual nondisclosure agreements (NDAs) may be negotiated and signed, and a discussion of Peering Policy and prerequisites then follows.
Note that NDAs are an optional step, and many ISPs do not require signed NDAs prior to discussions.
Traffic engineering discussions and data disclosure may be needed to justify the peering relationship. Each ISP typically has a set of requirements for peering. During these discussions both sides explore the peering prerequisites such as peering at some number of geographically distributed locations, or peering at specific public exchange points.
Traffic volume is often a key determinant. The decision rule hinges upon whether or not there is sufficient value from peering to justify spending time and money. A Bilateral Peering Agreement (BLPA) is the legal form that details each party’s understanding of acceptable behavior, and it legally defines the arms-length (loosely coupled) relationship that each side agrees to.
In some cases the effect on the business is also examined. Might this “peer” be a customer? How will peering impact the network?
After this initial discussion, either party may decide to walk away from the peering discussions until certain criteria are met. If both parties agree that their requirements are sufficiently met, they discuss interconnect methods as described next.
Regional interconnection has two dominant forms: The Direct-Circuit Peering model and the Internet Exchange Point peering model.
Definition: Direct-Circuit Peering is peering using a point-to-point circuit.
This peering model doesn’t require additional Points of Presence (POPs), nor does it require the deployment of additional hardware; the two peers simply purchase a circuit between their existing POPs. If there is a failure in the interconnection, only three parties are involved: the two ISPs and the circuit provider.
The Direct-Circuit Peering model has two dominant uses:
First, Metro-area Direct-Circuit Peering is used between two parties that seek interconnection with only a few other parties within the metro area. If more than one interconnect region is required, the two parties often split the interconnect costs by alternating who pays for each additional regional interconnection (Figure 4-10).
Figure 4-10. Metro-area Direct-Circuit Peering.
The second place you will see the Direct-Circuit Peering model is with “transoceanic half-circuits”, interconnections between the large incumbent ISPs that seek interconnection without the foreign peer having a local presence in their home markets. This interconnection model is sometimes called “half-circuits.”
Notes from the field.
Benefits of Peering < Benefits of Protecting Home Market
John Milburn brought up an excellent point about half-circuit peering. He said, “the benefits of peering are dwarfed by the desire to protect one’s home market”. By this he means that it may cost more money to build into another region and peer there. It may cost more money to pay for half of a transoceanic cable and peer in the middle of the ocean. However, the primary motivation for these ISPs is to “own” the home market and not make it easy for other ISPs to build in to their home market and compete with them. If they peer abroad, a peer could still build into their home market and offer Internet Transit services, but traffic to the incumbent’s customers would have to traverse an ocean twice before being delivered to the incumbent’s customers. This offering is not a very compelling one in a market where most traffic is between customers in the home market. If the ISP peered in its home market, then the peer could easily undercut the price and provide a reasonably low-latency service within the home market.
The Direct-Circuit Peering cost model scales linearly; the more interconnections one has, the more it costs to interconnect. Every additional peer incurs the cost of an additional point-to-point metro circuit.
The alternative to the Direct-Circuit Peering model is to peer at an Internet Exchange Point.
Definition: An Internet Exchange Point (IXP) is a place where multiple ISPs interconnect their networks together (Figure 4-11).
Potentially many peering sessions can be established across a single well-populated IXP peering fabric.
Figure 4-11. The IXP model.
Notes from the field.
Tier 1 ISPs Migrate From Metro Interconnects to Carrier-Neutral Exchange Points
In the early days of the U.S. Internet Peering Ecosystem, the largest ISPs in the U.S. interconnected their networks together using direct circuits. They ordered circuits between their POPs in a region from a local carrier. These circuits were sometimes delayed up to 18 months past the due date. Traffic management for the two parties became a very tough challenge, as they had to route their peering traffic along a more circuitous, uncongested path until the new regional circuits were in place. This situation led to a large-scale build into carrier-neutral data centers and explains why the IXP model is now the method predominantly used for interconnection at the core of the Internet.
The cost of peering at an IXP usually involves the following cost components (Figure 4-12):
Figure 4-12. The costs of the IXP model.
Definition: Transport Fees refer to the monthly recurring expenses associated with a physical/data link layer media interconnection into a peering location.
Unlike transit service, transport is not metered; it is sold as a fixed-capacity circuit that costs the same regardless of the amount of traffic exchanged over it.
Notes from the field.
Ray the Hoster
A counter example came up during the hard times of the last economy collapse. During the downturn, an entrepreneur purchased a data center for pennies on the dollar. He “un-mothballed” the generators; Uninterruptible Power Supply (UPS); Heating, Venting, and Air Conditioning (HVAC) systems; security systems; etc. and was able to negotiate a metered transport pipe, something very rarely made available. He explained to Qwest that there was no traffic yet, but the gear was already in the building, and if a deal could be arranged he would start paying something for which there was $0 revenue today. The deal was struck. This situation is the only time I have heard of the circuit provider allowing the customer to meter itself.
Definition: Colocation Fees are the (typically monthly) expenses associated with operating network equipment in a data center suitable for operating telecommunications equipment.
Not only do colocation facilities provide the operations environment necessary for the equipment, but the better ones also make it easy and cost-effective for their population to interconnect with each other. They understand their customers’ businesses and seek to establish and grow a community of participants. A handful of these colocation centers provides much more than space, power, and cross-connects. They facilitate peering.
Definition: Equipment Fees are the amortized costs of the networking equipment used for Internet Peering.
(In modeling the Internet exchange later on, we will calculate the total cost of peering at an exchange, including equipment, and make assumptions about pricing, depreciating the deployed router gear.)
Definition: Peering Port Fees are the monthly recurring costs associated with peering across a shared peering fabric.
Together, these fees are the monthly cost of peering. These fees are typically the same monthly recurring cost regardless of the amount of traffic that is exchanged over the infrastructure.
For the cost of the interconnection, both parties can then send and receive as much traffic as can fit across the transport circuit and peering fabric.
The interconnections at the IXP take one of two forms: Private Peering or Public Peering.
Definition: Private Peering is peering across a dedicated layer 2 circuit between exactly two parties, typically using a fiber cross-connect or a VLAN between two parties at an IXP.
Private Peering is the same as the Direct-Circuit Peering model but within a building or set of interconnected buildings. There is typically a nominal cost to Private Peering (a few hundred dollars per month for a fiber cross-connect, for example), where circuits are typically more expensive (a few thousand dollars per month for a 10G circuit, for example).
Notes from the field.
Pricing of Colocation Cross Connects in U.S. and Europe
In the U.S. the Private Peering cross-connects are run by the colocation provider with a nominal (~$300/month) recurring monthly charge. In Europe it is somewhat more common for ISPs to run the cross-connects themselves with perhaps a one-time fee. After tours of dozens of IXPs around the globe, I have seen some of the very messy results of decades of ad hoc cross-connect runs.
For a colocation center to charge ~$300/month for a fiber cross-connect that costs $40 to deploy may seem excessive. It is a large-margin product, but it is also a high-value product. This topic is discussed more in “Chapter 13 - The IXP Playbook“—you will see that this high-margin cross-connect product can motivate the IXP operator to build that population of peering customers.
Definition: Public Peering is peering across a shared fabric such as an Ethernet switch.
Public Peering is the dominant method of peering in the peering ecosystems we studied, although many support both Public and Private Peering.
The basic and important question comes next.
We will make the business case for peering in the next chapter, but to make this chapter independently complete we will answer this question in brief.
Answer: From a strictly financial view…
Peering makes sense when it is cheaper to send traffic to peers than through a transit provider.
Some companies offer a derivative service called Paid Peering.
Definition: A Paid Peering relationship is a peering relationship with an exchange of compensation from one party to the other (Figure 4-13).
Figure 4-13. The graphical notation for Paid Peering.
The compensation may take the metered form of $/Mbps. In other cases, one side might cover more of the peering costs than the other.
When does peering become Paid Peering?
My litmus test: If the peering is not a settlement-free and no-strings-attached peering relationship, then it is Paid Peering.
Notes from the field.
Paid Peering Examples
Here are a few examples of Paid Peering observed in the field:
Paid Peering for Cash: Comcast offers a Paid Peering service for those who do not meet their settlement-free Peering Policy. The metered rate is rumored to be in the $2–4/Mbps price range, approximately the same as the market price of transit. Why would you pay the market transit price when you receive only the Comcast routes? Peering generally provides a better performing path to the eyeballs, and if it costs the same and performs better, why wouldn’t you prefer to buy Paid Peering?
Paid Peering via Barter. PSINet in the early 1990s freely peered, but you had to meet PSINet at its data center; you paid for all of the costs of peering, including building into its location. The fact that there was an asymmetric allocation of the costs of peering makes this arrangement Paid Peering by my litmus test.
Paid Peering via Bundling/Barter. Both AOL and Comcast have worked out complex business relationships with U.S. Tier 1 ISPs in which other services such as fiber, colocation, large-volume transit relationships, etc. were contingent upon the inclusion of settlement-free peering. This peering is not true settlement-free peering since there appears to be some form of broader business arrangement (strings attached) involved in the transaction.
Note that Paid Peering can in fact be created using transit by filtering route announcements and being selective in the routes that you accept. Since there is no indication in the global Internet routing table for “paid routes” and “free peering” routes, most people would not be able to infer the relationship from the routing tools. At the same time, from a practical perspective, the ISP community all knows who is peering and who is most likely paying. There is a long history of bar stories between people who are paid to talk with each other.
Here are a few discussion questions from the Internet Peering Workshop.
Good discussion question for class. Should Paid Peering be priced the same as Internet Transit? What is the case for its being priced cheaper than transit, and what is the case for pricing it higher than the price of transit?
Answers to these questions along with related FAQs and supplemental materials are here:
http://DrPeering.net/books/The-Internet-Peering-Playbook/peering.php
DrPeering.net ©2014 | Privacy Policy | Terms and Conditions | About Us | Contact Us
The 2014 Internet Peering Playbook is now available on the iPad at the Apple Store and for the Amazon Kindle. The Kindle, ePub and PDF form are also perpetually updated on the DrPeering DropBox share.
