The Evolution of the U.S. Internet Peering Ecosystem

Abstract

A new Internet Peering Ecosystem is rising from the Ashes of the 1999/2000 U.S. Telecommunications Sector crash. Global Internet Transit Providers have gone bust and a critical broadband infrastructure provider has failed, leaving in their wake a large set of Internet players to fend for themselves to provide their customers with Internet services. A broad set of Service Providers that were once focused only on growing their market share (at any cost) now are bending down to shave pennies off of their cost structure. Those who can not prove the viability of their business model while satisfying their customer demands are out of business.

In this paper we share research carried out over the last four years with hundreds of Peering Coordinators to document the recent chaotic evolution of the Peering Ecosystem. We do this by first defining the notion of an Internet Peering Ecosystem as a set of autonomous Internet Regions, each with three distinct categories of participants. Each of these groups of participants has their own sets of characteristics, motivations and corresponding behaviors and interconnection dynamics. We describe four classes of Peering Inclinations as articulated in Peering Policies.

The bulk of the paper however focuses on the Evolution of the U.S. Peering Ecosystem. Several key players, some abandoned by their service providers, have entered into the Peering Ecosystem and caused a significant disruption to the Ecosystem. Peer-to-Peer application traffic has grown to represent a significant portion of their expense. We describe five major events and three emerging evolutions in the Peering Ecosystem that have had and continue to have a significant disintermediation effect on the Tier 1 ISPs.

In the appendix we share a simple mathematical Internet Peering Model that can be used to demonstrate this Peering Ecosystem evolution. While not complete or by any means precise, it does allow us to demonstrate the affect of these disruptions in the Peering Ecosystem.


The Global Internet Peering Ecosystem

The description of the evolution of the U.S. Internet Peering Ecosystem requires a set of terms and definitions. With this lexicon we can more precisely describe the evolution that has been occurring over the last few years.

The Internet consists of millions of network devices (routers, servers, workstations, etc.) operated by a wide variety of network operators, content providers, and end users. We will call these operators of network devices “Internet players”. These “players” are independent (but interconnected) members of a system we will call an “Ecosystem”; each player has a definable role, an associated set of motivations and corresponding behaviors. We will focus in this paper only on the core of the Internet Ecosystem : the network operators that make up the Internet Peering Ecosystem.

The Internet as a Global Internet Peering Ecosystem

Definition: The “Internet Peering Ecosystem” is a community of loosely affiliated network service providers that interact and interconnect their networks in various business relationships.

There are many Internet Peering Ecosystems around the world, each with their own set of network operators collectively providing Internet access to the Internet Region.

There are vast differences between each region of the Internet Peering Ecosystem , so we model the Global Internet Peering Ecosystem as a loosely coupled set of “Internet Regions” each with its own Peering Ecosystem.

Internet Regions

In order for an ISP to provide access to the global Internet, it must get attached to the Global Internet. It does so with either Transit or Peering relationships, or a combination of both within an Internet Region.

Definition: An Internet Region is a portion of the Internet, usually defined by geographical boundaries (country or continent borders), in which an Internet Peering ecosystem is contained.

Internet Peering Ecosystem Image

Figure 1 – Internet Players within an “Internet Region”

Each Peering Ecosystem is generally composed of three general classes of Internet players: Tier 1 ISPs, Tier 2 ISPs, and Content Providers / Enterprise Companies. Each Player within these classes hold similar power positions within the Peering Ecosystem and therefore tend to have similar motivations and exhibit similar behaviors within the Peering Ecosystem. We will describe each of these players next.

Definition: Internet Peering is the business relationship whereby companies reciprocally provide access to each others’ customers.

This definition applies equally to Internet Service Providers (ISPs), Content Distribution Networks (CDNs), and Large Scale Network Savvy Content Providers.

To illustrate peering, considerthe figure below showing a much simplified Internet; an Internet with only three ISPs: WestNet, MidNet, and EastNet.

Peering Diagram

WestNet is an ISP with green customers, MidNet is an ISP with blue customers, and EastNet is an ISP with red customers.

WestNet is in a Peering relationship with MidNet whereby WestNet learns how to reach MidNet's blue customers, and MidNet reciprocally learns how to reach WestNet's green customers.

EastNet is in a Peering relationship with MidNet whereby EastNet learns how to reach MidNet's blue customers, and MidNet reciprocally learns how to reach EastNet's red customers.

Peering is typically a free arrangement, with each side deriving about the same value from the reciprocally arrangement. If there is not equal value, sometime one party of the other pushes for a Paid Peering relationship.

Important as well is that peering is not a transitive relationship. WestNet peering with MidNet and EastNet peering with MidNet does not mean EastNet customers can reach WestNet customers. WestNet only knows how to get to blue and green customers, and EastNet knows how to reach only blue and red customers. The fact that they both peer with MidNet is inconsequential; peering is a non-transitive relationship.

 

 

Ecosystem Player: The Regional Tier 1 ISP

The phrase “Tier 1 ISP” has been an overloaded and misused term in the industry, but for this discussion we will use the following working definition:

Definition: A Regional Tier 1 ISP is an ISP that has access to the entire Internet Region routing table solely through Peering relationships.

Model of Tier 1 ISP

The figure above models a Tier 1 ISP graphically as selling transit to downstream customers, peering traffic away off to the side, and operating backbone links to offload the traffic elsewhere in their network.

Through these free peering connections, the Tier 1 ISPs all enjoy settlement-free access to all destinations within the Internet Region. Some in the Peering Community refer to the set of Tier 1 ISPs as the "The Tier 1 club". The Tier 1 ISPs sell access to the region as part of a transit service as well, often as a wholesale transit service, to Tier 2 ISPs, CDNs, and Large Scale Network Savvy Content Providers.

Tier 1 ISP club

While this "Tier 1 ISP" definition has been criticized as being almost impossible to prove it is an important distinction as it pertains to ISP motivations and behaviors in the Internet Region. Tier 1 ISPs are not motivated to peer to reduce the cost of transit since, by definition, Tier 1 ISPs don’t pay for transit.

Tier 1 ISPs may peer in many geographically diverse locations for technical reasons. Peering has the benefit of lower latency, better control over routing, and may therefore lead to lower packet loss. For ISPs that charge on a per-Mbps basis, this leads to financial benefits as customers use more bandwidth and therefore pay more money. This is a function of TCP: lower latency and lower packet loss means that the TCP window opens more quickly. This results in greater usage and therefore greater customer revenue. Most Tier 1 ISPs around the world exchange enough traffic with each other to require multiple points of interconnect.

A Tier 1 Peering Policy generally includes a minimum number of locations as a requirement for peering.

Motivation and Behavior.

Tier 1 ISPs do not need to buy transit since by definition they can reach all the networks in the Internet Peering Ecosystem solely through free peering relationships. Stated most eloquently by Waqar Khan (Qwest, a Tier 1 ISP in the U.S.), “We have all the peering we need.” It is not surprising that Tier 1 ISPs see the rest of the players in the Internet Peering Ecosystem players as potential customers, and do not seek peering with them. In fact, James Spenceley (Comindico) went through the process of negotiating peering with the Tier 1 ISPs in Australia and described the Tier 1 peering negotiation tactics in a one word acronym: MILD: Make It Long and Difficult . This reflects the ISP’s underlying Peering Inclination, perhaps articulated in their Peering Policy .

Discussion

Regional Tier 1 ISPs are at the top of the hierarchy and don’t have to pay transit fees (since by definition they have access to the entire Internet Region solely through peering relationships with the other Tier 1 ISPs). All other ISPs operating in the region are required to purchase transit from one or more of the Regional Tier 1 ISPs (or indirectly from their downstream ISPs) in order to reach all destinations in the Internet Region.

This Regional Tier 1 definition is important for this discussion as it allows us to explain some of the Peering motivations.

Regional Tier 1 ISP Model. The figure below is the graphical representation of the Tier 1 ISP Peering Presence, a set of routers. There are three essential elements of the Tier 1 ISP Peering presence: Downstream Transit Link(s) to Transit Customers, Peering Interconnections , and Backbone Backhaul links to the ISP’s other core routers. We will discuss each of these interconnections in turn.

Tier 1 ISP Model

Figure 2 – The Tier 1 ISP Model

Regional Tier 1 ISP Transit Links to Transit Customers. Shown attached lines underneath the ISP core routers, transit customers connect either directly to the core routers, or more commonly, to tributary routers that feed up into the core router. In either case, these Transit Attachments are often referred to as “Downstream Customers.”

Regional Tier 1 ISP Backbone Backhaul. On one side of the core router are backbone interconnections (called “backhaul connections”) linking a Tier 1 ISP’s core routers together. These core routers and core links together represent the core of a Tier 1 ISP transit network, and must be sufficiently large to carry the aggregate traffic back to the core and on to the end-customers.

Regional Tier 1 ISP Regional Peering Interconnections. The interconnections between the Tier 1 ISPs tends to be full-mesh across each Interconnect Regions . Each interconnect link needs to have enough capacity to handle the aggregate traffic to and from the other Tier 1 ISPs. The generalized Interconnection Region is shown graphically below.

Tier 1 Interconnect Model

Figure 3 - Generalized Tier 1 Interconnection Model

Regional Tier 1 ISP Peering Motivation. By definition, Tier 1 ISPs don’t pay transit fees, so they are only motivated to peer with each other as a way to provide connectivity for their customers.

Regional Tier 1 ISP Relationship with non-Tier 1 ISPs. Since non-Tier 1 ISPs represent current or prospective customers, or customers of other Tier 1 ISPs, there is little incentive to peer with them. Peering policies tend to reflect this inclination – we will discuss this more in the section on Peering Policies.

We can group the Tier 1 ISPs into a cluster as shown below showing this group collectively provides access to the entire Internet region.

Interconnect Regions. Some Internet Regions are large enough to require multiple “Interconnect Regions” to distribute the load of peering across multiple points.

Regional Tier 1 ISPs in the United States

To make this model more concrete, the figures below shows some of the legacy United States Tier 1 ISPs and their full mech Interconnection Region Peering.

Tier 1 ISPs in the U.S.

Figure 5 – The U.S. Tier 1 Interconnect Region

These U.S. Tier 1 ISPs generally peer with each other in eight Interconnection Regions across the United States Internet Region, using private peering circuits or cross connect meshes as shown below .

U.S. Interconnect Areas

Figure 6 - Tier 1 Interconnection Regions in the U.S.

Notes and Sources...

Definition: An Interconnection is a general term that refers to a connection between two ISPs.

An Interconnection can refer to a Peering or Transit relationship. Peering and customer Transit traffic.

This statement is roughly true; there may be delays before all Tier 1 ISPs migrate into each Interconnect Region.

A Cross connect is a dedicated fiber run directly between pieces of networking equipment for the purposes of private peering (or for transit/transport purchases). New York City Area, Washington DC Area, Atlanta, Chicago, Dallas, Seattle, Bay Area, Los Angeles. Source: Cable & Wireless Peering Policy http://www.cw.com/template_05.jsp?ID=us_10_02

Ecosystem Player: The Tier 2 ISP

Roughly speaking, all other (non-Tier 1) ISPs can be broadly categorized as Tier 2 ISPs in the Peering Ecosystem.

Definition: A Tier 2 ISP is an Internet Service Provider that purchases (and therefore resells) transit within an Internet Region.Model of Tier 2 ISP

The figure above models a Tier 2 ISP graphically position as purchasing transit from an upstream provider, selling transit to downstream customers, peering traffic away off to the side, and operating (optional) backbone links to offload the traffic elsewhere in their network.

A Tier 2 ISP will often embrace peering to reduce transit costs, improve performance, and potentially even increase revenue. You will find Tier 2 ISPs participating actively in Peering Forums - this is an expected bahavior from a player looking to increase the amount of peering they have. Some of the larger Tier 2 ISPs will be able to peer away towards 70% of their traffic.

 

Tier 2 ISP Model. The Tier 2 ISP model is similar to the Tier 1 model but includes one (or more) “Upstream” transit connections as shown graphically below.

Tier 2 ISP Model

Figure 7 – The Tier 2 ISP model

Tier 2 ISP Transit Links and Customers. Just as with the Tier 1 ISPs, Tier 2 ISPs sell transit to their customer base, using a variety of techniques to target market and differentiate themselves from the Tier 1 ISPs. These market focus techniques include serving underserved or rural areas, focusing on niche markets or serving specific industry sectors.

Tier 2 ISP Backbone Backhaul. A subset of Tier 2 ISPs that operate in multiple markets have backbone links constituting their core backbone network. Other Tier 2 ISPs operate in a single market and do not require any backbone backhaul. As with the Tier 1 ISPs, these links are drawn on the side of the graphic.

Tier 2 ISP Peering Motivations. Tier 2 ISPs have several motivations for Peering:

  1. Reduce Transit Costs. By directly peering with willing players, traffic is exchanged directly with the peer, and typically is done so settlement-free. This peering traffic therefore reduces the load on and therefore the cost of transit. For non-Tier 1 ISPs, Peering is one easy way to reduce transit fees.
  2. Improved Performance. Traffic that flows directly between players has lower latency than traffic that first traverses a transit providers network before being handed off to the peer.
  3. Greater Control over Routing. Some ISPs prefer to have tight control over the path and performance of the traffic . If a poor performing path is preferred by the routing protocols , an alternative path can be configured.
  4. Regulatory Reasons. In the U.S. for example, the consent decree prevents RBOCs from interconnecting their backbone nodes across LATA boundaries. This restriction makes it impossible for the RBOCs to meet the geographic peering requirements of many of the Tier1 ISPs. Therefore, one way for the RBOCs to reduce the traffic exchange costs is to peer openly regionally at least until regulatory relief is granted.

Tier 2 ISP Regional Peering Interconnections.

The Tier 2 ISPs generally interconnect with each other across shared Peering fabrics located in an Interconnection Region. These interconnections have historically required smaller capacity interconnects than the Tier 1 ISP Peering interconnects. As a result, public peering (using a shared peering fabric like Ethernet) has been a cost effective method to interconnect with a lot of players and aggregate traffic over a single router port.

Tier 2 ISPs tend to peer only in Interconnect Regions in which they sell services. This creates a diverse population of peering Tier 2 ISPs in each Interconnection Region. There is a set of larger Tier 2 ISPs that peer at many exchange points in many Interconnect Regions.

Tier 2 Interconnect Model

Figure 8 – Generalized Tier 2 Interconnection Model

As a result, the category of Tier 2 ISPs is much broader and more diverse category than that of the Tier 1 ISPs. There is a wider range of peering policies, and the Tier 2 interconnect mesh tends to be more sparse than that of the Tier 1 ISPs. The Tier 2 ISPs after all have an alternative to peering (transit!) For Tier 2 ISPs, Peering is a local routing optimization.

Tier 2 ISPs Relationship with each other. At the same time there is great heterogeneity of this Tier 2 ISP group, there is also great cooperation, driven by aligned interests. Tier 2 ISPs by definition all purchase transit and therefore generally are interested in peering with each other.

Tier 2 ISPs Relationship with Tier 1 ISPs. Tier 1 ISPs are transit providers, as well as competitors for the really big customers. Since the Tier 2 ISPs know that they will be referred to as the “middle man” in competitive situations with the Tier 1 ISPs, they tend not to mention that they purchase transit at all. Hence Sean Donelan’s comment that “Everyone is a Tier 1 ISP .”

If we model the Tier 2 ISPs as a group, we see an aggregate set of traffic that is exchanged within the group, a set that gets exchanged up through the transit provider(s), and a set of customers being provided with transit services as shown below.

 

Notes, Sources ...

Some Peers have to carefully control Peering Traffic Exchange Ratios in order to maintain peering relationships.

BGP as it is deployed today uses a set of criteria for path selection that doesn’t automatically take into account the cost of traffic going along a particular path, nor the performance of the path.

Ren Nowlin – SBC, paraphrased. See the “Interconnection Strategies for ISPs” for a comparison of these two interconnection techniques.

Available from the author: wbn@DrPeering.net and on the DrPeering web site: http://www.DrPeering.net .

The “Tier 1” label has varied in meaning from “A Top Shelf ISP”, to a “Global Player”, to a “large national player” to “whatever we decide Tier 1 means!”

Since definitions shape thought and meaningful consistent dialogue, it is a necessary prerequisite to consistently use terms and definitions.

This in part has led to some confusion over the Peering Ecosystem players.

This is significant because ISPs often will not peer with an entity they consider a potential transit customer. By not being in the transit sales business, there is no such disincentive to peer.

Ecosystem Player: The Content Provider

Definition: Content Providers are all companies that operate an Internet Service but do not sell transit within the Internet Peering Ecosystem. Model of Content Provider in the Internet Peering Ecosystem

The model above shows this player simply purchasing transit from an upstream provider while creating content as their core business.

 

Large-scale content players have always purchased transit to deliver their content to the end-users on the Internet. Therefore, their role and behavior in the Peering Ecosystem is similar to the Tier 2 ISPs with the exceptions that most Content Players:

Content Provider Model. The Content Provider model is a simple model. On the bottom of the model is content connected to the Internet via one or more transit (upstream) connections.

ContentProvider Model

Figure ## - The Content Provider Model

Content Players include companies like Amazon.com, eBay, and Staples.com. Enterprises include Avon.com, Hertz, Agilent, and General Electric.

Content Providers Relationship with Each Other. Generally speaking, there is no relationship between the content providers. They either compete with each other (so don’t communicate) or they have no interest in each other (so don’t communicate). There are two exceptions: some e-mail providers exchange e-mail freely and directly, and some content providers interact using extranets in supply chain integration forms. None of these however has been directly seen to influence the peering ecosystem.

Content Providers Relationship with the other Players. The Content Providers generally see the ISPs as vendors, and select vendors along widely varying parameters, increasingly looking at the ISPs peering infrastructure .

Sources and Notes ...

This is significant because ISPs often will not peer with an entity they consider a potential transit customer. By not being in the transit sales business, there is no such disincentive to peer.

The Peering Ecosystem

Pulling these individual player models together into a Regional Peering Ecosystem leads to a generalized picture below.

Generalized Peering Ecosystem

 

Figure 10 - Generalized Peering Ecosystem, consisting of Tier 1 ISPs, Tier 2 ISPs, and Content/Enterprise Players.

Grouping these three models of Ecosystem participants (Tier 1 ISPs, Tier2 ISPs, Content Providers) we see the relative roles in the hierarchy of traffic exchange as shown below.

Ecosystem_images/Tier 1Tier 2 Content Providers in the Ecosystem

Figure 11 - Tier 1 ISPs, Tier 2 ISPs, and Content Providers in the Peering Ecosystem

Peering Inclinations and Peering Policies

We can explain an ecosystem participant's Peering Behavior by identifying their position and corresponding Peering Inclination as articulated in the form of a Peering Policy .

Definition: A Peering Inclination is a predisposition towards or against peering as demonstrated by Peering behavior in an Internet Peering Ecosystem.

Definition: A Peering Policy is an articulation of the Peering Inclination; it documents and defines the prerequisites to peering.

The 4 Categories of Internet Peering Policy

Our research has identified four categories of peering policies:

Definition: A Restrictive Peering Policy is an articulation of an inclination not to peer with any more entities.

A Tier 1 ISP may have a posted peering policy indicating that they will peer with ISPs of similar size and scale that meet them in a large number of locations in a certain number of countries with a certain large traffic volume. In this research, several Tier 2 ISPs have expressed frustration that once they met these stated requirements, the requirements were immediately adjusted upwards just out of reach. This demonstrates the Restrictive Peering Policy underlying foundation - an inclination not to peer.

Definition: A Selective Peering Policy is an articulation of an inclination to peer, but with some conditions.

A Selective Peering Policy details prerequisites that must be met, and when met, will generally lead to peering. Some ISPs for example require multiple interconnect points across a country with a minimum amount of traffic in order for it to be worth the engineer’s time to set up the peering session. Large Tier 2 ISPs often have Selective Peering Policies; they want to peer, but have a threshold to pass in order for it to be worthwhile. The difference between the Selective and the Restrictive peering policy is somewhat subjective but is based on the underlying intent; a restrictive peer does not want to peer with anyone else so sets requirements arbitrarily high such that almost no one is qualified to peer with them.

Definition: An Open Peering Policy is an articulation of an inclination to peer with anyone.

Brokaw Price from Yahoo! calls this “pulse peering; if you have a pulse we will peer with you!” Content Players and most Tier 2 ISPs seek to reduce their transit fees by peering broadly by announcing to the peering community that they have an Open Peering Policy.

While these broad categories of players and their peering policies do not capture all facets of the markets, they do provide the broad strokes needed to understand some of the International peering interplay that we will discuss in this paper.

Definition: A No-Peering Policy is an articulation of an inclination not to peer at all.

Those with a no-peering policy simply do not see peering as an important strategy for for them to employ to get connectivity to the global Internet.

Examples of Peering Inclinations and Peering Policies.

To apply these definitions to the Internet Peering Ecosystem, we see in the field that most Tier2 ISPs and peering Content Providers have an Open Peering Inclination, and may or may not even have an articulated Peering Policy. Yahoo! for example has a one word Peering Policy: “Yes! ”

Some Tier 2 ISPs, particularly those deployed across many markets and Interconnection Regions, have a Selective Peering Inclination intended to manage traffic effectively and perhaps to ensure that they are getting equal value from a peering relationship. They may design a peering policy to make sure that the peer adheres to engineering criteria (adequate backbone scale, multiple interconnection regions, etc.) or operations criteria (a 24/7 NOC and updated contact and pager information, etc.). There are a wide range of Selective Peering Policies reflecting a Selective Peering Inclination.

Tier 1 ISPs generally have a Restrictive Peering Inclination articulated in their Restrictive Peering Policies. Since they are only required to peer to ensure connectivity for their customers, and they forego revenue by peering , there is not much incentive to accept additional peers . The Sprint Peering Policy is not available on the web but was cited to include an OC-48 capacity to Europe and an OC-12 to Asia .

Now that we have generalized the Peering Ecosystem today, and touched on some of the Peering Behaviors of the players, we will move on to some recent significant changes in the U.S. Peering Ecosystem.

In order to show the evolution of the Peering Ecosystem, we will use the generalized representation below to show the peering within these groups and transit between these groups of players.

Generalized Representation of the Regional Peering Ecosystem

Figure 12 - Generalized Representation of the Regional Peering Ecosystem

Sources and Notes ...

Ren Nowlin (SBC) described the conflicts a Peering Coordinator faces.

On one hand, there are some peering relationships that can not be divulged due to non-disclosure agreements.

One the other hand, the prospective customers genuinely care about the degree of interconnection and therefore connectedness of the ISP.

Generalized from the authors four years of research with the Peering Coordinator Community. Eyeball networks don’t want to carry traffic across the world for free…

See both sides of this argument at: http://www.cctec.com/maillists/nanog/historical/9808/msg00138.html This is difficult to prove but it is nonetheless a view widely held in the Peering Coordinator community.

Note that there are business-rational motivations for this stance. The “Art of Peering” (freely available from the author) white paper documents some tactics and business arrangements that have led to peering with Tier 1 ISPs.

Some Tier 1 ISP Peering coordinators made the case that “What is out of reach for some is reasonable and reachable for another.”

This is a valid criticism of this categorization, but there are many stories from the field supporting this categorization. Many conversations with Brokaw Price (Yahoo! Peering Coordinator).

See the SBC Peering Policy at http://www.sbcbackbone.net/peering and AOL’s Selective Peering Policy at http://www.atdn.net/settlement_free_int.shtml for good examples of well articulated Selective Peering Policies. Other Peering Policies can be viewed at http://vision.opentransit.net/docs/peering_policy/peering_policy.html , http://global.mci.com/uunet/peering/

A denied peer may become a direct or indirect customer. However, recently, there has been greater government scrutiny over Tier 1 Peering Policies.

The threat of government regulation is resulting in the Tier 1 ISPs applying their Peering Policies with consistency.

Source: Several NANOG conversations. It is interesting that requirements for U.S. peering include a global footprint. Others have commented that what is restrictive to one ISP, is reachable to another ISP.

Evolution of the U.S. Peering Ecosystem

During the last few years, five significant events contributed to a dramatic change on the U.S. Peering Ecosystem. In particular:

  1. The 1999/2000 Economic Collapse of the Telecom Sector has led to the bankruptcy of several major Tier 1 ISPs , many Tier 2 ISPs , as well as a long list of content companies. This has generally caused Internet players to ensure that their business model and practices are financially sound. (The author notes that this has led to an increased interest in the Economics of Peering White Papers .)
  2. The Growth of the Used Networking Equipment Market (selling equipment reaped from the bankrupt companies) allows ISPs to reduce the capital cost of Peering lowering the breakeven point for peering . Michel Py points to a used Cisco 12000 series router loaded available on eBay for $25,000 as an example.
  3. The Upstream Provider for the Cable Companies (@Home) Went Bankrupt.. This forced each North American cable company to negotiate emergency multi-Gigabit per second transit connections with several of the Tier 1 ISPs.
  4. Peer-to-Peer File Sharing Networks grow exponentially in popularity and Traffic between Access Network Providers. Napster (and later) Kazaa, Morpheus, etc. Peer-2-Peer file sharing users around the world have shifted from sharing 4MB music files to sharing 700MB movies on the Internet. These applications relentlessly consume all of the end user’s available Internet bandwidth attempting to download chunks of the files from any sources on-line at the time. It may take weeks, but eventually, the music files or video files are downloaded. Since many Kazaa users leave their PCs on 24/7, the access networks are filling up 24/7, resulting in significant costs to the access-heavy ISPs (Cable Companies and DSL Providers in particular).
  5. Transit Rates Drop and Transport Rates Drop. At the same time that transit prices plummeted, the transport prices into exchange points dropped. The revenue the ISPs can expect to extract from their deployed equipment has diminished. The cost for customers to get transport into an exchange point for peering has dropped dramatically. This further motivates companies to peer with each other. This has led some to peer as a risk mitigation strategy; peer to mitigate the risk of transit prices rising, buy transit to mitigate rise transport and peering costs .

As a result of these forces we are seeing three major shifts of the Peering Ecosystem.

 

Evolution #1: The Cable Companies are Peering

Most Cable Companies were forced to replace the Internet Service that @Home once provided. They had 30 days to establish multi-gigabit-per-second transit relationships with Tier 1 ISPs! After some initial analysis, they found that about 40% of this Cable Company traffic is Peer2Peer traffic , and ultimately destined for other cable companies .

Cable Company Model

Figure 13 - Cable Company Model

As a result of this analysis and as a routing optimization, the cable companies migrated into exchange points to peer with each other as shown graphically below.

Cable Company Model

Figure 14 - Cable companies Peer with each other with huge traffic volumes

The Peering Cable Companies and their Peer2Peer traffic represent a significant change in the Peering Ecosystem because:

Other Broadband Players. One question that often comes up when describing the Peering Ecosystem is: Why do you pick out the Cable Companies and not the RBOCs and other DSL providers? The RBOCs and many of the DSL Providers sell transit to other companies in the Peering Ecosystem, and therefore may have a disincentive to peer with them. Aside from providing large amount of access bandwidth, they generally have a Selective Peering Inclination, and are not changing the Peering Ecosystem as dramatically as the Open Peering Cable Companies.

Sidebar: A related Kazaa story involves a grandmother in Australia that had her grandkids over during the holidays. In Australia, end users are charged for Internet access on a per-Megabytes-downloaded basis. When the grandkids wanted to hear the latest Britney Spears song on grandma’s computer, they installed Kazaa and downloaded the song. By default Kazaa shares all files it downloads, and soon grandma’s computer became the preferred source for the Britney Spears songs! Grandma’s bill from Telstra was several hundred dollars as opposed to the usual $25 she pays per month! In the U.S. where broadband is typically a flat fee there is no such disincentive to use all the bandwidth available.

Evolution #1 - Cable Companies Peer

Figure 15 - Evolution #1 - Cable Company Peering.

 

Sources and Notes ...

WorldCom, Genuity, Global Crossing are among a few of the Tier 1 ISPs that have gone bankrupt. Covad, Broadwing, Williams, etc. Also see Bill Woodcock’s papers on Peering at http://www.pch.net/resources/papers/ See “Do ATM-based Internet Exchanges Make Sense Anymore” to see how the used equipment market affects the Peering Breakeven Analysis. Capital expense is a major concern for many Peers: “Nobody buys new anymore!”. Source: Michel Py. Eric Bell – Time Warner Telecom white paper walk through July 2003.

Joe Klein (Adelphia) demonstrated this at the Chicago Gigabit Peering Forum in 2002 in his talk “Peer-to-Peer Traffic Growth and its Impact on Peering”. This statistics was validated (roughly) with the other cable companies, and has been seen as high as 75% by a large scale overseas ISP. Note that this traffic is ultimately destined to any Access Heavy ISP with end users running Peer2Peer Applications. At the time Adelphia estimated 4 gigabit per second of traffic went to their upstream ISPs. If 40% of that is peering traffic to the other Cable Companies, we have 1.6Gbps of peering traffic! Source: Conversations with Joe Klein, North American Cable Company Peering Conference Call 4/16/2002. Tier 1 ISPs are peering with each other in each Interconnect Region using multiple OC-12s (n*622Mbps) and OC-48s (2.5Gbps). Source: Anonymous Tier 1 Operator, June 1, 2003. Source: I$P HO$TING ReportMay 2003, Volume VII, Number 5, Page 8, Article "Dialup Bakeoff - Is it really so bad at EarthLink? Or so good at United?" citing The Bridge & Leichtman Research. Shaw and Rogers estimates are from the author conversations with these companies. A few of the Cable Companies are refining their peering policies. Cox for example has articulated a rather Selective Peering Policy: http://www.cox.com/INETPeering/ and AOL/Roadrunner has documented a selective peering policy at http://www.atdn.net/ . This is proving to be an example of a strong Peering gravitational pull. See http://www.atdn.net for Peering Policies and Interconnection Regions for the AOL Transit Data Network. Shared with the audience at the Los Angeles Gigabit Peering Forum, Feb 13, 2003. Read about this story on the web at: http://www.smh.com.au/articles/2003/06/16/1055615717838.html Note that there are costs associated with peering here including transport fees, colo fees, port fees, etc.

 

Evolution #2: Network Savvy Large Scale Content Companies are Peering

A select group of network savvy Content Providers are emerging as large volume peers. Most of them peer bi-coastally, some peer more broadly across the country. The dominant motivations to peer are similar to the Tier 2 ISP motivations to Peer:

  1. To Reduce Transit Costs. Any traffic that can be sent directly to the Access-Heavy (also called Eyeball) Networks is traffic that doesn’t have to go over the metered transit connection. Likewise, E-Mail Service Providers such as MSN HotMail and Yahoo! can exchange E-Mail without incurring any transit charges .
  2. To Improve the end-user experience. Yahoo! for example applies considerable resources monitoring (in real-time!) the application performance along the various Internet paths to identify congestion points and to alter the network path to improve the end-user experience.
  3. They need a move to a new collocation area anyway. Many collocation and datacenters have gone out of business. Given the choice, some content players prefer a location where they can peer as well as buy transit .

The model for the Network Savvy Large Scale Content Provider is shown in the figure below.

Large Scale Network Savvy Content Provider enters Ecosystem

Figure 16 - Network Savvy Large-Scale Content player Model

These players still purchase transit (generally from the Tier1 ISPs), but supplement this by peering openly with anyone and everyone. Examples of these Large Scale Content Players are Yahoo!, Google, Microsoft, Amazon, Walmart.com, Apple, Electronic Arts, and Sony Online.

This represents a significant change to the Peering Ecosystem because:

We can see an example of this effect in the graphic below.

LargeScale Network Savvy CP Peers

Figure 17 – Content Company Peering Effect

Sources and Notes ...

Note that there are costs associated with peering here including transport fees, colo fees, port fees, etc. Cook Report: Nov/Dec 2002 Issue. Several people reached the same conclusion in this peering study. Brokaw Price (Yahoo!) actively participates in NANOG Peering BOFs and industry Peering Forums such as the Gigabit Peering Forums.

Evolution #3: Cable Companies Peer Directly with Content Companies

The peering of the Network Savvy Large Scale Content Companies with the Cable Companies puts the most popular content on the Internet on the same network as the largest set of subscribed broadband eyeballs. The end result is a major disruption in the Internet Ecosystem, motivated by network performance between the customers of both and significant cost savings from peering!

The Network Savvy Large Scale Content Companies and the Cable Companies can be shown as a substrate above the other Content Companies, demonstrating different behavior (peering, operating networks, detailed network flow analysis, etc.). The degree to which traffic is exchanged below the Tier 1 clouds, is the degree to which the traffic and revenue is being pulled away from the Tier 1 players as shown in the graphic below.

Cable Companies Peer with Large Scale Network Savvy Content Providers

Figure 18 – Evolution #3 : Cable Companies peer with Large Scale Network Savvy Content Providers

This represents a significant dynamic shift in the Peering Ecosystem since the Tier 1 ISPs are being cut out of a high volume traffic exchange loop. The Tier 1 ISP significance has decreased as can be measured in the growth of the Tier 1 ISP traffic. A hierarchy has evolved to a mesh with the Tier 1 transit providers required only for the route of last resort. Traffic destined to locations too far away or too expensive to reach will ultimately need the services of an international transit provider. Other than that the Tier 1 transit providers have lost their grip on the U.S. Internet Peering Ecosystem.

 

Summary

This paper introduced some terms including the Internet Region and Interconnect Region as an environment in which the Internet Peering Ecosystem exists. Tier 1 ISPs are at the top of the Peering Ecosystem providing transit services to the rest of the Internet Region. Tier 2 ISPs are distinct from the Tier 1 ISPs in that they must purchase transit from one or more ISPs to reach the Regional Internet. The Content Providers have traditionally purchased transit.

We are seeing some interesting and dramatic changes in the Peering Ecosystem as the Network Savvy Large-Scale Content Companies and the Cable Companies are getting into Peering with large volume peering sessions.

We introduced four categories of Peering Policies that reflect Peering Inclinations. Open Peers will peer with anyone anywhere. Selective Peers will peer with some prerequisites, detailed in a Peering Policy. Restrictive Peers generally do not peer. Content Providers may have a No Peering inclination, preferring solely to buy transit. Recent shifts in the Peering Inclinations can be summarized in the graph below:

Peering Inclination Changes over time

Here we see some of the Network Savvy Large Scale Content Providers shifting from a NoPeering Peering Inclination to an Open Peering Inclination in order to reduce Transit Fees and better control traffic and the end user experience. Tier 2 ISPs for the most part remain Open Peers or NoPeering. Some Tier 2 ISPs, particularly those that expand beyond a few Interconnect Regions have a more Selective Peering Policy. The RBOCs remain split into two categories: Selective Peers and those who choose NoPeering because they prefer to only buy transit. The Tier 1 ISPs are highly Selective or Restrictive in their Peering Inclination. The Foreign ISPs follow a pattern: first applying their perceived market strength in the country, and then realizing that they don’t have market power, are reduced to peering Openly with the Tier 2 ISPs.

Follow On Research Topics

We have laid out the Ecosystem and the players, their behavior and motivations in a way that leads nicely into a Simulation or Modeling system. Can we further refine the models to identify prospective peering candidates and why they should want to peer with each other? Can we pull in data (such as the Internet Exchange Point participant’s lists ) to populate an Ecosystem model with currently peering Tier 2 ISPs? Comparing the network topology maps for these ISPs, can we infer their role in the Peering Ecosystem?

It would be interesting to map out the Cable Company territories across the Interconnect Regions and see if they are peering in the local Exchange Point. Do they have Peering Policies on their web sites, and are they indeed evolving from Open to Selective? Will AOL, the largest of the Cable Companies, become a Tier 1 ISP, and will their Peering Policy change when that happens ? Is AOL incented then to de-peer its Cable Company peers?

Can we quantify the effect of the Cable Companies peering with the Content companies? Is there evidence of the reduction of load on the Tier 1 ISP networks, or at least a slowing of growth as the traffic prefers the more direct path? Can we more precisely define and quantify the notion of value to the Peering Ecosystem? Is the model of eyeball networks peering scalable or will a hierarchy develop?

Acknowledgements

The following people have greatly contributed to and influenced the development of this white paper by providing insight, advice, and/or allowing me to walk them through the early, very rough, drafts of this document, providing critical feedback and suggestions: Ren Nowlin (SBC), Christina Chu (NTT/Verio), Joe Zhu (China Telecom USA), Joe Provo (RCN), Terry Manderson (APNIC–formerly from Optus), Dave McGaugh (PCCW/BTN), Philip Smith (Cisco), Keith Mitchell (ExchangePoint), Fumio Terashima (Japan Telecom America), Geoff Huston (Telstra), Patrick Gilmore (Akamai), Jonathan Chou (Equinix), Phil Grasso (NTT), James Spenceley (Comindico), Chris Chaudry (Comindico), Gavin Tweedie (WAIX), Lloyd Taylor (Keynote Systems), Tatsuo Sato (KDDI America), Sam Habash (SuperNews), Joe McGuckin (Via.Net), Mike Leber (Hurricane Electric), Morgan Snyder (Equinix), Eric Bell (Time Warner Telecom), Peter Cohen (Telia), Dan Golding (AOL), Alexander Tudor and Lance Tatman (Agilent Labs), Eric Aupperle (President Emeritus, Merit Network), Dave Meyer (Sprint, Cisco), Michel Py, Keith Young, Nenad Trifunovic (MCI), Sean Butler, Marcellus Nixon (XO), Mark Seery.

Sources and Notes ...

See http://www.ep.net for a list of Regional Interconnect Points (Exchange Points) which typically have their customers listed.

See the presentation from the GPF VII in Ashburn, VA: “The Road to Tier 1 Nirvana”, Vikas Mehta, AOL. https://ecc.equinix.com/peering/agendas.htm


Presentation Materials

 

  1. Evolution of the U.S. Peering Ecosystem.ppt

Mentions

  1. http://www.mendeley.com/collections/569921/Next-Generation-Networks/ An agent-based model for the evolution of the internet ecosystem, Amogh Dhamdhere Georgia Institute of Technology Constantine Dovrolis Georgia Institute of Technology, Proceedings of the First international conference on COMmunication Systems And NETworks
  2. Interconnection and Peering among Internet Service Providers, InterIsle Consulting Group, Lyman Chapin, Chris Owens, 2005

Related Documents

 

Internet Transit Pricing Historical and Projections

 

Index of other white papers on peering

About the Author

Who is William B. Norton?

Executive Director, DrPeering International

Chief Strategy Officer, International Internet Exchange (IIX)

WIlliam B. Norton is the author of The Internet Peering Playbook: Connecting to the Core of the Internet, a highly sought after public speaker, and an international recognized expert on Internet Peering. He is currently employed as the Chief Strategy Officer and VP of Business Development for IIX, a peering solutions provider. He also maintains his position as Executive Director for DrPeering.net, a leading Internet Peering portal. With over twenty years of experience in the Internet operations arena, Mr. Norton focuses his attention on sharing his knowledge with the broader community in the form of presentations, Internet white papers, and most recently, in book form.

From 1998-2008, Mr. Norton’s title was Co-Founder and Chief Technical Liaison for Equinix, a global Internet data center and colocation provider. From startup to IPO and until 2008 when the market cap for Equinix was $3.6B, Mr. Norton spent 90% of his time working closely with the peering coordinator community. As an established thought leader, his focus was on building a critical mass of carriers, ISPs and content providers. At the same time, he documented the core values that Internet Peering provides, specifically, the Peering Break-Even Point and Estimating the Value of an Internet Exchange.

To this end, he created the white paper process, identifying interesting and important Internet Peering operations topics, and documenting what he learned from the peering community. He published and presented his research white papers in over 100 international operations and research forums. These activities helped establish the relationships necessary to attract the set of Tier 1 ISPs, Tier 2 ISPs, Cable Companies, and Content Providers necessary for a healthy Internet Exchange Point ecosystem.

Mr. Norton developed the first business plan for the North American Network Operator's Group (NANOG), the Operations forum for the North American Internet. He was chair of NANOG from May 1995 to June 1998 and was elected to the first NANOG Steering Committee post-NANOG revolution.

William B. Norton received his Computer Science degree from State University of New York Potsdam in 1986 and his MBA from the Michigan Business School in 1998.

Read his monthly newsletter: http://Ask.DrPeering.net or e-mail: wbn (at) TheCoreOfTheInter (dot) net

Click here for Industry Leadership and a reference list of public speaking engagements and here for a complete list of authored documents

About the White Papers...

The Peering White Papers are based on conversations with hundreds of Peering Coordinators and have gone through a validation process involving walking through the papers with hundreds of Peering Coordinators in public and private sessions.

While the price points cited in these papers are volatile and therefore out-of-date almost immediately, the definitions, the concepts and the logic remains valid.

If you have questions or comments, or would like a walk through any of the paper, please feel free to send email to consultants at DrPeering dot net

Please provide us with feedback on this white paper. Did you find it helpful? Were there errors or suggestions? Please tell us what you think using the form below.

 

Contact us by calling +1.650-614-5135 or sending e-mail to info (at) DrPeering.net

 

 

Back To DrPeering Home

DrPeering.net ©2014 | Privacy Policy | Terms and Conditions | About Us | Contact Us

The 2014 Internet Peering Playbook is now available on the iPad at the Apple Store and for the Amazon Kindle. The Kindle, ePub and PDF form are also perpetually updated on the DrPeering DropBox share.

Sponsors