Netflix Secretly Holds Subscribers Hostage to Gain Favorable FCC Internet Regulations

Posted by | September 16, 2014 | Broadband Internet | 11 Comments

A stunning revelation is buried in a lengthy Netflix filing at the Federal Communications Commission (FCC): Netflix used its subscribers as pawns in a Machiavellian game of regulatory chess designed to win favorable Internet regulations.

The filing reveals that Netflix knowingly slowed down its video streaming service with the intention of blaming Internet service providers (ISPs). Specifically, Netflix used its relationships with Internet ‘backbone’ providers (e.g., Level 3, Cogent) to deliberately congest their peering links with ISPs, and at the same time, started publishing ‘ISP speed rankings’ to make it appear that ISPs were causing the congestion. It appears that Netflix cynically held its subscribers hostage to reduced service quality in order to pressure the FCC into adopting favorable Internet regulations that would permanently lower Netflix’s costs of doing business.

Netflix’s plan to frame ISPs for sabotaging its service has been surprisingly successful so far. Some subscribers have blamed their ISPs for the service disruptions Netflix itself caused, which prompted the FCC to open an investigation of the Internet backbone market. Now all Netflix needs is for the FCC to adopt new regulations.

This post attempts to shed some light on Netflix’s hostage strategy before it’s too late. After a brief summary of the Internet backbone market, it describes the origins of the strategy and explains how Netflix succeeded in manipulating public opinion.

The Internet Backbone Market

Unlike many other segments of the communications marketplace, the FCC has never regulated the prices or terms of exchanging traffic among the backbone networks that connect ISPs to the global Internet. The competitive Internet backbone market has successfully developed industry norms for exchanging commercial traffic through privately negotiated agreements for (1) transit and (2) peering.

Transit. Providers of ‘edge’ services on the Internet have traditionally used transit to reach end users. Transit agreements establish a simple, paid customer-supplier relationship in which one network sells access to all Internet destinations, typically on a metered basis.

Peering. Peering relationships traditionally developed between large Internet backbone providers who exchange substantial amounts of traffic. Under peering arrangements, networks provide transport and reciprocal access to each other’s customers, typically without payment (‘settlement-free’). Most peering agreements are settlement-free because they are barter transactions in which networks trade the use of their infrastructure to exchange roughly equivalent traffic. If any party to a peering agreement perceives that the benefit derived from peering has become asymmetric, that party may seek payment for peering or terminate the peering relationship.

Netflix’s Goal

Internet transit is often a substantial component of an edge provider’s operating costs. Netflix, which accounts for 34 percent of Internet traffic during peak hours, has particularly strong incentives to improve its bottom line by reducing its Internet transit costs. Firms usually cut costs by increasing operational efficiencies or obtaining better terms from suppliers. In the highly regulated communications market, however, firms have the additional option of asking the FCC to intervene, which is the strategy Netflix has decided to pursue.

Netflix wants the FCC to give it the benefits of settlement-free peering relationships while retaining the rights enjoyed by customers in transit relationships. Put another way, Netflix wants to stop paying for transit, and it wants the government to make it happen. If Netflix gets its way, it would put an end to industry norms that have governed the Internet backbone market for decades.

The Level 3 Gambit

The origins of Netflix’s hostage strategy go back to 2010, when Netflix decided to switch a significant portion of its streaming video traffic from Akamai’s content delivery network (CDN) to Level 3.

In a press release touting the switch, Netflix expressed confidence in Level 3’s “ability to quickly scale to meet demand,” and Level 3 claimed it could “better control the performance of [its] CDN” as compared to its competitors. Such puffery isn’t unusual in a press statement. But, the notion that Level 3 could somehow “better control” the performance of its CDN than Akamai is particularly notable in this context. CDNs like Akamai exercise control over the performance of their services through transit agreements with ISPs, which places CDNs and ISPs in a customer-supplier relationship. As a transit customer, a CDN can request that an ISP provision an additional port whenever the CDN requires additional capacity, and the ISP has every incentive, as well as a contractual duty, to fulfill the request.

If the Netflix/Level 3 deal had been motivated solely by the performance of Level 3’s CDN services, the deal would have passed into obscurity. Unlike Akamai, however, Level 3 is also an Internet backbone provider that has settlement-free peering agreements with many major ISPs. Level 3’s status as an Internet peer offered Netflix something that Akamai could not — an opportunity to create the conditions for a successful regulatory challenge to industry norms governing the delivery of CDN traffic. That opportunity explains why Level 3 proposed to send its newly acquired Netflix traffic to Comcast under their settlement-free peering agreement immediately after the Netflix/Level 3 deal was announced.

Comcast’s rejection of Level 3’s proposal would not have been a surprise to Level 3. The proposal would have resulted in a significantly asymmetric traffic exchange ratio between Level 3 and Comcast, respectively. Based on established industry norms for settlement-free peering relationships and the fact that Level 3 itself had objected to similarly unbalanced relationships in the past, Level 3 would have expected Comcast to respond with a request for paid peering or a transit arrangement.

It is telling that Level 3 capitulated only three days after the ‘dispute’ began (immediately after Comcast described its request for transit as a “take it or leave it” offer). Though Level 3 ostensibly accepted Comcast’s terms in order “to ensure customers did not experience any disruptions,” it doesn’t appear that service disruptions were imminent. Netflix had retained its relationship with Akamai, and the transfer of Netflix’s traffic to Level 3’s network was intended to occur over time. In these circumstances, it would have been passing strange for Level 3 to concede a commercial dispute so easily — unless it had already achieved its real purpose. Three days was all Level 3 needed to lure Comcast into “stat[ing] a position” that would support a regulatory complaint.

A subsequent statement by Level 3’s legal counsel removed any doubt that the ‘dispute’ was merely a pretext for seeking to extend net neutrality regulations to the Internet backbone market: “It’s not about the money that we’re now being forced to pay Comcast. It’s about the precedent.”

The FCC did not fall for Level 3’s gambit. The agency’s previous Chairman determined that the FCC’s net neutrality rules did not apply to the Internet backbone market, and he showed no inclination to propose additional regulations.

The Netflix Hostage Strategy

After the failure of the Level 3 gambit, Netflix realized that an obscure industry complaint wouldn’t be enough to convince the FCC to regulate the competitive backbone market without evidence of consumer harm. So Netflix adjusted its strategy. It decided to create consumer disruption by taking its own subscribers hostage to reduced service quality and blaming the results on ISPs.

According to its Petition to Deny the Comcast/Time Warner Cable merger, “in early 2012, Netflix began to transition its traffic off of CDNs and onto transit providers with settlement-free routes into Comcast’s network.” Based on industry norms with respect to peering and its experience with the Level 3 gambit, Netflix knew that transferring its traffic to settlement-free routes would result in network congestion by dramatically altering the ratio of traffic exchanged.

Netflix also knew that ISPs were likely to balk at providing additional settlement-free ports. As noted above, settlement-free peering agreements are barter relationships based on mutual benefits, not ordinary customer-supplier relationships. If a settlement-free peer exceeds the scope of the agreement unilaterally, its peering partner has no obligation to provide additional services on a settlement-free basis. The peers must reevaluate their bargain when determining how to alleviate the congestion.

That’s why Netflix started publishing ‘ISP speed rankings’ in 2012 that specifically measure the speed of Netflix’s connections with ISPs. If Netflix was going to blame ISPs for the congestion it was about to create, it needed to establish baseline performance metrics for the delivery of its traffic that would paint ISPs as the culprits. The initial baseline metrics established in 2012 included traffic Netflix had traditionally delivered through CDNs with paid transit arrangements. Netflix knew that, once it created congestion by shifting its traffic to settlement-free routes, the initial baselines in its ‘ISP speed rankings’ would start to drop.

Implementing the Plan

With respect to Comcast and Verizon, Netflix initially transitioned its traffic to Cogent, who had settlement-free peering relationships with those ISPs. Predictably, Cogent’s settlement-free routes became congested.

Netflix admits that, in 2013, congestion on these settlement-free routes steadily increased to a level that adversely affected Netflix’s subscribers. Yet Netflix did nothing. It could have alleviated the congestion by shifting some of its traffic back to CDNs or other transit arrangements, but it decided to watch and wait instead. To succeed in pressuring the FCC, Netflix needed to let its service quality decline until it generated some headlines.

It didn’t take long for the tech community to oblige. Based on the first Netflix ‘ISP speed rankings’ issued after the D.C. Circuit Court of Appeals overturned the FCC’s net neutrality rules, Consumerist speculated that Comcast and Verizon “might be throttling some of the traffic moving through them,” and the allegation became widespread. Netflix had finally succeeded in persuading the public to link the Internet backbone market with net neutrality.

It wasn’t hard for Netflix to stoke speculation about ISP throttling. As Ars Technica noted when it analyzed claims that Verizon was throttling, end users typically can’t determine what is causing network congestion. Netflix’s ‘ISP speed rankings’, which suggest that ISPs are solely responsible for the speed of Netflix downloads, were designed to take advantage of this fact. Though Netflix acknowledges that its rankings are affected by many factors, including factors that are entirely within Netflix’s control, it also says “these factors cancel out when comparing across ISPs.”

That isn’t true with respect to peering and transit facilities. When it decided to send its traffic solely through settlement-free transit providers in 2012, Netflix knew full well that its decision to congest peering and degrade the quality of its service would affect some ISPs very differently than others. Netflix had also started its own CDN in 2012, and some ISPs — those who do not operate Internet backbone networks, and thus would otherwise have to pay transit fees to receive Netflix traffic — had already agreed to the unusual arrangement of directly interconnecting with Netflix’s CDN without charge. Those ISPs would not be adversely affected by Netflix’s hostage strategy, and Netflix knew it. Netflix intended to cast blame only on the ISPs that operate Internet backbones, and thus would not realize the same cost savings from Netflix’s CDN as the ISPs who do not operate backbone networks.

Netflix’s ‘ISP speed rankings’ made no effort to account for this fundamental difference or to inform the public that Netflix’s internal transit decisions would have a disparate impact on different ISPs. Netflix nevertheless used its misleading rankings to publicly castigate ISPs that operate Internet backbones for choosing to adhere to long-standing and well-accepted industry norms regarding Internet peering practices.

Contrary to Netflix’s allegations, recent research suggests that these norms are economically efficient. In their case study of the congestion that occurred between Cogent and Comcast in 2013, a group of computer scientists from the Massachusetts Institute of Technology and the Center for Applied Internet Data Analysis at the University of California, San Diego, found that “congestion can more or less instantly shift (in a day or so) from one path to another.”

[T]hese rapid shifts strongly suggest that the correct response to growing congestion is not always to add more capacity. On the contrary, adding capacity to a link might be a poor investment if a content provider can shift a huge fraction of the traffic from that link to another link overnight.

This conclusion also indicates that Netflix could exercise a substantial amount of control over the timing and the extent of the congestion that occurred during the implementation of its hostage strategy.

The FCC investigation

Netflix’s false ‘consumer crisis’ prompted the FCC to open an investigation “to understand whether consumers are being harmed.”

The FCC should start by asking Netflix why it continued to shift its traffic to settlement-free peering routes even after its service quality began to decline. Why didn’t Netflix take action to protect its subscribers from reduced service quality when, according to Netflix, its transit costs are “so small” that they aren’t worth accounting for in its pricing decisions and aren’t “customer affecting”?

It should also ask Cogent and Level 3 why they didn’t agree to paid peering or alternative transit arrangements with Comcast and Verizon once their settlement-free peering ports became congested, which would have been the typical practice under industry norms. Did Cogent’s and Level 3’s agreements with Netflix prohibit them from paying Comcast and Verizon for additional ports?

Finally, it’s curious that Netflix’s speed drops began to peak when the D.C. Circuit Court of Appeals was deciding the legality of the FCC’s net neutrality rules, yet never reached such an egregious level that Netflix risked losing a substantial number of customers. Did Netflix intentionally manipulate its traffic flows in a manner that would cause noticeable speed drops during this period while avoiding any severe slowdowns that might actually cause it to lose subscribers?

Every Netflix subscriber should want to know the answers to these questions.

11 Comments

  • Telecomsense says:

    Really good piece, Fred! Here’s my question: do we know if the non-Netflix traffic carried by the backbone companies went “along for the ride?”

    Level 3 is one of the very largest CDNs, and carries traffic for a lot of companies. Imagine if you experienced Skype problems, or excessive “glitching” on your PlayStation network, at the same time Netflix was congesting the ISP’s peering point?. You, and the companies whose services you were using, would not be thrilled to know that services you are paying for were being degraded as part of this misguided, advocacy-oriented, traffic manipulation. Not only would you be angry, but the person paying for the “CDN” service (like Microsoft or Sony) probably has a cause of action–under contract and under Section 208.

    You’d hope that the Netflix-only traffic was treated separately by the backbone carriers. But, if it was, this only underscores your point. If the backbone provider is using dedicated facilities on its own network to handle Netflix’s traffic, it seems patently unreasonable to expect the ISP to add its own dedicated facilities to accommodate traffic that was never “peered” at any other point in its delivery path–for the sole reason of accommodating another carrier’s “CDN” customer.

  • Robert Enger says:

    In most cases, Netflix and others use geolocation technology to select a content-server closest to the requesting customer. Thus, the source is often in the SAME CITY as the customer requesting the content. NO LONG HAUL backbone service is required from the last-mile network operator (Comcast or Verizon). Literally, the last-mile provider (comcast, Verizon, et al) are only being asked to carry traffic from one point in the city to the customer’s home in the SAME CITY. They are REFUSING to do that. What exactly is the residential customer paying for, if their last-mile provider won’t carry traffic from one side of the city to another??
    And the traffic-balance argument is a fiction. The underlying technology of most last-mile networks (eg GPON, ADSL) are asymmetric. And many last-mile providers forbid their residential customers from operating servers or other equipment that would generate large amounts of upstream traffic (to balance the OTT video coming downstream).
    Last-mile providers are carefully pulling the wool over the eyes of the commission. And the consumer is getting screwed.

    Finally, the note makes reference to “metered” usage. HAH! No big telecom provider charges by-the-byte on internal network pipes. Its 95th-percetile 5-minute peak flow RATE that is charged. This reflects the actual burden on the provider (they have to build-out equipment to support RATE, not bytes sent). And the same is true of last-mile networks. Billing for total bytes-sent is a complete scam to disadvantage OTT video. It takes no account of peak rates (or even time of day of the use). It prevents pre-push of video during off-peak hours. Again, the last-mile provider is pulling the wool over the commission and consumer-protection regulators.

    If a last-mile provider won’t carry traffic from one side of the city to another without extorting the content-supplier for a piece of the action, what exactly is the customer paying for each month? Maybe the fight should leave the FCC and move to FTC or DoJ.

    • Fred Campbell says:

      Your comment doesn’t reflect how industry norms with respect to Internet traffic exchanges work. For an ISP that doesn’t own backbone facilities, the fact that the ISP doesn’t generate large amounts of upstream traffic means that the ISP would have to pay transit fees in order to receive Netflix traffic (though it’s my understanding that the Netflix CDN doesn’t adhere to this industry norm). Your implication that transit fees always go one way and that ISPs thus have an incentive to avoid upstream traffic in order to gain an advantage in the transit market is simply false.

      As a point of clarification, the post’s reference to metering was referring to the 95th percentile method of measuring traffic, which is a form of metered use.

  • Bob says:

    Can you post where in the filing this was discovered? It is quite a lengthy piece, and some direction or citation to the relevant sections would be appreciated. Thank you.

    • Fred Campbell says:

      The key admission, that “in early 2012, Netflix began to transition its traffic off of CDNs and onto transit providers with settlement-free routes into Comcast’s network,” is on page 55.

  • […] is your reminder that Netflix is sabotaging its own customers to lobby for extreme cronyist […]

  • […] said that Netflix allegations against Comcast and Verizon Communications throttling data had been proved false by information found in the video streaming company’s August filing with the […]

  • […] recently explained how Netflix held its subscribers hostage to reduced service quality while blaming Comcast. […]

  • […] its own customers hostage during negotiations with ISPs. A number of independent sources cite evidence that Netflix deliberately slowed its own traffic in an attempt to gain attention from the FCC, […]