Testing QUIC Against TCP for Streaming Video Delivery: An SVTA Proof-of-Concept

Americans streamed almost 15 million years’ worth of content in 2021, and while 62% of all US households currently subscribe to some form of streaming service, 46% spend more than $20 a month on one or more premium services.

Services including Netflix and Amazon Prime have grown organically, not only into global content delivery services, but also as prominent entertainment studios and content creators.

Traditional broadcasters including Disney and CBS have also embraced this new distribution model by building their own OTT services and competing for eyeballs and dollars with pure-play streaming companies. In the case of Disney, they also started releasing movies online at the same time they are in the theater.

To keep pace with this demand, HTTP and TCP have evolved, but the changes to those protocols have resulted in potential performance degradation for streaming services. The Quick UDP Internet Connections (QUIC) protocol, standardized as IETF RFC9000, provides solutions to those TCP issues while optimizing the user experience for consuming OTT services.

The SVTA has published an in-depth technical brief on the QUIC protocol, its advantages and disadvantages, infrastructure and network requirements and its applicability to streaming media, therefore this will not be repeated in this document.

Some SVTA QUIC Pre-PoC Thoughts and Hypotheses

We believe that, at least on paper, QUIC promises to improve the user experience when consuming high value content delivered from the CSP or CDN edge and could help cultivate additional value for consumers and revenue for content owners. CSP’s and Public CDN’s also cannot wait for QUIC to become ubiquitous in other on-line applications before deciding whether to experiment – the time to learn is “now”.

The SVTA QUIC PoC Purpose and Objectives

Based on that hypothesis, the main purpose behind this PoC is to validate some of the assumptions related to QUIC within the context of streaming media, and to illustrate how the protocol could be leveraged in existing or new OTT platforms. Additionally, this PoC will establish best practices for organizations that are contemplating their own QUIC implementation, allowing them to assess the effort required and the overall impact of delivering a QUIC-based streaming service.

Important deliverables will include:

• a reference deployment architecture that will serve as a foundation for content owners or CSP’s that intend to run their own lab or consumer trials
• a detailed analysis outlining the functional and performance gains (or deficits) that were observed between QUIC and TCP deployments,
• a set of recommendations and best practices that should be considered by the industry when including QUIC as a part of their overall strategy and delivery infrastructure.

The target audience for these deliverables is predominantly corporate technology leadership including CTO organizations and their technical staff who are focused on implementing and supporting caching and network infrastructure within CSP’s, CDN’s, and global hyperscalers; however, it is anticipated that product and industry futurists will also find the results of this PoC informative when setting mid- to long-term corporate strategy.

Methodology

The SVTA will conduct the QUIC PoC based on a two phased approach.

The first phase will focus on a deployment within the confines of a lab environment. The second phase will extend the distribution of content into the public domain using fixed access and wireless infrastructure provided by SVTA member companies.

Each phase will include several iterations. During each iteration the data collected will be examined and the PoC design and test cases evaluated to improve PoC environment, approach, and methodology. These revisions will form the foundation of recommendations and best practices deliverables.

Phase 1

Phase 1 testing will place an emphasis on two distinct aspects of the QUIC implementation,  functional validation and overall performance, with three key deliverables:

• Reference architecture
• QUIC functional and performance testing
• Infrastructure performance testing

Reference Architecture

The initial goal is to establish a reference architecture and implementation as well as to document a baseline for the infrastructure and applications that are required to support QUIC.

QUIC Functional and Performance Testing

Functional and performance testing will be based on the reference architecture and focus principally on the key features anticipated to improve the streaming experience. This will include 1-RTT session establishment, 0-RTT session continuance, session portability, latency improvements, head of line (HOL) blocking improvements, and QPACK performance.

These features will be compared with corresponding features (if available) provided by TCP to determine if there is a quantifiable improvement.

Infrastructure Performance Testing

Server, client, and infrastructure performance will also be assessed to determine if QUIC places additional demands on the streaming environment, and the underlying congestion controls will also be evaluated. The POC will assess the impact and implementation flexibility stemming of shifting the congestion control functionality from the kernel space to the user space. It will also determine whether modern algorithms such as BBR and PCC provide an improvement over legacy TCP CUBIC of the IETF’s recommended TCP NEW RENO implementation.

Phase 1 Lab Deployment

The following diagram provides an overview of the lab deployment in Phase 1.

The infrastructure will consist of an edge proxy acting as a cache for content fragments and manifests. It will fulfill both TCP/HTTP2 and QUIC/HTTP3 client requests. This cache will proxy missed cache requests from the client to an upstream origin server that will act as an ingest point for live streams and a repository for on-demand content.

Video players will support both TCP/HTTP2 and QUIC/HTTP3 requests, allowing concurrent testing and KPI measurement for both protocols. The player requests fragments from the cache to simulate a real world CDN deployment.

As the delivery method used has significant impact on the performance of the streaming services, the POC will assess multiple delivery methods, the support for TCP CUBIC, BBR, and PCC is included for both protocols and implemented at the system level for TCP and in the user space for QUIC

A traffic shaper will sit between the video player and the edge proxy and inject loss and latency into the network to simulate real world conditions. The goal is to run both TCP and QUIC tests under variant network conditions (bandwidth fluctuations, latency, packet loss, different network buffer size, etc.) to determine the effect on each protocol.

Finally, an analytics platform will be used to collect and process data from both the client and the cache. Client data will be gathered using a standard CDN analytics platform while data from the edge infrastructure will be collected using an agent deployed on the cache and uploaded to a cloud service. This approach allows both sets of data to be combined and provides a much richer set of KPI’s.

Phase 2

Phase 2 testing extends the PoC by adding content delivery over fixed access and LTE/5G networks through a set of network adaption functions. Functional validation and overall performance will again be evaluated, however additional KPI’s will be added and correlated to the underlying delivery network infrastructure.

The following diagram provides an overview of the Phase 2 testing environment.

KPI’s for subjective testing will be added to test scenarios executed by a group of SVTA volunteers. Testers will watch content and assign a score for each test scenario. These results will be combined with objective test results from the player and the network/edge cache level and will be combined to provide an overall Mean Opinion Score (MOS). Additional testing focused on reliability, capacity, and scalability will also be included in the Phase 2 scenarios.

For each test scenario in both phases the following can be adjusted:

• Test Protocol: HTTPS (TCP) and QUIC (UDP)
• Object Size: Limiting access based on Bitrate and Resolution (SD, HD, 4K.
• Fragment size
• Network conditions: upstream and downstream bandwidth, latency, packet loss, etc. This will be completed leveraging a network emulator in the lab phase but reflect real-world conditions in Phase 2

Test Content

Test content will include both live and on-demand and will be encoded as H.264/AAC-HE and packaged as DASH/ISOBMFF. Live sources will be encoded directly from acquisition feeds and will include a range of content complexities from live sports to “talking heads”. On-demand content will be delivered from common origin services within the lab.

There will be no DRM and the player buffer will be configured at 4 seconds. Bitrate ladders will be limited to 4 profiles to help force bitrate switching

Test Cases and Test Case Hierarchy

Phase one test cases will be limited to the fixed access network infrastructure available within the lab environment leveraging HD and 4K MPEG-DASH content as encoded in bitrate ladder table previously.

Phase two testing will be further refined based on lessons learned from phase 1; however, it is assumed that the test cases will be ostensibly the same but conducted over public infrastructure.

Initial testing will focus primarily on validating QUIC features and functionality as outlined in the table below.

A second round of testing will compare the performance of QUIC and TCP.

The third and final round will evaluate how QUIC and TCP perform with different standardized congestion control algorithms that are common in IP networks.

Test Clients

All test cases will leverage two sets of clients that will support subject and objective measurements.

Test subjects will access content through a SVTA branded web page implementing Clappr and Shaka in Javascript. Subjects can view content and provide a “happiness” score based on their subjective observed experience. Subjects viewing VR/360-degree content will leverage the same environment that was used for the PoC recently conducted in the VR Working Group

Test clients will also be made available for automated, objective testing, leveraging Clappr and Shaka but embedded within a command line client that is deployed on the user’s equipment.

Key Performance Indicators

The following KPI’s will be collected and analyzed for each test scenario:

Client

• Average Bitrate
• Rebuffering Ratio
• Startup time / Buffer Fill
• Duration/Play Length
• Lag Length
• Lag Ratio
• Device Data

Network

• RTT
• Throughput
• Packet Loss
• Latency
• % Reordering (Optional)

Edge Cache

• CPU Utilization
• System Load
• Load Average/CPU ratio
• Memory Utilization
• Memory/Processes ratio
• Swap/Paging
• Disk Utilization & IO
• Inode Usage
• Network throughput
• Context Switches

It is anticipated that this data will provide the source for the functional compliance, performance analysis, and the recommendations and best practices documents.

Some Final Thoughts

We believe that experimenting with QUIC, in parallel with production streaming delivery over standard TCP, is a low-cost endeavor. CSPs and ISPs can do so even while waiting for the results of our PoC testing. But doing so now will also help start seeding the edge with a foundation for QUIC and providing empirical data that correlates to performance and quality improvements. Once our PoC is done and the documents produced, CSPs and ISPs can compare their own findings and make optimizations to their QUIC implementations based on our findings.

Although it is relatively unknown what the outcome of our testing effort will be, anecdotal evidence from trial deployments have shown genuine performance gains. But it will take a broad representation of streaming ecosystem participants to establish empirical baselines.  The SVTA would happily welcome CDN, caching, player, analytics, and encoding vendors as part of this POC effort.

Networking and Transport Working Group

Website