I understand the argument: Since there is so much TCP (HTTP anything, etc. etc.) this is what the user experiences.
That argument has several issues. First, we need to discuss the problems with using TCP at all as there are technical reasons the idea does not align with the goal of speed tests. Second, the argument is aging very poorly and very quickly, especially now that we have the emergence of QUIC over UDP.
At the end of the day, TCP should never be used for Speed Tests. Let me justify this.
There are two associated articles you may be interested in reading:
These articles offer the speed test sites that most people refer to, as well as the results of some testing we did using those sites.
Let’s begin by clarifying the main purpose of Speed Tests.
What is the main Purpose of a Speed Test?
I would offer that speed tests are used by folks to measure the speed of their access to the Internet. For example, if I am paying for Internet Service at my home, or on a plane, I want to know if I am getting what I paid for. Simple. That’s it.
I will acknowledge that Speed Test sites gather data for marketing purposes, but using or not using a particular protocol does not matter, except that if the wrong protocol is used, the data has limited value.
Translated into implementation, the user wants to know if I am paying for a 400 Megabit/second service – think of it as the speed of a conveyor belt – is the conveyor belt running at that speed? Am I getting what I paid for? Is it consistent? Can I depend on that service?
Most ISP’s have put a speed test server in their CO – essentially 1 hop away from the user to limit the number of conveyor belts in the equation. This is smart, however it does not eliminate the question about the use of TCP.
Why is TCP the Wrong Protocol?
TCP (Transmission Control Protocol) is not ideal for speed tests primarily due to its inherent design characteristics that prioritize reliability over speed.
Here’s a detailed explanation of why TCP may not give an accurate reflection of your network’s true speed:
1. Overhead and Variations from Connection Setup (Three-Way Handshake)
TCP requires a connection to be established before any data can be transferred. This connection setup involves a three-way handshake, which adds latency to the start of the data transmission. For a speed test, this additional time can skew results, especially over short tests or high-latency networks. The delay from setting up the connection can reduce the perceived throughput, especially in scenarios where the speed test only lasts a few seconds.
If the Speed Test software and protocol stack uses different settings in Maximum Segment Size (MSS), options such as Selective Acknowledgement (SACK) or Window scaling, or initial Window sizes, then the test measurements will vary.
2. Congestion Control and Flow Control
TCP includes mechanisms like congestion control and flow control to ensure that packets are delivered reliably and without overwhelming the network or the receiving device. These mechanisms throttle the data transmission rate to avoid packet loss and congestion, but this can result in artificially lower speeds during a speed test.
- Congestion Control: TCP dynamically adjusts the speed at which data is sent, especially when network conditions are congested. It starts slow and ramps up (this is called slow start) until it reaches an optimal rate, but this ramping period can make the speed appear slower at the start.
- Flow Control: If the receiving device signals that it is becoming overwhelmed (buffer limits, for example), TCP slows down the transmission, which may not reflect the true capacity of the network.
- Impact: These controls can reduce the peak throughput measured by the speed test, even if the network could handle higher speeds in ideal conditions.
These algorithms make TCP unpredictable as to performance.
3. Packet Retransmissions
If packets are lost or delayed, TCP will retransmit those packets to ensure reliable delivery. While this is important for applications like file transfers or web browsing, it can artificially reduce the speed reported during a speed test, as it lowers the goodput (the amount of usable data delivered). In networks experiencing packet loss or jitter, TCP’s need to retransmit lost packets can lead to slower speed test results, even though the actual capacity of the network might be higher.
4. Buffering and Window Size
TCP uses a sliding window mechanism to control the amount of data sent before receiving an acknowledgment. If the window size is too small, or if there is significant delay in receiving acknowledgments (due to latency or packet loss), the throughput will be limited. In high-latency or high-bandwidth networks (such as long-distance or satellite connections), the TCP window size can become a bottleneck, preventing the full bandwidth from being utilized during a speed test.
5. Head-of-Line Blocking
TCP delivers packets in order, and if a packet is lost or delayed, the receiver has to wait for the missing packet before processing subsequent packets. This can result in delays and lower perceived speed, even if the majority of the network is functioning well. Head-of-line blocking can delay throughput and lead to an artificially lower speed test result, especially if there are any issues with packet delivery during the test.
6. Limited Burst Transmission
Because TCP prioritizes smooth and reliable data transmission, it avoids sending large bursts of data all at once. Instead, it regulates the flow to ensure packets arrive in order and without loss. This can lead to slower speed measurements compared to other protocols that might send larger bursts of data. If a speed test needs to assess the maximum possible bandwidth in a short period, TCP’s conservative approach might underreport the actual capacity.
7. Acknowledgment Overhead
TCP requires that every segment of data sent is acknowledged by the receiver. This adds extra traffic in the form of ACK packets and introduces delays, as the sender often needs to wait for these acknowledgments before sending more data (depending on the window size and congestion control). The time spent waiting for acknowledgments, along with the overhead introduced by additional packets, can reduce the overall throughput measured by the speed test.
8. Impact of Latency
Since TCP’s performance is tightly coupled to the round-trip time (RTT) of the connection, the higher the latency, the slower TCP will operate, especially in terms of ramping up its transmission rate. This is particularly noticeable in long-distance or satellite connections. If you’re testing speed over a high-latency connection, TCP’s slow-start mechanism and other latency-related controls will lead to a slower speed test result compared to protocols that aren’t as latency-sensitive.
9. There is no version number in TCP – so which version are we using?
That’s right. Yet there are lots of different versions:
TCP CUBIC, New Reno and Compound TCP are the most popular (highlighted), but that does not mean that is the exact version in play on a given connection. If you want to read more on this, click here.
Can we just stop there? You would think I am making this up, but I am not. So what are our options?
Comparison with Other Protocols for Speed Tests
There is only one other protocol in the Transport Layer – that is UDP.
UDP (User Datagram Protocol)
Many speed tests use UDP instead of TCP because UDP is connectionless and does not have the same overhead or congestion control mechanisms as TCP. UDP allows for more accurate measurement of the maximum possible throughput, because:
- No Congestion Control: UDP does not adjust the speed of data transmission based on network conditions. This means it can more accurately reflect the maximum available bandwidth.
- No Packet Retransmission: UDP does not resend lost packets, so it measures the network’s ability to transmit data without the need for error correction.
- Minimal Overhead: UDP has minimal header and acknowledgment overhead compared to TCP, leading to a clearer measurement of raw bandwidth.
However, the tradeoff with UDP is that it does not guarantee delivery, so it is not suitable for applications where reliability is important, but for pure speed measurement, UDP is often preferred.
What if we used QUIC over UDP?
QUIC (Quick UDP Internet Connections) is considered an excellent protocol for speed tests because it is designed to improve performance, particularly in the areas of latency and connection establishment, making it a great fit for evaluating network speed. Here are the key reasons why:
1. Fast Connection Establishment
QUIC reduces the overhead of setting up connections. Unlike traditional TCP, which requires a three-way handshake (SYN, SYN-ACK, ACK), QUIC combines connection establishment and encryption setup into a single step. This means the connection is ready to transmit data much faster, which is crucial for speed tests where minimizing setup time helps focus on actual data transfer speeds.
2. Reduced Latency
QUIC is designed to reduce latency, especially in environments where connection quality fluctuates or where network paths are long (i.e., high-latency networks). Since speed tests aim to measure data throughput and responsiveness, having a low-latency protocol like QUIC helps get more accurate results in real-world conditions.
3. Built-In Encryption
QUIC integrates Transport Layer Security (TLS) by default, which means that all data is encrypted from the get-go. This is not just good for security, but it also removes the extra round trips required for negotiating encryption in protocols like TCP. The faster setup allows for more immediate testing.
4. Multiplexing and No Head-of-Line Blocking
Unlike TCP, QUIC allows multiple streams of data to be sent over a single connection without head-of-line blocking. In TCP, if one packet is lost, the entire stream is held up until the missing packet is retransmitted. QUIC, being based on UDP, doesn’t have this issue, which results in better performance during speed tests as packet loss doesn’t degrade throughput as much.
5. Better Adaptation to Network Conditions
QUIC was designed to adapt better to changing network conditions. It has more advanced congestion control algorithms and uses forward error correction (FEC) to recover from packet losses without relying on retransmissions as much as TCP. This adaptability leads to more consistent results in speed tests, especially in environments where the connection quality fluctuates.
6. Resilience to Network Middleboxes
QUIC is more resilient to interference from middleboxes (e.g., routers, firewalls) that can tamper with or delay TCP traffic. By operating over UDP, QUIC sidesteps some of the legacy issues that affect TCP-based speed tests, leading to more accurate results.
7. Seamless Connection Migration
QUIC supports connection migration, meaning that a connection can continue uninterrupted if a user’s IP address changes (e.g., moving from Wi-Fi to cellular). In speed tests, this ensures that performance is measured continuously even if there are slight changes in the network setup.
These features make QUIC a great protocol for speed tests, as it ensures quick connection setup, reduced latency, better handling of packet loss, and more efficient throughput, leading to more accurate and real-world performance measurement.
What Did We Find in our Testing?
I refer you back to our prior articles (links at the top of the page). We got a myriad of results. That said, what protocols did we see?
Here is what we found: Spoiler Alert — THEY ALL USE TCP!!!!!
| Test Site | Download Speed | Protocols Used | From the 3-way Handshake MSS (client) (server) SACK (y/n) Stabilized Calculated Window Size Window Scale (client) (server) |
| speedtest.net | 103.36 Mbps | TCP | 1460 (client) 1460 (server) y 64128 x256 (client) x128 (server) |
| Google Speed Test | 108.3 Mbps | TCP (why don’t they do QUIC?) | 1460 (client) 1460 (server) y 64128 x256 (client) x128 (server) |
| Fast.com | 89 Mbps | TCP | 1460 (client) 1460 (server) y 1050112 x256 (client) x512 (server) |
| nperf | 135.9 Mbps | TCP | 1460 (client) 1460 (server) n 64512 x256 (client) x512 (server) |
| Testmy.net | 59.9 Mbps | TCP | 1460 (client) 1460 (server) y 43008 x256 (client) x1024 (server) |
| Speedofme | 56.54 Mbps | TCP | 1460 (client) 1460 (server) y 75264 x256 (client) x512 (server) *interesting to note they set the do not fragment flag on all packets from the server |
| Mlab | 103.07 Mbps | TCP | 1460 (client) 1460 (server) y 64128 x256 (client) x128 (server) |
| OpenSpeedTest | 160.71 Mbps | TCP | 1460 (client) 1460 (server) y 32256 x256 (client) x512 (server) |
| Xfinity | 89.4 Mbps | TCP | 1460 (client) 1460 (server) y 36864 x256 (client) x4096 (server) |
Conclusions
Now that we have dug a bit deeper into the speed test experiment, looked at what protocol is used and what TCP settings were experienced, can we draw any conclusions?
Apart from TCP throwing us curve balls with its options and then the invisible stuff we can’t control in its algorithms, the part we should have looked at was number of hops away by running and recording a trace route to each of the servers. We did not do that. My suspicion is that the slower speed tests resulted in multiple conveyor belts being involved, albeit ones that are much faster in totality, they are being shared by many other users, and these other conveyors become the bottleneck, plus I am no longer just testing my service providers access speed.
Which brings up a point: what are we really testing? None of the speed tests explain anything about this, so most users don’t stop and think about it.
I should point out that many managed Wi-Fi systems now run speed tests of their own, from the access point to a server in the local central office (essentially one hop away). That is the best of scenarios from a conveyor belt standpoint, but again, what protocols are being used? How busy is that conveyor with other traffic?
If you ever hear me say, speed tests should have never been invented, and they aren’t designed correctly, you now have the detail as to why. The problem is they were, and the “cat is out of the bag”. So we should at least do speed testing correctly, so that the variations in results, and the legitimacy of the results is grounded in good network science.
I should also point out, there is one speed test software that at least allows you to select UDP called Tamosoft. See this page. It requires a server and client installation, of course. Hopefully some of these other speed test sites will get their act together one day.
Comments are welcomed below from registered users. You can also leave comments at our Discord server.
If you would like to see more content and articles like this, please support us by clicking the patron link where you will receive free bonus access to courses and more, or simply buying us a cup of coffee!