VPN testing methodology

When it comes to recommending VPNs, I’m very particular. I wouldn’t suggest anyone use a service that I haven’t thoroughly tested. Of course, since we’re dealing with digital privacy, it’s important to show any workings. Below, I’ll walk you through Privacy.net’s VPN testing process and explain what I look for in a Virtual Private Network.

Finding out if a VPN is suitable for a particular task is more complex than simply getting connected and trying it out for an hour or two. There’s often a degree of fact-checking involved, too. Some VPN providers stretch the truth about what, in reality, are less-than-ideal services. Here’s a list of questions I ask when testing a new VPN:

  • Can this service adequately protect my privacy?
  • Is my internet traffic properly secured?
  • Is this VPN fast and reliable enough to meet my needs?
  • Will I be able to securely access my usual services while traveling?
  • Does this VPN have what it takes to bypass online censorship?
  • Can I get help with problems in a reasonable amount of time?
  • Which devices does the VPN provider make apps for?
  • How much am I willing to pay for this service?

As you might expect, none of these questions are easy to answer. To really address them, I have to dive deep into each aspect of the service. I’ve included a breakdown of these areas below, and will highlight the things that I consider non-negotiable, as well as any red flags that might indicate you’re considering a sub-par VPN.

Privacy

VPN privacy

When I mention privacy, I’m talking about a VPN’s ability to keep your identity and browsing habits hidden from third parties such as your Internet Service Provider or the government. Notably, the VPN provider itself shouldn’t be able to see what you do online either. Strong security features have a part to play here, but I’ll discuss those a little later. Instead, this section will focus on the administrative processes a VPN uses to shield you from anyone who comes looking.

Anonymous payment

The first of these presents itself before you’ve even signed up. If a service allows you to pay in gift cards, cryptocurrencies such as Bitcoin, or other non-standard currencies (such as jars of honey), it’s possible to register without revealing any information about yourself that a credit card would. This includes your real name, bank of choice, and the town you live in.

Home country

Where a VPN is based makes a huge difference. This determines which country’s laws the service must obey with regard to data collection, third-party access, and liability. In Russia, for instance, the government must be able to access a VPN’s servers, whereas Switzerland has a history of passing very pro-privacy legislation.

I strongly prefer services based in countries that are not known for their strong surveillance capabilities. This rules out the US, the UK, and all of the other 14 Eyes countries, as well as titans of censorship like China and Iran.

Logging policies

This is arguably the most important thing to consider from a VPN privacy perspective. A VPN’s logging policy lets you know what kind of information the service stores as you browse. Ideally, you’d use a VPN with a true no-logs policy (sometimes called a zero-logs policy). This ensures that the service you’re using doesn’t keep records of:

  • Your original IP address, or that of the server you’re using
  • The time that you connected to the VPN
  • How long your session lasted
  • Which websites you visit, what you search for, and who you talk to

So why is this important? In a nutshell, it can be used to personally identify you. For instance, if a VPN can say exactly which user logged into a specific server at a specific time, it becomes much easier to discern who accessed what. This problem is only compounded if your real name and address are linked to your user account, which is why I prefer VPNs that let you sign up anonymously. 

If you can’t find a privacy policy, or it’s vague about the kind of information that is logged, that’s a strong sign that you’re dealing with an untrustworthy VPN.  

It’s worth mentioning that not all logs are bad for your privacy. Some reputable providers record how much data you’ve transferred, which lets them offer a free service with limited monthly bandwidth. They may also record the date (but not time) of your connection and your chosen location, which helps debug any issues that might occur.

See also: Best VPNs for Tor

Security

Most people know that VPNs encrypt their traffic, but there’s much more to concealing your activities than this, and digital cryptography is a field you could feasibly spend years studying. In the interest of time, here’s a quick overview of the technology your VPN uses to conceal your activities from anyone who might be watching, and the things I consider to be mandatory for proper security:

  • Encryption: This encodes your traffic, rendering it unreadable to any observers who don’t have the required decryption key. 128-bit AES keys are the bare minimum, but I prefer 256-bit keys as they’re even harder to crack.
  • Protocols: Different protocols route and conceal your traffic in different ways. A service has to support at least one up-to-date, secure protocol like OpenVPN, L2TP, or WireGuard. If it has its own fully-audited, open-source protocol, so much the better.
  • Key exchange: To create a secure connection, your device has to exchange keys with your chosen server. Of course, these have to be encrypted, or else anyone could intercept them. Diffie-Hellman keys are considered the gold standard, provided they’re at least 2048 bits in length, since this algorithm allows us to take advantage of perfect forward secrecy (more on this below). I expect to see SHA3 adoption in the near future.
  • Authentication: VPNs use hashing algorithms to verify whether communications are really coming from a trusted sender or not, and whether the contents of data have been modified. Most VPNs use SHA keys that are at least 256 bits long (confusingly, these are often called SHA-2 keys, even if they’re 384 or 512 bits in length), which is good because SHA-1 is now deprecated and unsafe.
  • A kill switch: This stops transferring data if your connection drops unexpectedly, preventing you from accidentally browsing unprotected.
  • Leak protection: Some VPNs encrypt your traffic but leave your IP address and DNS requests visible to anyone who knows where to look. That just won’t do, which is why all of the services I recommend have IPv6, DNS, and WebRTC leak protection built-in.
  • Perfect forward secrecy: Essentially, this changes the session’s private key periodically so that even if your key is compromised, an attacker can only see a small section of your activities. They’ll be locked out again as soon as the key changes.
  • Servers: I prefer servers to be physically located in their respective countries instead of simply existing virtually. This way, users know exactly which country’s laws apply. Several VPNs have also introduced RAM-only servers, which clear all stored data the moment they’re turned off — these aren’t mandatory but they’re certainly nice to have and provide a little extra peace of mind.

The security features laid out above are just a baseline. If a provider offers a wider range of tools, like traffic obfuscation, automatic wifi protection, or tracker-blocking, that’s even better.

Be wary of services that refuse to provide detailed information about their security features. It’s easy to use terms like “military-grade encryption” because these sound impressive and rarely have an agreed-upon definition, whereas “256-bit AES encryption” means something very specific.

Speed

Anyone who’s used a VPN knows that their speeds can vary quite a bit depending on the time of day you connect, where your chosen server is located (and how busy it is), and which protocol you’re using. In fact, a provider could have some servers that are much faster than others. As such, there’s no guarantee that the ones I test will continue to provide the same level of performance. As you might expect, these factors make it fairly tricky to rank services in terms of speed.

The best way I’ve found of doing so is to connect to a server in North America, one in Asia, and one in Europe at three different times of the day. On each occasion, I perform a speed test and once all nine tests are finished, I consider the average result to be a fair indicator of a provider’s overall speed. Note that I’m on a 1 Gbps connection, so the VPNs I’m testing won’t be bottlenecked by a slow base speed.

VPN speed testing

There are a couple of things I can do to make things as fair as possible. Firstly, I always use the same speed testing app: Ookla Speedtest.net. I close my browser and use the desktop app to ensure that the test isn’t impacted by any open tabs.

I also make sure to use the fastest protocol on offer, provided it’s still considered safe to use. Historically, this has been OpenVPN or IKEv2, but more providers are making the jump to WireGuard (which is significantly faster), so this will likely become the default moving forward.

Now, high speeds on paper don’t mean much if they don’t translate to real-world application. As part of Privacy.net’s testing, I also try streaming high-resolution video and playing fast-paced online games to make sure that they’re suitable for day-to-day use.

Streaming capabilities

Just about every major streaming service uses some form of geo-blocking these days. It might be that they completely prevent access to their platform outside of a specific region, or simply change the available content. However, given the frequent disparity between content libraries, it should be no surprise that many people have begun using VPNs to regain access to their usual streaming services as they travel.

Streaming providers know this, though, and have taken steps to identify VPN users based on their traffic patterns. As part of my testing, I check to see if a particular VPN can securely access services like:

  • Netflix
  • HBO Max
  • Amazon Prime Video
  • BBC iPlayer
  • Disney+
  • Peacock
  • Hulu
  • Paramount Plus

If possible, I’ll also test whether the VPN lets you use a service’s mobile app or just the web browser. Where additional functionality is available (such as the ability to download shows for offline viewing), I’ll attempt to take advantage of this as well.

It’s a big red flag if a VPN tries to convince you that it can access every streaming platform. Even industry-leading providers have difficulty with certain services, and will readily admit so in their website’s support section.

Evading censorship

Unfortunately, there are countries where online freedom isn’t respected in the way we are accustomed to. For instance, China has a collection of tools known as the Great Firewall that help the government restrict access to huge swathes of content including social media platforms, foreign news sources, and privacy-related topics.

Privacy.net strives to accurately determine whether a given VPN can bypass this type of country-wide blocking. However, these situations are fluid, and a previously-working service may be blocked without warning. Usually, major providers can find a workaround a short time later, but whatever happens, I’ll update the review to reflect any changes.

While less serious, web-filtering also takes place on a smaller scale at universities, schools, and workplaces. You might not be able to browse Facebook on your lunch break, for instance, due to local internet policies. The right VPN should be able to get around these limitations without breaking a sweat.

Network

VPN server network

As I mentioned above, it’s difficult to quantify server performance. This is one reason why a VPN with a larger network isn’t necessarily faster or more reliable. Rather, it’s better to consider the locations a provider lets you connect to. There are a few questions to ask yourself here, such as:

  • How many countries does the network cover?
  • Does the VPN have servers in the country I’m in?
  • Are all of the servers in a country clustered in one region?
  • Is there a server in my home country that I can use to access my usual services while traveling?
  • Do I have to use specific servers for torrenting? Where are these located?

It’s also a good idea to consider whether your provider uses virtual servers. While traditional VPN servers are physically held in racks in a warehouse, virtual servers only exist in the cloud. Users shouldn’t notice any difference in performance, but it does become much more difficult to tell which country’s laws apply. On the plus side, virtual servers make it possible for VPN providers to support locations where maintaining a physical presence would be too expensive or dangerous.

Help and support

No judgment here: it’s entirely possible you’ll hit a snag and have to ask for help. You might be sick of trying to figure out which servers work with a given streaming service, for example, or just looking for assistance installing the VPN on your home router.

While writing a review, I always make sure to get in touch with a VPN’s customer support team. I note down what forms of support are available (live chat, email, ticket systems, and so on), as well as whether these are only available to subscribers or not. I also keep a record of how long support staff take to reply, what hours they operate, and whether their response adequately addresses the issue raised.

Can’t find out how to contact your VPN provider? Most reputable services have an email address or live chat window on their website. If yours doesn’t, and its only form of support is a frequently asked questions section, you may have chosen an unscrupulous provider.

Multi-platform support

What’s the point in a VPN if it only protects some of your devices? Almost every major provider has its own apps for iOS, Android, MacOS, and Windows. Some go even further, with command line-based installers for Linux systems, native apps for Amazon Fire TV, and browser extensions for Chrome and Firefox.

Some devices don’t actually support VPNs, usually because they have limited app stores. This includes most games consoles, Chromecasts, Apple TVs, and many non-Android Smart TVs. However, you can install a VPN directly onto your router, which will reroute traffic from every device on your network. As a bonus, this will also allow you to bypass your VPN’s connection limit. Since a router only counts as one VPN connection, you can connect as many devices as you like.

VPN Pricing

I’ve seen all kinds of pricing structures. These range from the standard flat monthly fee to build-your-own packages that charge by the server. What I’m more concerned with, though, is whether a service is priced fairly. In other words, how do its features hold up against other VPNs of a similar price?

I’ll also consider whether the service offers any discounts on its longer-term subscriptions, whether there’s a money-back guarantee or free trial available, and whether it charges extra for things like a static IP address.

While researching a service, I keep an eye out for unsavory business practices too. This could be a “no refunds” policy, terms of use designed to prevent you from getting your money back, or even something as simple as automatic subscription renewal.