Examples & benchmarks / #8 · 2026-09-30 · Benchmark · Medium · for web scraping
A bandwidth diet for residential proxies: what blocking really saves
Count every byte a news homepage pulls through your proxy, then block images, media, fonts and third parties — and see where the money actually goes.
Run on production on 2026-09-30: ✓ Node.js ✓ Python ✓ Java ✓ C# ✓ Go
A benchmark: the results below are what this setup measured on the dates shown, on our own test runs. Sites, networks and browsers change, so rerun the code before you rely on the numbers.
The problem
Residential proxies bill per gigabyte, so a scraper's cost is mostly bytes. The usual advice is "block images". How much does that save on a real, ad-heavy page — and what else is worth blocking?
What we used, and why
| What | Why |
|---|---|
context.route('**/*') + route.abort() | Drops requests before they leave the browser, so they never cost proxy bandwidth. |
request.resourceType() | Classifies requests: image, media, font, script, xhr, … |
| A first-party list | The site's own domains and CDNs (bbc.com, bbc.co.uk, bbci.co.uk); everything else is third-party (ads, analytics, trackers). |
request.sizes() on requestfinished | Exact bytes on the wire per request: request and response headers plus the encoded body. |
| A fresh context per variant | Empty cache each time, so every variant pays full price. |
How it works
- Load the page with nothing blocked; count requests and bytes for DOM-ready plus 5 seconds.
- Again, blocking images, media and fonts.
- Again, also blocking every third-party host.
- Convert to proxy cost per 100,000 page loads at $3/GB.
The code
The same program in five languages (also on GitHub, with the raw output). Set these environment variables first:
CDPFLEET_API_KEY— your API key (dashboard → API keys)PROXY_URL— your proxy, e.g.http://user:[email protected]:8000
// npm install [email protected]
// env: CDPFLEET_API_KEY, PROXY_URL
import { chromium } from 'playwright';
const KEY = process.env.CDPFLEET_API_KEY;
const PAGE = 'https://www.bbc.com/news';
const FIRST_PARTY = ['bbc.com', 'bbc.co.uk', 'bbci.co.uk']; // the site's own domains and CDNs
const PRICE_PER_GB = 3; // a typical residential proxy price, USD
const res = await fetch('https://starter.cdpfleet.com/chromium/session', {
method: 'POST',
headers: { 'x-api-key': KEY, 'content-type': 'application/json' },
body: JSON.stringify({ proxy: process.env.PROXY_URL, headless: true }),
});
if (!res.ok) throw new Error(`launch: ${res.status} ${await res.text()}`);
const { wsUrl } = await res.json();
const browser = await chromium.connect(wsUrl, { headers: { 'x-api-key': KEY } });
const HEAVY = new Set(['image', 'media', 'font']);
// Load the page in a fresh context (empty cache) and count every byte on the wire.
async function measure(label, block) {
const context = await browser.newContext();
if (block) {
await context.route('**/*', (route) => {
const r = route.request();
if (HEAVY.has(r.resourceType()) || (block === 'first-party' && !FIRST_PARTY.some((d) => new URL(r.url()).hostname.endsWith(d)))) {
return route.abort();
}
return route.continue();
});
}
const page = await context.newPage();
let bytes = 0;
let requests = 0;
let blocked = 0;
const count = async (req) => {
const s = await req.sizes();
bytes += s.requestHeadersSize + s.requestBodySize + s.responseHeadersSize + s.responseBodySize;
requests++;
};
page.on('requestfinished', count);
page.on('requestfailed', () => { blocked++; });
const t = Date.now();
// DOM ready, then a fixed 5 s for the rest to arrive: the same window for every variant
// ('load' can wait forever on a blocked video).
await page.goto(PAGE, { waitUntil: 'domcontentloaded', timeout: 90000 });
const readyMs = Date.now() - t;
await page.waitForTimeout(5000);
const title = await page.title();
page.off('requestfinished', count); // stop counting before closing
await context.close();
return { variant: label, title, requests, blocked, kilobytes: Math.round(bytes / 1024), dom_ready_ms: readyMs };
}
try {
const rows = [
await measure('everything', null),
await measure('no images, media or fonts', 'heavy'),
await measure('…and first-party only', 'first-party'),
];
const full = rows[0].kilobytes;
for (const r of rows) {
r.saved = `${Math.round((1 - r.kilobytes / full) * 100)}%`;
r.proxy_cost_per_100k_pages = `$${((r.kilobytes * 100000) / 1024 / 1024 * PRICE_PER_GB).toFixed(0)}`;
}
console.log(JSON.stringify(rows, null, 2));
} finally {
await browser.close();
}
# pip install playwright==1.60.0 requests
# env: CDPFLEET_API_KEY, PROXY_URL
import json
import os
import time
from urllib.parse import urlparse
import requests
from playwright.sync_api import sync_playwright
KEY = os.environ["CDPFLEET_API_KEY"]
PAGE = "https://www.bbc.com/news"
FIRST_PARTY = ("bbc.com", "bbc.co.uk", "bbci.co.uk") # the site's own domains and CDNs
PRICE_PER_GB = 3 # a typical residential proxy price, USD
HEAVY = {"image", "media", "font"}
res = requests.post("https://starter.cdpfleet.com/chromium/session", headers={"x-api-key": KEY},
json={"proxy": os.environ["PROXY_URL"], "headless": True}, timeout=60)
res.raise_for_status()
def measure(browser, label, block):
"""Load the page in a fresh context (empty cache) and count every byte on the wire."""
context = browser.new_context()
if block:
def handle(route):
r = route.request
third_party = not urlparse(r.url).hostname.endswith(FIRST_PARTY)
if r.resource_type in HEAVY or (block == "first-party" and third_party):
return route.abort()
return route.continue_()
context.route("**/*", handle)
page = context.new_page()
stats = {"bytes": 0, "requests": 0, "blocked": 0}
def finished(req):
s = req.sizes()
stats["bytes"] += s["requestHeadersSize"] + s["requestBodySize"] + s["responseHeadersSize"] + s["responseBodySize"]
stats["requests"] += 1
page.on("requestfinished", finished)
page.on("requestfailed", lambda req: stats.__setitem__("blocked", stats["blocked"] + 1))
t = time.time()
# DOM ready, then a fixed 5 s for the rest to arrive: the same window for every variant
# ('load' can wait forever on a blocked video).
page.goto(PAGE, wait_until="domcontentloaded", timeout=90000)
ready_ms = round((time.time() - t) * 1000)
page.wait_for_timeout(5000)
title = page.title()
page.remove_listener("requestfinished", finished) # stop counting before closing
context.close()
return {"variant": label, "title": title, "requests": stats["requests"], "blocked": stats["blocked"],
"kilobytes": round(stats["bytes"] / 1024), "dom_ready_ms": ready_ms}
with sync_playwright() as p:
browser = p.chromium.connect(res.json()["wsUrl"], headers={"x-api-key": KEY})
try:
rows = [
measure(browser, "everything", None),
measure(browser, "no images, media or fonts", "heavy"),
measure(browser, "…and first-party only", "first-party"),
]
full = rows[0]["kilobytes"]
for r in rows:
r["saved"] = f"{round((1 - r['kilobytes'] / full) * 100)}%"
r["proxy_cost_per_100k_pages"] = f"${r['kilobytes'] * 100000 / 1024 / 1024 * PRICE_PER_GB:.0f}"
print(json.dumps(rows, indent=2, ensure_ascii=False))
finally:
browser.close()
// Maven: com.microsoft.playwright:playwright:1.60.0, com.google.code.gson:gson:2.11.0
// Run with PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD=1. env: CDPFLEET_API_KEY, PROXY_URL
import com.google.gson.*;
import com.microsoft.playwright.*;
import com.microsoft.playwright.options.*;
import java.net.URI;
import java.net.http.*;
import java.util.*;
import java.util.function.Consumer;
public class Main {
static final String KEY = System.getenv("CDPFLEET_API_KEY");
static final String PAGE = "https://www.bbc.com/news";
static final List<String> FIRST_PARTY = List.of("bbc.com", "bbc.co.uk", "bbci.co.uk"); // the site's own domains and CDNs
static final Set<String> HEAVY = Set.of("image", "media", "font");
static final double PRICE_PER_GB = 3; // a typical residential proxy price, USD
// Load the page in a fresh context (empty cache) and count every byte on the wire.
static JsonObject measure(Browser browser, String label, String block) {
BrowserContext context = browser.newContext();
if (block != null) {
context.route("**/*", route -> {
Request r = route.request();
String host = URI.create(r.url()).getHost();
boolean thirdParty = FIRST_PARTY.stream().noneMatch(host::endsWith);
if (HEAVY.contains(r.resourceType()) || (block.equals("first-party") && thirdParty)) route.abort();
else route.resume();
});
}
Page page = context.newPage();
long[] bytes = {0};
int[] counts = {0, 0}; // requests, blocked
Consumer<Request> count = req -> {
Sizes s = req.sizes();
bytes[0] += s.requestHeadersSize + s.requestBodySize + s.responseHeadersSize + s.responseBodySize;
counts[0]++;
};
page.onRequestFinished(count);
page.onRequestFailed(req -> counts[1]++);
long t = System.currentTimeMillis();
// DOM ready, then a fixed 5 s for the rest to arrive: the same window for every variant
// ('load' can wait forever on a blocked video).
page.navigate(PAGE, new Page.NavigateOptions().setWaitUntil(WaitUntilState.DOMCONTENTLOADED).setTimeout(90000));
long readyMs = System.currentTimeMillis() - t;
page.waitForTimeout(5000);
String title = page.title();
page.offRequestFinished(count); // stop counting before closing
context.close();
JsonObject row = new JsonObject();
row.addProperty("variant", label);
row.addProperty("title", title);
row.addProperty("requests", counts[0]);
row.addProperty("blocked", counts[1]);
row.addProperty("kilobytes", Math.round(bytes[0] / 1024.0));
row.addProperty("dom_ready_ms", readyMs);
return row;
}
public static void main(String[] args) throws Exception {
String body = "{\"proxy\": " + new Gson().toJson(System.getenv("PROXY_URL")) + ", \"headless\": true}";
HttpResponse<String> res = HttpClient.newHttpClient().send(HttpRequest.newBuilder(URI.create("https://starter.cdpfleet.com/chromium/session"))
.header("x-api-key", KEY).header("content-type", "application/json")
.POST(HttpRequest.BodyPublishers.ofString(body)).build(), HttpResponse.BodyHandlers.ofString());
if (res.statusCode() != 200) throw new RuntimeException("launch: " + res.statusCode() + " " + res.body());
String wsUrl = JsonParser.parseString(res.body()).getAsJsonObject().get("wsUrl").getAsString();
try (Playwright playwright = Playwright.create()) {
Browser browser = playwright.chromium().connect(wsUrl, new BrowserType.ConnectOptions().setHeaders(Map.of("x-api-key", KEY)));
try {
List<JsonObject> rows = List.of(
measure(browser, "everything", null),
measure(browser, "no images, media or fonts", "heavy"),
measure(browser, "…and first-party only", "first-party"));
double full = rows.get(0).get("kilobytes").getAsDouble();
JsonArray out = new JsonArray();
for (JsonObject r : rows) {
double kb = r.get("kilobytes").getAsDouble();
r.addProperty("saved", Math.round((1 - kb / full) * 100) + "%");
r.addProperty("proxy_cost_per_100k_pages", "$" + Math.round(kb * 100000 / 1024 / 1024 * PRICE_PER_GB));
out.add(r);
}
System.out.println(new GsonBuilder().setPrettyPrinting().disableHtmlEscaping().create().toJson(out));
} finally {
browser.close();
}
}
}
}
// dotnet add package Microsoft.Playwright --version 1.60.0
// env: CDPFLEET_API_KEY, PROXY_URL
using System.Diagnostics;
using System.Net.Http.Json;
using System.Text.Encodings.Web;
using System.Text.Json;
using System.Text.Json.Nodes;
using Microsoft.Playwright;
const string PageUrl = "https://www.bbc.com/news";
string[] firstParty = { "bbc.com", "bbc.co.uk", "bbci.co.uk" }; // the site's own domains and CDNs
var heavy = new HashSet<string> { "image", "media", "font" };
const double PricePerGb = 3; // a typical residential proxy price, USD
var key = Environment.GetEnvironmentVariable("CDPFLEET_API_KEY")!;
using var http = new HttpClient();
http.DefaultRequestHeaders.Add("x-api-key", key);
var res = await http.PostAsJsonAsync("https://starter.cdpfleet.com/chromium/session",
new { proxy = Environment.GetEnvironmentVariable("PROXY_URL"), headless = true });
if (!res.IsSuccessStatusCode) throw new Exception($"launch: {(int)res.StatusCode} {await res.Content.ReadAsStringAsync()}");
var wsUrl = (await res.Content.ReadFromJsonAsync<JsonElement>()).GetProperty("wsUrl").GetString()!;
using var playwright = await Playwright.CreateAsync();
var browser = await playwright.Chromium.ConnectAsync(wsUrl, new() { Headers = new Dictionary<string, string> { ["x-api-key"] = key } });
// Load the page in a fresh context (empty cache) and count every byte on the wire.
async Task<JsonObject> Measure(string label, string? block)
{
var context = await browser.NewContextAsync();
if (block != null)
{
await context.RouteAsync("**/*", route =>
{
var r = route.Request;
var thirdParty = !firstParty.Any(d => new Uri(r.Url).Host.EndsWith(d));
return heavy.Contains(r.ResourceType) || (block == "first-party" && thirdParty) ? route.AbortAsync() : route.ContinueAsync();
});
}
var page = await context.NewPageAsync();
long bytes = 0;
int requests = 0, blocked = 0;
var pending = new List<Task>();
async Task Count(IRequest req)
{
var s = await req.SizesAsync();
Interlocked.Add(ref bytes, s.RequestHeadersSize + s.RequestBodySize + s.ResponseHeadersSize + s.ResponseBodySize);
Interlocked.Increment(ref requests);
}
EventHandler<IRequest> onFinished = (_, req) => { lock (pending) pending.Add(Count(req)); };
page.RequestFinished += onFinished;
page.RequestFailed += (_, _) => Interlocked.Increment(ref blocked);
var sw = Stopwatch.StartNew();
// DOM ready, then a fixed 5 s for the rest to arrive: the same window for every variant
// ('load' can wait forever on a blocked video).
await page.GotoAsync(PageUrl, new() { WaitUntil = WaitUntilState.DOMContentLoaded, Timeout = 90000 });
var readyMs = sw.ElapsedMilliseconds;
await page.WaitForTimeoutAsync(5000);
var title = await page.TitleAsync();
page.RequestFinished -= onFinished; // stop counting before closing
Task[] counting;
lock (pending) counting = pending.ToArray();
await Task.WhenAll(counting);
await context.CloseAsync();
return new JsonObject
{
["variant"] = label, ["title"] = title, ["requests"] = requests, ["blocked"] = blocked,
["kilobytes"] = (long)Math.Round(bytes / 1024.0), ["dom_ready_ms"] = readyMs,
};
}
try
{
var rows = new[]
{
await Measure("everything", null),
await Measure("no images, media or fonts", "heavy"),
await Measure("…and first-party only", "first-party"),
};
var full = rows[0]["kilobytes"]!.GetValue<long>();
foreach (var r in rows)
{
double kb = r["kilobytes"]!.GetValue<long>();
r["saved"] = $"{Math.Round((1 - kb / full) * 100)}%";
r["proxy_cost_per_100k_pages"] = $"${Math.Round(kb * 100000 / 1024 / 1024 * PricePerGb)}";
}
Console.WriteLine(new JsonArray(rows.ToArray<JsonNode?>()).ToJsonString(new JsonSerializerOptions { WriteIndented = true, Encoder = JavaScriptEncoder.UnsafeRelaxedJsonEscaping }));
}
finally
{
await browser.CloseAsync();
}
// go get github.com/playwright-community/[email protected]
// Driver: build playwright-core 1.60.0 from npm and set PLAYWRIGHT_DRIVER_PATH (see /docs/quickstart).
// env: CDPFLEET_API_KEY, PROXY_URL
package main
import (
"bytes"
"encoding/json"
"fmt"
"io"
"log"
"math"
"net/http"
"net/url"
"os"
"strings"
"sync"
"time"
"github.com/playwright-community/playwright-go"
)
var key = os.Getenv("CDPFLEET_API_KEY")
const pageURL = "https://www.bbc.com/news"
const pricePerGB = 3.0 // a typical residential proxy price, USD
var firstParty = []string{"bbc.com", "bbc.co.uk", "bbci.co.uk"} // the site's own domains and CDNs
var heavy = map[string]bool{"image": true, "media": true, "font": true}
func launch(name string, options map[string]any) (map[string]any, error) {
body, _ := json.Marshal(options)
req, _ := http.NewRequest("POST", "https://starter.cdpfleet.com/"+name+"/session", bytes.NewReader(body))
req.Header.Set("x-api-key", key)
req.Header.Set("content-type", "application/json")
res, err := http.DefaultClient.Do(req)
if err != nil {
return nil, err
}
defer res.Body.Close()
if res.StatusCode != http.StatusOK {
msg, _ := io.ReadAll(res.Body)
return nil, fmt.Errorf("launch %s: %s %s", name, res.Status, msg)
}
var session map[string]any
return session, json.NewDecoder(res.Body).Decode(&session)
}
type row struct {
Variant string `json:"variant"`
Title string `json:"title"`
Requests int `json:"requests"`
Blocked int `json:"blocked"`
Kilobytes int64 `json:"kilobytes"`
DomReadyMs int64 `json:"dom_ready_ms"`
Saved string `json:"saved"`
ProxyCost string `json:"proxy_cost_per_100k_pages"`
}
// Load the page in a fresh context (empty cache) and count every byte on the wire.
func measure(browser playwright.Browser, label, block string) row {
context, err := browser.NewContext()
if err != nil {
log.Fatal(err)
}
defer context.Close()
if block != "" {
context.Route("**/*", func(route playwright.Route) {
r := route.Request()
u, _ := url.Parse(r.URL())
thirdParty := true
for _, d := range firstParty {
if strings.HasSuffix(u.Hostname(), d) {
thirdParty = false
}
}
if heavy[r.ResourceType()] || (block == "first-party" && thirdParty) {
route.Abort()
} else {
route.Continue()
}
})
}
page, _ := context.NewPage()
var mu sync.Mutex
var bytesSeen int64
requests, blocked := 0, 0
count := func(req playwright.Request) {
s, err := req.Sizes()
if err != nil {
return
}
mu.Lock()
bytesSeen += int64(s.RequestHeadersSize + s.RequestBodySize + s.ResponseHeadersSize + s.ResponseBodySize)
requests++
mu.Unlock()
}
page.OnRequestFinished(count)
page.OnRequestFailed(func(playwright.Request) { mu.Lock(); blocked++; mu.Unlock() })
t := time.Now()
// DOM ready, then a fixed 5 s for the rest to arrive: the same window for every variant
// ('load' can wait forever on a blocked video).
if _, err := page.Goto(pageURL, playwright.PageGotoOptions{WaitUntil: playwright.WaitUntilStateDomcontentloaded, Timeout: playwright.Float(90000)}); err != nil {
log.Fatal(err)
}
readyMs := time.Since(t).Milliseconds()
page.WaitForTimeout(5000)
title, _ := page.Title()
page.RemoveListener("requestfinished", count) // stop counting before closing
mu.Lock()
defer mu.Unlock()
return row{Variant: label, Title: title, Requests: requests, Blocked: blocked,
Kilobytes: int64(math.Round(float64(bytesSeen) / 1024)), DomReadyMs: readyMs}
}
func main() {
session, err := launch("chromium", map[string]any{"proxy": os.Getenv("PROXY_URL"), "headless": true})
if err != nil {
log.Fatal(err)
}
pw, err := playwright.Run(&playwright.RunOptions{SkipInstallBrowsers: true})
if err != nil {
log.Fatal(err)
}
defer pw.Stop()
browser, err := pw.Chromium.Connect(session["wsUrl"].(string), playwright.BrowserTypeConnectOptions{Headers: map[string]string{"x-api-key": key}})
if err != nil {
log.Fatal(err)
}
defer browser.Close()
rows := []row{
measure(browser, "everything", ""),
measure(browser, "no images, media or fonts", "heavy"),
measure(browser, "…and first-party only", "first-party"),
}
full := float64(rows[0].Kilobytes)
for i := range rows {
kb := float64(rows[i].Kilobytes)
rows[i].Saved = fmt.Sprintf("%.0f%%", (1-kb/full)*100)
rows[i].ProxyCost = fmt.Sprintf("$%.0f", kb*100000/1024/1024*pricePerGB)
}
var buf bytes.Buffer
enc := json.NewEncoder(&buf)
enc.SetEscapeHTML(false)
enc.SetIndent("", " ")
enc.Encode(rows)
fmt.Print(buf.String())
}
What we got
| Variant | Requests | Blocked | KB | Saved | Proxy cost / 100k pages | DOM ready (ms) |
|---|---|---|---|---|---|---|
| everything | 80 | 0 | 1662 | 0% | $476 | 4314 |
| no images, media or fonts | 84 | 28 | 1533 | 8% | $439 | 5030 |
| …and first-party only | 53 | 31 | 791 | 52% | $226 | 2688 |
From the Node.js run on 2026-09-30. IP addresses are replaced with placeholders (203.0.113.x); equal addresses stay equal. The other languages produced the same findings.
Raw output (Node.js)
[
{
"variant": "everything",
"title": "BBC News - Breaking news, video and the latest top stories from the U.S. and around the world",
"requests": 80,
"blocked": 0,
"kilobytes": 1662,
"dom_ready_ms": 4314,
"saved": "0%",
"proxy_cost_per_100k_pages": "$476"
},
{
"variant": "no images, media or fonts",
"title": "BBC News - Breaking news, video and the latest top stories from the U.S. and around the world",
"requests": 84,
"blocked": 28,
"kilobytes": 1533,
"dom_ready_ms": 5030,
"saved": "8%",
"proxy_cost_per_100k_pages": "$439"
},
{
"variant": "…and first-party only",
"title": "BBC News - Breaking news, video and the latest top stories from the U.S. and around the world",
"requests": 53,
"blocked": 31,
"kilobytes": 791,
"dom_ready_ms": 2688,
"saved": "52%",
"proxy_cost_per_100k_pages": "$226"
}
]Takeaways
- Images, media and fonts are a smaller share than you'd think on a script-heavy news page — at most about a fifth, and on a single load it can even look worse: the ad slots fill differently every time.
- Third parties are the bigger lever: ads, analytics and consent tooling pull scripts, pixels and beacons from dozens of hosts. Keeping only first-party requests cut the bytes much further.
- Blocking can also make pages faster — fewer requests competing for the proxy. But test it: some sites need a third-party script (consent, anti-bot) to render at all.
loadmay never fire when a video is blocked; measure withdomcontentloadedplus a fixed window, as here.- Numbers move between runs (ad auctions differ per load); the proportions hold.