Examples & benchmarks / #30 · 2026-10-09 · Example · Easy · for web scraping

Five pages at once: parallel tabs in one browser session

Open five unrelated pages concurrently in one session and compare with loading them one by one — same thread, same proxy, a fraction of the wall time.

Chromium

Run on production on 2026-10-21: ✓ Node.js ✓ Python ✓ Java ✓ C# ✓ Go

The problem

You need something from several unrelated pages. The obvious loop opens one, extracts, closes, and moves on — and most of each wait is latency through the proxy while the browser sits idle. A session can hold many tabs, and each language has its own way to wait on several at once. How much does that save, and what does it cost?

What we used, and why

What	Why
`chromium`, `headless: "new"`	One session (1 thread) for all five pages.
Five independent URLs	books.toscrape.com, quotes.toscrape.com, example.com, a Wikipedia article and the Hacker News front page: public, no login, safe to load at once.
`browser.newPage()` per URL	Each tab has its own navigation; all share the session's proxy and connection.
`Promise.all` · `asyncio.gather` · `Task.WhenAll` · goroutines	Each language's own way to wait on five loads at once. Java starts all five navigations and then waits for each — no threads.
`waitUntil: "domcontentloaded"`	Wait for the HTML, not every image and script: one slow asset would otherwise set the pace.
`page.title()`	The smallest proof a page loaded; put your extraction here.

How it works

Launch one Chromium session.
Sequential: open each URL in a new tab, read its title, close it — timed.
Parallel: open all five at once, read the titles, close the tabs — timed.
Do both three times, alternating, and keep each method's best time: a single page that stalls in the proxy (we saw 14–28 s now and then) would otherwise decide one comparison on its own.
Print both timings, the speedup, every round and every title.

The code

The same program in five languages (also on GitHub, with the raw output). Set these environment variables first:

CDPFLEET_API_KEY — your API key (dashboard → API keys)
PROXY_URL — your proxy, e.g. http://user:[email protected]:8000

// npm install [email protected]
// env: CDPFLEET_API_KEY, PROXY_URL
import { chromium } from 'playwright';

const KEY = process.env.CDPFLEET_API_KEY;
const URLS = [
  'https://books.toscrape.com/',
  'https://quotes.toscrape.com/',
  'https://example.com',
  'https://en.wikipedia.org/wiki/Web_scraping',
  'https://news.ycombinator.com/',
];

const res = await fetch('https://starter.cdpfleet.com/chromium/session', {
  method: 'POST',
  headers: { 'x-api-key': KEY, 'content-type': 'application/json' },
  body: JSON.stringify({ proxy: process.env.PROXY_URL, headless: 'new' }),
});
if (!res.ok) throw new Error(`launch ${res.status} ${await res.text()}`);
const { wsUrl } = await res.json();
const browser = await chromium.connect(wsUrl, { headers: { 'x-api-key': KEY } });

async function extract(browser, url) {
  // One retry, in a fresh tab: over 30 loads through a rotating proxy, one occasionally fails.
  for (let attempt = 1; ; attempt++) {
    const page = await browser.newPage();
    try {
      await page.goto(url, { waitUntil: 'domcontentloaded', timeout: 60000 });
      return { url, title: await page.title() };
    } catch (err) {
      if (attempt === 2) throw err;
    } finally {
      await page.close();
    }
  }
}

try {
  // Three rounds of each, alternating; the best time of each method counts. One page that
  // stalls for 20 s in the proxy would otherwise decide a single comparison on its own.
  const rounds = [];
  let sequential = [];
  let parallel = [];
  for (let round = 0; round < 3; round++) {
    const t1 = Date.now();
    sequential = [];
    for (const url of URLS) {
      sequential.push({ ...await extract(browser, url), method: 'sequential' });
    }
    const seqSeconds = Math.round((Date.now() - t1) / 10) / 100;

    const t2 = Date.now();
    parallel = await Promise.all(URLS.map((url) => extract(browser, url).then((r) => ({ ...r, method: 'parallel' }))));
    const parSeconds = Math.round((Date.now() - t2) / 10) / 100;
    rounds.push({ sequential_seconds: seqSeconds, parallel_seconds: parSeconds });
  }
  const seqSeconds = Math.min(...rounds.map((r) => r.sequential_seconds));
  const parSeconds = Math.min(...rounds.map((r) => r.parallel_seconds));

  console.log(JSON.stringify({
    sequential_seconds: seqSeconds,
    parallel_seconds: parSeconds,
    speedup: `${Math.round(seqSeconds / parSeconds * 10) / 10}x`,
    rounds,
    results: [...sequential, ...parallel],
  }, null, 2));
} finally {
  await browser.close();
}

# pip install playwright==1.60.0 requests
# env: CDPFLEET_API_KEY, PROXY_URL
# Playwright's sync API isn't thread-safe, so the parallel loads use the async API.
import asyncio
import json
import os
import time

import requests
from playwright.async_api import async_playwright

KEY = os.environ["CDPFLEET_API_KEY"]
URLS = [
    "https://books.toscrape.com/",
    "https://quotes.toscrape.com/",
    "https://example.com",
    "https://en.wikipedia.org/wiki/Web_scraping",
    "https://news.ycombinator.com/",
]


async def main():
    res = requests.post("https://starter.cdpfleet.com/chromium/session",
        headers={"x-api-key": KEY},
        json={"proxy": os.environ["PROXY_URL"], "headless": "new"}, timeout=60)
    if not res.ok:
        raise SystemExit(f"launch {res.status_code} {res.text}")

    async with async_playwright() as p:
        browser = await p.chromium.connect(res.json()["wsUrl"], headers={"x-api-key": KEY})
        try:
            async def extract(url):
                # One retry, in a fresh tab: over 30 loads through a rotating proxy, one occasionally fails.
                for attempt in (1, 2):
                    page = await browser.new_page()
                    try:
                        await page.goto(url, wait_until="domcontentloaded", timeout=60000)
                        return {"url": url, "title": await page.title()}
                    except Exception:
                        if attempt == 2:
                            raise
                    finally:
                        await page.close()

            # Three rounds of each, alternating; the best time of each method counts. One page that
            # stalls for 20 s in the proxy would otherwise decide a single comparison on its own.
            rounds = []
            for _ in range(3):
                t1 = time.time()
                sequential = [{**(await extract(url)), "method": "sequential"} for url in URLS]
                seq = round(time.time() - t1, 2)

                t2 = time.time()
                parallel = [{**r, "method": "parallel"} for r in await asyncio.gather(*(extract(url) for url in URLS))]
                par = round(time.time() - t2, 2)
                rounds.append({"sequential_seconds": seq, "parallel_seconds": par})
            seq_seconds = min(r["sequential_seconds"] for r in rounds)
            par_seconds = min(r["parallel_seconds"] for r in rounds)

            print(json.dumps({
                "sequential_seconds": seq_seconds,
                "parallel_seconds": par_seconds,
                "speedup": f"{round(seq_seconds / par_seconds, 1)}x",
                "rounds": rounds,
                "results": sequential + parallel,
            }, indent=2, ensure_ascii=False))
        finally:
            await browser.close()


asyncio.run(main())

// Maven: com.microsoft.playwright:playwright:1.60.0, com.google.code.gson:gson:2.11.0
// Run with PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD=1. env: CDPFLEET_API_KEY, PROXY_URL
import com.google.gson.*;
import com.microsoft.playwright.*;
import com.microsoft.playwright.options.WaitUntilState;
import java.net.URI;
import java.net.http.*;
import java.util.*;
import java.util.concurrent.*;

public class Main {
  static final String KEY = System.getenv("CDPFLEET_API_KEY");
  static final Gson GSON = new GsonBuilder().serializeNulls().create();
  static final String[] URLS = {
      "https://books.toscrape.com/",
      "https://quotes.toscrape.com/",
      "https://example.com",
      "https://en.wikipedia.org/wiki/Web_scraping",
      "https://news.ycombinator.com/",
  };

  record Result(String url, String title, String method) {}

  static Result extract(Browser browser, String url) {
    // One retry, in a fresh tab: over 30 loads through a rotating proxy, one occasionally fails.
    for (int attempt = 1; ; attempt++) {
      Page page = browser.newPage();
      try {
        page.navigate(url, new Page.NavigateOptions().setWaitUntil(WaitUntilState.DOMCONTENTLOADED).setTimeout(60000));
        return new Result(url, page.title(), null);
      } catch (PlaywrightException e) {
        if (attempt == 2) throw e;
      } finally {
        page.close();
      }
    }
  }

  public static void main(String[] args) throws Exception {
    String body = "{\"proxy\": " + GSON.toJson(System.getenv("PROXY_URL")) + ", \"headless\": \"new\"}";
    HttpResponse<String> res = HttpClient.newHttpClient().send(HttpRequest.newBuilder(URI.create("https://starter.cdpfleet.com/chromium/session"))
        .header("x-api-key", KEY).header("content-type", "application/json")
        .POST(HttpRequest.BodyPublishers.ofString(body)).build(), HttpResponse.BodyHandlers.ofString());
    if (res.statusCode() != 200) throw new RuntimeException("launch: " + res.statusCode() + " " + res.body());
    String wsUrl = JsonParser.parseString(res.body()).getAsJsonObject().get("wsUrl").getAsString();

    try (Playwright playwright = Playwright.create()) {
      Browser browser = playwright.chromium().connect(wsUrl, new BrowserType.ConnectOptions().setHeaders(Map.of("x-api-key", KEY)));
      try {
        // Three rounds of each, alternating; the best time of each method counts. One page that
        // stalls for 20 s in the proxy would otherwise decide a single comparison on its own.
        JsonArray rounds = new JsonArray();
        List<Result> sequential = new ArrayList<>();
        List<Result> parallel = new ArrayList<>();
        double seqSeconds = Double.MAX_VALUE;
        double parSeconds = Double.MAX_VALUE;
        for (int round = 0; round < 3; round++) {
          long t1 = System.currentTimeMillis();
          sequential = new ArrayList<>();
          for (String url : URLS) {
            Result r = extract(browser, url);
            sequential.add(new Result(r.url, r.title, "sequential"));
          }
          double seq = Math.round((System.currentTimeMillis() - t1) / 10.0) / 100.0;

          // Playwright for Java isn't thread-safe: start all five navigations, then wait for each.
          long t2 = System.currentTimeMillis();
          List<Page> pages = new ArrayList<>();
          for (String url : URLS) {
            Page page = browser.newPage();
            page.evaluate("u => { location.href = u; }", url);
            pages.add(page);
          }
          parallel = new ArrayList<>();
          for (int i = 0; i < URLS.length; i++) {
            Page page = pages.get(i);
            page.waitForURL(u -> !u.startsWith("about:"), new Page.WaitForURLOptions().setWaitUntil(WaitUntilState.DOMCONTENTLOADED).setTimeout(60000));
            parallel.add(new Result(URLS[i], page.title(), "parallel"));
            page.close();
          }
          double par = Math.round((System.currentTimeMillis() - t2) / 10.0) / 100.0;
          JsonObject r = new JsonObject();
          r.addProperty("sequential_seconds", seq);
          r.addProperty("parallel_seconds", par);
          rounds.add(r);
          seqSeconds = Math.min(seqSeconds, seq);
          parSeconds = Math.min(parSeconds, par);
        }

        JsonObject out = new JsonObject();
        out.addProperty("sequential_seconds", seqSeconds);
        out.addProperty("parallel_seconds", parSeconds);
        out.addProperty("speedup", Math.round(seqSeconds / parSeconds * 10) / 10.0 + "x");
        out.add("rounds", rounds);
        JsonArray results = new JsonArray();
        for (Result r : sequential) results.add(GSON.toJsonTree(r));
        for (Result r : parallel) results.add(GSON.toJsonTree(r));
        out.add("results", results);

        System.out.println(new GsonBuilder().serializeNulls().setPrettyPrinting().disableHtmlEscaping().create().toJson(out));
      } finally {
        browser.close();
      }
    }
  }
}

// dotnet add package Microsoft.Playwright --version 1.60.0
// env: CDPFLEET_API_KEY, PROXY_URL
using System.Diagnostics;
using System.Net.Http.Json;
using System.Text.Encodings.Web;
using System.Text.Json;
using System.Text.Json.Nodes;
using Microsoft.Playwright;

string[] urls = [
    "https://books.toscrape.com/",
    "https://quotes.toscrape.com/",
    "https://example.com",
    "https://en.wikipedia.org/wiki/Web_scraping",
    "https://news.ycombinator.com/",
];

var key = Environment.GetEnvironmentVariable("CDPFLEET_API_KEY")!;
using var http = new HttpClient();
http.DefaultRequestHeaders.Add("x-api-key", key);
var res = await http.PostAsJsonAsync("https://starter.cdpfleet.com/chromium/session",
    new { proxy = Environment.GetEnvironmentVariable("PROXY_URL"), headless = "new" });
if (!res.IsSuccessStatusCode) throw new Exception($"launch: {(int)res.StatusCode} {await res.Content.ReadAsStringAsync()}");
var wsUrl = (await res.Content.ReadFromJsonAsync<JsonElement>()).GetProperty("wsUrl").GetString()!;

using var playwright = await Playwright.CreateAsync();
var browser = await playwright.Chromium.ConnectAsync(wsUrl, new() { Headers = new Dictionary<string, string> { ["x-api-key"] = key } });
try
{
    async Task<JsonObject> Extract(string url)
    {
        // One retry, in a fresh tab: over 30 loads through a rotating proxy, one occasionally fails.
        for (var attempt = 1; ; attempt++)
        {
            var page = await browser.NewPageAsync();
            try
            {
                await page.GotoAsync(url, new() { WaitUntil = WaitUntilState.DOMContentLoaded, Timeout = 60000 });
                return new JsonObject { ["url"] = url, ["title"] = await page.TitleAsync() };
            }
            catch (PlaywrightException) when (attempt == 1)
            {
            }
            finally
            {
                await page.CloseAsync();
            }
        }
    }

    // Three rounds of each, alternating; the best time of each method counts. One page that
    // stalls for 20 s in the proxy would otherwise decide a single comparison on its own.
    var rounds = new JsonArray();
    var sequential = new List<JsonObject>();
    JsonObject[] parallelResults = [];
    double seqSeconds = double.MaxValue, parSeconds = double.MaxValue;
    for (var round = 0; round < 3; round++)
    {
        var sw1 = Stopwatch.StartNew();
        sequential = new List<JsonObject>();
        foreach (var url in urls)
        {
            var r = await Extract(url);
            r["method"] = "sequential";
            sequential.Add(r);
        }
        var seq = Math.Round(sw1.Elapsed.TotalSeconds, 2);

        var sw2 = Stopwatch.StartNew();
        var tasks = urls.Select(async url =>
        {
            var r = await Extract(url);
            r["method"] = "parallel";
            return r;
        }).ToArray();
        parallelResults = await Task.WhenAll(tasks);
        var par = Math.Round(sw2.Elapsed.TotalSeconds, 2);
        rounds.Add(new JsonObject { ["sequential_seconds"] = seq, ["parallel_seconds"] = par });
        seqSeconds = Math.Min(seqSeconds, seq);
        parSeconds = Math.Min(parSeconds, par);
    }

    var results = new JsonArray();
    foreach (var r in sequential) results.Add(r);
    foreach (var r in parallelResults) results.Add(r);

    var output = new JsonObject
    {
        ["sequential_seconds"] = seqSeconds,
        ["parallel_seconds"] = parSeconds,
        ["speedup"] = $"{Math.Round(seqSeconds / parSeconds, 1)}x",
        ["rounds"] = rounds,
        ["results"] = results,
    };
    Console.WriteLine(output.ToJsonString(new JsonSerializerOptions { WriteIndented = true, Encoder = JavaScriptEncoder.UnsafeRelaxedJsonEscaping }));
}
finally
{
    await browser.CloseAsync();
}

// go get github.com/playwright-community/[email protected]
// Driver: build playwright-core 1.60.0 from npm and set PLAYWRIGHT_DRIVER_PATH (see /docs/quickstart).
// env: CDPFLEET_API_KEY, PROXY_URL
package main

import (
	"bytes"
	"encoding/json"
	"fmt"
	"io"
	"log"
	"math"
	"net/http"
	"os"
	"sync"
	"time"

	"github.com/playwright-community/playwright-go"
)

var key = os.Getenv("CDPFLEET_API_KEY")

var urls = []string{
	"https://books.toscrape.com/",
	"https://quotes.toscrape.com/",
	"https://example.com",
	"https://en.wikipedia.org/wiki/Web_scraping",
	"https://news.ycombinator.com/",
}

func launch(name string, options map[string]any) (map[string]any, error) {
	body, _ := json.Marshal(options)
	req, _ := http.NewRequest("POST", "https://starter.cdpfleet.com/"+name+"/session", bytes.NewReader(body))
	req.Header.Set("x-api-key", key)
	req.Header.Set("content-type", "application/json")
	res, err := http.DefaultClient.Do(req)
	if err != nil {
		return nil, err
	}
	defer res.Body.Close()
	if res.StatusCode != http.StatusOK {
		msg, _ := io.ReadAll(res.Body)
		return nil, fmt.Errorf("launch %s: %s %s", name, res.Status, msg)
	}
	var session map[string]any
	return session, json.NewDecoder(res.Body).Decode(&session)
}

type result struct {
	URL    string `json:"url"`
	Title  string `json:"title"`
	Method string `json:"method"`
}

func extract(browser playwright.Browser, url string) (string, string) {
	// One retry, in a fresh tab: over 30 loads through a rotating proxy, one occasionally fails.
	for attempt := 1; ; attempt++ {
		title, err := load(browser, url)
		if err == nil {
			return url, title
		}
		if attempt == 2 {
			log.Fatal(err)
		}
	}
}

func load(browser playwright.Browser, url string) (string, error) {
	page, err := browser.NewPage()
	if err != nil {
		return "", err
	}
	defer page.Close()
	if _, err := page.Goto(url, playwright.PageGotoOptions{WaitUntil: playwright.WaitUntilStateDomcontentloaded, Timeout: playwright.Float(60000)}); err != nil {
		return "", err
	}
	return page.Title()
}

func main() {
	session, err := launch("chromium", map[string]any{"proxy": os.Getenv("PROXY_URL"), "headless": "new"})
	if err != nil {
		log.Fatal(err)
	}
	pw, err := playwright.Run(&playwright.RunOptions{SkipInstallBrowsers: true})
	if err != nil {
		log.Fatal(err)
	}
	defer pw.Stop()
	browser, err := pw.Chromium.Connect(session["wsUrl"].(string), playwright.BrowserTypeConnectOptions{Headers: map[string]string{"x-api-key": key}})
	if err != nil {
		log.Fatal(err)
	}
	defer browser.Close()

	// Three rounds of each, alternating; the best time of each method counts. One page that
	// stalls for 20 s in the proxy would otherwise decide a single comparison on its own.
	type timing struct {
		SequentialSeconds float64 `json:"sequential_seconds"`
		ParallelSeconds   float64 `json:"parallel_seconds"`
	}
	var rounds []timing
	var sequential, parallel []result
	seqSeconds, parSeconds := math.MaxFloat64, math.MaxFloat64
	for round := 0; round < 3; round++ {
		t1 := time.Now()
		sequential = nil
		for _, u := range urls {
			url, title := extract(browser, u)
			sequential = append(sequential, result{URL: url, Title: title, Method: "sequential"})
		}
		seq := math.Round(time.Since(t1).Seconds()*100) / 100

		t2 := time.Now()
		parallel = make([]result, len(urls))
		var wg sync.WaitGroup
		for i, u := range urls {
			wg.Add(1)
			go func(i int, u string) {
				defer wg.Done()
				url, title := extract(browser, u)
				parallel[i] = result{URL: url, Title: title, Method: "parallel"}
			}(i, u)
		}
		wg.Wait()
		par := math.Round(time.Since(t2).Seconds()*100) / 100
		rounds = append(rounds, timing{seq, par})
		seqSeconds = math.Min(seqSeconds, seq)
		parSeconds = math.Min(parSeconds, par)
	}

	all := append(sequential, parallel...)
	out := struct {
		SequentialSeconds float64  `json:"sequential_seconds"`
		ParallelSeconds   float64  `json:"parallel_seconds"`
		Speedup           string   `json:"speedup"`
		Rounds            []timing `json:"rounds"`
		Results           []result `json:"results"`
	}{
		SequentialSeconds: seqSeconds,
		ParallelSeconds:   parSeconds,
		Speedup:           fmt.Sprintf("%.1fx", math.Round(seqSeconds/parSeconds*10)/10),
		Rounds:            rounds,
		Results:           all,
	}
	var buf bytes.Buffer
	enc := json.NewEncoder(&buf)
	enc.SetEscapeHTML(false)
	enc.SetIndent("", "  ")
	enc.Encode(out)
	fmt.Print(buf.String())
}

What we got

Method	Pages	Seconds
Sequential	5	12.77
Parallel	5	9.39
Speedup		1.4x

From the Node.js run on 2026-10-21. IP addresses are replaced with placeholders (203.0.113.x); equal addresses stay equal. The other languages produced the same findings.

Raw output (Node.js)

{
  "sequential_seconds": 12.77,
  "parallel_seconds": 9.39,
  "speedup": "1.4x",
  "rounds": [
    {
      "sequential_seconds": 16.61,
      "parallel_seconds": 30.7
    },
    {
      "sequential_seconds": 15.14,
      "parallel_seconds": 9.39
    },
    {
      "sequential_seconds": 12.77,
      "parallel_seconds": 13.3
    }
  ],
  "results": [
    {
      "url": "https://books.toscrape.com/",
      "title": "All products | Books to Scrape - Sandbox",
      "method": "sequential"
    },
    {
      "url": "https://quotes.toscrape.com/",
      "title": "Quotes to Scrape",
      "method": "sequential"
    },
    {
      "url": "https://example.com",
      "title": "Example Domain",
      "method": "sequential"
    },
    {
      "url": "https://en.wikipedia.org/wiki/Web_scraping",
      "title": "Web scraping - Wikipedia",
      "method": "sequential"
    },
    {
      "url": "https://news.ycombinator.com/",
      "title": "Hacker News",
      "method": "sequential"
    },
    {
      "url": "https://books.toscrape.com/",
      "title": "All products | Books to Scrape - Sandbox",
      "method": "parallel"
    },
    {
      "url": "https://quotes.toscrape.com/",
      "title": "Quotes to Scrape",
      "method": "parallel"
    },
    {
      "url": "https://example.com",
      "title": "Example Domain",
      "method": "parallel"
    },
    {
      "url": "https://en.wikipedia.org/wiki/Web_scraping",
      "title": "Web scraping - Wikipedia",
      "method": "parallel"
    },
    {
      "url": "https://news.ycombinator.com/",
      "title": "Hacker News",
      "method": "parallel"
    }
  ]
}

Takeaways

Tabs overlap the waiting: 1.4–4.4× faster in the 2026-10-21 runs, taking each method's best of three rounds (6.9–12.8 s one by one, 1.7–9.4 s in parallel) — the wall time approaches the slowest page.
…which is also the catch: in 3 of the 15 rounds one page stalled in the proxy (up to 30 s) and the parallel batch came out slower than loading one by one. Give each load a timeout and a retry, and compare methods over several rounds, not one.
No extra threads or proxy slots: five tabs are one session — the plan meter counts one thread.
The gain depends on page time, not page count: fast pages gain little; slow ones through a distant proxy gain most.
Don't share Playwright across threads: its Python sync API and its Java API aren't thread-safe (a thread pool fails with Target closed / NoSuchElementException). Use Python's async API, and in Java start every navigation before waiting on any.
Wait for the HTML, not the whole page: with load, one page with a slow asset (18 s through our proxy) made the parallel run as slow as the sequential one.
Close tabs when done — every open page holds memory in the remote browser.