Examples & benchmarks / #7 · 2026-09-30 · Example · Medium · for web scraping

Different sites, different proxies, one browser: proxy_rules

Route one host through a SOCKS5 proxy, another through a round-robin pool and everything else through residential — and the IP-lookup hosts only Camoufox keeps on the default proxy.

Chromium

Run on production on 2026-10-17: ✓ Node.js ✓ Python ✓ Java ✓ C# ✓ Go

The problem

Residential bandwidth is expensive; datacenter proxies are cheap but get blocked on the sites that matter. A common pattern: send the protected site through residential, static assets or APIs through cheap proxies. proxy_rules does this inside one browser — let's check each rule really exits where it should.

What we used, and why

What	Why
`proxy`	The default route: every host no rule matches (residential here).
`proxy_rules` rule 1	`.ident.me` and `ident.me` → one SOCKS5 proxy. Host patterns take `` wildcards.
`proxy_rules` rule 2	`httpbin.org` → a list of two proxies, used round-robin per connection.
`proxy_rules` rule 3	`api.ipify.org` → the same SOCKS5 proxy as rule 1. An IP-lookup host: rules for it apply on every browser but Camoufox — see below.
A new context per reading	Each context has its own connection pool, so every reading is a new connection and the round-robin shows.

How it works

Launch with a default proxy and three rules.
Read the exit IP from each echo service, each time in a fresh context (one retry if a proxy drops the connection).
Compare each exit with the proxy its rule names.

The code

The same program in five languages (also on GitHub, with the raw output). Set these environment variables first:

CDPFLEET_API_KEY — your API key (dashboard → API keys)
PROXY_URL — your proxy, e.g. http://user:[email protected]:8000
SOCKS_PROXIES — comma-separated socks5://user:pass@host:port proxies

// npm install [email protected]
// env: CDPFLEET_API_KEY, PROXY_URL, SOCKS_PROXIES (comma-separated socks5:// URLs, 3 or more)
import { chromium } from 'playwright';

const KEY = process.env.CDPFLEET_API_KEY;
const [socksA, socksB, socksC] = process.env.SOCKS_PROXIES.split(',');

const options = {
  proxy: process.env.PROXY_URL, // everything not matched below
  proxy_rules: [
    { hosts: ['*.ident.me', 'ident.me'], proxy: socksA },
    { hosts: ['httpbin.org', '*.httpbin.org'], proxy: [socksB, socksC] }, // round-robin
    // An IP-lookup host: rules for it apply on every browser but Camoufox.
    { hosts: ['api.ipify.org'], proxy: socksA },
  ],
  headless: true,
};

const res = await fetch('https://starter.cdpfleet.com/chromium/session', {
  method: 'POST',
  headers: { 'x-api-key': KEY, 'content-type': 'application/json' },
  body: JSON.stringify(options),
});
if (!res.ok) throw new Error(`launch: ${res.status} ${await res.text()}`);
const { wsUrl } = await res.json();
const browser = await chromium.connect(wsUrl, { headers: { 'x-api-key': KEY } });

// Each context has its own connection pool, so each reading is a fresh connection.
// Proxies drop a connection now and then: one retry.
async function exitIpVia(url) {
  for (let attempt = 1; ; attempt++) {
    const context = await browser.newContext();
    try {
      const page = await context.newPage();
      const text = await (await page.goto(url, { timeout: 30000 })).text();
      return text.match(/\d{1,3}(\.\d{1,3}){3}/)?.[0] ?? `(no IP in ${url})`;
    } catch (err) {
      if (attempt === 2) return `(failed: ${err.message.split('\n')[0]})`;
    } finally {
      await context.close();
    }
  }
}

try {
  const readings = [];
  for (const url of ['https://v4.ident.me/', 'https://httpbin.org/ip', 'https://httpbin.org/ip', 'https://httpbin.org/ip', 'https://api.ipify.org/', 'https://www.cloudflare.com/cdn-cgi/trace']) {
    readings.push({ url, exit_ip: await exitIpVia(url) });
  }
  console.log(JSON.stringify({
    rules: {
      '*.ident.me': 'SOCKS proxy A',
      'httpbin.org': ['SOCKS proxy B', 'SOCKS proxy C'],
      'api.ipify.org': 'SOCKS proxy A',
      '(everything else)': 'residential PROXY_URL',
    },
    readings,
  }, null, 2));
} finally {
  await browser.close();
}

# pip install playwright==1.60.0 requests
# env: CDPFLEET_API_KEY, PROXY_URL, SOCKS_PROXIES (comma-separated socks5:// URLs, 3 or more)
import json
import os
import re

import requests
from playwright.sync_api import sync_playwright

KEY = os.environ["CDPFLEET_API_KEY"]
socks_a, socks_b, socks_c = os.environ["SOCKS_PROXIES"].split(",")[:3]

options = {
    "proxy": os.environ["PROXY_URL"],  # everything not matched below
    "proxy_rules": [
        {"hosts": ["*.ident.me", "ident.me"], "proxy": socks_a},
        {"hosts": ["httpbin.org", "*.httpbin.org"], "proxy": [socks_b, socks_c]},  # round-robin
        # An IP-lookup host: rules for it apply on every browser but Camoufox.
        {"hosts": ["api.ipify.org"], "proxy": socks_a},
    ],
    "headless": True,
}
res = requests.post("https://starter.cdpfleet.com/chromium/session", headers={"x-api-key": KEY}, json=options, timeout=60)
res.raise_for_status()


def exit_ip_via(browser, url):
    # Each context has its own connection pool, so each reading is a fresh connection.
    # Proxies drop a connection now and then: one retry.
    for attempt in (1, 2):
        context = browser.new_context()
        try:
            text = context.new_page().goto(url, timeout=30000).text()
            m = re.search(r"\d{1,3}(\.\d{1,3}){3}", text)
            return m.group(0) if m else f"(no IP in {url})"
        except Exception as err:
            if attempt == 2:
                return f"(failed: {str(err).splitlines()[0]})"
        finally:
            context.close()


with sync_playwright() as p:
    browser = p.chromium.connect(res.json()["wsUrl"], headers={"x-api-key": KEY})
    try:
        urls = ["https://v4.ident.me/", "https://httpbin.org/ip", "https://httpbin.org/ip", "https://httpbin.org/ip",
                "https://api.ipify.org/", "https://www.cloudflare.com/cdn-cgi/trace"]
        print(json.dumps({
            "rules": {
                "*.ident.me": "SOCKS proxy A",
                "httpbin.org": ["SOCKS proxy B", "SOCKS proxy C"],
                "api.ipify.org": "SOCKS proxy A",
                "(everything else)": "residential PROXY_URL",
            },
            "readings": [{"url": u, "exit_ip": exit_ip_via(browser, u)} for u in urls],
        }, indent=2))
    finally:
        browser.close()

// Maven: com.microsoft.playwright:playwright:1.60.0, com.google.code.gson:gson:2.11.0
// Run with PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD=1.
// env: CDPFLEET_API_KEY, PROXY_URL, SOCKS_PROXIES (comma-separated socks5:// URLs, 3 or more)
import com.google.gson.*;
import com.microsoft.playwright.*;
import java.net.URI;
import java.net.http.*;
import java.util.*;
import java.util.regex.*;

public class Main {
  static final String KEY = System.getenv("CDPFLEET_API_KEY");
  static final Pattern IP = Pattern.compile("\\d{1,3}(\\.\\d{1,3}){3}");


  // Each context has its own connection pool, so each reading is a fresh connection.
  // Proxies drop a connection now and then: one retry.
  static String exitIpVia(Browser browser, String url) {
    for (int attempt = 1; ; attempt++) {
      BrowserContext context = browser.newContext();
      try {
        String text = context.newPage().navigate(url, new Page.NavigateOptions().setTimeout(30000)).text();
        Matcher m = IP.matcher(text);
        return m.find() ? m.group() : "(no IP in " + url + ")";
      } catch (PlaywrightException err) {
        if (attempt == 2) return "(failed: " + err.getMessage().split("\n")[0] + ")";
      } finally {
        context.close();
      }
    }
  }

  public static void main(String[] args) throws Exception {
    String[] socks = System.getenv("SOCKS_PROXIES").split(",");
    Gson gson = new Gson();
    String body = """
        {
          "proxy": %s,
          "proxy_rules": [
            {"hosts": ["*.ident.me", "ident.me"], "proxy": %s},
            {"hosts": ["httpbin.org", "*.httpbin.org"], "proxy": [%s, %s]},
            {"hosts": ["api.ipify.org"], "proxy": %s}
          ],
          "headless": true
        }""".formatted(gson.toJson(System.getenv("PROXY_URL")), gson.toJson(socks[0]),
        gson.toJson(socks[1]), gson.toJson(socks[2]), gson.toJson(socks[0]));
    // Rule 2 round-robins over two proxies. Rule 3 is an IP-lookup host: rules for it
    // apply on every browser but Camoufox.
    HttpResponse<String> res = HttpClient.newHttpClient().send(HttpRequest.newBuilder(URI.create("https://starter.cdpfleet.com/chromium/session"))
        .header("x-api-key", KEY).header("content-type", "application/json")
        .POST(HttpRequest.BodyPublishers.ofString(body)).build(), HttpResponse.BodyHandlers.ofString());
    if (res.statusCode() != 200) throw new RuntimeException("launch: " + res.statusCode() + " " + res.body());
    String wsUrl = JsonParser.parseString(res.body()).getAsJsonObject().get("wsUrl").getAsString();

    try (Playwright playwright = Playwright.create()) {
      Browser browser = playwright.chromium().connect(wsUrl, new BrowserType.ConnectOptions().setHeaders(Map.of("x-api-key", KEY)));
      try {
        JsonObject rules = new JsonObject();
        rules.addProperty("*.ident.me", "SOCKS proxy A");
        JsonArray pool = new JsonArray();
        pool.add("SOCKS proxy B");
        pool.add("SOCKS proxy C");
        rules.add("httpbin.org", pool);
        rules.addProperty("api.ipify.org", "SOCKS proxy A");
        rules.addProperty("(everything else)", "residential PROXY_URL");
        JsonArray readings = new JsonArray();
        for (String url : List.of("https://v4.ident.me/", "https://httpbin.org/ip", "https://httpbin.org/ip", "https://httpbin.org/ip",
            "https://api.ipify.org/", "https://www.cloudflare.com/cdn-cgi/trace")) {
          JsonObject r = new JsonObject();
          r.addProperty("url", url);
          r.addProperty("exit_ip", exitIpVia(browser, url));
          readings.add(r);
        }
        JsonObject out = new JsonObject();
        out.add("rules", rules);
        out.add("readings", readings);
        System.out.println(new GsonBuilder().setPrettyPrinting().disableHtmlEscaping().create().toJson(out));
      } finally {
        browser.close();
      }
    }
  }
}

// dotnet add package Microsoft.Playwright --version 1.60.0
// env: CDPFLEET_API_KEY, PROXY_URL, SOCKS_PROXIES (comma-separated socks5:// URLs, 3 or more)
using System.Net.Http.Json;
using System.Text.Encodings.Web;
using System.Text.Json;
using System.Text.Json.Nodes;
using System.Text.RegularExpressions;
using Microsoft.Playwright;

var key = Environment.GetEnvironmentVariable("CDPFLEET_API_KEY")!;
var socks = Environment.GetEnvironmentVariable("SOCKS_PROXIES")!.Split(',');

var options = new
{
    proxy = Environment.GetEnvironmentVariable("PROXY_URL"), // everything not matched below
    proxy_rules = new object[]
    {
        new { hosts = new[] { "*.ident.me", "ident.me" }, proxy = socks[0] },
        new { hosts = new[] { "httpbin.org", "*.httpbin.org" }, proxy = new[] { socks[1], socks[2] } }, // round-robin
        // An IP-lookup host: rules for it apply on every browser but Camoufox.
        new { hosts = new[] { "api.ipify.org" }, proxy = socks[0] },
    },
    headless = true,
};
using var http = new HttpClient();
http.DefaultRequestHeaders.Add("x-api-key", key);
var res = await http.PostAsJsonAsync("https://starter.cdpfleet.com/chromium/session", options);
if (!res.IsSuccessStatusCode) throw new Exception($"launch: {(int)res.StatusCode} {await res.Content.ReadAsStringAsync()}");
var wsUrl = (await res.Content.ReadFromJsonAsync<JsonElement>()).GetProperty("wsUrl").GetString()!;

using var playwright = await Playwright.CreateAsync();
var browser = await playwright.Chromium.ConnectAsync(wsUrl, new() { Headers = new Dictionary<string, string> { ["x-api-key"] = key } });

// Each context has its own connection pool, so each reading is a fresh connection.
// Proxies drop a connection now and then: one retry.
async Task<string> ExitIpVia(string url)
{
    for (var attempt = 1; ; attempt++)
    {
        var context = await browser.NewContextAsync();
        try
        {
            var page = await context.NewPageAsync();
            var text = await (await page.GotoAsync(url, new() { Timeout = 30000 }))!.TextAsync();
            var m = Regex.Match(text, @"\d{1,3}(\.\d{1,3}){3}");
            return m.Success ? m.Value : $"(no IP in {url})";
        }
        // .NET reports navigation timeouts as TimeoutException, not PlaywrightException.
        catch (Exception err) when ((err is PlaywrightException or TimeoutException) && attempt < 2) { }
        catch (Exception err) when (err is PlaywrightException or TimeoutException) { return $"(failed: {err.Message.Split('\n')[0]})"; }
        finally { await context.CloseAsync(); }
    }
}

try
{
    var readings = new JsonArray();
    foreach (var url in new[] { "https://v4.ident.me/", "https://httpbin.org/ip", "https://httpbin.org/ip", "https://httpbin.org/ip",
                                "https://api.ipify.org/", "https://www.cloudflare.com/cdn-cgi/trace" })
        readings.Add(new JsonObject { ["url"] = url, ["exit_ip"] = await ExitIpVia(url) });
    var result = new JsonObject
    {
        ["rules"] = new JsonObject
        {
            ["*.ident.me"] = "SOCKS proxy A",
            ["httpbin.org"] = new JsonArray("SOCKS proxy B", "SOCKS proxy C"),
            ["api.ipify.org"] = "SOCKS proxy A",
            ["(everything else)"] = "residential PROXY_URL",
        },
        ["readings"] = readings,
    };
    Console.WriteLine(result.ToJsonString(new JsonSerializerOptions { WriteIndented = true, Encoder = JavaScriptEncoder.UnsafeRelaxedJsonEscaping }));
}
finally
{
    await browser.CloseAsync();
}

// go get github.com/playwright-community/[email protected]
// Driver: build playwright-core 1.60.0 from npm and set PLAYWRIGHT_DRIVER_PATH (see /docs/quickstart).
// env: CDPFLEET_API_KEY, PROXY_URL, SOCKS_PROXIES (comma-separated socks5:// URLs, 3 or more)
package main

import (
	"bytes"
	"encoding/json"
	"fmt"
	"io"
	"log"
	"net/http"
	"os"
	"regexp"
	"strings"

	"github.com/playwright-community/playwright-go"
)

var key = os.Getenv("CDPFLEET_API_KEY")

func launch(name string, options map[string]any) (map[string]any, error) {
	body, _ := json.Marshal(options)
	req, _ := http.NewRequest("POST", "https://starter.cdpfleet.com/"+name+"/session", bytes.NewReader(body))
	req.Header.Set("x-api-key", key)
	req.Header.Set("content-type", "application/json")
	res, err := http.DefaultClient.Do(req)
	if err != nil {
		return nil, err
	}
	defer res.Body.Close()
	if res.StatusCode != http.StatusOK {
		msg, _ := io.ReadAll(res.Body)
		return nil, fmt.Errorf("launch %s: %s %s", name, res.Status, msg)
	}
	var session map[string]any
	return session, json.NewDecoder(res.Body).Decode(&session)
}

var ipPattern = regexp.MustCompile(`\d{1,3}(\.\d{1,3}){3}`)


// Each context has its own connection pool, so each reading is a fresh connection.
// Proxies drop a connection now and then: one retry.
func exitIPVia(browser playwright.Browser, u string) string {
	for attempt := 1; ; attempt++ {
		context, err := browser.NewContext()
		if err != nil {
			log.Fatal(err)
		}
		page, _ := context.NewPage()
		res, err := page.Goto(u, playwright.PageGotoOptions{Timeout: playwright.Float(30000)})
		var text string
		if err == nil {
			text, err = res.Text()
		}
		context.Close()
		if err == nil {
			if ip := ipPattern.FindString(text); ip != "" {
				return ip
			}
			return "(no IP in " + u + ")"
		}
		if attempt == 2 {
			return "(failed: " + strings.SplitN(err.Error(), "\n", 2)[0] + ")"
		}
	}
}

type reading struct {
	URL    string `json:"url"`
	ExitIP string `json:"exit_ip"`
}

func main() {
	socks := strings.Split(os.Getenv("SOCKS_PROXIES"), ",")
	session, err := launch("chromium", map[string]any{
		"proxy": os.Getenv("PROXY_URL"), // everything not matched below
		"proxy_rules": []map[string]any{
			{"hosts": []string{"*.ident.me", "ident.me"}, "proxy": socks[0]},
			{"hosts": []string{"httpbin.org", "*.httpbin.org"}, "proxy": []string{socks[1], socks[2]}}, // round-robin
			// An IP-lookup host: rules for it apply on every browser but Camoufox.
			{"hosts": []string{"api.ipify.org"}, "proxy": socks[0]},
		},
		"headless": true,
	})
	if err != nil {
		log.Fatal(err)
	}
	pw, err := playwright.Run(&playwright.RunOptions{SkipInstallBrowsers: true})
	if err != nil {
		log.Fatal(err)
	}
	defer pw.Stop()
	browser, err := pw.Chromium.Connect(session["wsUrl"].(string), playwright.BrowserTypeConnectOptions{Headers: map[string]string{"x-api-key": key}})
	if err != nil {
		log.Fatal(err)
	}
	defer browser.Close()

	readings := []reading{}
	for _, u := range []string{"https://v4.ident.me/", "https://httpbin.org/ip", "https://httpbin.org/ip", "https://httpbin.org/ip",
		"https://api.ipify.org/", "https://www.cloudflare.com/cdn-cgi/trace"} {
		readings = append(readings, reading{u, exitIPVia(browser, u)})
	}
	out, _ := json.MarshalIndent(map[string]any{
		"rules": map[string]any{
			"*.ident.me":        "SOCKS proxy A",
			"httpbin.org":       []string{"SOCKS proxy B", "SOCKS proxy C"},
			"api.ipify.org":     "SOCKS proxy A",
			"(everything else)": "residential PROXY_URL",
		},
		"readings": readings,
	}, "", "  ")
	fmt.Println(string(out))
}

What we got

URL	Exit IP	Rule
https://v4.ident.me/	203.0.113.1	SOCKS proxy A
https://httpbin.org/ip	203.0.113.2	SOCKS proxy B or SOCKS proxy C
https://httpbin.org/ip	203.0.113.3	SOCKS proxy B or SOCKS proxy C
https://httpbin.org/ip	203.0.113.4	SOCKS proxy B or SOCKS proxy C
https://api.ipify.org/	203.0.113.1	SOCKS proxy A
https://www.cloudflare.com/cdn-cgi/trace	203.0.113.5	default proxy

From the Node.js run on 2026-10-17. IP addresses are replaced with placeholders (203.0.113.x); equal addresses stay equal. The other languages produced the same findings.

Raw output (Node.js)

{
  "rules": {
    "*.ident.me": "SOCKS proxy A",
    "httpbin.org": [
      "SOCKS proxy B",
      "SOCKS proxy C"
    ],
    "api.ipify.org": "SOCKS proxy A",
    "(everything else)": "residential PROXY_URL"
  },
  "readings": [
    {
      "url": "https://v4.ident.me/",
      "exit_ip": "203.0.113.1"
    },
    {
      "url": "https://httpbin.org/ip",
      "exit_ip": "203.0.113.2"
    },
    {
      "url": "https://httpbin.org/ip",
      "exit_ip": "203.0.113.3"
    },
    {
      "url": "https://httpbin.org/ip",
      "exit_ip": "203.0.113.4"
    },
    {
      "url": "https://api.ipify.org/",
      "exit_ip": "203.0.113.1"
    },
    {
      "url": "https://www.cloudflare.com/cdn-cgi/trace",
      "exit_ip": "203.0.113.5"
    }
  ]
}

Takeaways

ident.me exited through its SOCKS5 proxy, httpbin.org alternated between its two, and everything else went out residential.
Round-robin is per connection, not per request — requests on an open keep-alive connection stay on its proxy. A pool member that fails is skipped, so a run can show the same proxy twice, and a pool member that rotates its own exit IP can show a third address.
IP-lookup hosts follow their rule here, on Chromium: api.ipify.org exited through rule 3's proxy, the same exit as ident.me. On Camoufox they stay on the default proxy: api.ipify.org, checkip.amazonaws.com, ipinfo.io, icanhazip.com, ifconfig.co and ipecho.net ignore proxy_rules there, because Camoufox's geoip uses them to match its location to the exit. To check a rule on Camoufox, use another echo service such as v4.ident.me or httpbin.org/ip. (Until 2026-10-17 they were pinned on every browser.)
Rules route through a proxy only; there's no "direct from cdpfleet" rule — every session browses from a proxy you choose.