Map

A site's URLs from its sitemaps and the start page's links, without loading every page — the quick first look before a crawl. POST https://cdpfleet.com/v1/functions/map → application/json.

curl -s https://cdpfleet.com/v1/functions/map \
  -H "x-api-key: $CDPFLEET_API_KEY" -H "content-type: application/json" \
  -d '{"url":"https://www.djangoproject.com/","max_urls":200,"include":["/weblog/**"]}' -o urls.json -D - | grep -i '^x-call-id'

// Node.js 18+: save as map.mjs, run node map.mjs
const res = await fetch('https://cdpfleet.com/v1/functions/map', {
  method: 'POST',
  headers: { 'x-api-key': process.env.CDPFLEET_API_KEY, 'content-type': 'application/json' },
  body: JSON.stringify({
    url: 'https://www.djangoproject.com/',
    max_urls: 200,
    include: ['/weblog/**'],
  }),
  signal: AbortSignal.timeout(30000),
});
if (!res.ok) throw new Error(`${res.status} ${await res.text()}`); // 429: wait Retry-After seconds
console.log(await res.text());

# Python 3 with requests (pip install requests): save as map.py, run python map.py
import os
import requests

r = requests.post(
    "https://cdpfleet.com/v1/functions/map",
    headers={"x-api-key": os.environ["CDPFLEET_API_KEY"]},
    json={
        "url": "https://www.djangoproject.com/",
        "max_urls": 200,
        "include": ["/weblog/**"],
    },
    timeout=30,
)
if not r.ok:
    raise SystemExit(f"{r.status_code} {r.text}")  # 429: wait r.headers["Retry-After"] seconds
print(r.text)

// Java 17+: save as Main.java, run java Main.java
import java.net.URI;
import java.net.http.*;
import java.time.Duration;

public class Main {
  public static void main(String[] args) throws Exception {
    String body = """
        {
          "url": "https://www.djangoproject.com/",
          "max_urls": 200,
          "include": ["/weblog/**"]
        }
        """;
    HttpResponse<byte[]> res = HttpClient.newHttpClient().send(
        HttpRequest.newBuilder(URI.create("https://cdpfleet.com/v1/functions/map"))
            .header("x-api-key", System.getenv("CDPFLEET_API_KEY"))
            .header("content-type", "application/json")
            .timeout(Duration.ofSeconds(30))
            .POST(HttpRequest.BodyPublishers.ofString(body))
            .build(),
        HttpResponse.BodyHandlers.ofByteArray());
    if (res.statusCode() != 200) { // 429: wait Retry-After seconds
      throw new RuntimeException(res.statusCode() + " " + new String(res.body()));
    }
    System.out.writeBytes(res.body()); // UTF-8 as it came
    System.out.println();
  }
}

// .NET 8+: dotnet new console, replace Program.cs with this, run dotnet run
using System.Text;

using var http = new HttpClient { Timeout = TimeSpan.FromSeconds(30) };
http.DefaultRequestHeaders.Add("x-api-key", Environment.GetEnvironmentVariable("CDPFLEET_API_KEY"));
var body = """
    {
      "url": "https://www.djangoproject.com/",
      "max_urls": 200,
      "include": ["/weblog/**"]
    }
    """;
var res = await http.PostAsync("https://cdpfleet.com/v1/functions/map",
    new StringContent(body, Encoding.UTF8, "application/json"));
if (!res.IsSuccessStatusCode) // 429: wait Retry-After seconds
    throw new Exception($"{(int)res.StatusCode} {await res.Content.ReadAsStringAsync()}");
Console.WriteLine(await res.Content.ReadAsStringAsync());

// Go 1.21+: save as main.go, run go run main.go
package main

import (
	"fmt"
	"io"
	"log"
	"net/http"
	"os"
	"strings"
	"time"
)

func main() {
	body := `{
	  "url": "https://www.djangoproject.com/",
	  "max_urls": 200,
	  "include": ["/weblog/**"]
	}`
	req, _ := http.NewRequest("POST", "https://cdpfleet.com/v1/functions/map", strings.NewReader(body))
	req.Header.Set("x-api-key", os.Getenv("CDPFLEET_API_KEY"))
	req.Header.Set("content-type", "application/json")
	res, err := (&http.Client{Timeout: 30 * time.Second}).Do(req)
	if err != nil {
		log.Fatal(err)
	}
	defer res.Body.Close()
	data, err := io.ReadAll(res.Body)
	if err != nil {
		log.Fatal(err)
	}
	if res.StatusCode != http.StatusOK { // 429: wait Retry-After seconds
		log.Fatalf("%s %s", res.Status, data)
	}
	fmt.Println(string(data))
}

Fields

Field	Type	Default	What it does
`url`	string	required	The page to load. A public `http` or `https` address: private, loopback, link-local, CGNAT and cloud-metadata addresses, `localhost`, `.local`/`.internal` names and names that resolve to them are refused (`400 invalid_url`).
`country`	string	`"random"`	Two-letter country for the exit IP and for the page's locale, timezone, geolocation and `Accept-Language` (see countries); `"random"` picks one. Unknown codes: `400 invalid_country`.
`max_wait`	number	`20`	Seconds to wait for the page to load, 1–60 (`400 invalid_option` otherwise). The whole call returns within about `max_wait` + 5 s, so a client timeout of `max_wait` + 10 s is safe. A page that is still loading then comes back partial (below); one that never started: `408 page_timeout`.
`proxy`	string or string[]	the Community proxy	Omit it to use the free Community proxy in `country`. Or: your own proxy URL (`http://`, `https://`, `socks5://`), a list (one is picked per call), a `cdpfleet-plan…` token, or a residential `cdpfleet-resi…` token (tier, country, device OS, sticky session) billed per GB to your residential balance (proxies). `proxy_rules` isn't supported yet (`400 invalid_option`).
`max_urls`	number	`500`	At most this many URLs, 1–5,000.
`include`	string[]	all	Keep only URLs matching one of these patterns. `` matches within a path segment, across segments; a pattern starting with `/` matches the path on any host, e.g. `"/blog/`*`"`.
`exclude`	string[]	none	Drop URLs matching one of these patterns (same syntax), e.g. `["/tag/", ".pdf"]`.
`same_site`	boolean	`true`	Only URLs on the start URL's site.

Good to know

Response: urls[] and sources {sitemap, links} — how many came from sitemaps (from robots.txt, including nested sitemap indexes) and how many from the start page's links.
For more than 5,000 URLs, or links followed deeper than the start page, add "job": true (paid plans): it runs as a map job with no limit.

Billing, errors and countries: Functions.