Web Unlocker

# Web Unlocker API Reference ## Table of Contents - [Overview](#overview) - [Authentication](#authentication) - [REST API (Recommended)](#rest-api-recommended) - [Proxy Interface](#proxy-interface) - [All Request Parameters](#all-request-parameters) - [Special Headers](#special-headers) - [Response Structure](#response-structure) - [Async Requests](#async-requests) - [Output Formats](#output-formats) - [Features](#features) - [Billing Model](#billing-model) - [Best Practices](#best-practices) - [Anti-Patterns](#anti-patterns) --- ## Overview Web Unlocker is Bright Data's HTTP-based unlocking proxy. It automatically selects the best proxy network, handles anti-bot protections, solves CAPTCHAs, and retries failed attempts. Use it for non-interactive data extraction where you need HTML, JSON, markdown, or screenshots of web pages via a simple HTTP request. **When to use Web Unlocker vs other APIs:** | Need | Use | |------|-----| | Scrape any webpage by URL | Web Unlocker | | Search engine results (Google, Bing) | SERP API | | Structured data from known platforms | Web Scraper API | | Click, scroll, fill forms, run JS | Browser API | | Browser automation libraries (Playwright/Puppeteer) | Browser API, NOT Web Unlocker | --- ## Authentication Get your API key from the Bright Data Control Panel → Account Settings, or from your zone's Overview tab. ```bash # Environment variable (recommended) export BRIGHTDATA_API_KEY="your-api-key" export BRIGHTDATA_UNLOCKER_ZONE="your-zone-name" ``` --- ## REST API (Recommended) **Endpoint:** `POST https://api.brightdata.com/request` **Header:** `Authorization: Bearer YOUR_API_KEY` ### Minimal Request ```python import requests response = requests.post( "https://api.brightdata.com/request", headers={"Authorization": f"Bearer {API_KEY}"}, json={ "zone": "YOUR_ZONE_NAME", "url": "https://example.com", "format": "raw" } ) print(response.text) ``` ```javascript const response = await fetch("https://api.brightdata.com/request", { method: "POST", headers: { "Authorization": `Bearer ${API_KEY}`, "Content-Type": "application/json" }, body: JSON.stringify({ zone: "YOUR_ZONE_NAME", url: "https://example.com", format: "raw" }) }); const html = await response.text(); ``` ```bash curl -H "Authorization: Bearer $BRIGHTDATA_API_KEY" -H "Content-Type: application/json" -d '{"zone":"'"$BRIGHTDATA_UNLOCKER_ZONE"'","url":"https://example.com","format":"raw"}' https://api.brightdata.com/request ``` --- ## Proxy Interface Alternative to the REST API — route requests through a proxy endpoint. - **Host:** `brd.superproxy.io` - **Port:** `33335` - **Credentials:** `brd-customer-{CUSTOMER_ID}-zone-{ZONE_NAME}:{ZONE_PASSWORD}` ```python import requests proxies = { "http": "http://brd-customer-CUSTOMER_ID-zone-ZONE_NAME:[email protected]:33335", "https": "http://brd-customer-CUSTOMER_ID-zone-ZONE_NAME:[email protected]:33335" } # Install Bright Data's SSL certificate to avoid SSL errors response = requests.get("https://example.com", proxies=proxies, verify="/path/to/brightdata-cert.crt") ``` Special proxy username flags (append to username string): - `-country-XX` → Geo-target to country (e.g., `-country-us`) - `-ua-mobile` → Use mobile user agent - `-debug-full` → Enable debug header in response --- ## All Request Parameters ### REST API Body Parameters | Parameter | Type | Required | Description | |-----------|------|----------|-------------| | `zone` | string | Yes | Zone name from your Control Panel | | `url` | string | Yes | Target URL (must include `http://` or `https://`) | | `format` | string | Yes | `"raw"` (HTML string) or `"json"` (structured JSON) | | `method` | string | No | HTTP verb, default `"GET"`. Use `"POST"` with `body` for POST requests | | `body` | string | No | POST body payload (use with `"method": "POST"`) | | `country` | string | No | 2-letter ISO country code for geo-targeting (e.g., `"us"`, `"gb"`). Auto-selected if omitted | | `data_format` | string | No | Transform output: `"markdown"` or `"screenshot"` | | `async` | boolean | No | Set `true` to use asynchronous mode. Returns `response_id` immediately | --- ## Special Headers Used with the **proxy interface**. Pass in proxy username or as custom request headers. | Header | Value | Description | |--------|-------|-------------| | `x-unblock-data-format` | `"markdown"` or `"screenshot"` | Convert response to markdown or PNG screenshot | | `x-unblock-expect` | CSS selector, text, or body content | Wait for this element/text before returning response. Prevents partial page loads | | `x-unblock-url-fragment` | The `#fragment` part of URL | Handle hash-fragmented URLs in a single request | | `x-unblock-city` | City name string | Amazon-specific: simulate city selection | | `x-unblock-zipcode` | ZIP code string | Amazon-specific: simulate ZIP code for region-specific content | **Proxy username flags** (appended to username): - `-ua-mobile` → Uses mobile user agents instead of desktop - `-debug-full` → Adds `x-brd-debug` response header with: request ID, traffic metrics, billing status, destination IP, headers used, peer info, render status --- ## Response Structure ### Synchronous (format: "json") ```json { "status": 200, "headers": { "content-type": "text/html", "...": "..." }, "body": "..." } ``` ### Synchronous (format: "raw") Returns the raw HTML/content as a string directly. ### Error Codes | Code | Meaning | Action | |------|---------|--------| | `200` | Success | Process response | | `400` | Bad Request | Check required fields: `zone`, `url`, `format` | | `401` | Unauthorized | Verify API key | --- ## Async Requests ### Setup Enable in Control Panel: Your Zone → Advanced Options → Toggle "Asynchronous requests" ON. ### Flow **Step 1: Submit request** ```python response = requests.post( "https://api.brightdata.com/unblocker/req", params={"zone": "YOUR_ZONE_NAME"}, headers={"Authorization": f"Bearer {API_KEY}"}, json={"url": "https://example.com"} ) response_id = response.json()["response_id"] ``` **Step 2: Poll for result** ```python import time while True: result = requests.get( "https://api.brightdata.com/unblocker/get_result", params={"response_id": response_id}, headers={"Authorization": f"Bearer {API_KEY}"} ) if result.status_code == 200: print(result.json()["body"]) break time.sleep(5) ``` ### Async Key Facts - Responses typically complete within **5 minutes**, up to **8 hours** during peak - Results stored for **48 hours** - Better for slow-responding sites or large-scale URL processing - Use webhooks for production: set webhook URL in zone settings to receive push notifications --- ## Output Formats ### HTML (default) Raw HTML string. Use `format: "raw"`. ### Markdown Clean markdown extracted from the page. Best for LLM consumption. **REST API:** ```json { "zone": "...", "url": "...", "format": "raw", "data_format": "markdown" } ``` **Proxy:** Add header: `x-unblock-data-format: markdown` ### Screenshot PNG screenshot of the rendered page. Useful for debugging and visual verification. **REST API:** ```json { "zone": "...", "url": "...", "format": "raw", "data_format": "screenshot" } ``` --- ## Features ### CAPTCHA Solving Enabled by default. Automatically solves CAPTCHAs encountered during requests. Can be disabled in Advanced Settings for a lightweight solution when CAPTCHA solving isn't needed. ### Premium Domains Certain high-difficulty websites require additional resources. Enable "Premium Domains" in zone settings during creation. Only requests targeting premium-classified domains are billed at the higher rate. ### Geolocation Targeting Auto-selects optimal IP location. Override with `country` parameter (REST API) or `-country-XX` (proxy). Country-level targeting only (not city-level—use Browser API for that). ### Mobile User Agent Append `-ua-mobile` to proxy username, or use via proxy username flag, to use mobile-specific user agents. ### Auto-Throttling System automatically adjusts based on success rates: - Default threshold: 70% success rate - Automatically applies better-performing configurations - When custom headers/cookies are enabled: customizable threshold ### Debug Header Add `-debug-full` to proxy username to receive `x-brd-debug` response header containing: request ID, traffic metrics, billing status, destination IP, headers used, peer info, render status. ### Success Rate API Query domain-specific success rates for the past 7 days: ```bash GET https://api.brightdata.com/unblocker/success_rate/?zone=YOUR_ZONE&domain=example.com Authorization: Bearer YOUR_API_KEY ``` --- ## Custom Headers & Cookies (Advanced) Override automatically-generated headers and cookies. **When to use:** When you need to pass specific session cookies, authentication headers, or custom values to reach a particular site version. **How to enable:** Control Panel → Your Zone → Advanced Options → Toggle "Custom Headers & Cookies" ON. **Critical billing note:** Enabling custom headers/cookies means you are **billed for 100% of requests** (both successful and failed), because you are taking control of request parameters. Standard mode only bills for successes. **Restrictions:** - Custom values must be from a compliance-pre-approved list - Unlisted values require approval from the compliance team - Cannot pass authentication/login credentials --- ## Billing Model | Mode | Billing | |------|---------| | Standard | CPM — billed per 1,000 **successful** requests only | | Custom Headers/Cookies enabled | Billed for **all** requests (successful + failed) | | Async collect/retrieve calls | **Not billed** — only the initial submission is billed | Monitor usage via the "Traffic" column in My Proxies (CPM = cost per 1,000 successful requests). --- ## Best Practices ### 1. Start with direct API endpoint targeting Many sites expose clean API endpoints. Try hitting the API directly first — it often succeeds without extra config and is cheaper. ```python # Try the API endpoint first response = requests.post( "https://api.brightdata.com/request", headers={"Authorization": f"Bearer {API_KEY}"}, json={"zone": ZONE, "url": "https://site.com/api/products.json", "format": "raw"} ) ``` ### 2. Fall back to main webpage if API fails If the direct API endpoint fails, scrape the primary webpage instead. ### 3. Use Browser API for complex JS-heavy scenarios When you need to execute JavaScript, click elements, or interact with the page — don't try to force Web Unlocker. Use Browser API. ### 4. Use `x-unblock-expect` to prevent partial loads For pages that load content progressively, specify an expected element to ensure the content you need is present. ```json { "zone": "...", "url": "https://example.com/products", "format": "raw", "headers": { "x-unblock-expect": ".product-list" } } ``` ### 5. Use markdown format for LLM pipelines When feeding scraped content to an LLM, use `data_format: "markdown"` to get clean, structured text without HTML noise. ### 6. Use async for bulk/large-scale processing Async improves stability for slow sites and enables processing large batches of URLs without blocking. ### 7. Use geolocation for region-restricted content Pass `country` parameter when you need region-specific content (prices, availability, localized pages). ```json { "zone": "...", "url": "https://example.com", "format": "raw", "country": "de" } ``` ### 8. Monitor success rates before custom header mode Check your domain success rates via the API before enabling custom headers (which changes billing to 100%). --- ## Anti-Patterns **DO NOT use Web Unlocker with browser automation libraries.** - Puppeteer, Playwright, Selenium → use **Browser API** instead - Chrome, Firefox, Edge → use **Bright Data proxy networks** (Residential, ISP, etc.) - Anti-detect browsers (Adspower, Multilogin) → use proxy networks Web Unlocker is optimized for singular HTTP requests, not browser sessions. **DO NOT enable custom headers unless necessary.** Enabling custom headers changes billing from success-only to 100% of all requests. **DO NOT ignore the `x-unblock-expect` header for paginated/dynamic content.** Without it, you may receive partial pages before JavaScript has populated content.