# DeepFetch
> Production scraping for JavaScript & anti-bot sites

Extract structured JSON from JavaScript-heavy and bot-protected sites. Headless Chrome, premium proxy, and live endpoints — try a scrape free, no sign-in required.

Official site: https://deepfetch.dev
Support: jack@deepfetch.dev
OpenAPI: https://deepfetch.dev/api/openapi.json
Human docs: https://deepfetch.dev/docs
Status: https://deepfetch.dev/status

---

## What DeepFetch is

DeepFetch is a production web scraping API and dashboard. One HTTP call renders pages with real headless Chrome (or a fast static fetch first), bypasses common anti-bot blocks with premium residential proxy, and returns HTML, Markdown, JSON, screenshots, or zero-config structured data.

Differentiators vs generic scraper APIs:
- **Live JSON endpoints** — publish scrape results as shareable cached URLs (no code required)
- **fetch_first** — cheap HTTP fetch before browser escalation (saves credits on static pages)
- **autoparse** — free heuristic JSON extraction (title, links, images, products, Open Graph) with no selector setup
- **llm_markdown + SSE streaming** — aggressive main-content Markdown tuned for LLM pipelines
- **Credit refunds** — failed or blocked requests are not billed
- **Block diagnostics** — every response includes `Sf-block-type`, `Sf-failure-reason`, proxy tier used

---

## Authentication

Pass your API key one of three ways:

```
?api_key=sk_live_xxx
Authorization: Bearer sk_live_xxx
X-API-Key: sk_live_xxx
```

Create keys in the dashboard: https://deepfetch.dev/dashboard/api

---

## Primary endpoint

```
GET|POST https://deepfetch.dev/api/v1
```

### Required
| Parameter | Type | Description |
|-----------|------|-------------|
| url | string | Target http(s) URL (fetch mode) |

### Rendering
| Parameter | Default | Description |
|-----------|---------|-------------|
| render_js | true | Headless Chrome. false = static HTTP only |
| fetch_first | true | When render_js=true, try cheap fetch before browser |
| wait | 0 | Fixed wait in ms after load (0–35000) |
| wait_for | — | CSS/XPath selector to wait for |
| wait_browser | domcontentloaded | load \| domcontentloaded \| networkidle \| commit |
| device | desktop | desktop \| mobile |
| timeout | 90000 | Navigation timeout ms (1000–140000) |

### Proxies & geo
| Parameter | Default | Description |
|-----------|---------|-------------|
| premium_proxy | false | Residential/premium proxy pool |
| country_code | — | Geo-target e.g. US, GB |
| own_proxy | — | BYO proxy URL override |

### Output formats
| Parameter | Default | Description |
|-----------|---------|-------------|
| output_format | html | html \| text \| markdown \| json |
| markdown | false | Shortcut for output_format=markdown |
| llm_markdown | false | Aggressive LLM-optimized Markdown (~90% smaller) |
| autoparse | false | **Free** structured JSON (title, links, images, products, OG tags) |
| extract_rules | — | CSS/XPath rules (+5 credits) |
| ai_query | — | Natural-language extraction (+5 credits) |
| ai_extract_rules | — | Schema-based AI extraction (+5 credits) |
| screenshot | false | PNG screenshot |
| screenshot_full_page | false | Full-page PNG |
| json_response | false | Wrap body + metadata in JSON envelope |

### Interaction & sessions
| Parameter | Description |
|-----------|-------------|
| js_scenario | JSON array of click/scroll/wait/fill/evaluate instructions |
| session_id | Reuse browser cookies across requests |
| create_session | Start a new persistent session |
| solve_captcha | Attempt CAPTCHA solve when blocked |

---

## Credit costs (successful requests only)

| Request type | Credits |
|--------------|---------|
| Static fetch (render_js=false) | 1 |
| JS rendering | 5 |
| Premium proxy, no JS | 10 |
| Premium proxy + JS | 25 |
| extract_rules / ai_query / ai_extract_rules | +5 |
| **autoparse** | **0 extra** (included in base tier cost) |

Failed, blocked, or empty results are **refunded automatically**.

---

## Response headers (Sf-*)

Every successful scrape includes:
- Sf-cost — credits charged
- Sf-credits-remaining — account balance
- Sf-resolved-url — final URL after redirects
- Sf-initial-status-code — target HTTP status
- Sf-attempts — retry count
- Sf-duration-ms — wall time
- Sf-proxy-used — true/false
- Sf-concurrency-limit — plan concurrent job cap
- Sf-concurrency-remaining — slots available now
- Sf-block-type — cloudflare, captcha, rate_limit, … (on failures)
- Sf-effective-proxy-tier — tier actually used after escalation

---

## Examples

### HTML with JS rendering
```bash
curl -G "https://deepfetch.dev/api/v1" \
  --data-urlencode "api_key=YOUR_KEY" \
  --data-urlencode "url=https://example.com" \
  --data-urlencode "render_js=true"
```

### Autoparse — zero-config JSON (no selectors, no AI surcharge)
```bash
curl -G "https://deepfetch.dev/api/v1" \
  --data-urlencode "api_key=YOUR_KEY" \
  --data-urlencode "url=https://example.com/product" \
  --data-urlencode "autoparse=true"
```

Returns JSON with: title, description, canonical, og tags, links, images, headings, product(s) from JSON-LD.

### LLM Markdown with streaming
```bash
curl -N "https://deepfetch.dev/api/v1/stream?api_key=YOUR_KEY&url=https://example.com/blog&llm_markdown=true"
```

### AI site search (66+ built-in templates)
```bash
curl -G "https://deepfetch.dev/api/v1" \
  --data-urlencode "api_key=YOUR_KEY" \
  --data-urlencode "mode=ai" \
  --data-urlencode "query=wireless keyboard" \
  --data-urlencode "site=amazon" \
  --data-urlencode "max_results=10"
```

### Store SERP APIs
```
GET https://deepfetch.dev/api/v1/store/google?search=YOUR+QUERY&api_key=YOUR_KEY
GET https://deepfetch.dev/api/v1/store/bing?search=YOUR+QUERY&api_key=YOUR_KEY
```

### Live JSON endpoints (dashboard)
Publish any dashboard scrape as a cached JSON URL:
```
GET https://deepfetch.dev/api/v1/endpoints/{id}?token={share_token}
GET https://deepfetch.dev/api/v1/endpoints/{id}?token={share_token}&fresh=true
```

---

## Webhooks (Zapier, Make, custom)

Configure an HTTPS webhook URL in https://deepfetch.dev/dashboard/settings.

Events:
- job.completed — dashboard scrape finished successfully
- job.failed — dashboard scrape failed

Payload (JSON POST):
```json
{
  "event": "job.completed",
  "endpointId": "abc123",
  "status": "completed",
  "site": "amazon",
  "query": "wireless keyboard",
  "resultCount": 10,
  "durationMs": 4200,
  "completedAt": "2026-06-25T12:00:00.000Z",
  "endpointDataUrl": "https://deepfetch.dev/api/v1/endpoints/{id}?token=...",
  "endpointFreshUrl": "https://deepfetch.dev/api/v1/endpoints/{id}?token=...&fresh=true",
  "dashboardUrl": "https://deepfetch.dev/dashboard/jobs/{id}",
  "dataPreview": [{ "title": "...", "price": "..." }]
}
```

Headers:
- Content-Type: application/json
- X-DeepFetch-Event: job.completed
- X-DeepFetch-Signature: sha256=... (when signing secret configured)
- X-Hook-Secret: ... (mirrors your secret for Zapier Catch Hook verification)

Zapier setup: Create Zap → Trigger "Webhooks by Zapier" → Catch Hook → paste URL into DeepFetch settings → complete a scrape → test trigger.

---

## Usage & limits

```
GET https://deepfetch.dev/api/v1/usage
```

Returns credits, plan, rate limit (RPM), max_concurrency, current_concurrency.

| Plan | RPM | Max concurrent |
|------|-----|----------------|
| Free | 30 | 2 |
| Starter | 60 | 5 |
| Pro | 120 | 15 |
| Enterprise | 600 | 50 |

---

## SDK

TypeScript/JavaScript: `@deepfetch/sdk` — see https://deepfetch.dev/docs and https://deepfetch.dev/api/openapi.json

---

## Legal & compliance

- Terms: https://deepfetch.dev/legal/terms
- Privacy: https://deepfetch.dev/legal/privacy
- Acceptable use: https://deepfetch.dev/legal/acceptable-use
- Trust & security: https://deepfetch.dev/trust
- SSRF protection on all targets; optional robots.txt respect (`respect_robots=true`)

---

## Documentation index

| Topic | URL |
|-------|-----|
| API reference | https://deepfetch.dev/docs/api |
| JavaScript rendering | https://deepfetch.dev/docs/javascript-rendering |
| Proxies | https://deepfetch.dev/docs/proxies |
| Dashboard guide | https://deepfetch.dev/dashboard/docs |
| Pricing | https://deepfetch.dev/upgrade |
| Web scraping API product | https://deepfetch.dev/web-scraping-api |
| Scrape bot product | https://deepfetch.dev/scrape-bot |

---

## Changelog & contact

For integration help: jack@deepfetch.dev
System status: https://deepfetch.dev/status

Last updated: 2026-06-25