> For the complete documentation index, see [llms.txt](https://cloudbrowser.gitbook.io/docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://cloudbrowser.gitbook.io/docs/configurations/scraping-with-ollama-and-crawl4ai.md).

# Scraping with Ollama and Crawl4AI

When you need to extract unstructured information from websites, a simple HTTP request is not enough — especially if the site uses JavaScript or dynamic rendering.\
In this tutorial, we'll walk through how to use **Crawl4AI**, **Ollama**, and **CloudBrowser** to automate the process, extract data, and format it into clean JSON.

***

### Why This Stack

* **Crawl4AI** — Orchestrates the crawling, handles concurrency, and chunks large content.
* **CloudBrowser** — Runs cloud-hosted Puppeteer browsers with Chrome DevTools Protocol (CDP) support.
* **Ollama** — Runs the LLM that turns HTML/Markdown into structured data based on your schema.

***

### Requirements

* Basic **Python** knowledge.
* Installed **Crawl4AI** and **Ollama**.
* A **CloudBrowser** account and API token.

***

### Environment Setup

#### 1. Install Crawl4AI

Follow the instructions from the official GitHub repository:

```bash
pip install crawl4ai
```

#### 2. Install and Run Ollama

If you don't have a remote Ollama server, install and run it locally:

```bash
brew install ollama
ollama pull gemma3:latest
ollama serve
```

This launches the **Gemma 3** model on `http://localhost:11434`.

#### 3. Request a CloudBrowser WebSocket

CloudBrowser sessions are opened by sending a POST request to the API. For example:

```bash
curl -X POST https://production.cloudbrowser.ai/api/v1/Browser/Open \
-H "Authorization: Bearer $CLOUDBROWSER_ACCESS_KEY" \
-d '{
  "Args": ["--no-sandbox", "--disable-gpu"],
  "IgnoredDefaultArgs": ["--enable-automation"],
  "Headless": true,
  "Stealth": true,
  "Browser": 2,
  "KeepOpen": 60,
  "Label": "app-worker01",
  "saveSession": true,
  "recoverSession": true
}'
```

The API will respond with JSON containing the `address` field:

```json
{
  "address": "browserprovider-worker-*.cloudbrowser.ai/devtools/browser/*",
  "status": 200
}
```

Use this `address` as your Chrome DevTools Protocol WebSocket URL.

***

### Example: Scraping Pricing Plans

The following Python script first requests a CloudBrowser session to get the WebSocket URL, then runs Crawl4AI.

```python
import asyncio
import os
import requests
from typing import List
from crawl4ai import *
from pydantic import BaseModel

# Step 1: Open a CloudBrowser session to get WebSocket URL
CLOUDBROWSER_API_KEY = os.environ.get("CLOUDBROWSER_ACCESS_KEY")
response = requests.post(
    "https://production.cloudbrowser.ai/api/v1/Browser/Open",
    headers={"Authorization": f"Bearer {CLOUDBROWSER_API_KEY}"},
    json={
        "Args": ["--no-sandbox", "--disable-gpu"],
        "IgnoredDefaultArgs": ["--enable-automation"],
        "Headless": True,
        "Stealth": True,
        "Browser": 2,
        "KeepOpen": 60,
        "Label": "app-worker01",
        "saveSession": True,
        "recoverSession": True
    }
)
ws_url = response.json()["address"]

# Step 2: Define the output schema
class Pricing(BaseModel):
    name: str
    href: str

class PricingPlans(BaseModel):
    pricing_plans: List[Pricing]

# Step 3: Configure CloudBrowser with the returned WebSocket
browser_config = BrowserConfig(
    headless=True,
    verbose=True,
    browser_mode="cdp",
    cdp_url=f"wss://{ws_url}"
)

# Step 4: Define the extraction strategy using Ollama + Gemma 3
extraction_strategy = LLMExtractionStrategy(
    llm_config=LLMConfig(provider="ollama/gemma3:4b", base_url="http://localhost:11434"),
    extraction_type="schema",
    schema=PricingPlans.model_json_schema(),
    instruction="Extract all the pricing plans JSON array containing their 'name' and 'price'.",
    chunk_token_threshold=1200,
    overlap_rate=0.1,
    apply_chunking=True,
    input_format="markdown",
    verbose=True
)

# Step 5: Configure and run the crawler
crawl_config = CrawlerRunConfig(
    extraction_strategy=extraction_strategy,
    cache_mode=CacheMode.BYPASS
)

async def main():
    async with AsyncWebCrawler(config=browser_config) as crawler:
        result = await crawler.arun(url="https://cloudbrowser.com/", config=crawl_config)

        if result.success:
            print("Extracted content:", result.extracted_content)
        else:
            print("Error:", result.error_message)

if __name__ == "__main__":
    asyncio.run(main())
```

***

### How It Works

1. **CloudBrowser API** creates a browser session and returns a WebSocket URL.
2. **Crawl4AI** connects to that WebSocket, navigates to the target page, and extracts DOM content.
3. **Ollama (Gemma 3)** formats the result into JSON based on your schema.
4. The final structured data is printed in your console.

***

### Conclusion

This approach ensures that you dynamically request a fresh CloudBrowser session before crawling, keeping the session alive only as long as you need it.\
**CloudBrowser + Crawl4AI + Ollama** provides a powerful, scala