RSS 中文
⌕

Chapter 25 | Web Proxy and Scraping MCP (WebFetch / Playwright)

20 MIN READ | UPDATED: 2026-06-07

🎯 Learning Objectives

  • Understand the importance and working principles of web proxies in scraping, and their application scenarios in Claude Code.
  • Master the method of performing lightweight, static web content scraping using the WebFetch tool in Claude Code.
  • Learn how to use the Playwright tool for complex dynamic web interactions, data extraction, and automated operations.
  • Understand how Claude Code integrates these external data collection capabilities to achieve an advanced workflow of gathering information from the internet and injecting it into code projects.

📖 Core Concepts Explained

25.1 Beyond the Local Disk: Why Claude Needs Web Access

Modern development often requires context from external documentation, API references, or open-source examples that aren't present in your local codebase. By enabling web tools, you allow Claude to:

  1. Search for Solutions: Look up error messages on StackOverflow or GitHub Issues.
  2. Read Latest Docs: Access the most recent documentation for libraries that might have updated since Claude's training cutoff.
  3. Analyze Competitors/Examples: Scrape specific websites to understand their structure or implementation details.

25.2 Lightweight Scraping with WebFetch

WebFetch is your go-to tool for simple GET requests. It's fast and doesn't require a browser engine.

# Example usage via Claude TUI
> "Read the content of https://example.com/docs/api-v2 and summarize the changes."

25.3 Dynamic Scraping with Playwright

When a website uses React, Vue, or other JS frameworks to render content, WebFetch might return an empty <div>. This is where Playwright comes in. It launches a headless browser, executes JS, and then extracts the data.

# Example usage via Claude TUI
> "Use Playwright to go to the login page of https://myapp.com, fill in 'test_user', and tell me if the error message appears."

🔧 Tools & Skills

Tool Purpose
WebFetch Fetches raw HTML or text from a URL.
Playwright Full browser automation (Click, Type, Wait, Scrape).
Search Performs a Google/Bing search to find relevant URLs first.

📝 Key Takeaways

  1. Static vs. Dynamic: Use WebFetch for speed and Playwright for complex JS-heavy sites.
  2. Privacy & Ethics: Always respect robots.txt and the terms of service of the websites you scrape.
  3. Data Injection: Claude doesn't just read the web; it can use that information to write new code or fix bugs in your project.