Web Fetch Strategy
Common web fetching strategy for anti-crawler handling.
Site Classification
| Type | Examples | Characteristics |
|---|---|---|
| Anti-crawler | Reddit, Twitter/X, LinkedIn | Need login or browser fingerprint |
| Regular | blog.rust-lang.org, docs.rs | No anti-crawler, direct fetch |
Fetch Priority
Anti-crawler sites: Local Chrome → crawl4ai MCP → give up and mark
Regular sites: WebFetch → crawl4ai MCP
Tools
1. Local Chrome (for anti-crawler)
User's real browser with login and normal fingerprint.
macOS:
# Open URL
osascript -e 'tell application "Google Chrome" to open location "URL"'
# Get page HTML
osascript -e 'tell application "Google Chrome" to execute front window'\''s active tab javascript "document.documentElement.outerHTML"'
2. crawl4ai MCP (fallback)
Strong anti-crawler bypass, needs Docker.
mcp__crawl4ai__scrape(url: "URL")
3. WebFetch (regular sites)
Built-in tool, simple and fast, no anti-crawler capability.
Site Routing
| Domain | Tool | Reason |
|---|---|---|
| reddit.com | Local Chrome | Strict anti-crawler |
| twitter.com / x.com | Local Chrome | Needs login |
| linkedin.com | Local Chrome | Strict anti-crawler |
| *.rust-lang.org | WebFetch | No anti-crawler |
| docs.rs | WebFetch | No anti-crawler |
| crates.io | WebFetch | No anti-crawler |
| this-week-in-rust.org | WebFetch | No anti-crawler |
| rustfoundation.org | WebFetch | No anti-crawler |
| github.com | WebFetch | Light rate limit |
Failure Handling
- Local Chrome fails → try crawl4ai
- crawl4ai fails → try WebFetch
- All fail → mark "Fetch failed: {reason}"
Validation
After fetch, check:
- Content is not empty
- Not an error page (403, 429, "blocked")
- Contains expected data
你可能感兴趣的文章
Agent Integration Test Scenarios
Ownership (m01) Test Scenarios