Crawlee handles the crawling logic — queues, retries, session pools, browser automation — but it doesn't solve the one thing that gets scrapers blocked fastest: sending every request from the same IP address. That's what a proxy layer is for, and Crawlee's ProxyConfiguration class makes wiring one in straightforward, whether you're on the Node.js/TypeScript version or Crawlee for Python.
This guide covers both. Pick the language section you need, or read both if your team runs a mixed stack.
Why Crawlee needs a proxy
Crawlee's session pool, retry logic, and autoscaling all assume you'll eventually get blocked and are built to recover gracefully — but none of that helps if every retry comes from the same IP. A proxy layer gives Crawlee:
- A rotating pool of IPs so rate limits and IP-based blocks don't stop a crawl after a few hundred requests
- Sticky sessions so a multi-step flow (login, then browse, then checkout) looks like one consistent visitor instead of a different IP on every request
- Geo-targeting so you can crawl a site the way a visitor from a specific country would see it — pricing pages, localized search results, region-locked content
Choosing the right Webshare proxy type
Webshare offers three proxy products, and which one fits depends on what your Crawlee crawler is actually doing:
If you're not sure, start with datacenter proxies — they're the cheapest way to confirm your Crawlee setup works, and you can upgrade specific crawlers to residential once you see where you're getting blocked.
Getting your Webshare proxy credentials
- Sign up at Webshare and open your dashboard — new accounts start with 10 free proxies, enough to test this setup before committing to a plan.
- Go to Proxy List and set Connection Method to Backbone Connection. This matters: the default Direct Connection view shows credentials in a different format that won't work with the examples below.
- Note your username, password, and the proxy address — Backbone connections always use
p.webshare.io, on port 80 (username/password auth also supports 1080, 3128, and 9999–19999).
That's the only setup needed before touching Crawlee.
Node.js / TypeScript
Install
npm install crawlee
Basic setup
ProxyConfiguration plugs into every crawler type — CheerioCrawler, HttpCrawler, JSDOMCrawler, PlaywrightCrawler, and PuppeteerCrawler — the same way:
import { CheerioCrawler, ProxyConfiguration } from 'crawlee';
const proxyConfiguration = new ProxyConfiguration({
proxyUrls: [
'http://username:password@p.webshare.io:80',
],
});
const crawler = new CheerioCrawler({
proxyConfiguration,
async requestHandler({ request, $, log, proxyInfo }) {
const title = $('title').text();
log.info(`${request.url} -> "${title}" (via ${proxyInfo?.url})`);
},
});
await crawler.run(['https://example.com']);
proxyInfo in the request handler confirms which proxy served each request — useful for debugging before you scale up.
Sticky sessions with country targeting
Webshare supports appending parameters to your username to target a country or pin a session: {username}-{country_code}-{session_id}. Combine that with Crawlee's newUrlFunction, which receives the session ID Crawlee is currently using, and every request tied to that session gets the same Webshare IP:
import { PlaywrightCrawler, ProxyConfiguration } from 'crawlee';
const proxyConfiguration = new ProxyConfiguration({
newUrlFunction: (sessionId) => {
const sid = sessionId ?? Math.floor(Math.random() * 1_000_000);
return `http://username-us-${sid}:password@p.webshare.io:80`;
},
});
const crawler = new PlaywrightCrawler({
proxyConfiguration,
useSessionPool: true,
sessionPoolOptions: { maxPoolSize: 50 },
async requestHandler({ page, request, proxyInfo, log }) {
await page.waitForSelector('body');
log.info(`${request.url} via ${proxyInfo?.url}`);
},
});
await crawler.run(['https://example.com']);
If Crawlee detects blocking, it automatically rotates to a new session ID — which, via newUrlFunction, rotates you to a new Webshare IP without any extra code.
Tiered proxies: cheap by default, escalate on blocking
tieredProxyUrls lets Crawlee start on your cheapest proxy type and automatically climb to a pricier tier only when it detects blocking on a given domain — a natural fit for mixing Webshare's three proxy types in one crawler:
const proxyConfiguration = new ProxyConfiguration({
tieredProxyUrls: [
['http://dc-username:password@p.webshare.io:80'], // Tier 0: datacenter
['http://isp-username:password@p.webshare.io:80'], // Tier 1: static residential
['http://rotating-username:password@p.webshare.io:80'], // Tier 2: rotating residential
],
});
Crawlee periodically re-checks lower tiers, so if a site's blocking eases up, it'll drop back down to the cheaper proxy automatically.
Python
Install
pip install crawlee
# Add extras depending on the crawler you need:
pip install 'crawlee[beautifulsoup]'
pip install 'crawlee[playwright]'
Basic setup
The same ProxyConfiguration class works across BeautifulSoupCrawler, HttpCrawler, and PlaywrightCrawler:
import asyncio
from crawlee.beautifulsoup_crawler import BeautifulSoupCrawler, BeautifulSoupCrawlingContext
from crawlee.proxy_configuration import ProxyConfiguration
async def main() -> None:
proxy_configuration = ProxyConfiguration(
proxy_urls=[
'http://username:password@p.webshare.io:80',
]
)
crawler = BeautifulSoupCrawler(proxy_configuration=proxy_configuration)
@crawler.router.default_handler
async def request_handler(context: BeautifulSoupCrawlingContext) -> None:
title = context.soup.title.string if context.soup.title else None
context.log.info(f'{context.request.url} -> "{title}"')
await crawler.run(['https://example.com'])
if __name__ == '__main__':
asyncio.run(main())
Sticky sessions with country targeting
Same pairing as the JS version: Webshare's {username}-{country_code}-{session_id} format plugged into Crawlee's new_url_function, which receives the active session_id:
import asyncio
from crawlee.playwright_crawler import PlaywrightCrawler, PlaywrightCrawlingContext
from crawlee.proxy_configuration import ProxyConfiguration
def get_webshare_url(session_id: str | None = None, request=None, proxy_tier=None) -> str:
sid = session_id or 'default'
return f'http://username-us-{sid}:password@p.webshare.io:80'
async def main() -> None:
proxy_configuration = ProxyConfiguration(new_url_function=get_webshare_url)
crawler = PlaywrightCrawler(proxy_configuration=proxy_configuration)
@crawler.router.default_handler
async def request_handler(context: PlaywrightCrawlingContext) -> None:
title = await context.page.title()
context.log.info(f'{context.request.url} -> "{title}"')
await crawler.run(['https://example.com'])
if __name__ == '__main__':
asyncio.run(main())
Tiered proxies
proxy_configuration = ProxyConfiguration(
tiered_proxy_urls=[
['http://dc-username:password@p.webshare.io:80'],
['http://isp-username:password@p.webshare.io:80'],
['http://rotating-username:password@p.webshare.io:80'],
]
)
A gotcha worth knowing: the port 80/443 bug
If you're on an older version of crawlee-python, watch for a proxy URL silently turning into something like p.webshare.io:None instead of :80. This was a real bug reported against crawlee-python — a proxy on port 80 or 443 got mangled because of how the underlying HTTP client read "default" ports as None rather than the actual number. It's since been fixed, but if you see a stray :None in a proxy error, upgrading your crawlee/crawlee-python version is the fix, not your Webshare credentials.
Verifying your setup
Before pointing Crawlee at a real target, confirm the proxy itself works:
curl -x http://username:password@p.webshare.io:80 https://ipinfo.io
If that returns an IP that isn't your own, your credentials are good — any connection issue after that is in the Crawlee config, not the proxy.
-1743061344.png)
