Main Website
Scraping
Updated on
July 21, 2026

How to Scrape Amazon (Without Getting Blocked)

TL;DR

  • Learn how to scrape Amazon product pages, search results, and reviews using Python, with complete code you can paste and run.
  • Understand what triggers Amazon’s block scrapers, and the three layers needed to get past IP bans, CAPTCHAs, and 503 errors.
  • Use Webshare rotating residential proxies to maintain access at scale. 
  • Follow end-to-end code requests with BeautifulSoup, Playwright, and Scrapy approaches.

Why Amazon Is Hard to Scrape (and Why People Try Anyway)

If you have tried scraping Amazon and hit a wall, you are not alone. Amazon is one of the most aggressively protected sites on the web, and a plain Python script that works fine on smaller targets can start failing after just a handful of requests.

Still, people keep attempting Amazon scraping because the data is genuinely valuable. It offers more than 350 million products, along with live prices, ratings, availability, reviews, and other product information. That data supports price monitoring, competitor intelligence, market research, and AI training datasets. However, there’s no single unrestricted public API that provides all of the data a scraper might need.

So you try to scrape the ecommerce platform, and encounter 503 errors, CAPTCHAs instead of the product, and honeypot traps. Meanwhile, the pricing data you need is missing from the raw HTML because it's dynamically loaded through JavaScript.

This guide explains how to scrape Amazon in 2026 (with copyable code), what tools you need, and what actually holds up in practice.

What Data Can You Scrape from Amazon?

Before writing any code, it helps to know which page you are targeting, because each Amazon page type holds different data and may require different selectors. 

Most scraping projects fall into four page types, and the table below maps each one to the fields worth extracting and the selectors commonly used to find them. 

Knowing this upfront lets you scrape Amazon product data, search results, and reviews without guessing at structure mid-run.

Page Type Data to Extract Key CSS Selectors / Notes
Product detail page (/dp/ASIN) Title, price, rating, review count, availability, ASIN, images, bullet points, description #productTitle, .a-price .a-offscreen, #acrCustomerReviewText, #availability, #imgTagWrapperId.

The price selector varies by product type.
Search results page (/s?k=query) Product titles, ASINs, prices, ratings, sponsored flags, page count [data-component-type="s-search-result"], [data-asin], .a-size-base-plus. Paginate with the &page=N param.
Reviews page (/product-reviews/ASIN) Review text, star rating, verified badge, date, helpful votes [data-hook="review"], [data-hook="review-body"], [data-hook="review-star-rating"], [data-hook="review-date"]. Paginate with the pageNumber parameter.
Seller page (/sp?seller=ID) Seller name, rating, feedback count, response rate #sellerName, #feedback-summary-table.

Often requires JavaScript rendering (Approach 3).

These page types also paginate differently. If you scrape Amazon search results, they use the page parameter, while to scrape Amazon reviews, you use pageNumber instead. Mixing up the two is a common early mistake. Amazon’s HTML can and has changed in the past, so it's important to validate that the selectors you’re using are still valid.

Why Amazon Blocks Scrapers — and How to Get Around It

Amazon runs one of the most layered anti-bot stacks of any consumer site, and understanding what it checks tells you exactly what your scraper has to get right. 

Here’s what trips up most scripts:

  • IP rate-limiting. Too many requests from one IP trigger a block within seconds. This is the single fastest way to get blocked, and it is why one machine hammering Amazon from a single address never lasts.
  • TLS fingerprinting. Amazon inspects the TLS handshake signature of each connection. Python’s requests library has a known fingerprint that does not match a real browser, so requests can be flagged as automated before a single byte of HTML is returned.
  • CAPTCHA challenges. These are triggered when your behavioral pattern deviates from human browsing. For example, identical timing between requests or a missing referer chain.
  • Honeypot traps. Amazon plants invisible links in the HTML that normal users would never interact with. A crawler that follows every link walks straight into them and flags its own IP.
  • User-agent detection. Requests carrying Python’s default user-agent string are blocked immediately, so a realistic browser user-agent is table stakes, not an optimization.
  • JavaScript rendering. Prices, availability, and some review data are injected into the DOM after the initial HTML loads. If you only read the raw response body, those fields come back empty, no matter how clean your request looks.

Another factor that can affect web scrapers is that Amazon changed its HTML in 2025. 

Legacy selectors like #priceblock_ourprice and #priceblock_dealprice  may no longer work, so newer selectors such as  .a-price .a-offscreen for prices and #productTitle for product titles are generally more useful. However, because Amazon periodically A/B tests its markup, wrap your selectors in error handling so your scraper can handle changes without failing

Why not use Amazon’s API?

Amazon’s APIs aren’t the best route for scraping because they weren’t built for data collection.

For many developers, the Product Advertising API was the go-to option, but Amazon deprecated it in May 2026 and replaced it with the Creators API. 

The Creators API, like its predecessor, is gated behind Amazon’s Associates program. This means you must meet the program’s requirements for the marketplace you want to access. This includes generating at least 10 qualified sales in the past 30 days to access the PA API. If you don’t meet that target, your API access can be paused until your sales recover.

Reviews aren’t included either. The API only gives you a star rating and a count, not the actual review text.

Also, access is gated by your affiliate performance rather than your project's needs. Under the legacy PA-API, this meant starting at one request per second (capped at 8,640/day) and earning one more request per second for every $4,320 in shipped affiliate orders, up to 10/sec. As of the Creators API migration (PA-API fully retired May 15, 2026), the baseline requirement shifted to a flat 10 qualifying sales in the trailing 30 days just to unlock API access at all — with rate limits scaling from there. Either way, the ceiling is tied to your sales performance, not your scraping needs.

If you’re scraping Amazon for price monitoring or review data, you have no audience to sell to, so you’d have to build an entirely unrelated affiliate business first just to unlock access to the work you came to do.

As we’ll learn in this guide and see how to apply it in practice, the fix involves realistic headers and request timing, Webshare rotating residential proxies, and a Playwright headless browser for JavaScript content.

Setup: Prerequisites and Installation

You’ll need Python 3.9 or newer. Check your version by running the command below:

python --version

Then, we’ll need the following libraries, so install them:

pip install requests beautifulsoup4 lxml playwright brotli
playwright install chromium

Next, sign up for a free Webshare account and copy your rotating residential proxy endpoint from the dashboard. The endpoint routes every request through an 80M+ IP pool and rotates you to a fresh IP on each request.

Paste it into your script like this:

http://username-rotate:password@p.webshare.io:80

To pull IPs from a specific country, append the country code to the username, for example, username-rotate-us for the United States. This keeps your Amazon data consistent because product prices, availability, shipping options, and search results can vary between marketplaces and regions.

Webshare’s free plan includes 10 proxies with no credit card required, which is enough to wire up and test these scripts before you scale. For sustained Amazon scraping in production, the rotating residential plan is what you want, since its IP pool is what keeps you unblocked at volume.

Approach 1: requests + BeautifulSoup + Webshare Proxies (Recommended Starting Point)

This is the primary code path for scraping Amazon product data, and where most Amazon scraper builds start. It pairs requests with Beautiful Soup, the combination that nearly every Amazon web scraper in Python begins with. 

If you’re new to scraping Amazon in Python, start here before reaching for anything heavier.

Below are the steps involved.

Step 1: Why a bare requests call fails on Amazon

A plain request is the fastest way to see the problem. Send a plain request to a product page with no headers and no proxy, and check the status code.

There are two possible responses you’ll get:

  1. A 503 Service Unavailable, which means Amazon rate-limited the request outright.
  2. A 200 OK whose body is actually a CAPTCHA challenge, not the product page.

Step 2: Add realistic headers (User-Agent, Accept-Language, Accept-Encoding, Referer)

One way to prevent requests from flagging is to make them look like they came from a real browser instead of scripts.

Real browsers send a set of headers that are absent from Python’s default request. These include:

  • User-Agent: Identifies the request as a real Chrome build instead of Python.
  • Accept and Accept-Language: Tells Amazon what content and locale you expect, the way a browser would.
  • Accept-Encoding: Tells Amazon you accept compressed responses (gzip, deflate, or brotli), like any real browser.
  • Referer: Makes the visit look like it came from within Amazon.

With these headers, simple, low-volume scrapes will go through. However, Amazon’s blockers will still be triggered at high-volume workloads because headers don’t solve the IP.

Enough requests from one address, and the platform’s rate limiter will kick in, meaning you still need a rotating proxy like what Webshare offers.

Step 3: Add Webshare rotating proxy

Routing every request through the Webshare rotating endpoint gives each one a different residential IP, so no single address carries enough traffic to trip the rate limiter. 

This way, the endpoint rotates automatically, and the proxy URL stays http:// even for HTTPS targets, since that refers to how you reach the proxy, not the site you are scraping.

Step 4: Parse with BeautifulSoup

A request to a product page returns the full HTML of that page as a single string. Pass that string to BeautifulSoup and let a parse_product helper pull out the title, price, rating, review count, and availability by their CSS selector.

Consider these factors  regarding the selectors:

  • price and rating break most often. If they return None, that means Amazon has changed its markup, and you’ll need to update them.
  • The star rating is inside a title attribute ("4.5 out of 5 stars"), not in the element’s text. This means it needs its own line to extract the attribute and grab the number rather than the text() helper that the other fields use.

Step 5: Build a function to scrape multiple ASINs with random delay between requests

In real workflows, you’re rarely fetching one product. Instead, you’re pulling hundreds or thousands. To gather them at once, wrap the request and parse in a function, then loop your ASIN (Amazon Standard Identification Number) list.

To keep this stable at volume, you need two things:

  1. Record failed requests instead of dropping them. Anything that isn't a clean 200 returns an error row, so you can retry it later rather than losing it in the run.
  2. Break up the timing. The random delay scatters the mechanical, evenly spaced request pattern that behavioral detection flags.

Step 6: Export to JSON / CSV

Finally, the results are written out to both JSON, which keeps every row including errors, and CSV, which gives you a flat file for a spreadsheet:

Here's the complete code:

import requests
import time
import random
import json
import csv
from bs4 import BeautifulSoup

# Step 2: headers that make the request look like a real Chrome session
headers = {
    "User-Agent": (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
        "(KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
    ),
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.9",
    "Accept-Encoding": "gzip, deflate, br",
    "Referer": "https://www.amazon.com/",
}

# Step 3: Webshare rotating residential endpoint, a fresh IP per request
proxies = {
    "http": "http://username-rotate:password@p.webshare.io:80",
    "https": "http://username-rotate:password@p.webshare.io:80",
}


# Step 4: pull the product fields out of the HTML
def parse_product(html, asin):
    soup = BeautifulSoup(html, "lxml")

    def text(selector):                    # stripped text, or None if the selector is missing
        el = soup.select_one(selector)
        return el.get_text(strip=True) if el else None

    # Rating lives in the #acrPopover title attribute, e.g. "4.5 out of 5 stars"
    rating_el = soup.select_one("#acrPopover")
    rating = rating_el["title"].split()[0] if rating_el and rating_el.has_attr("title") else None

    return {
        "asin": asin,
        "title": text("#productTitle"),
        "price": text(".a-price .a-offscreen"),      # fragile: varies by product type, changes often
        "rating": rating,
        "review_count": text("#acrCustomerReviewText"),
        "availability": text("#availability"),
    }


# Steps 1-3: request a product page through the headers and rotating proxy
def scrape_product(asin):
    url = f"https://www.amazon.com/dp/{asin}"
    response = requests.get(url, headers=headers, proxies=proxies, timeout=30)
    if response.status_code != 200:        # record anything that is not a clean 200
        return {"asin": asin, "error": response.status_code}
    return parse_product(response.text, asin)


# Step 5: scrape multiple ASINs with a random delay between requests
asins = ["B0CX23V2ZK", "B08N5WRWNW", "B09B8V1LZ3"]
results = []
for asin in asins:
    results.append(scrape_product(asin))
    time.sleep(random.uniform(2, 5))       # human-like gap between requests

# Step 6: export results to JSON (everything) and CSV (successful rows only)
with open("products.json", "w", encoding="utf-8") as f:
    json.dump(results, f, indent=2, ensure_ascii=False)

fieldnames = ["asin", "title", "price", "rating", "review_count", "availability"]
with open("products.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.DictWriter(f, fieldnames=fieldnames, extrasaction="ignore")
    writer.writeheader()
    writer.writerows(row for row in results if "error" not in row)

Get your free Webshare proxy credentials → 10 proxies, no credit card, works immediately with the code above.

Approach 2: Scraping Amazon Search Results Pages

If you don’t have a list of ASINs, you can build one by scraping Amazon search results. Each search results page follows a predictable web address pattern, https://www.amazon.com/s?k=QUERY&page=N, where QUERY is your keyword, and N is the page number.

Every product in the results carries its ASIN in a data-asin attribute, so a single search page hands you a batch of them at once.

From each result, the scraper pulls:

  • The ASIN, straight from the data-asin attribute.
  • The title and price, with a fallback title selector since Amazon uses more than one layout.
  • The rating, read from the aria-label text.
  • A sponsored flag, so you can filter paid placements out of your organic results.

You’ll need to paginate, because a single search can return hundreds of products spread across many pages. First, read the total result count off the first page, divide it by the results per page to get your page total, then loop through with &page=N.

Route each request through your Webshare rotating proxy, since a run of sequential requests to the same search from one IP is exactly the pattern the rate limiter watches for.

Here's the complete code:

import requests
import time
import random
import re
import math
from urllib.parse import quote_plus
from bs4 import BeautifulSoup

# Reuses the headers and proxies dict from Approach 1

def scrape_search(query, page=1):
    url = f"https://www.amazon.com/s?k={quote_plus(query)}&page={page}"
    response = requests.get(url, headers=headers, proxies=proxies, timeout=30)
    soup = BeautifulSoup(response.text, "lxml")

    results = []
    for item in soup.select('[data-component-type="s-search-result"]'):   # one block per product
        asin = item.get("data-asin")
        if not asin:                          # skip banners and non-product rows
            continue

        title_el = item.select_one("h2 a span") or item.select_one(".a-size-base-plus")
        price_el = item.select_one(".a-price .a-offscreen")
        rating_el = item.select_one("[aria-label*='out of 5 stars']")
        sponsored = item.select_one(".puis-sponsored-label-text") is not None  # sponsored flag

        results.append({
            "asin": asin,
            "title": title_el.get_text(strip=True) if title_el else None,
            "price": price_el.get_text(strip=True) if price_el else None,
            "rating": rating_el["aria-label"].split()[0] if rating_el else None,
            "sponsored": sponsored,
        })
    return results


def get_page_count(query, per_page=48):
    url = f"https://www.amazon.com/s?k={quote_plus(query)}"
    response = requests.get(url, headers=headers, proxies=proxies, timeout=30)
    soup = BeautifulSoup(response.text, "lxml")

    # Read "1-48 of X results" from the results bar, then divide to get the page total
    # Fragile: Amazon changes this bar's wording and class, so guard it and cap the result
    info = soup.select_one('[data-component-type="s-result-info-bar"]')
    match = re.search(r"of(?:\s+over)?\s+([\d,]+)\s+results", info.get_text()) if info else None
    if not match:
        return 1
    total = int(match.group(1).replace(",", ""))
    return min(math.ceil(total / per_page), 20)   # cap at 20 pages; Amazon rarely serves more


# Detect the page count, then loop through each page with &page=N
query = "wireless earbuds"
pages = get_page_count(query)
all_products = []
for page in range(1, pages + 1):
    all_products.extend(scrape_search(query, page=page))
    time.sleep(random.uniform(2, 5))    # pause between pages

print(f"Collected {len(all_products)} products across {pages} pages")

Approach 3: Playwright + Webshare for JavaScript-Rendered Content

requests works when the data you need is sitting in the raw HTML. However, scraping Amazon with Playwright becomes necessary when the content is loaded dynamically through JavaScript, which on Amazon covers:

  • Prices that render after the initial page load.
  • Live review counts and dynamic availability.
  • Sections like “Frequently bought together.”

Playwright opens an actual Chromium browser in the background, so the JavaScript executes just like it would for a real visitor.

With Playwright, you can:

  • Route your Webshare proxy through Playwright’s own proxy option.
  • Disable the automation flag with --disable-blink-features=AutomationControlled.
  • Set navigator.webdriver to false, the value a real browser reports during normal browsing.
  • Set a realistic user-agent and viewport so the browser looks like a normal visitor.

Here’s the complete code:

import asyncio
from playwright.async_api import async_playwright

async def scrape_product_js(asin):
    async with async_playwright() as p:
        browser = await p.chromium.launch(
            headless=True,
            proxy={                                 # Webshare rotating residential proxy
                "server": "http://p.webshare.io:80",
                "username": "username-rotate",
                "password": "password",
            },
            args=[
                "--no-sandbox",
                "--disable-blink-features=AutomationControlled",   # drop the automation fingerprint
            ],
        )
        context = await browser.new_context(
            user_agent=(
                "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
                "(KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
            ),
            viewport={"width": 1366, "height": 768},
        )
        # Report navigator.webdriver as false, the value a real browser shows
        await context.add_init_script(
            "Object.defineProperty(navigator, 'webdriver', {get: () => false})"
        )

        page = await context.new_page()
        await page.goto(f"https://www.amazon.com/dp/{asin}",
                        wait_until="domcontentloaded", timeout=60000)

        # Dismiss the cookie banner if it appears (only shows in some regions)
        try:
            await page.click("#sp-cc-accept", timeout=5000)
        except Exception:
            pass

        # Wait for the JS-rendered price to attach before reading anything
        await page.wait_for_selector(".a-price .a-offscreen", timeout=15000)

        data = {
            "asin": asin,
            "title": await page.locator("#productTitle").inner_text(),
            "price": await page.locator(".a-price .a-offscreen").first.inner_text(),
            "availability": await page.locator("#availability").inner_text(),
        }
        await browser.close()
        return data

print(asyncio.run(scrape_product_js("B0CX23V2ZK")))

You also need to choose between sticky sessions and per-request rotation.

Sticky sessions are ideal when you’re driving a browser, since a single page load fires dozens of sub-requests, and holding one IP across all of them keeps the page from looking like it came from many visitors at once. Webshare lets you hold that IP for a window you set, from 1 minute to 24 hours.

Per-request rotation is the better fit for the stateless requests approach, where each request stands on its own, and a fresh IP address each time spreads the load.

Approach 4: Scraping Amazon Reviews

Amazon keeps reviews on a separate page from the product listing. 

To scrape Amazon reviews, target amazon.com/product-reviews/ASIN?pageNumber=N. Navigate through by iterating pageNumber from 1 to the calculated max (total reviews ÷ 10).

Note that review pages are slightly less protected than product pages, but still need proxies at scale.

The code below fetches various review-related details.

import requests
import json
import math
import re
from bs4 import BeautifulSoup

# Reuses the headers and proxies dict from Approach 1

def get_review_pages(asin):
    url = f"https://www.amazon.com/product-reviews/{asin}?pageNumber=1"
    response = requests.get(url, headers=headers, proxies=proxies, timeout=30)
    soup = BeautifulSoup(response.text, "lxml")

    # Read the review count off the page, then divide by 10 for the page total
    # Fragile: Amazon changes this element, so guard it and cap the result
    info = soup.select_one('[data-hook="cr-filter-info-review-rating-count"]')
    match = re.search(r"([\d,]+)\s+with reviews", info.get_text()) if info else None
    if not match:
        return 1
    total = int(match.group(1).replace(",", ""))
    return min(math.ceil(total / 10), 50)   # cap at 50 pages

def scrape_reviews(asin):
    reviews = []
    for page in range(1, get_review_pages(asin) + 1):
        url = f"https://www.amazon.com/product-reviews/{asin}?pageNumber={page}"
        response = requests.get(url, headers=headers, proxies=proxies, timeout=30)
        soup = BeautifulSoup(response.text, "lxml")

        # Each review card carries its fields in data-hook attributes
        for review in soup.select('[data-hook="review"]'):
            reviewer = review.select_one(".a-profile-name")
            body = review.select_one('[data-hook="review-body"]')
            rating = review.select_one('[data-hook="review-star-rating"] .a-icon-alt')
            date = review.select_one('[data-hook="review-date"]')
            helpful = review.select_one('[data-hook="helpful-vote-statement"]')
            verified = review.select_one('[data-hook="avp-badge"]') is not None

            reviews.append({
                "asin": asin,
                "reviewer_name": reviewer.get_text(strip=True) if reviewer else None,
                "text": body.get_text(strip=True) if body else None,
                "rating": rating.get_text(strip=True).split()[0] if rating else None,
                "date": date.get_text(strip=True) if date else None,
                "verified_purchase": verified,
                "helpful_votes": helpful.get_text(strip=True) if helpful else None,
            })
    return reviews

# Scrape every review page and save the results to JSON
data = scrape_reviews("B0CX23V2ZK")
with open("reviews.json", "w", encoding="utf-8") as f:
    json.dump(data, f, indent=2, ensure_ascii=False)

print(f"Collected {len(data)} reviews")

Approach 5: Scrapy spider + Webshare proxy middleware

Once you’re pulling tens of thousands of pages, looping requests by hand gets slow and fragile. Scrapy handles that scale for you, with built-in concurrency and retries.

The only Scrapy-specific part is wiring in the proxy. One way is using a middleware that attaches your Webshare endpoint to every request. You can switch this on in a file (settings.py in the code below).

Here’s the whole thing, middleware and spider together:

# settings.py
# Turn on the proxy middleware (lower number = runs earlier)
DOWNLOADER_MIDDLEWARES = {
    "amazon_scraper.middlewares.WebshareProxyMiddleware": 350,
}

DOWNLOAD_DELAY = 3                 # wait a few seconds between requests
RANDOMIZE_DOWNLOAD_DELAY = True    # vary that wait so it doesn't look robotic


# middlewares.py
class WebshareProxyMiddleware:
    def process_request(self, request, spider):
        # Send every request out through your Webshare endpoint
        request.meta["proxy"] = "http://username-rotate:password@p.webshare.io:80"


# spiders/products.py
import scrapy

class ProductsSpider(scrapy.Spider):
    name = "products"
    asins = ["B0CX23V2ZK", "B08N5WRWNW"]

    def start_requests(self):
        for asin in self.asins:
            yield scrapy.Request(f"https://www.amazon.com/dp/{asin}",
                                 cb_kwargs={"asin": asin})

    def parse(self, response, asin):
        yield {
            "asin": asin,
            "title": response.css("#productTitle::text").get(default="").strip(),
            "price": response.css(".a-price .a-offscreen::text").get(),
            "review_count": response.css("#acrCustomerReviewText::text").get(),
        }

If you’d rather not write the middleware, the scrapy-rotating-proxies package does the same job: drop your endpoints into a ROTATING_PROXY_LIST and the package handles the rotation for you.

If you prefer SOCKS5 over HTTP proxies, Webshare supports both protocols at no additional cost, so you can use the same proxy pool with Scrapy without changing your proxy plan.

Approach Comparison and When to Use Each

Each approach trades complexity for block resistance. Use the lightest one that clears your target:

Approach Complexity Block Risk Best For Webshare Integration
requests + BeautifulSoup (no proxy) Low Very high Learning only Not recommended at scale; shown first so you see why it fails
requests + BeautifulSoup + Webshare rotating proxies Low–Medium Medium Product page scraping at moderate volume Primary path. Webshare HTTP endpoint + rotating residential
Playwright + Webshare residential proxies Medium Low JS-rendered pages, reviews, dynamic pricing Secondary path. Webshare proxy config in the Playwright launch options
requests + BeautifulSoup + Webshare rotating proxies (reviews) Low–Medium Medium Pulling review text, ratings, and reviewer details at volume Review path. Webshare rotating residential proxies once you go past a few requests
Scrapy spider + Webshare proxy middleware High Low–Medium Large-scale bulk product/category scraping Advanced path. Webshare in a Scrapy downloader middleware

Common Amazon Scraping Errors and How to Fix Them

There are some errors you’ll likely experience regardless of how well-built your Amazon scraper is: 

  • 503 Service Unavailable: Amazon has rate-limited your IP. Switch to rotating residential proxies and add delays between requests so no single address stands out due to a high request volume.
  • CAPTCHA page instead of the product: You have been flagged as a bot. The fix is residential proxies plus realistic headers together. Datacenter IPs trigger CAPTCHAs almost immediately on Amazon, which is why the datacenter product is the wrong choice here.
  • Empty price field: Price is loaded via JavaScript and is not in the raw HTML. Move that scrape to Playwright, which renders the page before you read it.
  • Selector returning None: Amazon often A/B tests its HTML. Wrap every selector in a try or a null check, log the failures, and update the selectors when a field consistently returns empty values.
  • Redirect to amazon.com/errors/validateCaptcha: A honeypot hit or too many requests from one IP. Rotate the proxy, add a random delay, and confirm your user-agent string is a real browser.
  • Inconsistent pricing across requests: Amazon shows different prices by region and IP. 
  • Use Webshare’s geo-targeted proxies to lock your request to a single country and obtain consistent results for a specific market.

Scaling Your Amazon Scraper

Here are some best practices that can help keep your data extraction runs stable at scale:

  • Rate limiting strategy: Keep request rates to roughly 1 to 2 requests per IP per minute for continuous scraping, and only burst to 5 to 10 when your pool is large enough to distribute the traffic. 
  • Webshare pool depth: Webshare’s rotating residential pool spans more than 80 million IPs, which, for any reasonable request volume, means effectively unlimited rotation, so the constraint is your pacing, not the number of available addresses. 
  • Parallel scraping: Use asyncio with Playwright or concurrent.futures with requests, so each worker has its own proxy session, and so they do not collide on the same IP. 
  • Retry logic: Wrap every request in retry logic that catches a 503 or 429, rotates the proxy, waits with exponential backoff, and retries up to three times before skipping the item.
  • ASIN-first approach: Build your ASIN list up front from search scraping or an external source, then scrape product pages in bulk. This is far more efficient than navigating from search each time.
  • Storage: For storage, JSON is fine for small datasets, but move to PostgreSQL or BigQuery once you are running a production pipeline that tracks how prices change over time.

Legal and Ethical Considerations

Amazon’s Terms of Service restrict automated access, so scraping is done at your own risk, particularly for commercial use.

On the legal side, under U.S. case law, the Ninth Circuit's ruling in hiQ Labs v. LinkedIn found that scraping publicly accessible data is unlikely to violate the Computer Fraud and Abuse Act (CFAA) - a preliminary injunction ruling, not a final judgment, since the parties settled privately in 2022 before the case concluded. Separately, a district court found hiQ still breached LinkedIn's Terms of Service through its scraping, a contract claim independent of the CFAA question. In short: public-data scraping is on relatively solid CFAA footing, but ToS violations remain a distinct legal risk.

One best practice to follow is respecting Amazon's robots.txt; don’t use scraped data to deceive customers or violate Amazon’s brand guidelines. Also, for commercial use cases, it’s best to speak with a lawyer.