Developers

Scrapy proxies.

A downloader middleware is enough. Put the gateway in settings or per-request meta. Do not share one sticky session across unrelated spiders.

settings.py

HttpProxyMiddleware is built in. Point it at the HTTP gateway.

# settings.py
DOWNLOADER_MIDDLEWARES = {
    "scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware": 750,
}
HTTPS_PROXY = "http://USER-country-US:PASS@GATEWAY:7777"
HTTP_PROXY = HTTPS_PROXY

Per-request country

Override proxy in Request.meta when a spider must hop countries.

yield scrapy.Request(
    url,
    meta={
        "proxy": "http://USER-country-FR:PASS@GATEWAY:7777",
    },
)

FAQ

Retry on 407 or 502?

407 is bad credentials. 502/503 from the gateway is often a dead exit — retry the request so rotation can pick another IP.

Budget vs Premium in Scrapy?

Same client code. Budget is the volume pool. Premium adds city/ASN and TCP/IP OS filter when the target is picky.