Web Scraper API
Get to know how Web Scraper API works and integrate it into your app.
Data Scraping and API Endpoint Integration
https://scrape.proxy-seller.com
Sync API
Sync API allows to receive immediate result using our Residential Proxy
Query Parameters
url (string, required)
Destination url to retrieve (url-encoded)
{"url":"google.com"}
default=null
api_key (string, required)
Web Scraper API key
{"api_key":"0de32912321"}
default=null
country_code (str, optional)
Proxy country code (geolocation)
{"country_code":"fr"}
default=null, options - us, gb, de, fr, cn, jp. View all codes
render_js (bool, optional)
Render JS on page
{"render_js":"true"}
default=false
return_json (bool, optional)
Return HTML if flagged "false"
{"return_json": "false"}
default=true
headers (JSON, optional)
Custom headers
{ "headers": {"user-agent": "Example user agent", "accept": "text/html,*/*"} }
default our headers
language (str, optional)
Language
{"language":"en-US"}
default=en-US, options - en-US, en-CA, es-ES, fr-CA. View all codes
selector (str, optional)
CSS Selector
{"selector":".className"}, {"selector":"#idName"}
default=null, is only used with render_js. Timeout <= 60 sec. CSS Selector Reference
Returns
200 (Success)
Request successful. Returns JSON with headers and html fields
{"headers":{}, 'html':""}
401 (Unauthorized)
API key is missing or wrong
{'error':'API key is missing or wrong'}
422 (Unprocessable Entity)
Error in query parameters
{'error':'Wrong query'}
504 (Timeout)
Site returned timeout after 3 attempts to reach it
{'error':'Timeout'}
API key:
'api_key': 'API_KEY', where API_KEY is api_key from your package
Country codes
If you want to define the geolocation for your request, you may set the country_code (string) parameter with one country code at the creation of the request.
Example: 'us', 'gb', 'fr', 'de', 'jp', 'cn', 'ru'
Supported country codes can be found in the collapsible table below:
You're also free to pass your own headers set to the request
Language codes
If you want to define the language in your request, you may set the language (string) parameter with one language tag at the creation of the request.
Language codes can be found in the collapsible table below:
Curl example
Curl (with headers)
Using headers:
If you’re trying to set your own headers - please make sure to pass all the necessary values including user agent in the correct order. In this case your target server will receive your headers without any modifications from our side. Setting incorrect headers might cause unexpected page results. Please use this only if you know what you are doing.
Async API
Async API allows to put multiple time-consuming requests to the queue and receive the results as soon as they are getting ready
POST https://scrape.proxy-seller.com/job
Payload parameters
Payload parameters you can find here in table.
Returns
200 (Success)
Request successful. Returns JSON with headers and html fields
{"id":'result_id'}
401 (Unauthorized)
API key is missing or wrong
{'error':'API key is missing or wrong'}
422 (Unprocessable Entity)
Error in query parameters
{'error':'Wrong query'}
504 (Timeout)
Site returned timeout after 3 attempts to reach it
{'error':'Timeout'}
Curl
Async API - receiving results
POST https://scrape.proxy-seller.com/job/<job_id>
Path Parameters
job_id (string, required)
Job ID
https://scrape.proxy-seller.com/job/0de32912321
default=null
Payload Parameters
api_key (string, required)
Web Scraper API key
{"api_key":"0de32912321"}
default=null
Returns
200 (Success)
Request successful. Returns JSON with headers and html fields
{ <br/> "status":"running", <br/> "statusUrl":"https://scrape.proxy-seller.com/job/0962a8a0-5f1a-4e14-bf8c-5efcc18f1953", <br/> "url":"http://httpbin.org/ip" <br/> }
401 (Unauthorized)
API key is missing or wrong
{'error':'API key is missing or wrong'}
422 (Unprocessable Entity)
Error in query parameters
{'error':'Wrong query'}
504 (Timeout)
Site returned timeout after 3 attempts to reach it
{'error':'Timeout'}
Curl
Credits and Requests
Your plan determines how many credits you can use. Each request you make costs some credits. The number of credits you use varies based on the domain and parameters of your request. Geotargeting is included in these credit costs.
Domains
We have built special scrapers for some sites. These scrapers will run when you scrape those domains, changing the credit cost. Scraping other domains costs 1 credit (without additional parameters).
credit cost
1
10
10
with render_js
10
20
20
caution
Normal - any other website if no additional parameters are added;
SERP - Google;
Ecommerce - Amazon, Booking;
LinkedIn - the cost will be 130 credits per request.
List will be updated as functionality is added.
Paid query parameters
These parameters provide you with additional features for the scraping.
{"render_js":"true"} – requests cost 10 credits.
Last updated