Web Scraper API
Get to know how Web Scraper API works and integrate it into your app.
Last updated
Get to know how Web Scraper API works and integrate it into your app.
Last updated
https://scrape.proxy-seller.com
url (string, required)
Destination url to retrieve (url-encoded)
{"url":"google.com"}
default=null
api_key (string, required)
Web Scraper API key
{"api_key":"0de32912321"}
default=null
country_code (str, optional)
Proxy country code (geolocation)
{"country_code":"fr"}
default=null, options - us, gb, de, fr, cn, jp. View all codes
render_js (bool, optional)
Render JS on page
{"render_js":"true"}
default=false
return_json (bool, optional)
Return HTML if flagged "false"
{"return_json": "false"}
default=true
headers (JSON, optional)
Custom headers
{ "headers": {"user-agent": "Example user agent", "accept": "text/html,*/*"} }
default our headers
language (str, optional)
Language
{"language":"en-US"}
selector (str, optional)
CSS Selector
{"selector":".className"}, {"selector":"#idName"}
200 (Success)
Request successful. Returns JSON with headers and html fields
{"headers":{}, 'html':""}
401 (Unauthorized)
API key is missing or wrong
{'error':'API key is missing or wrong'}
422 (Unprocessable Entity)
Error in query parameters
{'error':'Wrong query'}
504 (Timeout)
Site returned timeout after 3 attempts to reach it
{'error':'Timeout'}
API key:
If you want to define the geolocation for your request, you may set the country_code (string) parameter with one country code at the creation of the request.
Example: 'us', 'gb', 'fr', 'de', 'jp', 'cn', 'ru'
Supported country codes can be found in the collapsible table below:
You're also free to pass your own headers set to the request
If you want to define the language in your request, you may set the language (string) parameter with one language tag at the creation of the request.
Language codes can be found in the collapsible table below:
Using headers:
If you’re trying to set your own headers - please make sure to pass all the necessary values including user agent in the correct order. In this case your target server will receive your headers without any modifications from our side. Setting incorrect headers might cause unexpected page results. Please use this only if you know what you are doing.
Async API allows to put multiple time-consuming requests to the queue and receive the results as soon as they are getting ready
POST https://scrape.proxy-seller.com/job
200 (Success)
Request successful. Returns JSON with headers and html fields
{"id":'result_id'}
401 (Unauthorized)
API key is missing or wrong
{'error':'API key is missing or wrong'}
422 (Unprocessable Entity)
Error in query parameters
{'error':'Wrong query'}
504 (Timeout)
Site returned timeout after 3 attempts to reach it
{'error':'Timeout'}
POST https://scrape.proxy-seller.com/job/<job_id>
job_id (string, required)
Job ID
https://scrape.proxy-seller.com/job/0de32912321
default=null
api_key (string, required)
Web Scraper API key
{"api_key":"0de32912321"}
default=null
200 (Success)
Request successful. Returns JSON with headers and html fields
{ <br/> "status":"running", <br/> "statusUrl":"https://scrape.proxy-seller.com/job/0962a8a0-5f1a-4e14-bf8c-5efcc18f1953", <br/> "url":"http://httpbin.org/ip" <br/> }
401 (Unauthorized)
API key is missing or wrong
{'error':'API key is missing or wrong'}
422 (Unprocessable Entity)
Error in query parameters
{'error':'Wrong query'}
504 (Timeout)
Site returned timeout after 3 attempts to reach it
{'error':'Timeout'}
Your plan determines how many credits you can use. Each request you make costs some credits. The number of credits you use varies based on the domain and parameters of your request. Geotargeting is included in these credit costs.
We have built special scrapers for some sites. These scrapers will run when you scrape those domains, changing the credit cost. Scraping other domains costs 1 credit (without additional parameters).
credit cost
1
10
10
with render_js
10
20
20
caution
Normal - any other website if no additional parameters are added;
SERP - Google;
Ecommerce - Amazon, Booking;
LinkedIn - the cost will be 130 credits per request.
List will be updated as functionality is added.
These parameters provide you with additional features for the scraping.
{"render_js":"true"} – requests cost 10 credits.
default=en-US, options - en-US, en-CA, es-ES, fr-CA.
default=null, is only used with render_js. Timeout <= 60 sec.
'api_key': 'API_KEY', where API_KEY is api_key from
Payload parameters you can find in table.