Website Crawler API Documentation

What is Website Crawler API?

Crawling is a resource intensive and difficult task. Website Crawler handles this with breeze. You might want to first know the capability of Websitecrawler.org. Our API enables developers to harness the capability of WebsiteCrawler.org through 3 simple endpoints which we have discussed below. Before your program starts making requests, you'll have to generate an API key.

Getting started

Register a new account or sign in to WebsiteCrawler with your Google Account. Navigate to the setttings page. Find the generate API key button and click it. That's it. Your API key will be generated instantly.

The 3 endpoints

WC provides 3 endpoints with which you can run the crawler, get the last URL processed by WC and get a JSON Array of the scraped data. Crawling is a difficult task. The base URL is https://www.websitecrawler.org/api. WC handles this efficiently. To run its crawler, simple call the follow endpoints

Endpoint 1: GET /crawl/start

Here are the required paramters:

• URL: A non-redirecting main URL (domain) of the website. For example https://www.websitecrawler.org
• Limit: Number of links WC should visit and crawl
• Key: The API key you have generated

Sample request

https://www.websitecrawler.org/api/crawl/start?url=MY_URL&limit=MY_LIMIT&key=YOUR_API_KEY    //replace MY_URL,YOUR_API_KEY,MY_LIMIT
                    

Sample response of the above request

                        
{
  "status": "Crawling"
}
                    

Endpoint 2: GET /crawl/currentURL

These are the required parameters:

URL: A non-redirecting main URL (domain) of the website. For example https://www.websitecrawler.org

Key: Your API key

Sample request

https://www.websitecrawler.org/api/crawl/start?currentURL=MY_URL&key=YOUR_API_KEY    //replace MY_URL,YOUR_API_KEY
                    

Sample response of the above request

                        
{
  "currentURL": "https://example.com"
}
                    

Endpoint 3: GET /crawl/cwdata

These parameters are required

URL: A non-redirecting main URL (domain) of the website. For example https://www.websitecrawler.org

Key: The API key you have generated

Sample request

https://www.websitecrawler.org/api/crawl/start?url=MY_URL&key=YOUR_API_KEY    //replace MY_URL,YOUR_API_KEY
                    

Sample response of the above request

[
  {
    "tt": "",
    "np": "12",
    "h1": "Pramod",
    "nw": "540",
    "h2": "",
    "ul___li___a___span": "",
    "h4": "",
    "h5": "",
    "atb": "",
    "sc": "200",
    "ELSC": "",
    "md": ""
 }
]