If you have been using our platform for a while, you may know that you can set the crawler type while creating a new project or via settings page. WebsiteCrawler supports two crawl modes – HTML and JavaScript. What is the difference between the two? Which one is ideal for your project? Here’s a brief difference between the two modes of crawling:
HTML
HTML crawler doesn’t does not process JavaScript generated content. It fetches the page and ignores everything generated/created by JavaScript. Because JS is not rendered, this crawler is much faster than the Chrome one. Use this mode if you’re interested only in basic information on the page or non JS data. Will the page data be missed if the crawling is fast? No, we’ve set a timeout for each page. WebsiteCrawler waits for the page data. Hence, the page will be processed only after WebsiteCrawler gets its source code. In case the page was not reachable, it will be recrawled automatically after a few seconds as our both crawlers are incorporated with the “retry” feature.
Chrome
If you choose the “Chrome” crawler, WebsiteCrawler makes a real Chrome browser render each page it has found on your site as if a real user is browsing the page. The following features are dependent on the Chrome:
Screenshots: WebsiteCrawler won’t take screenshots of pages. You won’t know how your website pages appear on the mobile, desktop, or table devices.
Console Errors: This feature relies on the Chrome browser. It lets you see complex errors identified by the Chrome browser.
CSS Usage and JS Usage: This feature enables users to find unused CSS and JavaScript code on your pages. Unless Chrome has rendered the page, it is impossible to know what percentage of JS code is unused on your page.
Render Blocking resources: Render blocking resources are difficult to detect manually. Depending on their blocking time, they may slow down the loading of pages.
Accessibility report: This report shows which pages on your site are non WCAG compliant by displaying issues and the element responsible for the issue.
Render vs Response: WebsiteCrawler compares the raw HTML with the Chrome rendered HTML. It shows which element was injected when the page was rendered by Chrome. This report won’t be available.
Accurate loading time: The page loading time calculated using the data obtained from the HTML crawler doesn’t take the JS execution into consideration. The loading time calculated while the page is being rendered in Chrome is much more reliable.
The Chrome crawler is available only to the users of Growth, Scale, Enterprise, and Starter plan. The users of these plans have access to the HTML crawler as well.
Leave a Reply