Tag: Sitemap

As you might know, Website Crawler – On Page SEO Checker is capable of extracting URLs from sitemap and processing them. From now on, you can now create XML sitemaps for any website with WC. For those who don’t know, a sitemap is a file that contains all links the search bots should crawl and index. If your site is well structured, search bots will be able to find the pages. If your site is poorly structured and doesn’t have a sitemap, search bots may not crawl and index some of the pages on your website.

To make sure that the important pages of your site are indexed, you should submit the URL of the sitemap file. Where to save the sitemap.xml file? Well, you can save the file to any folder of your website’s directory.

WebsiteCrawler sitemap generator

Once Website Crawler crawls your website, head over to the projects section, and click the “XML Sitemap” option.

Now, you’ll see a form with several fields. In the first textbox, enter the number of URLs you want the sitemap to have. If you don’t want to see URLs containing specific words or characters in the XML file, enter the word in the textbox 2.

URLs in sitemaps can have priority. URL with higher priority might be crawled/indexed first. With WC, you can create multiple sitemaps. Each sitemap can have URLs with different priorities.

Once the sitemap is generated, you’ll see a URL. Click this link to see the XML file. Once the browser opens the file, right-click on the file’s data and select the “Save As” option.

Sometimes, you may update a page/post on your site. To make sure that bots learn about the change and index the newer version of the page, you must either add the modified date or changefreq to the sitemap. As of now, WC doesn’t support dates. However, it can create sitemaps with changefreq option. If you set the URL changefreq to “weekly”, bots will crawl the URL on a weekly basis. Likewise, setting changefreq to “Always” will make the search bot visit the URL often.

A sitemap, as you may already know, is the most important part of a website. It contains a list of links of a website and helps search engines in crawling/indexing pages which it may not find it. Today, I have good news for the users of Website Crawler. WC can now crawl the links it finds in the sitemap file. Yes, that’s right. This feature was on our checklist and we have rolled it out today.

In case you’re wondering how to use the new XML crawl feature of Website Crawler, here’s a tutorial that you can follow.

How to make Websitecrawler crawl sitemap links?

Enter the direct link to the XML format sitemap file of your website in the large text box you see on the Website Crawler’s homepage. Don’t worry about the file’s size. WC can analyze sitemaps containing 25000+ URLs.

Once you enter the URL, enter the number of URLs you want the website crawler to crawl in the “Depth” textbox. Now, click the submit button.

Note: Free account of WC supports 550 URLs i.e. no matter how big the sitemap file is, only up to 550 URLs will be processed. Our Silver (paid) plan supports 2500 URLs.

When you hit this button, WC will start crawling your website URLs. To see the status i.e. current URL being crawled and the list of processed URLs, click the “Status” button.

Once WC finishes processing the sitemap links, you’ll see a new form that asks you to enter your email address. Enter your email ID and click the submit button. WC will now send a verification email to your inbox. Enter the 3 digit code in the textbox and hit the button “Submit”. WC will now create your websitecrawler.org account through which you can see the On-Page SEO reports of your website.

Conclusion: If you want Website Crawler to analyze a fixed set of URLs or the links of the sitemap files only, enter the link to the sitemap file instead of a website URL in the large text box that you’ll find on WC’s home page.