Pramod is the developer/founder of Website Crawler. He loves building web applications.
Logfile generated by web-servers such as Nginx, Apache, etc contains information that the widely used Analytics tools i.e. Google Analytics won’t display. For example, Google Analytics won’t display the IP address of the bot or the user. Log files contain a timestamp, HTTP protocol, request type, URL, status code, IP address, etc. Analyzing the log […]
As you might know, Website Crawler – On Page SEO Checker is capable of extracting URLs from sitemap and processing them. From now on, you can now create XML sitemaps for any website with WC. For those who don’t know, a sitemap is a file that contains all links the search bots should crawl and […]
“rel” is one of the attributes supported by hyperlinks. Before Google introduced the “sponsored” value for the “rel” attribute this month, nofollow and dofollow were the commonly used values. Links using the rel=”nofollow”/”dofollow” attributes are widely known as nofollow or dofollow URLs. DF links pass page rank to linked documents. Searchbots follow these URLs. NF […]
Thin content is one of the major reasons why websites suffer from Google Panda algorithm penalty. For those who are not aware, Google’s Panda algorithm has been designed to lower the rankings of websites that have On-Page SEO issues. Thin content is one of several On-Page search engine optimization issues. The Panda algorithm is now […]
When a page of a website is unreachable, the visitor will see an error message in the browser’s tab. These messages are reported because of the following reasons: The database server isn’t working: All nonstatic websites save data to the database. If the database server isn’t working, the page won’t won’t be able to get data […]
A sitemap, as you may already know, is the most important part of a website. It contains a list of links of a website and helps search engines in crawling/indexing pages which it may not find it. Today, I have good news for the users of Website Crawler. WC can now crawl the links it finds […]
Google dropping 1000s of pages from its index is a nightmare for bloggers, webmasters, developers, and online business owners. One of my sites has around 28000 pages. Google had indexed around 12000 pages of this site but in the last few months, it started dropping the pages from its index. If you’re following SEO news […]