Website Crawler Blog

Tag: Logs

  • Get WebsiteCrawler alerts to your Slack channel

    WebsiteCrawler integration for Slack enables users to receive messages to the Slack channel of the users choice leveraging Slack’s powerful feature called channels.

    Why use integration for Slack? Although you can group mails, web applications don’t send messages via different email addresses. For example, we send most emails through our help@websitecrawler.org address. We don’t send SSL expiry alerts through some different email alias. A user may get emails from many different sources. Hence, the mails can get lost in the crowd. Slack allows you to create dedicated channels. The WebsiteCrawler integration for Slack lets users receive important messages to these dedicated channels. Thus, if you create a channel dedicated to our alerts, you can easily find all important messages sent by WebsiteCrawler at the same place.

    Types of alerts WebsiteCrawler will send:

    Downtime alerts: Our platform monitors uptime of sites. Depending on your chosen subscription plan, our platform run uptime checks every 5, 3, and 1 minute. If your site is unreachable after a check, WebsiteCrawler will immediately send an email to your registered email address and a message to your Slack channel.

    SSL Expiry alerts: If the time left for the SSL certificate for the site to expire is less than a week, WebsiteCrawler will send Slack message to the channel along with the email.

    Crawl completion alerts: If you run a crawl, you don’t have to keep your eyes glued on the list of processed URLs displayed by WebsiteCrawler. Once the crawl job is complete, you will immediately get an alert message in your chosen Slack channel.

    Integrating Slack with WebsiteCrawler

    Our platform enables you to register a new account and sign in with your login credentials and sign in with Google. Log in to your WebsiteCrawler account and visit the settings page. Click the “API Integration” menu and find the button “Connect to Slack”.

    Click this button and complete the OAuth authentication. Once the OAuth authentication is successful, you’ll see a button “See channels”. Click this button. WebsiteCrawler will now show a list of channels available in your workspace.

    Select the channel where you’d live to receive our messages and click the “Save” button. The message “Connected to Slack” will convert to “Connected to Slack and sending messages to channel with id XXXXXX” where XXXXXX is your chosen Slack channel’s id.

    How will the messages look?

    The sent messages are easy to interpret. Slack will log every message sent by our platform in your channel.

    To know what data we collect and how long we retain it, please refer our Privacy Policy. You can also visit our Terms of Service page to read the TOS.

    If you’re facing any difficulties while using this feature or want to know more about this feature, you can write an email to us at help@websitecrawler.org

  • Analyze Log Files online

    Analyze Log Files online

    Logfile generated by web-servers such as Nginx, Apache, etc contains information that the widely used Analytics tools i.e. Google Analytics won’t display. For example, Google Analytics won’t display the IP address of the bot or the user.

    Log files contain a timestamp, HTTP protocol, request type, URL, status code, IP address, etc. Analyzing the log file data manually can be time-consuming. Here’s when the Website Crawler tool comes into the picture.

    With the log file analyzer tool of WC, you can see what URLs search bots are crawling or the IP addresses of the users that have visited the website. You can also see the URLs that people/bots are visiting the most.

    The Website Crawler’s Log File Analyzer tool displays the following important information:

    • Links with HTTP status code 200, 404, etc.
    • The number of times bots have visited your site.
    • Number of URLs present in the log file, and more.

    How to use the log file analyzer tool?

    Click the “Choose File” option, and select the access log file on your PC. If you don’t have the file, get it from the server. Once you choose the file, click the “Process” button. You’ll now see some vital details, a table containing the log file data, and a textbox.

    Log File Analyzer’s filter

    You can filter data by entering a word in the textbox you’ll below the file upload option. For example, if you want to see the list of URLs crawled by Googlebot, enter “Googlebot” in the textbox that you’ll find below the file selector/upload option. Once you enter the word, you’ll see the filtered data.

    Although WC is capable of reading large files, it will process only 100 lines of log files uploaded by free account holders/unregistered users. If you’ve got a silver account, WC will process 2500 URLs.

    Note: Website Crawler won’t save the file data or the entire file on the server. The file will be discarded once you’ve exited the page/closed the tab in which you’ve opened the tool.