An answer engine cannot cite a page that its bot cannot read. The Crawlability page shows you, in one view, if the AI bots can crawl your website. Genezio reads the robots.txt file and the sitemaps of each website of your brand. You do not have to upload a file or to ask your infrastructure team for data.

​
Why crawlability is important

A bot that your site blocks does not read your pages. An answer engine that does not read your pages cannot cite them. Often, a block is an old rule that nobody remembers. A firewall setting can also refuse the AI bots with no one noticing. The page shows these problems before they cost you citations.

​
Open the page

Open Crawlability in the report of your brand. When your brand has more than one website, the page has one tab for each website. Genezio makes the first check when you add a new brand. After this, it checks each website every week. To check at once, select Check again. When the brand has no check yet, select Run first check.

​
Read the verdict

Each website gets one verdict:

​
See the status of each bot

The page lists the AI bots of the answer engines. Each bot has a type:
  • Training: the bot collects pages to train the AI models.
  • Search: the bot indexes pages, so that the AI search engines can find and cite them.
  • User query: the bot opens a page when a person asks an AI assistant a question about it.
Each bot has one status:
A blocked training bot is not always a mistake. Some brands choose to keep their content out of the training of the models. Make sure that this is your decision, and not an old rule.
You can filter the list by bot type, by status, and by platform. Select View robots.txt to read the file as Genezio read it.

​
Check the sitemap

A sitemap gives the bots a list of your pages. The page shows each sitemap that Genezio found, and its state: Valid, Empty, Not a sitemap, or Not found. A sitemap that is not valid does not help the bots to find your pages.

​
See how the result changed

Select History to open the older checks of the website. Each check shows its date and its reason: the first check of the brand, a weekly check, or a manual check. Open an older check to see the website as it was at that date. Thus, you can see when a change in robots.txt began to block a bot.