CDN log integration

The CDN log analysis uses your own server-side access logs to track AI crawler activity. It shows how answer engines really use your website:

• Which pages they fetch.
• When they fetch them.
• If these pages agree with the pages that they cite.

This analysis adds first-party evidence from your own infrastructure to the citation data of Genezio. You upload a log on the Agent Analytics page of your brand. Refer to Upload your logs.

​

Why CDN logs show how answer engines use your site

The citation tracking tells you which of your pages appear in the answers of answer engines. Your logs tell you the other half: what answer engines do on your site first. The logs show two different behaviors. The main purpose of the analysis is to keep these two behaviors separate.

​

1. Training crawls

Answer engines crawl the web to make and update the models behind their answers. Your logs show which of your pages they visit for training. Thus, you see which parts of your site the model knows and which parts it ignored.

​

2. Real-time answer enrichment

This signal has more value. Answer engines also fetch pages in real time, while they answer a specific question. They use these pages to add current data to the answer. When a page has this type of access, an answer engine adds your content to a live answer at that time. Thus, you see which pages are important for the answers of answer engines now. This data is very different from “a crawler visited this page at some time”, and you can act on it more easily.

​

Questions that the analysis answers

• Do answer engines crawl my most important content, or do they ignore it?
• Which pages do answer engines fetch in real time to support live answers?
• Which high-value pages do AI bots never visit?
• Which pages do crawlers visit frequently, but answer engines never cite?

The analysis uses your own server logs. Thus, it is direct, first-party evidence, not an estimate.

​

Use the CDN logs with GA4 data

The log data is most useful when you read it together with your Google Analytics (GA4) data. Your CDN logs show what the answer engines did: the crawls and the real-time fetches. GA4 shows what persons did. When you compare the two, you get a much fuller view of these items:

• How answer engines use the content that you publish
• What the persons who come to your site from these answers do

​

What log data Genezio ingests

Genezio ingests your CDN or web server access logs. These are the standard logs that your CDN or your origin server already records for each request:

• The URLs and the paths of the requests
• The user agents that identify the source of each request, which include AI crawlers
• The time of each request, which makes the real-time answer enrichment visible

Usually, your infrastructure team can export these logs from your CDN or your web server. You do not have to add special instrumentation.

​

What Genezio analyzes in the logs

Raw logs are long lists of separate requests. Alone, they tell you very little. Genezio analyzes them and gives you reports on three items.

​

Clustering by keyword

Genezio clusters your log entries by keyword. It puts the requested paths into themes. Thus, Genezio reports the activity at the level of topics, not of single URLs. When a different view is useful, Genezio can cluster the same logs in more than one way.

​

Crawler breakdown

Genezio makes a crawler breakdown. It identifies the bots and the crawlers that access your site. It gives special attention to AI crawlers, which are the crawlers that answer engines use to fetch content. This keeps the traffic of answer engines separate from search engines, scrapers, and usual visitors.

​

AI-crawler traffic compared with citations

The analysis with the most value connects the two halves. It compares the AI-crawler traffic with your citations. It examines if the pages that answer engines fetch agree with the pages that they really cite in their answers. This analysis shows the gaps. Examples are important pages that answer engines ignore, or pages that they fetch frequently but never cite.

​

Upload your logs

You upload an access log on the Agent Analytics page of your brand. Genezio reads the file in the background, and the page shows the AI bots and the people that visited your site. For a custom report that adds your GA4 data, contact Genezio.

​

Read Agent Analytics

The Agent Analytics page has three tabs. They use the same filters.

• Overview: the visits of the AI bots, the page views of people, their countries, and the people that the AI answers sent to your site (the AI referrals).
• Detailed insights: the pages that the bots visit, in the folders of your site. For each folder and page, you see the platforms, the citations, and the citations for each visit. A page with many visits and no citations shows where a bot reads you and the answers do not use you.
• Settings: your uploads, the setting for unverified traffic, and the removal of your data.

Select a page, a folder, a type of bot, or a country to open a panel with its detail. The panel of a page also shows its citations by answer engine, after its bots. While Genezio reads an upload, the page shows a summary of it. When the read ends, the summary shows the result.

​

Verified and unverified visits

Any client can send the name of an AI bot. Thus, Genezio checks the IP address of each visit against the lists that the AI companies publish. Each visit is verified, unverified, or unknown. A bot with no published list is unknown. By default, the charts show only the verified visits, and the Overview shows the share of verified visits. To see the unverified visits too, change the setting in the Settings tab. The chart then shows them as a dashed line.

​

Remove your data

In the Settings tab, you can remove an upload with its visits. You can also remove all the data of Agent Analytics for the brand.

​

Related pages

• Access log format
• Citations
• Actionable insights
• How LLMs select sources