Access Log Format for AI Bot Analytics

Agent Analytics reads an access log of your website. It shows which AI bots visit your pages, what they read, and how your site answers them. It also shows how many people the AI answers send to your site, and the countries of the people who read your pages. This page tells your infrastructure team how to export the log. Give this page to the team that operates your CDN or your web server.

What Agent Analytics reads

Agent Analytics reads two types of lines from the log:

The requests of AI bots. Genezio identifies the bot from the user agent, for example GPTBot, ClaudeBot, or PerplexityBot. It counts each visit by day, bot, page, and status code.
The page views of people. A page view is a line from a browser, with the method GET, a status of 2xx or 304, and a path that is not a static file. Genezio counts the page views by day, page, country, and AI platform.

An AI referral is a page view that an AI assistant sent. The referer names the site of the assistant, for example chatgpt.com, perplexity.ai, or claude.ai. When the referer gives no AI assistant, Genezio reads the utm_source parameter of the URL. For example, ChatGPT adds utm_source=chatgpt.com to its links. Genezio ignores the other lines: search engines, scripts, monitors, and the static files of a page, such as images, scripts, and fonts.

The country of a bot visit is the country of the server of the bot, not of a person. Thus, Agent Analytics shows the countries of the people only.

What Genezio keeps

Genezio keeps only counts and the requests of the AI bots:

Genezio does not keep an IP address. It reads the IP address of a bot only to verify the bot.
Genezio does not keep the referer. It reads the referer only to find the AI platform that sent a person.
Genezio removes the query from each URL, because a query can hold a token or a personal value.
Genezio does not keep the requests of people. It keeps only the count of page views for each day, page, and country.

File requirements

Requirement	Value
Format	CSV with a header, or the Combined Log Format (CLF)
Extension	`.csv`, `.log`, or `.txt`
Size	50 MB or less
Compression	None. Decompress a `.gz` file before you upload it.
Encoding	UTF-8
Lines	One request on each line. A line longer than 16 KB is ignored.
Dates	The last 400 days. Genezio ignores a line that is older, or more than one day in the future.

When your log is larger than 50 MB, divide it into more files. Make sure that each day is in one file only. Refer to Upload more than one file.

The CSV format

The first line of the file is the header. The names of the columns are not case-sensitive, and the order of the columns is free. Genezio ignores a column that is not in this table.

Column	Required	Value
`timestamp`	Yes	The time of the request. Refer to Time formats.
`request_url`	Yes	The full URL or only the path, for example `https://example.com/pricing` or `/pricing`.
`response_status`	Yes	The HTTP status code, for example `200`.
`user_agent`	Yes	The `User-Agent` header of the request.
`request_method`	No	The HTTP method, for example `GET`.
`client_ip`	No	The IP address of the client. Genezio uses it to verify the AI bots.
`country_code`	No	The ISO 3166 code of two letters of the country of the client, for example `US` or `ro`.
`referer`	No	The `Referer` header of the request. Genezio uses it to find the AI referrals.

Without client_ip, Genezio cannot verify a bot. Without country_code, the page shows no country of people. Without referer, Genezio finds an AI referral only from utm_source.

Example

timestamp,request_method,request_url,response_status,client_ip,user_agent,country_code,referer
2026-10-03T12:01:14Z,GET,https://example.com/pricing,200,20.171.207.2,"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.2; +https://openai.com/gptbot",US,
2026-10-03T12:02:40Z,GET,https://example.com/pricing?utm_source=chatgpt.com,200,86.124.10.7,"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36",RO,https://chatgpt.com/

The first line is a visit of GPTBot. The second line is a page view of a person from Romania, and ChatGPT sent the person. Genezio removes the query of the URL. Put a value between double quotes when it holds a comma. A user agent usually holds a comma.

Time formats

The timestamp column accepts these formats:

Format	Example
ISO 8601	`2026-10-03T12:01:14Z` or `2026-10-03T14:01:14+02:00`
Unix time in seconds	`1791028874`
Unix time in milliseconds	`1791028874000`
CLF time	`03/Oct/2026:12:01:14 +0000`

A time without a time zone is UTC. Genezio groups the visits by the UTC day.

The Combined Log Format

Genezio also reads the Combined Log Format of nginx and Apache. Each line holds the user agent at the end:

20.171.207.2 - - [03/Oct/2026:12:01:14 +0000] "GET /pricing HTTP/1.1" 200 5120 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.2; +https://openai.com/gptbot"

nginx writes this format with the predefined combined format of access_log.
Apache writes this format with LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined.

The Common Log Format has no user agent. Genezio cannot identify a bot without the user agent, thus it refuses this format. A CLF line has no country. To see the countries of the people, export a CSV with the column country_code.

Export the log from your CDN

Export the log from the CDN, not from the origin server. Behind a CDN, the origin server sees the IP address of the CDN, not of the client. Each CDN uses its own names for the fields. Select the fields in this table, and then rename them to the columns of Genezio.

Genezio column	Cloudflare (Logpush, HTTP requests)	Amazon CloudFront (standard logs)	Fastly (VCL variables)	Akamai (DataStream 2)
`timestamp`	`EdgeStartTimestamp`	`date` and `time`	`time.start`	`reqTimeSec`
`request_method`	`ClientRequestMethod`	`cs-method`	`req.method`	`reqMethod`
`request_url`	`ClientRequestURI`	`cs-uri-stem`	`req.url`	`reqPath`
`response_status`	`EdgeResponseStatus`	`sc-status`	`resp.status`	`statusCode`
`user_agent`	`ClientRequestUserAgent`	`cs(User-Agent)`	`req.http.User-Agent`	`UA`
`client_ip`	`ClientIP`	`c-ip`	`client.ip`	`cliIP`
`country_code`	`ClientCountry`	Not available	`client.geo.country_code`	`country`
`referer`	`ClientRequestReferer`	`cs(Referer)`	`req.http.Referer`	`referer`

The names of the fields can change. Examine the documentation of your CDN before you start the export.

CloudFront writes its standard logs as text with tabs, and it encodes the user agent. Convert the file to CSV before you upload it. The standard logs of CloudFront have no country.

Convert a JSON export to CSV

Many CDNs write the log as JSON, with one object on each line. This script converts such a file to the CSV of Genezio. Change COLUMNS to the field names of your CDN. The example uses the fields of Cloudflare.

import csv
import json
import sys

# The column of Genezio, and the field of the CDN.
COLUMNS = {
    "timestamp": "EdgeStartTimestamp",
    "request_method": "ClientRequestMethod",
    "request_url": "ClientRequestURI",
    "response_status": "EdgeResponseStatus",
    "user_agent": "ClientRequestUserAgent",
    "client_ip": "ClientIP",
    "country_code": "ClientCountry",
    "referer": "ClientRequestReferer",
}

writer = csv.writer(sys.stdout)
writer.writerow(COLUMNS)
for line in sys.stdin:
    if line.strip():
        record = json.loads(line)
        writer.writerow(record.get(field, "") for field in COLUMNS.values())

Run the script:

python3 to_genezio_csv.py < cloudflare_logs.json > access_log.csv

Cloudflare can write EdgeStartTimestamp as RFC 3339, as Unix seconds, or as Unix nanoseconds. Select RFC 3339 or Unix seconds. Genezio cannot read nanoseconds.

Upload the file

Open Agent Analytics

In the dashboard, open your brand, and then open Agent Analytics.

Open the settings

Select the Settings tab.

Upload the log

Drag the file to the upload area, or select Upload log. Genezio checks the first line of the file at once.

Wait for the result

Genezio reads the file in the background. A summary shows the visits of the AI bots and the page views of people.

A brand can have 3 uploads in progress at one time.

Upload more than one file

Each upload is one source of data. When two sources hold the same day, the charts use the newest source for that day. Thus, a log that you upload two times does not count a visit two times. This rule has one effect. When one day is in two files, the charts show only the part of the day that is in the newest file. Make sure that each file holds full days.

Errors

Message	Cause	Fix
The first line is not a CSV header and not a line of the Combined Log Format.	The file is not a log, or its first line is not a header.	Add the header line, or export the Combined Log Format.
The CSV header has no column …	The header does not have a required column.	Rename the column of your CDN to the Genezio name.
The log has no user agent.	The file uses the Common Log Format.	Export the Combined Log Format.
The file is larger than 50 MB.	The file is too large.	Divide the file by day.
The file is compressed. Extract the log and upload the text file.	The file is a `.gz` or a `.zip` file.	Decompress the file.
The file is not a text file.	The file holds binary data.	Export the log as text.

Start here

Platform

Analyze and improve

Learn

Administration

Export an access log for Agent Analytics

What Agent Analytics reads

What Genezio keeps

File requirements

The CSV format

Example

Time formats

The Combined Log Format

Export the log from your CDN

Convert a JSON export to CSV

Upload the file

Upload more than one file

Errors

​What Agent Analytics reads

​What Genezio keeps

​File requirements

​The CSV format

​Example

​Time formats

​The Combined Log Format

​Export the log from your CDN

​Convert a JSON export to CSV

​Upload the file

​Upload more than one file

​Errors

​Related pages

What Agent Analytics reads

What Genezio keeps

File requirements

The CSV format

Example

Time formats

The Combined Log Format

Export the log from your CDN

Convert a JSON export to CSV

Upload the file

Upload more than one file

Errors

Related pages