AI Crawlers Visited E-commerce Sites 2.4 Times More Often Than Google in September Traffic Analysis

AI crawlers from OpenAI, Anthropic, Perplexity and Meta made 6,826 requests to two e-commerce websites between September 1 and 27, 2026, while Googlebot made 2,810 requests across the same period, according to server log analysis published today by Tripster Developers. The 2.4-to-1 ratio means AI assistants now read online stores more than twice as often as Google, yet most retailers cannot see these visits because standard analytics tools do not track them.

Pratham Jani, founder of Tripster Developers, analyzed raw server logs from two company-operated websites to identify who was actually reading the pages, according to the analysis published on SecurityBrief. Of the 6,826 AI crawler requests, 859 came from assistants fetching pages in real time because a person had asked ChatGPT, Claude or Perplexity a question. “Each one is a potential customer asking an AI tool for a recommendation, and the tool reading a business website to decide what to say,” Jani wrote in the analysis.

The finding challenges the twenty-year focus on Google optimization that has defined e-commerce SEO strategy for Australian retailers. Businesses that cannot be read by AI assistants drop out of recommendations even when their Google rankings remain strong.

Analytics Blind Spot Hides AI Traffic From Most Retailers

Google Analytics and similar platforms do not display AI crawler visits because most AI systems do not execute the JavaScript tracking scripts that analytics tools depend on, the analysis found. The requests appear only in server access logs, which few business owners review regularly.

Tripster Developers operates its own e-commerce sites alongside merchant stores, giving the firm access to unfiltered server data. The September log review revealed traffic patterns that standard analytics would classify as zero visits. For retailers relying exclusively on Google Analytics dashboards, AI assistant activity remains invisible even as it accounts for the majority of machine-readable page requests.

The visibility gap means businesses make optimization decisions based on incomplete data. A site performing well in Google Analytics may be completely unreadable to AI assistants due to blocking rules or technical errors that analytics platforms never surface.

One in Ten AI Requests Were Attackers Borrowing Crawler Names

The server log analysis identified three patterns affecting how AI systems interact with e-commerce sites. First, approximately 10 percent of requests carrying AI company names were not reading pages at all—they were probing for configuration files, passwords and known software vulnerabilities, according to the findings. Attackers have discovered that many websites allow unrestricted access to anything identifying itself as an AI crawler, so hostile bots borrow the names.

Second, nearly 20 percent of genuine AI crawler requests ended in errors. Some reached deleted pages, some followed broken internal links, and some were blocked by security rules or rate limits designed to stop attackers. Each error represents a moment when an AI assistant tried to gather information about a business and returned empty.

Third, blocking settings are often enabled without explicit decision-making. At least one major content delivery network blocks AI crawlers by default on new websites, and many hosting platforms offer one-click blocking that remains active indefinitely. The website continues to function normally for human visitors, so no one notices the blocking is active. Meanwhile AI assistants receive nothing and exclude the business from answers.

Default Blocking Rules Separate Training Crawlers From Live Assistants

Jani drew a distinction between AI crawlers that gather training data and assistants that fetch pages to answer live customer questions. “The important distinction is between crawlers that gather training data and assistants that fetch a page to answer a live question,” the analysis stated. “Many businesses block both without realising they are different, and lose the second, which is the one that sends customers.”

Businesses with original research or premium content may choose to block training crawlers as a commercial decision. For most retailers, however, being readable is the objective—a product an assistant cannot access is a product it cannot recommend. Several AI companies publish the network address ranges their legitimate crawlers use, allowing site operators to verify traffic against those lists rather than trusting user-agent strings alone.

The analysis recommended retailers audit their robots.txt files and security settings together, as the robots.txt file might permit AI crawlers while a firewall or CDN blocks them. Both systems must agree for AI assistants to successfully read pages. Related guidance on technical SEO infrastructure requirements for AI search platforms covers how site architecture affects machine readability.

Six-Point Audit Framework for Retailers Checking AI Readability

Tripster Developers outlined six checks retailers should complete this quarter. First, review access logs or request log analysis from website managers, counting AI crawler visits, requested pages and error rates. Second, verify that robots.txt and security configurations align rather than contradict each other.

Third, match traffic against published crawler address ranges from AI companies, treating unverified “AI” traffic as hostile. Fourth, fix errors AI tools encounter—broken links, removed products without redirects, and timeout issues cost visibility with assistants as much as with human shoppers. Fifth, structure key facts for easy extraction, as assistants favor pages stating clearly what a product is, its price, who it suits, and shipping details.

Sixth, test results by asking major AI assistants about products in the business’s category and checking whether the site appears and whether information cited is accurate. The framework parallels recent brand AI search visibility audit methodologies published by digital marketing firms tracking citation rates across generative platforms.

Research published this month found product page copy remains critical despite AI agents reading feeds and schema, as assistants extract factual, specific text over vague marketing language when generating recommendations.

Business Implications

Australian e-commerce businesses optimizing exclusively for Google now operate with incomplete visibility into how potential customers find products. The September traffic analysis demonstrates AI assistants have become the primary machine reader of online stores, yet standard analytics infrastructure was built for an earlier era when Google dominated machine traffic. Retailers who cannot track AI crawler activity cannot optimize for it, creating a structural disadvantage against competitors auditing server logs and fixing readability barriers.

The finding carries particular weight for businesses in sectors where AI assistants influence purchase decisions—electronics, home goods, travel and professional services categories where customers routinely ask ChatGPT or Perplexity for recommendations. A retailer with strong Google rankings but broken AI readability loses citation opportunities to competitors whose technical infrastructure accommodates both channels. The business implications extend beyond traffic share to recommendation inclusion, as products AI assistants cannot read drop out of consideration sets entirely.

The six-point audit framework requires no platform purchases or agency retainers—only access to server logs and several hours of review time. Retailers who complete the audit gain visibility into a traffic source now twice the size of Google and can fix technical barriers before competitors do. Those who continue relying on Google Analytics alone will optimize for a shrinking share of machine-readable traffic while AI assistants direct customers elsewhere.