A new empirical study provides deep insights into how generative AI is disrupting traditional web search paradigms, specifically comparing Google's search engine, its AI Overview (AIO) feature, and Gemini Flash 2.5 in terms of information retrieval and presentation.
The research employed a public benchmark dataset comprising 11,500 real-user queries to evaluate and contrast the search results from these three distinct mechanisms. Key findings highlight the profound impact of generative search on the information access ecosystem.
Firstly, the study revealed that for 51.5% of representative user queries, AIOs are generated and prominently displayed above traditional organic search results. It was also noted that controversial questions frequently trigger the generation of an AIO.
Secondly, a substantial divergence in retrieved sources was observed across the different search engines, with an average Jaccard similarity of less than 0.2. Traditional Google Search predominantly retrieves information from popular or institutional websites, such as those in government or education sectors. In contrast, generative search engines like AIO and Gemini show a significant propensity to retrieve Google-owned content.
Thirdly, a crucial finding indicates that websites actively blocking Google's AI crawler (e.g., Google-Extended) are significantly less likely to be retrieved by AIOs, even when their content remains accessible via traditional indexing methods.
Finally, the study found that AIOs exhibit less consistency when processing two runs of the same query and demonstrate lower robustness to minor query edits, meaning slight modifications to a query can lead to vastly different outcomes.
These findings carry significant implications for understanding website visibility, the effectiveness of generative engine optimization (GEO) techniques, and the quality of information users ultimately receive. The researchers advocate for the establishment of new revenue frameworks to foster a sustainable and mutually beneficial ecosystem for publishers and generative search providers.