From LEXICAL to Predictive Search: Evolution or Involution?

Abstract

We explore the paradigm shift in information retrieval from classic semantic search to the emerging dominance of neural information retrieval (Neural IR) and Generative AI. Based on the Sarada Ranganathan Endowment Lecture I gave in November 2025, our analysis questions whether this transition represents true technological evolution or an “involution”—a regression in which structural logic and verifiable truth are sacrificed for statistical prediction. We start by detailing the architecture of classic semantic search, emphasising its reliance on explicit knowledge resources such as ontologies and linguistic rules to “understand” the user’s truly semantic intent. This is contrasted with the “pseudosemantic search” of modern neural models, which utilise vector embeddings and Retrieval Augmented Generation (RAG) to mimic understanding. Significant attention is given to the societal and ethical risks of this shift, including the “illusion of understanding” in Large Language Models (LLMs), the dangers of anthropomorphising AI, and the deepening digital divide caused by language inequality. We conclude by advocating for a hybrid future that reintegrates the logic, reasoning, common sense, and explicit knowledge of semantic systems into the powerful predictive capabilities of generative AI.