LLMs and Search Engines
A set of notes on how search engines can leverage Large Language Models (LLMs) and generative AI for significant customer and business benefits.
Part 1
Several recent papers have delved into the realms of knowledge editing and domain knowledge acquisition for Large Language Models (LLMs). I posit that a critical stride toward enhancing vertical search engines’ efficacy in serving customers lies in their ability to cater to diverse information needs comprehensively. This advancement not only facilitates users in finding desired information but also translates to increased conversion rates and revenue for businesses. Imagine a traveler using a search engine to plan a vacation, who should be able to inquire , for example, about various aspects such as suitable kite-surfing beaches in Maui, recommended fish restaurants, and the optimal kite-surfing season to make the right decisions. Similarly, someone purchasing a laptop on the online shopping site, for example, should be empowered to ask about the necessary memory and processor specifications for tasks like active photo and video editing alongside office work requirements if they are needed by this user to make the right decisions. To effectively support users throughout their search journey, search engines must possess in-depth domain knowledge (e.g., geographical, recreational activities, local amenities etc for vacation planners; electronics, home furniture, fashion etc for online shoppers), surpassing the capabilities of generic LLMs. Significant efforts are underway to enhance LLMs’ domain specificity through knowledge editing, tailoring them for specific types of factual queries, and imbuing them with timeliness (e.g., incorporating real-time weather updates or major events for travel search engines).
https://lnkd.in/g6FQqpZB
https://lnkd.in/gY8bniV2
https://lnkd.in/g68MTuBW
Part 2
In recent discussions, we considered the potential of Generative AI to significantly enhance search engine capabilities by integrating additional knowledge bases and question-answering functionalities. This could expedite user search processes and improve decision-making efficiency. For instance, in a travel search domain, AI could analyze and convey which beaches in Maui are best suited for kite surfing during specific seasons, or assist a potential camera buyer in understanding the relevance of ISO settings for landscape versus sports photography, including comparative analyses between products like Sony and Canon mirrorless cameras.
Another critical area for improvement pertains to handling generic search queries such as “52 inch TV,” “Italian restaurants in Scottsdale,” or “lawn mowers.” (they constitute a large part of traffic of search engines) Presently, the diversity in search results reflects the varied preferences of consumers, yet the traditional list format provided by search engines often fails to facilitate the optimal choice. Consequently, users frequently resort to multiple platforms and review sites, resulting in potential customer attrition for search engines — a situation detrimental both to users and search providers.
Historically, the challenge of managing complex information has been mitigated by sites that provide comprehensive reviews and descriptions, aiding users in making informed decisions. When examining a query like “Italian restaurants in Scottsdale,” for example, pages that offer comparative and relevant information prove particularly valuable (examples, “Italian restaurant Scottsdale” for local search vertical, https://lnkd.in/geAJws-x “coffeemakers” for online shopping https://lnkd.in/gFn9MUMh). See example, from Gen AI experiences of Bing and Google (pictures attached)
Utilizing the capabilities of Large Language Models (LLMs), search engines could dynamically adjust the richness and type of information provided based on multiple variables such as the query specifics, user profile, location, and time. This adaptive response could determine the appropriate balance between comparative and direct information, tailoring content to meet individual user needs, including expert opinions versus general reviews etc. Such customization would likely vary significantly among different types of queries (query understanding), with some necessitating a straightforward “10 links” approach, while others require more elaborate, contextually relevant information.
This strategic use of LLMs could profoundly transform the utility of search engines, especially within specialized verticals, shifting from mere conduits to other information sources to becoming essential decision-support tools. This advancement stands to redefine user engagement and retention, significantly enhancing search engine efficacy in the digital ecosystem.
Part 3
I’m really impressed by the work from Amazon’s science team on translating objective e-commerce product attributes into customer intents. This allows customers to search for products in a way that suits them, which is especially crucial because search engines often struggle with this issue. The paper highlights significant challenges, but there are many more variations of mismatches that weren’t discussed. Addressing these could significantly improve customer experience and boost conversion rates.
Large Language Models (LLMs), when integrated with additional knowledge sources, can play a vital role in resolving these mismatches. For example, in my experience, when customers search for strollers, search engines often suggest filtering by frame size. However, most people don’t know whether they need a stroller that’s 42 or 38 inches. Instead, they are aware of more practical considerations like the model of their car (that defines the size of their car’s trunk), whether they live in an apartment with an elevator or a house, all of which dictate their actual needs regarding stroller size.
Asking customers directly about the frame size isn’t very helpful. However, if the search engine were to ask about their living conditions and car type, it could then deduce the appropriate frame size for them. This principle applies across many other product categories where customers might not know the specific attributes listed in the catalog but can provide information about their conditions and requirements that define those attributes. This approach allows the search engine to ask relevant clarifying questions, leading to a more efficient and satisfying shopping experience.
it’s true for any search experience (travel, local, real estate, financial services, healthcare etc)
LLM and Search 4
Leveraging Large Language Models in Conversational Recommender Systems, Google Research
Can Large Language Models (LLMs) be harnessed to gain a deep understanding of user interests over both long and short terms through conversational engagement? Additionally, can LLMs offer users an interactive and personalized conversational experience that refines their search processes, utilizes a broad spectrum of domain-specific knowledge, and provides expert-level interactive guidance to navigate the search space effectively?
In a recent publication from Google Research concerning Search and Language Model technologies, the paper delineates the nuances of a conversational recommender system, utilizing YouTube videos as a case study. Although categorized under “Recommender Systems,” the blurred boundaries allow its inclusion under the broader umbrella of search technologies. This type of system is imperative across various search domains such as travel, retail, local services, finance, fashion, and healthcare, where the complexity and multitude of product choices substantially influence consumer decision-making processes.
The challenges inherent in these domains stem not only from the complex nature of the products but also from the sheer volume of available options — ranging from millions to billions of items. Such environments require sophisticated mechanisms to decode user requests and parse through vast item sets to identify those that align with user interests, thereby facilitating informed decision-making.
Conversational recommender systems excel in these contexts by engaging with users through interactive dialogues, effectively capturing both long-term preferences and immediate needs. This interaction is crucial, particularly in scenarios where users may not have fully formed their search intents or find it challenging to articulate their needs through a single query. By dynamically refining understanding through ongoing conversation, these systems offer a tailored search experience that adjusts in real-time to user feedback.
Furthermore, the paper discusses methodologies for updating and modifying user preferences as new information is gleaned throughout the search process. This capability ensures that the search remains relevant and tightly aligned with user interests. The dialogic nature of these systems also helps maintain control over the search conversation, guiding it strategically to meet the user’s evolving expectations and needs.
In conclusion, the long-term vision for search engines, as suggested by the paper, is to evolve into systems that rival the best human experts, agents, or merchants. This evolution aims to enhance efficiency and decision quality in user interactions, personalized to individual preferences and contextual needs — a significant leap forward in the field of search technology and user interaction.
https://arxiv.org/abs/2305.07961
interesting follow ups on this and related topics
https://arxiv.org/abs/2307.02046
Part 5. Search as Task Solving
LLMs and Search
DOLOMITES: Domain-Specific Long-Form Methodical Tasks
Google DeepMind and the University of Pennsylvania have developed a new benchmark focused on systematically defining tasks, drawing from expertise across various fields. This benchmark is intended to evaluate the performance of large language models (LLMs) by presenting them with a diverse array of tasks. The initiative is poised to significantly influence the capabilities of LLMs, enhancing their role as problem-solvers and assistants in numerous domains.
The implications of this work are particularly promising for search and recommender systems. Such systems could evolve to not only respond to but actively assist with complex user queries like planning a 2–3 day backpacking trip in Canyonlands National Park, cooking pasta at home, or organizing a two-week vacation in California that includes sightseeing, hiking, and cultural events. By breaking down these tasks into searchable recommendations and actionable steps, grounding them into buy/book/rent/ activities on site, these platforms could offer a more integrated and user-focused service, transforming how users interact with search and recommendation engines and benefit from AI technologies.
https://arxiv.org/abs/2405.05938
Part 6 Conversational Search with Search engine as a third agent
LLMs and Search: Introducing CoSearchAgent, a lightweight collaborative search agent with large language models.
This search engine agent participates in conversations with multiple human parties, assisting them with information-seeking requests as needed.
Similar work was done with our startup Ozlo (https://lnkd.in/gnh8F9Gt https://lnkd.in/g6XEBpu), a conversation agent that helped with local search within human-human conversations. Ozlo was acquired by Facebook in 2017.
With the use of LLMs, there is a lot of additional power in handling conversations better in 2024 than in 2017. Check out the review section of the paper for some useful references about understanding context in multi-turn conversational search.
Learn more about CoSearchAgent’s capabilities in the full paper: https://lnkd.in/gbJR4FTs
Part 7
One of the most important papers on Search and LLM is written by authors both from Google and OpenAI (and the University of Massachusetts Amherst)
it’s quite interesting especially as everyone writes their guesses what will be announced next week
FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation
https://arxiv.org/abs/2310.03214
Ref
Search and Recommendation engines chapter in LLM Evaluation Compendium
