ChatGPT as a Search Engine vs Google
Part 1
(see the original and discussion, the LinkedIn post got 30 thousand views within a few days after it was published)
There are a lot of discussions if chatGPT can compete with major search engines
Some thoughts.
For consumer search engines, a large amount of search requests is about very fresh information — fresh up to several days. Being capable to answer such queries requires indexing hundreds of millions of pages every day, billions weekly. Can LLM be retrained with that speed? Not now. For example, asking several product queries, i see the answers are on text from 2021, there is no 2022-based information
To make a search engine commercially successful, one needs to run 10⁴ and more queries per 1 dollar, with the current price of chatGPT answers several cents per query (reply by S Altman) — unachievable
Many answers are factually incorrect. it’s impossible to judge correctness because the origin of the information is not provided (for example, it answers “the Oyster House” as a restaurant in the Richmond beach area), and there is no such restaurant here. How LLMs are trained now, the origin of the information is not preserved through the process. Being capable to provide links to the original information is important to judge correctness, and is also a key product property for many search applications, people do not want just to see an ‘answer’, they want to see a lot of information associated with the answer to getting more information. In many cases, the link or a set of links is a very important product feature, which is needed for customers, rather than a dry factual answer.
A large amount of consumer search requests are long tail, answering which requires indexing of tens of billions of pages. I do not see them in the chatGPT model now (for example, try name requests for non-celebrities)
Nevertheless, the quality of answers for some queries is well beyond the quality of answers from search engines. I believe a big challenger in vertical search engines can be built within one year
Part 2
I was saying that chatGPT will not be a competitor for a generic internet search (see my post a few days ago here) but can be a very substantial player in search verticals
Just look at this interaction. There are several turns in the dialogue, with typical imprecise questions from the searcher (me) but quite good informative relevant answers of the system, which understands the well full scope of each question including information from the previous turns. Playing with it more, there are several weak points (mostly incomplete information) — but certainly, it’s very promising. it can be a great local search engine, significantly better than other search engines for this vertical. If integrated, with scheduling, and ordering systems, — almost perfect. The same for eCommerce, movies/media almost any other vertical
Question: What are the best fish restaurants in Seattle
chatGPT Answer: Seattle is known for its seafood, and there are many excellent fish restaurants to choose from. Here are a few highly rated options that might be worth checking out:
The Crab Pot: This popular seafood restaurant is known for its generous portions of seafood served in a basket or on a platter. The Crab Pot has locations in both Seattle and Bellevue.
Ivar’s Acres of Clams: This long-established seafood chain is a local favorite for its fresh, sustainably-sourced seafood and waterfront views.
Taylor Shellfish Oyster Bar: This seafood spot is known for its fresh, locally-sourced oysters and other seafood dishes.
The Walrus and the Carpenter: This popular oyster bar in the Ballard neighborhood has a reputation for serving some of the best oysters in the city.
The Fishermen’s Restaurant: This family-owned seafood spot in West Seattle is known for its fresh, locally-caught seafood and friendly atmosphere.
These are just a few examples, and there are many other excellent fish restaurants to choose from in Seattle. It’s always a good idea to do your own research and read reviews to find a restaurant that meets your personal preferences and taste.
Question: tell me more
chatGBT Answer: Sure! Here are a few more details about some of the fish restaurants I mentioned earlier:
The Crab Pot: This seafood restaurant is known for its casual, laid-back atmosphere and generous portions of seafood served in a basket or on a platter. The menu features a variety of seafood options, including crab, shrimp, mussels, and more. The Crab Pot also has a full bar with a selection of beers, wines, and cocktails.
Ivar’s Acres of Clams: This seafood chain has been a local favorite for over 80 years, and is known for its fresh, sustainably-sourced seafood and waterfront views. The menu features a variety of seafood options, including oysters, clams, crab, and more. Ivar’s also has a full bar with a selection of beers, wines, and cocktails.
Taylor Shellfish Oyster Bar: Located in the Capitol Hill neighborhood, this seafood spot is known for its fresh, locally-sourced oysters and other seafood dishes. The menu features a variety of oyster options, as well as other seafood dishes like clams, mussels, and crab. Taylor Shellfish also has a full bar with a selection of beers, wines, and cocktails.
The Walrus and the Carpenter: This popular oyster bar in the Ballard neighborhood has a reputation for serving some of the best oysters in the city. The menu features a variety of oyster options, as well as other seafood dishes like mussels, clams, and crab. The Walrus and the Carpenter also has a full bar with a selection of beers, wines, and cocktails.
The Fishermen’s Restaurant: This family-owned seafood spot in West Seattle is known for its fresh, locally-caught seafood and friendly atmosphere. The menu features a variety of seafood options, including oysters, clams, mussels, and more. The Fishermen’s Restaurant also has a full bar with a selection of beers, wines, and cocktails.
I hope this information helps! Let me know if you have any other questions.
Question: which of them is the best for oysters
chatGPT: It’s difficult to say which of these fish restaurants is the best for oysters, as personal preferences for oysters can vary greatly. Some people prefer a more traditional, briny oyster, while others prefer a sweeter, more fruity oyster. That being said, here are a few of these restaurants that are known for their oyster offerings:
Taylor Shellfish Oyster Bar: Located in the Capitol Hill neighborhood, this seafood spot is known for its fresh, locally-sourced oysters and other seafood dishes.
The Walrus and the Carpenter: This popular oyster bar in the Ballard neighborhood has a reputation for serving some of the best oysters in the city.
The Fishermen’s Restaurant: This family-owned seafood spot in West Seattle is known for its fresh, locally-caught seafood, and the menu features a variety of oyster options.
It’s always a good idea to do your own research and read reviews to find a restaurant that meets your personal preferences and taste. You might also consider contacting the restaurants directly to ask about their oyster offerings and which types of oysters they currently have available.
Part 3
Some thoughts while reading more articles about chatGPT
Some time ago commenting about AI code generators ( another hot area , models generating code for the task described in natural language , openAI codex , GitHub copilot , etc. chatGPT can do it too ) I noticed that advanced code verification will be required. The model may generate the code. Does it comply to the description. For many tasks verification is easier than generation. Verification will help to generate the correct code by detecting incorrect versions or finding inconsistencies. ( the formal verification of programs , formal methods is well established and well advanced AI area which may now intersect with nlp and llm and deep learning ). Most probably future products based on chatGPT will use fact verification programs.
Customer asks a question.
LLM such as chatGPT generates answers
Are they correct ? Which of them should be refuted ? Which can be presented to the customer as correct answers ?
Generated answers can be verified by another ai model. Again , in most cases verification is easier than generation ( think about it as NP vs P problem. It’s hard to generate a proof. It’s easy to verify given proof. The Same principle is valid in many areas)
In most published examples of wrong chatGPT answers , I see the falsehood could be detected by the algorithm on the top of chatGPT ( it’s not a trivial machine , it must be capable to retrieve information, do nlp extraction , do logical reasoning. But it’s doable. And it’s simpler than llm such as chatGPT)
The question is chatGPT can be expressive enough to generate many answers with high probability containing a correct information. Looking into llm research, I think yes
Part 4
I think work on RLHF will be one of the big areas of future work on improvements of LLM for practical tasks (such as search engines and other)
Is the current OpenAI ChatGPT can be a competitor to search engines?
A big part of the answer to this question: is LLM “tuned” to the functionality people expect from a search engine.
Certainly, the current OpenAI version of RLHF is better suited for this task, than old LM tuning based on old traditional LM metrics, but is it exactly the task for SE response? I still doubt about it
The same about using LLM for many other business tasks. Making right RLHF sets will be a big part of tuning LLM (+ of course, very expensive task of tuning it, once the set is constructed)
Part 5
Some thoughts during reading articles about chatGPT vs major search engines ‘competition’ (cont. https://lnkd.in/gJ8_qpU6 )
There is an assumption made by many authors that a natural language answer is always better than ‘just links’. Which is not true. Links are what people need for the majority of queries. Queries such as “Best Indian restaurants in Seattle” or “cooking polenta with seafood” — one does not need to get some natural language response about a certain restaurant, or certain recipe, but a list of results, illustrated by maps in one case, video in another, mix of results from different corpora (books, youtube, maps, newspaper reviews) and that mix make “links” much more valuable for information seeking rather than one natural language paragraph response. Certainly, NL answer (assuming it’s correct, comprehensive, etc) is good for some queries, but not for all and even not for majority of typical consumer queries
Other important aspects
It looks like the typical latency of chatGPT is 2–3 seconds. Compare it vs 100 ms of the major search engine. How many nodes one would need to run to support 100000 qps operations with such latency, of a neural network machine, with very limited throughput per node (and each node must be at least 8 A100 per node? ), and what cost to run such an engine?
