Has Search Gotten Worse? (No, but)

People over on Mastodon are arguing that web search has gotten worse. They are disappointed in Google and DuckDuckGo claiming that “you can't find anything anymore”. And that feels correct, doesn't it? Google in the past year has gotten exponentially worse. Or was it web search in generally that got just gradually worse in the past 10 years? What are we even talking about? I can't find good vegan recipes on the web and someone has to be responsible!

Okay, maybe the problem is more complex than the “truth” we immediately felt. Have cars gotten slower in the past 10 years? Feels like it, with all the traffic and speed limits and my driving skills getting better and seeing cars shot into space (I wanna do that!) and other people having faster cars than me, … Okay, so there's lots of factors contributing to us thinking search has gotten worse. Let's look at them individually. I don't have any data for them either, but maybe we can get closer to a truth.

The Search Algorithms

Consider the search algorithm. A search engine is supposed to serve you the most relevant results for your search. Some algorithm takes your search string and goes through the index to serve results.

Google started out 26 years ago1 with PageRank. They are closed source, but they are doing AB-Testing and I would venture the guess that they are not actively worsening the algorithm, favouring random results over relevant ones. I'd say the same of their competitors.

Okay, maybe they are placing not-so-relevant results higher, namely ads and sponsored content. Most search engines make their money through ads. Google does the tracking-kind, DuckDuckGo the non-tracking ones, but in the end they serve you ads and need you to click on them. Placing the ads where they would naturally appear in their “benevolent relevancy rating” doesn't make sense, why would an advertiser pay for that? So they get washed to the top where they aren't as relevant to the user as the first result.

The business of search seems to have gotten worse. With many people using ad blockers ad supported search engines have to get more aggressive in serving ads. Additionally, Google has grown so big that they are unable (or unwilling) to review the ads that are placed leading to malvertising2. So yes, the experience of web search has gotten worse for people without ad blockers in this regard.

Online Content

If search is the application of a search string to a corpus, then worsening the corpus will also decrease the quality of search results. From the early days SEO has led to worse content being produced. The favourite example being these annoying life stories prefacing recipes on the internet. Nobody wants to read them, nobody wants to write them, but Google's algorithm values them so they are there. With LLMs making it easy to generate this kind of content the internet is (not so) slowly being filled with it.

Search has gotten harder as the content is getting bigger in volume and is trying to trick search into showing less relevant results. So even a steady quality of search results would mean that search has gotten better, and if you feel that the quality of results has gotten worse than you'd have to measure it against the quality of all content on the web to arrive at a “pure search quality rating”.

Users

Maybe it's not search, maybe it's you. This last variable has a lot of different sides that could lead to worse results or worse felt results. Firstly, in conjunction with the above it might just feel worse to get a result that you didn't look for. In the early days you'd still get the hand written page of someone trying to make a cool site. That's neat, even if it's not what you were looking for. A wrong click now will serve you viruses and auto-generated word vomit and signs you up to a couple newsletters.

Secondly, if you're complaining about search qualities, you're probably not the type of person Google has been targeting in the past 10 years. You're probably into computers a lot. You might use your PC through a terminal, know how to code, and browse the docs of coding languages for fun. Ten years ago you google things for family members, because their search queries were whole questions that did not even include the keyword they should. To grow Google needed to reach these people instead of proving vim key bindings for you.

Thirdly, our expectations have grown. Social media platforms have shown us is that while we are unique there's still millions of people like us. We enjoy our filter bubbles, but for all the personalization Google does we still get confronted with a world we're not familiar with. When I google “cake recipe”, why does Google show me non-vegan recipes? When I look for “Deadpool review”, why do I get positive ones from Marvel fans?

Lastly, our search skills might have deteriorated. Just like people are crafting the “perfect” prompts for LLMs we used to craft perfect queries using +positive and -negative filters etc. Now we dump typo-riddled half-formed questions into the text field labelled “Search” and expect a brief thought out answer. When you wanted to know the year when Pachelbel's Canon was written you searched “Pachelbel's Canon”, went to the Wikipedia page and skimmed it for the year. Now you vomit into the search bar “When wedding canon written” and expect the result without another click. And the most shocking thing is that you probably get it.

Conclusion

Has search gotten worse? No, probably not. In a bigger sea of (mis-)information more average users can use worse search queries to find answers. “Adele Hello” still gives you the YouTube upload of her song as the first result. The first result for “Brown bear” is still Wikipedia's entry on brown bears and “Facebook.com” still yields facebook.com. Is that good enough for a service that you paid exactly 0,00€ for in the past 26 years? Maybe not. I'm now paying 10€/mo for Kagi, not because search has gotten worse, but because I want better search.

Footnotes

[1] 1997. ↑ [2] Probably not entirely their fault as the online scamming business has also grown to incentivise these groups to be more aggressive themselves. ↑

___ Reach out via Mastodon @Optional@dice.camp or shoot me an email