Reddit promised its users an open internet. Then it closed the door.

Reddit Inc.’s privacy policy tells every person who posts there: "By using the Services, you are directing us to share this information publicly and freely."

That is not just a clause in a policy; it’s the fundamental promise Reddit has made multiple times to its users, and that openness is one of the primary reasons Reddit earned lasting popularity. And Reddit's user agreement is just as clear about who owns the posts: you do. Reddit holds a license. A non-exclusive one.

For most of Reddit's history, that deal held. Anyone could read Reddit without an account. Every search engine could find it. That openness is why a Reddit thread is so often the most useful result you get when you search for a real answer to a real question—someone else's honest experience, voted up by other people who found it useful too. It’s one of the reasons I’ve loved Reddit and why I use it almost every day. Reddit calls itself the most human place on the internet. Over many years, it earned that reputation.

Then Reddit decided its reputation was worth more than the advertising revenue it was already bringing in. To capture that value, it chose to alter the terms of the deal it made with its users.

In February 2024, Reddit filed to go public and announced an expanded partnership with Google. Both companies described it as a win for the open internet. A few months later, Reddit served a file to every crawler on the internet that said, in effect, go away—to everyone except Google. Bing lost access. DuckDuckGo lost access. Mojeek and Qwant lost access. Reddit had promised the Internet Archive would be spared. A year later, the Archive was cut off too.^[1]

Technically, robots.txt is governed by an official internet standard called RFC 9309, published by the Internet Engineering Task Force, that allows service owners to control whether and how automatic clients called crawlers may access their content and services. In essence, Reddit is relying on the good faith of these third parties to respect robots.txt while at the same time claiming that restricting access to just one crawler made the internet somehow more “open.”

But remember, Reddit has tried to shoehorn the facts to fit its copyright claims. Robots.txt isn’t a copyright protection mechanism. Instead, Reddit cites Google’s deployment of SearchGuard. If this had been about protecting anyone's copyright, the protection would have come first. It didn't. By Reddit's own account in this case, SearchGuard, the technology it now says was protecting its users' copyrights, wasn't deployed until January 2025—roughly six months after everyone but Google had already been locked out.

So what was it about? Its executives have said out loud that Reddit's conversations are "like oil for the modern internet," that scarce assets become valuable, and that they didn't want to give that value away for free. That is a candid description of a business decision. It is also a description of taking something that belongs to millions of people, making it artificially hard to reach, and selling the exception.

We think that's a problem, and not only for us. It challenges the basic principles of the open internet and the culture that created and maintained Reddit.

As a founder and CEO of my own business, I have no problem at all with Reddit evolving its operating model to capture more revenue. It is a public company now and has an obligation to serve its shareholders. But as a Reddit user, I completely reject the way they’ve gone about it: hampering competition in the search market, blocking access by organizations like the Internet Archive, and monopolizing my content without my knowledge. I believe in a fair and open internet, and I post on public forums like Reddit assuming my content is available to all, not a select few. 

I want to be precise about what SerpApi does, because it gets described carelessly. We access and organize search engine results data, and we are building our own search engine. We do not sell access to Reddit content. Our customers already have access to Reddit content—the same access you have when you run a search. What we sell is search data structure: the work of turning public results into something a program can actually read. The open internet is enormous and disorganized, and that work is harder than it looks. It is why researchers, universities, developers, and companies can build on what the web already makes public (e.g., white papers using SerpApi as a tool to understand misinformation). It is also, increasingly, how AI systems reach current information about the world.

As SerpApi expands beyond giving structure to existing search engine results pages like Google, Bing, and Yahoo to building our own search engine, Reddit’s exclusive agreement with Google is hindering innovation and blocking SerpApi and others from a chance to compete fairly in this important market.

Friday, we answered Reddit's lawsuit, and we filed counterclaims of our own. We've asked a court to look at what Reddit did: who it shut out, what it promised, when it promised it, and what it charged. The full filing is posted below, and as always, we encourage you to read it.

This is just one more step in the legal process and Reddit will have its chance to respond. And let me be clear that this isn't aimed at Reddit's communities, which are one of the genuinely good things on the internet. That's the point. The people in those communities wrote all of it, they own it, and they said to share it freely.

That promise meant something to me, as a Reddit user. It should mean something for anyone who believes in an open internet.

Questions? Contact press@serpapi.com.

^[1] In transparency: The Internet Archive has been a recipient of charitable contributions from SerpApi.