How Would You Organize 180 Million Websites?

There are currently about 180 million active websites on the internet. Finding what you need is going to be a challenge. Finding the website that meets your needs exactly and gives you a great experience is even harder. Organizing and finding stuff on the web has become a massive industry, with Google, Facebook, and Twitter battling for your precious time to best give you what you're looking for.

It's an extremely hard problem, and Google's method works pretty well for me so I was intrigued when I came across this post by Roy Pessis on how Google is killing the web. He laments about all the awesome websites he finds and how hard it is to find them and recall them when you need them:

Every week I find at least one site that blows my mind. I get excited about how this service could evolve into something big, it’s potential to grow into a billion dollar business, and how it can change the face of the Internet.

But you won’t find these great sites on the first page of Google results—you might not find them on the first 10. As a result, these services, some of them genuinely life-changing, get lost in the dark recesses of the Internet. Even when you find these gems, you probably won’t think to access them the next time you log on. Their biggest challenge is finding a large enough audience to create a habit around their product.

It's a commendable goal to want to improve the web experience and connect people with the companies that can best help them with their needs. If a service could show me the websites that would most efficiently and effectively help me do what I want right now, that would be beyond excellent.

This article really got me thinking about how the web could be better, but then a funny thing happened. I got stuck on the enormity of the problem. There is not one, but three main challenges to overcome—challenges that the big internet companies are attacking in various ways and doing a pretty good job of solving already. Any new solution is going to have to do better at all of these issues than the solutions that are currently out there, and that's a lot more difficult than convincing people that there should be a better way to find what they're looking for on the web.

How do you find exactly what you're looking for?

Finding the handful of websites that would best help you among the 180 million websites out there is hard enough, but to do it quickly, billions of times per day for hundreds of millions of users is shockingly difficult. Every user's idea of what they're looking for has its own context. Different websites will align better with different users' needs, even when they deal with the same topics. Finding the best match for everyone has a significant amount of irreducible complexity.

Each of the major internet companies deals with this complexity in a different way. Google attempts to match people to websites with keyword search. They index the web, find the keywords you're looking for in text and links, and return a ranked list of results. The whole process is much more complicated than that, of course, but it's a logical way to look for something in such a massive amount of information.

Facebook takes a different tact. They figure you'll be interested in the same types of things that your friends are interested in. You're likely to want to read or watch the things your friends find, create, and post, so your Facebook feed attempts to show you things from your friends' posts that are likely to interest you. This is not so much directed searching as finding what you're looking for through serendipity. You can find a lot of things you're interested in this way, but not likely what you're looking for right now.

Twitter uses yet another approach. It's similar to Facebook in that you follow other people and see a feed of their posts, but it's much more transient and you see all of the posts, as well as posts by others that respond to those posts. Choosing who to follow based on what you want to see is much more important here. If you carefully select who you follow, you'll have a well-curated feed of highly relevant links, comments, and discussions related to your interests. You do have to put in the time, and like Facebook, you probably wouldn't look to Twitter as a resource for immediate problems. But you can find a lot of valuable stuff this way over time.

Things aren't completely segmented along these lines, and each of these companies uses elements of the other approaches to help you find what you're looking for. Each of them provides a markedly different experience and makes different choices for the trade-offs involved. While none of them are perfect, they all get the job done fairly effectively, and each of them works better in certain situations.

How do you remember what you've already found?

Once you find something valuable on the web, you probably want to save it for later use. If you found it through Google, you may be able to use the same search terms to find it again the next time, if you can remember how you did it. It's even harder to find old stuff on Facebook, and it's nearly impossible on Twitter.

If you want to use the web like your desktop or tablet and store things for frequent use, then you need to "install" the websites you use most with bookmarks or a website like delicious.com. Personally, I use Firefox bookmarks, and they work pretty well. I keep them organized in folders, and I have access to them on any device that has Firefox installed. I can see how they don't scale well, though, and with hundreds of bookmarks, I'm starting to depend more on the search feature.

I don't know how to make bookmarks scale better, but desktops and tablets suffer from the same problem with installed apps. I know people who have installed 200+ apps on their smartphone and are in the same predicament. They can't find what they need when they need it. They need search. Having all of your apps on your desktop, just one click away, doesn't help if you can't find the ones you need in the sea of apps you never use. The desktop isn't really a solved problem. It's a different problem. Trying to make the web more like the desktop isn't going to solve any of the web's problems.

The real problem here is that once you get past a few dozen apps or bookmarks or whatever, it's hard to remember where you put them when you need them unless you've done a great job organizing them yourself. At a certain number of things, it's easier to resort to search. The web is way past that number, so the default is search.

I find that I use search more on the desktop now because it works so well for the web. I reserve the prime real estate on my taskbar for the dozen programs I use the most, and similarly, I have less than a dozen pinned tabs in Firefox for my most-used websites. Keeping more things than this available at once just isn't useful.

How do the best sites get noticed?

I'm sure we've all had the same experience of finding an awesome website, and then wondering why it was so hard to find or why we didn't find it sooner. These websites should be easy to find, right? Everyone should be using them because they're so awesome! But everyone has a different idea of what makes a great website, and there are a lot of different interests out there.

The most popular websites gained their popularity over time, and lots of websites benefit from network effects. They become more useful as more people use them. Sites like Facebook, Twitter, Amazon, and Stackoverflow depend on the sheer volume of users to make the sites better. It takes time and effort to build a site from small beginnings, and a site with lots of potential is much different than a site with millions of users. Not every awesome website is going to make that transition.

There's also the issue of competing in a world where the power law rules. Ben Thompson has an excellent article on how newspapers are suffering from the availability of content:

Most of what I read is the best there is to read on any given subject. The trash is few and far between, and the average equally rare.
This, of course, is made possible by the Internet. No longer are my reading choices constrained by time and especially place.

This property applies to all websites, though. It's hard to get noticed unless you're the best because people don't have time to look at much more than a few sources for any given topic. They're going to devote their precious time to the sites that are the most likely to give them good returns on their time investment. That typically means it's the popular sites that get the traffic. To get popular, sites need to have great content and great promotion strategies, or they'll get lost in the sea of other sites.

Every once in a while a new channel comes along that allows new websites to promote themselves easily and get popular, but that only works until the new channel gets saturated. Facebook and Twitter are recent examples. It may seem like app stores are a good model that could be used to promote websites because they've worked so well for smart phone and tablet apps. They've got reviews and ratings, and if you get promoted by Apple or Google, your app can really make it big. But there's still a lot of crappy apps out there with a small amount of great apps to find. iOS now has over 1.2 million apps and Android has well over 1.3 million apps. At those numbers it's not much easier to get noticed in an app store than it is on the web, no matter what the app store is like.

I would absolutely love a better web browsing experience. I think everyone would. I would love to find the best sources on any given topic or task instantaneously without any search effort. Who wouldn't? But who is judging what "best" is? My definition of best is almost guaranteed to be different than anyone else for a large selection of things. Aggregating opinions through ratings can go a long way, but what about the websites that go unnoticed that might be perfect for me? I wouldn't know unless I tried all of the options, and I don't have the time or the inclination to do that for most things.

I'm willing to give up some choice to Google or Amazon in exchange for expediency and something that satisfies my needs—something that is good enough. Taking into account the magnitude of content that is being sifted through, the current browser experience is more than good enough. I would welcome a better solution, but it's probably not going to replace the ones that are already out there. A new solution is going to have to make its own choices on the trade-offs, and it's going to have to first figure out how to organize those 180 million websites.