Feed Standardization Will Commoditize Feed Aggregation, So Let’s Create The Semantic Web!

Below is some of my personal theory on the semantic web and feed aggregation. Please excuse the rambling .
As companies race to effectively aggregate our content an interesting thing is taking place: suddenly an aggregation of all our web-based activities are available via multiple sites on the web. Personally, I have information aggregated at MyBloglog, FriendFeed, Facebook, and many of my activities also show up on Twitter. So with this information everywhere where’s the value add?

At one point, aggregation services that found you news or information related to your friends provided a valuable service, incentivizing you to return to the site in order to get the latest information about what’s going on in your world. Unfortunately, much of that information in now aggregated all over the web and the aggregation service is no longer the core value.

Instead the core value is the automated filtering of that information. If you have the best filter, then users will come back to you. Take Techmeme for example. The site aggregates the latest technology news that it believes is most relevant to the readers and most important for the readers to know. It’s partially automatic and partially manual.

Basic Aggregation Is A Commodity, Filtering Is The Added-Value

As we move toward a standardized way of presenting feed stories to aggregators, the value provided by the aggregators is essentially commoditized. Suddenly we will be able to choose any site on the web to aggregate all of our information but as many Facebook users are rapidly finding out, aggregating all that information can rapidly become inefficient.

Many Twitter users experience the same “feed fatigue” that Facbeook users have experienced over the past week. “Feed fatigue” is synonymous with content overload and so far few systems have been developed to handle the problem. Previously, Facebook would automatically choose what stories to display. Now the most popular stories are displayed within the “highlights” section of the homepage.

FriendFeed also provides an automated filtering system but so far, none of the companies have perfectly determined an automated system for finding what matters most to us. Gabe Rivera of Techmeme claims that no perfectly automated system can be developed. At the end, he suggests, there always needs to be some form of human interaction. To Gabe I would say: I agree but I don’t think it’s the editor. Instead, I think it’s the user that will provide the human interaction.

So What’s The Solution To Aggregation?

Whether it’s news about our friends and families or news about the world, there is still no perfect aggregator. If none of the systems developed so far are perfect at finding us all the relevant information we desire (e.g. we only need to visit one site to consumer all our information), what new models can be developed to handle this aggregation?

Many believe that search is the ultimate aggregator. For example, Google should be able to theoretically display all the information we want at any given moment without even having to query it. This would be a perfect system but unfortunately it doesn’t exist and probably won’t for a long time. It is also aligned with the concept of the semantic web which is still a relatively distant idea since all web data is still not perfectly structured.

I’ve been thinking more about feed aggregation recently and in my own opinion, having control of our filters is probably the most efficient way for the time being. When I reference filters, I don’t just mean “Friend Lists” and applications as Facebook currently provides. Instead I mean granular filters which include various sources, time stamps etc. Think of a stock screener but for general content.

Every content item has a source, a date, an interest category or folksonomy, and other various types of meta-data. You could then filter that information based on a variety of factors which will be defined by the data set. At the end of the day, certain information has a different meaning for each consumer of that information.

Rather than saying that there is one “true result” to a query, perhaps it’s better to let the users define how that query is structured in the first place, overtime building their own efficient filtering system (or aggregator).

Would Users Take The Time To Build Their Aggregator?

Many Facebook users that I’ve spoken with said that they took the time to vote on the various feed items that were displayed in their feed. In other words, they were willing to take the time to make the machine find more relevant information. If we assume that basic feed aggregation (content aggregation; the sum of which amounts to an entire “search index”) will become commoditized within a short period of time, isn’t it best that we begin spending time on developing efficient filtering systems for these aggregators?

At the end of the day, value is maximized through more efficient custom filtering services that the users can participate in creating. That means we need the systems to make the results more efficient. Facebook currently provides basic features (you can add or remove individuals from your feed as well as filter by friend list) and Google includes a few as well (SearchWiki and our clicks on search results), but neither have gotten very far.

I believe there should be much more user input in the search results including input into the filters that are being used. What do you think the next steps are for developing more efficient content discovery systems?