And all this business about apps is a real spoiler for suspension of disbelief. I’m clicking a link, expecting to learn more about what I was reading (that was certainly the author’s intent) but instead I get an ad for an app. If I seriously consider it, I’ve lost my train of thought. If I actually take the detour and install it, I’ve lost bigtime. The best way to minimize the loss is hit the Back button and skip it. But that’s a loss too. I clicked the link for a reason. And that was thwarted.

The main concept here is that if you provide links in your mobile application, then you should be linking to some information about the topic you were reading, not an advertisement for a different app. By itself, this complaint seems to be a bit of a stretch. However, he follows with another post a few weeks later that really explains his issue:

Every time around the loop, since then, the Internet has served as the antidote to the controls that the tech industry would place on users. Every time, the tech industry has a rationale, with some validity, that wide-open access would be a nightmare. But eventually we overcome their barriers, and another layer comes on.

Basically, Dave is seeing these mobile apps as trying to kill the web. If the app is trying to control interactivity by only supplying a “walled garden”, similar to what Facebook is doing, then the user really loses the power of the web. The core idea of linking gets lost in this scenario. However, what if the web is more than just links?

Prior to the last few years, people were very document-centric and reading HTML pages from a PC. Now, we have mobile devices that allow you to use the web whenever you want no matter where you are. We also had web applications that did not allow you to take your information elsewhere or integrate with other applications. Now, many applications use your Twitter or Facebook login in order to authenticate, and then post statuses or even create events in Facebook.

In order to support all of this integration, the web is evolving. The web is no longer just HTML. Last summer, I wrote about the rise of the API:

As the web evolves, much of that evolution will be powered by application APIs. Some of the APIs will be application specific, like the Twitter API, and others will be more generic like the various semantic web standards. All of these available APIs allow programmers to create more interesting applications, and potentially a new API layer on top of what already exists. What this means in the long term is that we are finally getting to the point where the semantic web had hoped to be, linking data between various applications and hopefully doing something interesting.

A web of links can be limiting when looking at applications. When looking at reading a news story, links make sense, but reading articles is only part of the web. By looking at the data available, we are starting to create a more interactive and informative web. Sarah Perez at TechCrunch thinks this could be moving towards a web of apps, but that post is more focused on mobile apps. As I said previously, mobile apps tend to be limiting in their own ways.

Think about the possibilities of using the various APIs that are available. I am not just talking about the social network APIs, I am talking about things like Freebase, a huge collection of data, or even data aggregator/providers like Gnip and DataSift. Using these services, developers can build interesting applications using data from various sources without worrying about what the Twitter API looks like.

These types of services provide a level of abstraction that developers are used to. First, you get the core APIs, then you start abstracting those APIs into something easier to use. Once concerns about low-level APIs are removed, developers can focus on being more productive with the new abstraction layer and start solving real problems. Once developers focus on real problems, that is when the real fun begins.

So, is the web just a bunch of links or are we finally getting a layer of abstraction on top of all of the web applications. This layer of abstraction is a normal evolution for developers, and we just need to ensure this is translated into more interesting applications.

Related articles from RegularGeek:

• Twitter Should Cut The SMS Cord
• A Look Back And A Look Ahead At 2012
• Plus Or Like, Choose Your Sharing Weapon

As a good example of technology without a solution, we can look at the semantic web. I am a big believer in semantic technologies, and they will have their day, but there is not a big solution for semantic technologies yet. The foundation is still being worked on, and even big advocates do not talk about solutions yet. Disappointingly, I found a recent post that talked about the added value of the semantic web:

If I ask the question: “What added value(s) does the Semantic Web brings on the table?”. So, what are the benefits that companies and organizations would get from using the Semantic Web? I am pretty sure that after asking this question, I would get answers such as:

• You will instantly be able to traverse graphs of relationships
• You will be able to infer facts (so create/persist new knowledge) from other existing facts
• You will be able to check to make sure that your knowledge base is consistent and satisfiable
• You will be able to modify your ontologies/vocabularies/schemas without impacting the description of your instance records or the usability of any software that use it (unlike relation databases)

Obviously, if you need to talk about productivity, you are still not in the solution stage. Talking about what you will be able to do, as opposed to what you are doing, should be a clue that we are looking at a technology and not a solution. Don’t get me wrong, there are probably plenty of custom business applications that take advantage of semantic web technologies, but we are not really talking about several widely available solutions. Solutions and widely available applications for the semantic web technologies will come, we just have to wait a little longer.

This leads me to my next point, theory is not reality. How many times have you hear technologists start a sentence with “In theory it should work…”? When you look at software development, there is a lot of theory and much of your daily work is based on those theories, like the theoretical performance of a lookup in a HashMap. However, when theory is thrown around at higher levels, it starts to get concerning. This came to mind when I read a post by Rob Williams. He was reflecting on the 10 year anniversary of the Agile Manifesto:

The ten year mark just passed on the Agile Manifesto. It brought out quite a lot of commentary from the Tribe Elders, and frankly, most of it was depressingly insipid. It reached something of a crescendo for me reading Jim Highsmith‘s piece in Dr. Dobbs …, he basically said “we won, and if what we won ends up getting spoiled, blame the idiots.” … What, exactly was won?

Seriously, what was won and what was the competition? Given that agile is a process, and it gets spoiled by idiots, did you really win? Part of the idea behind agile development processes is knowledge sharing, meaning no single person should be the keeper of all of the knowledge about a particular area of the code. In theory, this should extend to the agile processes, like Scrum, as well. If that is true, can you really say that you have won if you things get “spoiled by idiots”? You may have won the theory competition, but in practice you need to make sure the idiots cannot spoil your efforts.

We see this in software development. In an application, you write code that should pass some QA, where this is just some unit tests or even a manual test plan. That code is expected to work for as long as the application runs. Developers know that they need to write code that is difficult to break, so there are plenty of guards in the application. Part of your job as a software developer is to ensure that the next person to touch the code can modify it easily and reliably. Code that is elegant but immensely difficult to maintain is not entirely good code. Part of the software lifecycle is maintenance, and you need to ensure that your code can be maintained. Idiots can always ruin the beauty of your code, but if it is easily maintainable, it becomes difficult for the code to be “spoiled by idiots”.

So, what was Highsmith talking about? Well, agile processes are now being heavily used and the waterfall model is being avoided in most cases. This is a good thing. However, the practice of agile still needs help. The theory of agile has been espoused for a decade, and now people need help implementing agile appropriately. We need to make agile processes easier for people to adopt, creating solutions that people can use to make the processes easier to adopt. Some solutions exist, and some have seen solid adoption rates, but until agile processes and solutions supporting those processes are in more than 80% of all software development teams, we still have work to do. Once the “agile solution” is assumed for a development team, you cannot say anything has been won.

As with any programming problem, different requirements and different contexts may call for different technologies. If you get stuck on saying that JSON is better than XML (or the other way around), you lose another tool in your toolbox.

Another interesting context problem is one regarding location, but not the current location, the geography of thought. In that post, the author is looking at the different context biases western and eastern cultures have:

Unlike the westerners who generally believe objects (in its sense of abstraction) being the fundamental building blocks of the world, for long time the easterners perceive relations being the true construction blocks of the world. The westerners think a noun holding its uniqueness regardless of the application, while the easterners think of a noun indeed with few uniqueness until we recognize its application environment. The easterners think verbs being unique since they describe the ways of tangling while nouns are the things happen to be tangled.

Obviously, this is a different type of context than what the type of context discussed when arguing about the better data format. In all applications, context is of the utmost importance. In applications like Facebook or Twitter, you are providing context in the form of status updates. In applications like FourSquare, the context is provided by the GPS software on your phone. In all of these applications, there is a huge amount of data being created, all of it being transported with the needed contextual information.

However, all of this information currently has no meaning because we are not putting the pieces together. Some of the applications that work with Facebook, Twitter and FourSquare are trying to collect the information, but they are not processing the information. Louis Gray talks about this problem in his We Need Tech Intelligence post:

What is needed now is a doubling down on intelligence, or at least a way to improve our abilities to be better people, better employees, or simply better educated. The promise of something like Quora is not that it’s another social network to chat, but that it has the potential to be a valuable resource for discovery…

The problem with all of this data is that even with context it does not always have much meaning or at least actionable information. Or the data is from only one service, like FourSquare, but there are other players in the location application space. By only taking data from one application, you are potentially limiting your information to one subset of the population. Does this mean that you need all location application information? Absolutely not, but you should get some good coverage. Continuing with the location data example, you want several different checkin services, like GoWalla, Google Latitude and Facebook Places. In addition to basic location data, ensure that you are capturing additional metadata like reviews or ratings from services like Yelp. This allows you to gather more data over a larger population. By doing this, trends that become apparent in the data are more generalized. Taking a trend from one service could mean that a specific type of subset of the population may fit the trend, but it does not apply to the mainstream. Think back to a year ago and what FourSquare was then. It was a haven only for early adopters. Trends from that time were not entirely useful for the general population. If you look at the location services now, more people are using them, and Facebook has thrust location into the mainstream with its Places features. That is a much different population than the early FourSquare users.

Putting It All Together

So, if you have all of this data from all of these different types of services what do you do with it? First, you may not be in the best position to collect the data. Thankfully, there are services that will do the data collection for you for a price. I mentioned some in my “A Look Ahead” post, but there is something missing from that portion of the post:

The one prediction I will make is that some data startup will become huge. We have some players already in Gnip and DataSift, but 2011 is really the year of data and one company will have massive growth.

Yes, these types of data collection services will have a big year, but in order for a data startup to become huge, they need to add value to the data. Thinking in the terms of the examples above, the data collection services provide the data and the context. There is minimal analysis and meaning in the data. Some of this can be done by adding semantic tagging or implementing concepts from Linked Data. By providing this relationship data, some meaning can be inferred or at the minimum you have provided more contextual information. The analysis and meaning can be extracted using various analysis and graphing tools, but in many cases tools and algorithms from machine learning and artificial intelligence will be incorporated.

The analysis of data and extracting meaning from the data is not an easy process. Data analysis, statistics, machine learning and artificial intelligence have been lurking next to the web for years. Web analytics solutions use many concepts from these areas, but now we are generating useful data that is outside of the traditional web analytics world. Imaging if one of these data startups provided analysis of the location data across various services. This may not mean much to the mass consumer, but what about the small business owner? They would love to know that people are often visiting a competitor near them, but those same people have been unhappy with service. The small business owner cannot do this type of data collection and analysis themselves, and they are not the only people that would benefit from these types of services.

The real question is are you ready? Do you have the right tools and services that you need?

First, let me present some background information. A few months ago, Google announced several different protocols and products (Wave, SPDY, Go, PubSubHubbub, Public DNS, Salmon and WebFinger). I have been talking about Twitter becoming infrastructure for over a year, and they recently announced plans for their SMS services. This is all in addition to the fact that most social services are hooking into Twitter now. Last week, Facebook announced a bunch of things regarding the infrastructure of the social web and real time streams. If you look at all of this news, you may realize that these companies are not direct competitors. Why would I say this? I will give you an analogy that is very technical in nature. Basically, these companies are rebuilding the web but they are attacking different layers of the network protocol stack.

If you try to relate this to the image on the right, Google is playing with the “Lower Layers”. Maybe this is the geek culture or the fact that their core product, Search, is a very low level feature. They are defining the Physical Layer with things like SPDY and Public DNS. They are defining the Network Layer with Wave, Salmon and WebFinger. They are defining the way that sites will talk to one another.

Facebook is focused on the “Upper Layers” and is now defining the Application and Presentation Layers with their developer platform. They are also getting into the Session and Transport Layers with OpenGraph and the various widgets and APIs they are providing for third party sites. Twitter spans both the Upper and Lower layers. Their focus on SMS keeps them in the Network Layer, and the Twitter API spans the Transport and Session Layers. Their API has enabled an entire ecosystem of applications for the Presentation and Application Layers.

Ignoring the potential privacy issues that go with owning these layers, and each company has privacy issues, we are seeing something that potentially changes the way the entire web works. Google sees bits being transferred faster. Twitter sees the messages being passed to whatever device or application you tend to use. Facebook sees the relationships among all of the activity that users create. The evolution of this new web has taken a different form, and it will quickly take shape due to the number of combined users these sites have.

The original plans for the semantic web were created more by academics and it was a grand plan. However, there was no way to get people to see that vision without thousands of developers pushing in that direction. Now, the combined forces of Facebook, Google and Twitter are making the evolution to a semantic web much simpler. Much to the chagrin of many bloggers, users are already part of this evolution. This is the only way to make this evolution happen, to forcibly move users that are typically resistant to change. Privacy issues can be resolved, but the core features and relationships delivered by these companies enable developers to create applications that are helpful and not just another tool for another job.

I doubt many people thought that a social network, a search engine and a microblogging application would eventually form the backbone of a semantic web. However, they are leading the charge. Now, let’s see what developers can make out of this.

The OSI Protocol Stack image is borrowed from (http://www.yourdictionary.com/computer/osi-model).

RSS

The first step is to learn more about RSS. There is a very detailed overview of the history of RSS on Wikipedia. I point you to the history because there are forks in RSS that are important to know about and a few versions that may be used. There are three versions of RSS that you really want to focus on, RSS 0.91, RSS 2.0 and Media RSS. You will want to review the history of RSS and the three versions to determine what you want to generate, or specifics on how to find what data you can expect to see.

ATOM

Atom is another XML based format, but it was developed specifically as an alternative to RSS. Again, Wikipedia has a very good article on Atom and some other comparisons. The specification for Atom can be found as IETF RFC 4287. The purpose of Atom is best described by the spec itself:

The primary use case that Atom addresses is the syndication of Web content such as weblogs and news headlines to Web sites as well as directly to user agents.

Atom is considered a better format for blog feeds than RSS because it was written with syndicating content as the purpose. RSS is much older and has more general purposes.

JSON

JSON can be considered the new kid on the block regardless of how long the technology has been around. It has been only a few years that JSON has really been accepted as a data transfer format. It stands for JavaScript Object Notation. The idea is that you pass data around in what looks like a JavaScript object. For more detail on what this format looks like, review JSON.org for a good overview and always remember to wrap your keys in quotes otherwise Internet Explorer will get very unhappy. Wikipedia also has a nice article regarding JSON, some interesting tidbits and how it compares to some other formats. I highly recommend becoming familiar with JSON as most sites that provide an API are supporting JSON as an output format.

RDF

RDF, or Resource Description Framework, is a much different beast. Wikipedia comes to our rescue again with their RDF article. Where RSS is a general format that people are using to “describe” a blog’s feed, RDF is a general format to “describe” almost anything. The main problem with RDF is that many people do not really understand it. The first paragraph of the Wikipedia overview can show why:

Basically speaking, the RDF data model^[1] is not different from classic conceptual modeling approaches such as Entity-Relationship or Class diagrams, as it is based upon the idea of making statements about resources, in particular, Web resources, in the form of subject-predicate-object expressions. These expressions are known as triples in RDF terminology.

Thankfully, RDF can be serialized into XML. The RDF for this post could be:

<rdf:RDF
    xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
    xmlns:dc="http://purl.org/dc/elements/1.1/">
        <rdf:Description rdf:about="http://regulargeek.com/2009/03/24/api-data-formats-you-need-to-know">
            <dc:title>API Data Formats You Need To Know</dc:title>
            <dc:publisher>Rob Diana</dc:publisher>
        </rdf:Description>
</rdf:RDF>

RDF is fairly popular within the semantic web technologies as it is much more descriptive and convenient than trying to define custom data formats for everything. RDF is already in use in the social media world as well in the form of FOAF (Friend of a Friend). There are many other applications of RDF in the wild, so check the Wikipedia article if you are interested.

XML

XML is the last resort in data transfer formats. Basically, XML is still used because so many people know how to read, parse and use XML in their programming language of choice. The benefit of using a custom XML format is that it is very expressive and flexible. However, there are problems with using XML. Every client that reads your data format needs custom code in order to handle the data transfer. Any changes in the definition of the data require changes on the client side as well.

SUP

Another new player is SUP, Simple Update Protocol, that was defined by our friends at FriendFeed. The initial blog post on SUP has a good high level description:

SUP is a simple and compact “ping feed” that web services can produce in order to alert the consumers of their feeds when a feed has been updated. This reduces update latency and improves efficiency by eliminating the need for frequent polling.

I know this just sounds like a simple ping, but they have done something interesting with it. The idea is that a site like FriendFeed has one URL that you can poll. Within the results of that poll request, you could have the SUP-ID of thousands of feeds. Essentially, you get a “meta-feed” that tells you what feeds have changed. It is an interesting idea and it does have some support, but it is not as widely implemented as some of the others.

Anything Else?

I am sure there are many other formats that you could find and try to learn, but these will get you started. For each of these technologies, there are probably freely available libraries in the language of your choice for consuming and producing these formats as well. However, that is probably a topic for another post.

If anyone can have access to any information at any time, what is then the value of that information? As transaction costs to produce, distribute and consume information drop to zero the question arises if the information value itself drops to zero too? My guess is that in many cases the data itself will have less value. That same data all platforms are now fighting a war over, the data that makes web 2.0 more important than the destinations of web 1.0.

Because I have small children, I tend to relate things to simple quotes from cartoons or kids’ movies. Alexander’s quote reminds me of something said by Dash in The Incredibles. He states that if everyone is special, that is the same as saying nobody is special. Both this quote and the quote from Alexander seem to generalize the idea a little too broadly. If everyone has access to information, that does not mean that people get the same things from the information. In the case of news information, some people just skim headlines to stay on top of the general happenings in the world. Other people will read several articles in order to gain knowledge about a topic. They still have access to the same information, but the end result is very different. In the end, knowledge is the differentiator. Alexander does talk about this in relation to Stephen Hawking and information about black holes. If you have all published information about black holes readily available, that does not mean that you understand them as much as Stephen Hawking. I totally agree with this idea.

The main problem I have is that information is not knowledge. If I understand Alexander’s post, he feels the same way. However, he does not take this a step further to determine what this real time access may become. Real time data is just data. The real-time part of it just means that we have access to it quicker. However, quicker access to information is not the problem. Without knowing what to do with the data, the data is useless. I am going to make the same comparison I did in my comment on Alexander’s post.

There is knowledge to be mined

We are currently aggregating all of this real time information into sites like FriendFeed. This is very similar to a trend we saw in the 90s with databases. Many major corporations had various departmental databases. So, marketing may have had some interesting information on the various advertising campaigns, but they did not have any sales information. In order to get the sales information, they had to make a special request to the sales department for a report on the sales during specific periods of time. The sales department had all of this data in their own database, and just needed to write a custom report for the marketing team. People found that the delay in getting these reports was fairly lengthy, and they wondered why the data could not be brought together.

So, data warehousing arouse as a way to centralize the data being generated by various loosely related departmental databases. The simple benefits were immediately obvious. That same report for the marketing group now took a few hours to generate instead of the team waiting for a week to get the report, convert it into a readable format and load it into their own database. On top of this, data mining started to become a more formalised discipline. Once the data from the various departments was aggregated, people noticed that their reports only contained a small subset of the data available. There was a large amount of information that they had never seen before. What did this other information tell them?

For example, large pharmaceutical companies run advertisements all the time. How do they know if they are effective? In the data warehouse model, they can review the sales information for the timeframe of the advertising campaign and the few months after the campaign. If there is a non-seasonal increase in sales of the drug, then the campaign was probably effective. The other information that they could find in the data warehouse is the golden nugget that they are after. The other information that became available is the actual prescription data. These companies can receive daily or monthly feeds of anonymized prescription data from various pharmacy chains. This data will tell them which areas of the country purchase a specific drug more often. In addition, this data can be correlated to the advertising campaign to see if the advertising helped in those areas or even if it helped in areas where a competitor’s drug is selling better.

What does this have to do with real-time information access? First, we are still in the aggregation stage with tools like FriendFeed. Once the aggregation problem is fundamentally solved, people will start clamoring for better tools to help them understand and filter this data. We are currently building our data warehouses of real-time information. We are still waiting for the effective reporting and data mining. Some of this could come from the semantic web technologies and other pieces we probably have not seen yet. However, the mining of the real-time data is the reason we need to collect it. The problem is that you have to collect the information before you can understand what is in it.

Semantic Web

The semantic web is the poster child for the future of the web. This is the big bang that everyone keeps waiting for. I hate to tell you, but the semantic web will not burst onto the scene. There is too much change required by people creating the content on the web for a big bang to occur. People need to “semantically enable” their content. This is getting easier, but most people are going to ignore it until there is significant acceptance of semantic web technologies. That being said, the ReadWriteWeb article had a fantastic comment regarding what will happen:

Aziz Poonawalla said “folksonomy, leveraged en masse, could render algorithmic search obsolete. you get Semantic web almost for free.”

Image via CrunchBaseThe idea is that as people adopt semantic technologies, they get integrated into everything. Microsoft bought PowerSet in an attempt to leverage their semantic technology. Will standard algorithmic search become obsolete? I do not think so, but it will evolve just like everything else. In 5 years will the search algorithms look anything like they did 3 years ago? No, but it will not be a big jump from one algorithm to another. It will start with semantic technologies being add-ons to our existing technology. People will see the benefits long before they realize that it was semantic technology underneath.

iPhone, Android and the Mobile Revolution

Image via CrunchBaseDo you want big bang change? The mobile space is as close as you can get. Regardless of the hype behind the original iPhone, it was not the first touch screen mobile phone. However, it is one of the first phones that has the techies drooling and has mass consumer appeal. With the first Google Android phone being delivered as I write this, the consumer mobile space is primed for rapid growth. BlackBerry is even making its’ phones more consumer friendly, starting with the Curve, and with new touchscreen devices coming as well. Now that the phones have a usable and larger interface, developers can actually work with the platform. We have already seen some iPhone applications making more than $100,000. The application stores that the mobile companies are now building allow people to buy the applications in a much simpler manner. Currently, if you have a BlackBerry, where do you buy applications? There may be some weird interface for a “mall” that you can get to from your phone, but the process is far too hard for most people. I do not buy applications for my BlackBerry because of this. Now, applications are readily available or soon will be through these central stores.

For consumers, this is a huge deal. Consumers tend to have less patience as they are not “trying out” a device. They want one that works, and maybe buy some ringtones and applications for it. The process has to be really simple in order to gain acceptance. With the iPhone, all of these ideas have converged. This has forced the hands of many device manufacturers to respond, and Google created Android in order to compete as well. This is a very exciting time to be into the mobile industry. However, do not expect a big bang of acceptance. These phones are still expensive for the average consumer. Prices will slowly drop and adoption of these devices will slowly increase. Eventually, you will realize you REALLY can not go anywhere without your phone.

The revolution begins when the mobile devices have access to the semantic-enabled applications that are now running on your phone. With technologies like GPS on your phone, you will be able to go into a city and if you get hungry, find a restaurant to eat at. However the restaurant you chose will be someplace that one of your friends recommended and happens to be within one mile of where you are. Right now, you have to login to a social network to find the recommendation of an Italian restaurant, map the location on the GPS-enabled phone and get directions.

What if your phone realizes that it is around dinner time and just displays a restaurant’s location based on your friend’s recommendation all without you asking for it? That would be cool.

FriendFeed does have some nice filtering capabilities that has made it a little easier, but you probably do not want to stop following people, or ignoring a service as a whole. As TechCrunch put it, Web 3.0 Will Be About Reducing Noise. I am not sure I agree entirely with their statement, however I believe they are on the right track. Ian Kennedy talks about how the Lifestream Filter Will Be The Next Great Algorithm War. I think he is a little closer to the target when he says:

The huge opportunity ahead is a filter to bubble up the things you need to know without missing anything you want to know.

Lastly, Social|median relaunched recently as a personalized news filter. They are merging the ideas of social networks, think Ning, and social news, think Digg. I say Ning for social networks, because the site allows you to create networks around the news that you want to read. It is an interesting idea, but I do not think it is enough either.

However, look at the three ideas presented. First, we want to reduce noise. Second, people like the lifestreaming services but better filtering is needed. Third, we want to read only what we want to know about. This could be where the semantic web idea has its sweet spot. Part of the semantic web is about data and its relationships across websites. That sounds a lot like what the lifestreaming services do. However, the semantic web really links the data as opposed to just dumping on your screen. Most of the semantic websites are talking about learning your interests based on what you read, like and share. If the technology is good enough, the noise gets reduced because the service will provide recommendations for you. So, what do we want?

My best guess is that I want Twine‘s technology, FriendFeed’s interface and the ability to import my friends from all of my other social services (probably Twine’s biggest drawback). If someone provides all of this, they could be the next Google.