And all this business about apps is a real spoiler for suspension of disbelief. I’m clicking a link, expecting to learn more about what I was reading (that was certainly the author’s intent) but instead I get an ad for an app. If I seriously consider it, I’ve lost my train of thought. If I actually take the detour and install it, I’ve lost bigtime. The best way to minimize the loss is hit the Back button and skip it. But that’s a loss too. I clicked the link for a reason. And that was thwarted.

The main concept here is that if you provide links in your mobile application, then you should be linking to some information about the topic you were reading, not an advertisement for a different app. By itself, this complaint seems to be a bit of a stretch. However, he follows with another post a few weeks later that really explains his issue:

Every time around the loop, since then, the Internet has served as the antidote to the controls that the tech industry would place on users. Every time, the tech industry has a rationale, with some validity, that wide-open access would be a nightmare. But eventually we overcome their barriers, and another layer comes on.

Basically, Dave is seeing these mobile apps as trying to kill the web. If the app is trying to control interactivity by only supplying a “walled garden”, similar to what Facebook is doing, then the user really loses the power of the web. The core idea of linking gets lost in this scenario. However, what if the web is more than just links?

Prior to the last few years, people were very document-centric and reading HTML pages from a PC. Now, we have mobile devices that allow you to use the web whenever you want no matter where you are. We also had web applications that did not allow you to take your information elsewhere or integrate with other applications. Now, many applications use your Twitter or Facebook login in order to authenticate, and then post statuses or even create events in Facebook.

In order to support all of this integration, the web is evolving. The web is no longer just HTML. Last summer, I wrote about the rise of the API:

As the web evolves, much of that evolution will be powered by application APIs. Some of the APIs will be application specific, like the Twitter API, and others will be more generic like the various semantic web standards. All of these available APIs allow programmers to create more interesting applications, and potentially a new API layer on top of what already exists. What this means in the long term is that we are finally getting to the point where the semantic web had hoped to be, linking data between various applications and hopefully doing something interesting.

A web of links can be limiting when looking at applications. When looking at reading a news story, links make sense, but reading articles is only part of the web. By looking at the data available, we are starting to create a more interactive and informative web. Sarah Perez at TechCrunch thinks this could be moving towards a web of apps, but that post is more focused on mobile apps. As I said previously, mobile apps tend to be limiting in their own ways.

Think about the possibilities of using the various APIs that are available. I am not just talking about the social network APIs, I am talking about things like Freebase, a huge collection of data, or even data aggregator/providers like Gnip and DataSift. Using these services, developers can build interesting applications using data from various sources without worrying about what the Twitter API looks like.

These types of services provide a level of abstraction that developers are used to. First, you get the core APIs, then you start abstracting those APIs into something easier to use. Once concerns about low-level APIs are removed, developers can focus on being more productive with the new abstraction layer and start solving real problems. Once developers focus on real problems, that is when the real fun begins.

So, is the web just a bunch of links or are we finally getting a layer of abstraction on top of all of the web applications. This layer of abstraction is a normal evolution for developers, and we just need to ensure this is translated into more interesting applications.

Related articles from RegularGeek:

• Twitter Should Cut The SMS Cord
• A Look Back And A Look Ahead At 2012
• Plus Or Like, Choose Your Sharing Weapon

First, here are the comments I had on each language:

1. Lua – The language is seeing some good activity on GitHub and StackOverflow. It has a solid Tiobe ranking, but more importantly, jobs for Lua are becoming available. The relative trend for Lua is a great indicator that it is ready to go mainstream.
2. R – With data analysis and big data becoming a part of every web startup, languages catering to the data crowd will become popular. The job trend data will be the most interesting part to watch here.
3. Clojure – While not popular within the Tiobe index, it is popular in some programming circles. In particular, its job trend growth is showing that it could be posed for a big year of adoption.
4. Go – Only Tiobe really shows this as a popular language. It is not ranked well in the Dataist Tier, and job trends are not very reliable yet. However, with a parent of Google it may not need purely organic growth.
5. Erlang – This has been around for a few years and has decent trends all around. With the continued growth of real-time technologies like XMPP (eJabberd specifically), PubSubHubbub and others, 2011 could be a year where it becomes the next Python.
6. Scala – This is one of the interesting trends. Scala has a very good showing in the Dataist Tier, but is not really ranked in the Tiobe index. With the solid job growth trends and popular adopters (i.e. Twitter), Scala could gain a serious amount of acceptance.
7. Groovy – More than anything, I believe Groovy missed its window of opportunity. It is a solid scripting language with decent job growth trends, but how can it differentiate itself from Ruby, Python, Erlang and Scala?
8. Scheme – For whatever reason, this language refuses to go away and has even picked up some interest. Without decent job growth, we could be another year away from Scheme breaking out again, at least 30 years after it broke out the first time.
9. ActionScript – ActionScript is included in this list because of its high ranking on Tiobe and the Dataist Tier. However, the job trends do not point to good things for the language. It could be a temporary slump, so you should keep an eye on it next year.

• A is for Algol 60, which can be considered the father of structured programming languages.
• B is for BCPL, which we can blame for the travesty that is C.
• C is for Clipper, and we hope you never used it.
• D is for Dylan, which is like LISP without parentheses.
• E is for Eiffel, which has some cool features that still are not in Java.
• F is for Factor and Forth, because everyone loves stacks.
• G is for Go, as proof of what happens when Rob Pike and Ken Thompson get bored.
• H is for Haskell, one of a host of functional languages.
• I is for Inform, giving life to interactive fiction.
• J is for Java, because so few languages start with J.
• K is for KRL, a frame-based knowledge representation language.
• L is for LISP, because we love our silly parentheses.
• M is for Modula-2, mainly because you probably didn’t know there was a Modula or a successor named Modula-3.
• N is for NATURAL, which I have never actually seen used, just discussed.
• O is for Oberon, because Objective-C gets too much publicity already.
• P is for Prolog, progamming logic never seemed so fun.
• Q is for Qt, although it is more of an application framework.
• R is for REXX, an interpreted glue language inflicted upon us by IBM.
• S is for SNOBOL, one of the first string processing and manipulation languages.
• T is for Tcl, which eventually grew a Tk appendage.
• U is for UNITY, because parallelism is cool and flow control is for the weak.
• V is for Verilog, a hardware description language.
• W is for WebGL, bringing OpenGL to JavaScript.
• X is for XQuery, because even XML needs a programming language.
• Y is for Yorick, and all those graphs and simulations.
• Z is for Zsh, one of the many Unix shells.

Lastly, many thanks to Wikipedia for the “History of Programming Languages”, “List of Programming Languages”, “Timeline of Programming Languages” and much information about the languages themselves.

First, you need to figure out whether NoSQL fits your problem:

• Do your data needs include massive scalability in a system that supports a highly distributed set of nodes resilient to network outages and inconsistency? Then you might consider NoSql alternatives.
• Do you need a DBMS that doesn’t try to force you into static data schemas? Do you need the utmost of data model flexibility? Then you might consider NoSql alternatives.
• Is your database going to be filled with a truly massive number of objects with very simply query requirements? Is throughput an absolute priority down to the milliseconds? Then you might consider NoSql alternatives.
• Is your database going to be hit with a truly fantastic amount of write activity? Then you might consider NoSql alternatives.
• Would your solution benefit greatly from the utmost in maintenance and administration simplicity? Then you might consider NoSql alternatives.

If you talk to any traditional RDBMS developer, they will say that none of these points really require a NoSQL solution or that some of the issues are more related to design problems instead of data storage issues. Those talking points aside, there is some reason that these types of issues keep coming up. Traditional RDBMS systems that are being used at scale require significant administration and an administrator who is very good at their job. One of the reasons that people are using NoSQL solutions in these scenarios is that you do not need to be an experienced Oracle cluster DBA, you can just be a well-read software engineer.

One difference between RDBMS and NoSQL solutions is that you really need to understand how you are going to use the database. For an RDBMS, this was not a concern because they are meant as general data storage. The various NoSQL solutions have different purposes. Document stores are different than object stores which are different than the other storage models. The topic of choosing the appropriate type of solution is worthy of its own blog post, so I won’t cover it here. The important thing to note is that each storage model has advantages and disadvantages, so choosing the right solution can make your implementation much smoother.

However, there are other things you need to think about when researching NoSQL solutions. NoSQL is an immature market, so there are other issues that you need to be aware of. First, there is the anonymous rant about MongoDB that has been heavily talked about this week. The following is a condensed version of the issues in the rant:

• MongoDB issues writes in unsafe ways by default in order to win benchmarks: If you don’t issue getLastError(), MongoDB doesn’t wait for any confirmation from the database that the command was processed.
• MongoDB can lose data in many startling ways: Recovery on corrupt database was not successful, pre transaction log. Replication between master and slave had gaps in the oplogs, causing slaves to be missing records the master had. Yes, there is no checksum, and yes, the replication status had the slaves current
• MongoDB requires a global write lock to issue any write
• MongoDB’s sharding doesn’t work that well under load: Adding a shard under heavy load is a nightmare. Mongo either moves chunks between shards so quickly it DOSes the production traffic, or refuses to more chunks altogether.
• mongos is unreliable: The mongod/config server/mongos architecture is actually pretty reasonable and clever. Unfortunately, mongos is complete garbage. Under load, it crashed anywhere from every few hours to every few days.
• MongoDB actually once deleted the entire dataset: MongoDB 1.6, in replica set configuration, would sometimes determine the wrong node (often an empty node) was the freshest copy of the data available. It would then DELETE ALL THE DATA ON THE REPLICA (which may have been the 700GB of good data) AND REPLICATE THE EMPTY SET… They fixed this in 1.8.
• Things were shipped that should have never been shipped: Things with known, embarrassing bugs that could cause data problems were in “stable” releases—and often we weren’t told about these issues until after they bit us… The response was to send up a hot patch and that they were calling an RC internally, and then run that on our data.
• Replication was lackluster on busy servers: Replication would often, again, either DOS the master, or replicate so slowly that it would take far too long and the oplog would be exhausted (even with a 50G oplog).

Obviously, this is a long list of problems, but you will notice that the version numbers are varied, while MongoDB v2.0.1 has already been released and v1.8 was originally released in February 2011, which points out that some of this information could just be old. More than anything, the post seems like a list of issues over several versions. And that is part of the problem when reading posts like this. If you try to think what Oracle was like prior to v7, you probably don’t remember, but if you ask an experienced DBA they can tell you horror stories. Things were not all that rosy when RDBMS systems were trying to mature.

This is also the nature of software, it has defects and typically severe defects when the software is first released. NoSQL is still leading edge technology, so you have to assume you are taking some risks when you try to implement your storage solution with one of those products. To put this into more obvious terms, one of the most successful pieces of software sucked until it got past v3.1 and that was Microsoft Windows.

Michael Bolton has an interesting post regarding confusion as an oracle:

James peered over his glasses. “When you’re confused,” he said, “that’s a sign that there’s something confusing going on. I gave you a confusing product to test. Confusion might not be fun, but it’s a natural consequence when you’re dealing with a confusing product.” James was tacitly suggesting that Jon’s confusion cound be used as an oracle—a heuristic principle or mechanism by which we recognize a problem.

When the QA team is confused, that points to potential issues and should be used as a heuristic. The confusion could be from a few sources. First, the test script could be out of sync with the user interface in such a way that is not obvious how the tester should proceed. This can happen with any application that is evolving throughout the development lifecycle. Second, you could have someone that has not been trained on the application as the tester. This can quickly be remedied by providing the appropriate information on how to use the application. It could also point to an incomplete test script, meaning someone new cannot execute the test script without additional instruction.

Another reason the QA team could be confused is because the application is confusing. This is not a good sign and should be seen as a warning to the development team. Most likely there is a usability defect in the application. For some teams, usability defects do not exist and usability issues get added to the list of things to do. This is a big mistake when developing web applications. Almost a year ago I wrote about Krug’s usability rules:

The first two Krug rules of usability are very related:

1. Don’t make me think – as far as is humanly possible, when I look at a web page it should be self-evident, obvious, self-explanatory.
2. It doesn’t matter how many times I have to click, as long as each click is a mindless unambiguous choice.

I love the “don’t make me think” rule. If you design your application with this rule in mind, you will get past most usability problems. However, if you are a startup and developing for a large userbase, you may have slightly more concerns. Mark Evans wrote recently about a rule of thumb “I get it”:

But if you push aside the entrepreneurial enthusiasm, a startup’s success prospects depend on a compelling idea and, as important, the ability to quickly get potential users to say ”Yes, I get it”. This means being crystal clear what the service or product does, and the value propositions/benefits being delivered.

The product/service needs to fill a need or convince users it meets a need they didn’t know they had. Getting users on board has to be user-friendly and efficient. And the product/service has to delight.

If you are following these basic principles, your product should not be confusing. I am a big believer in making things simple for users, because it means they will enjoy (or at least not hate) using the system. This is obviously a good thing, but more importantly, users will look for other ways to use your application.

So, why should a “confusing” application have usability defects tracked? If an application is confusing, that means it is going to be difficult to use. When an application is difficult to use, people stop using it because they can not see the benefits to using the system.

Usability is also hard to define, but if you look at Krug’s rules you will see that the user should not have to find the right thing to do, it should be fairly obvious. But, what about prior to production? You don’t have typical users on the system, so how do you determine usability? One thing I have seen is that the length of the test script can be a good trigger for something being too difficult.

Let’s assume you have manual testing being completed. Ignoring the test case setup instructions, how many steps are required to test one feature? This is similar to the number of clicks heuristic, but it is the number of actions a user must take in order to complete a task. You could say the number is something like 7 but that is not entirely useful. I have used “Why” as my metric of choice. If you ask “why do I need to complete this step?” and there is no obvious or definable reason, then you likely are introducing unnecessary steps to the task. If you can eliminate these issues, your application usability should improve.

To really make your application simpler, make sure your QA team starts questioning things in the beginning of the process. Someone writes the test script, and it could be the QA tester themselves. The developer and the tester should go through the test script and ask “why” for each step. This will proactively remove any stumbling blocks in the plan and have a much more usable system after it passes QA testing.

Are there any tips or tricks you have for improving usability?

1. Your blog content can be syndicated to a site like DZone?
2. What is DZone?

First, blog content all over the world is syndicated to other sites, even the New York Times and Washington Post do it. DZone has several blogs that they syndicate including RegularGeek.com. The second question is more blasphemy in the development community. For many Java developers, Javalobby was the required reading site. Some time ago, Javalobby became the Java Zone at DZone. DZone has a bunch of other zones as well, like the Agile Zone, Web Builder Zone and Architects Zone.

I read a ton of content every day, and I pull several feeds from DZone. So, it came as a good surprise that DZone was launching an HTML5 Microzone. Again, you probably have two questions:

1. What is a Microzone?
2. Why is an HTML5 Microzone a good thing?

A Microzone is an area of DZone that does not quite warrant a full subsite yet. There are other cool Microzones, like DevOps and Solr-Lucene. Microzones are interesting because it allows you to find interesting topical content that may get lost in one of the bigger zones like Javalobby.

The HTML5 Microzone is a very cool addition because it is the hottest development area, and it is receiving a lot of coverage. The problem with HTML5 is that it does not fit cleanly with typical delineations of content. HTML5 is a conglomeration of new HTML tags, JavaScript APIs and CSS3 improvements. There is also the benefit that HTML5 is more mobile-friendly due to the combination of these new specifications, so there is a lot of hype regarding HTML5 on smartphones.

So, if you are looking to find good content on HTML5 technologies, take a look at the HTML5 Microzone at DZone. And maybe you will see some RegularGeek content there too.

As a good example of technology without a solution, we can look at the semantic web. I am a big believer in semantic technologies, and they will have their day, but there is not a big solution for semantic technologies yet. The foundation is still being worked on, and even big advocates do not talk about solutions yet. Disappointingly, I found a recent post that talked about the added value of the semantic web:

If I ask the question: “What added value(s) does the Semantic Web brings on the table?”. So, what are the benefits that companies and organizations would get from using the Semantic Web? I am pretty sure that after asking this question, I would get answers such as:

• You will instantly be able to traverse graphs of relationships
• You will be able to infer facts (so create/persist new knowledge) from other existing facts
• You will be able to check to make sure that your knowledge base is consistent and satisfiable
• You will be able to modify your ontologies/vocabularies/schemas without impacting the description of your instance records or the usability of any software that use it (unlike relation databases)

Obviously, if you need to talk about productivity, you are still not in the solution stage. Talking about what you will be able to do, as opposed to what you are doing, should be a clue that we are looking at a technology and not a solution. Don’t get me wrong, there are probably plenty of custom business applications that take advantage of semantic web technologies, but we are not really talking about several widely available solutions. Solutions and widely available applications for the semantic web technologies will come, we just have to wait a little longer.

This leads me to my next point, theory is not reality. How many times have you hear technologists start a sentence with “In theory it should work…”? When you look at software development, there is a lot of theory and much of your daily work is based on those theories, like the theoretical performance of a lookup in a HashMap. However, when theory is thrown around at higher levels, it starts to get concerning. This came to mind when I read a post by Rob Williams. He was reflecting on the 10 year anniversary of the Agile Manifesto:

The ten year mark just passed on the Agile Manifesto. It brought out quite a lot of commentary from the Tribe Elders, and frankly, most of it was depressingly insipid. It reached something of a crescendo for me reading Jim Highsmith‘s piece in Dr. Dobbs …, he basically said “we won, and if what we won ends up getting spoiled, blame the idiots.” … What, exactly was won?

Seriously, what was won and what was the competition? Given that agile is a process, and it gets spoiled by idiots, did you really win? Part of the idea behind agile development processes is knowledge sharing, meaning no single person should be the keeper of all of the knowledge about a particular area of the code. In theory, this should extend to the agile processes, like Scrum, as well. If that is true, can you really say that you have won if you things get “spoiled by idiots”? You may have won the theory competition, but in practice you need to make sure the idiots cannot spoil your efforts.

We see this in software development. In an application, you write code that should pass some QA, where this is just some unit tests or even a manual test plan. That code is expected to work for as long as the application runs. Developers know that they need to write code that is difficult to break, so there are plenty of guards in the application. Part of your job as a software developer is to ensure that the next person to touch the code can modify it easily and reliably. Code that is elegant but immensely difficult to maintain is not entirely good code. Part of the software lifecycle is maintenance, and you need to ensure that your code can be maintained. Idiots can always ruin the beauty of your code, but if it is easily maintainable, it becomes difficult for the code to be “spoiled by idiots”.

So, what was Highsmith talking about? Well, agile processes are now being heavily used and the waterfall model is being avoided in most cases. This is a good thing. However, the practice of agile still needs help. The theory of agile has been espoused for a decade, and now people need help implementing agile appropriately. We need to make agile processes easier for people to adopt, creating solutions that people can use to make the processes easier to adopt. Some solutions exist, and some have seen solid adoption rates, but until agile processes and solutions supporting those processes are in more than 80% of all software development teams, we still have work to do. Once the “agile solution” is assumed for a development team, you cannot say anything has been won.

Over the course of my career, I have dealt with outsourcing in many projects, and a common theme tends to appear. In an in-house project, if you make a few mistakes you can likely recover and deliver a successful project. This is typically due to the fact that communication is immediate, and solutions can be crafted with the help of many people. With outsourcing, and in particular offshoring, that communication immediacy normally does not exist. This creates problems by itself, but when you already have made some mistakes it becomes very difficult to deliver a successful project. So, how do you avoid making a ton of problems for yourself? First, there are plenty of blog posts extolling the virtues of communication with offshore teams, so I won’t go into details. Without open communication, most projects suffer and time differences with offshore teams just makes it more problematic.

From the more technical perspective, how do you ensure that you get what you were asking for? When using waterfall development processes, you need to take the time to create detailed functional specifications that the outsource team can translate into small tasks. Unless you are very familiar with the outsource team, meaning you probably hired them and have worked with them extensively, you need to have the work completed in smaller tasks in order to understand progress as well as to have a better handle on reviewable design and code. This means that the functional specifications will take some time to develop properly.

You may be wondering why I keep talking about waterfall processes. This is really to set a baseline expectation. Many companies still use these processes and have not transitioned to agile processes. Part of this problem is that people know and understand the waterfall model, but they do not entirely understand agile processes. When dealing with outsourcing, agile processes may actually make more sense.

For example, when you are using agile processes, Scrum in my examples, you have a short sprint, maybe 2 weeks, where you plan to complete a small number of stories. These stories will be broken down into tasks in order to complete the actual development. These tasks could easily replace the detailed specifications from waterfall processes because they are typically small effort problems. In most agile processes, a task is meant to be something that can be completed in a few hours. This is likely a good size for an outsourced resource as well. There will not be a lot of clarification needed for a 4-hour task. The daily standup or scrum meeting is the perfect opportunity to communicate with the outsource team as well, thus eliminating many of the communication problems that typically plague outsourced projects.

Another benefit of agile when outsourcing is the sprint itself. By having a 2 week sprint, you get code that you can demo and test. This allows you to have a much better idea of how a project is progressing, and is one of the major benefits of agile processes anyway. With outsourcing, the management of the project and status of progress is always difficult. Completed sprints ease the pain of the outsourced project management. Again, it is a forced communication point that you normally do not have with outsourced projects.

With the benefits of agile processes, it seems like outsourced projects would actually gain more than in-house projects. So, why it is that we do not see more agile processes for outsourcing?