As a software engineer, I was trained to think about the worst case. What is the worst possible performance for the algorithm used as specified in O(n), “big Oh“, notation? If you had formal education in computer science, you have been asked this question. Given that we are supposed to think about the worst case, why are there so many problems in the software development industry?

One post that got me thinking about this problem comes from Alex Payne, where he demands that people don’t be a hero.

Every team I’ve ever worked on has had a hero. You’ve probably worked with one too: the guy or gal eager and willing to pull all-nighters, work weekends, and take over on-call duty when nobody else wants to…

Because of this type of effort, this person has praise heaped upon them. Sometimes the company may even acknowledge their efforts publicly. Sometimes this praise is a good idea because that person may have save the company millions of dollars. However, should the hero always be praised? Alex does not think so:

The hero is celebrated. The hero is a role model…

Here’s the thing: the hero is the most damaging person on a team, particularly on a team that’s supposed to be writing high-availability or otherwise mission-critical software.

I may not entirely agree that the hero is the most damaging person on a team, but there is a significant problem with the hero. The hero treats the symptoms of the project, like lateness or defects, but the hero ignores the disease. Why are there so many defects? Why is the project so late? In some cases, it could just be a bad team. Sometimes a team just does not work well together or can not be productive. It happens. There may also be unproductive or unskilled members of your team. Typical hiring practices do not always give you an insight to the type of skill someone has or what type of worker they are. The hero will help the late project suddenly get back on schedule. This means that when a project review is completed, people will just remember that heroic effort saying “Without that we would have been late!”

Why Do We Need A Hero?

So, why do we so often need a hero for our projects to be completed successfully? There are some cases where people have severely underestimated a task, or there is a new technology that took longer to learn than expected. In my experience, those issues do arise but not as often as poor scheduling. Poor scheduling comes in many forms. There is the simple bad estimation problem which all engineers have. Sometimes we use different units of work when we estimate as compared to how the work gets done. For example, a task may take 8 hours, so we add it to the plan as taking an engineer one calendar day to complete. In almost all cases, the engineer does not code for 8 hours per day. The real number of coding hours is probably 6 per day or even as low as 4 depending upon the company. Obviously, blindly adding task estimates to a calendar does not provide a good or valid plan.

Estimation and planning has become such an issue for software development that people have written blog posts that were meant to be humorous but contain too much truth to really be funny. I was reminded of one such post by a colleague last week. Liz Keogh wrote about Estimation Anti-Patterns about two months ago. Much of the post is somewhat funny, but I have been involved in projects that used some of these types of estimation methods. In Liz’s post, underestimating tasks is typically due to Fractal Estimation:

Some stories may be considered too large to fit into an iteration, so the team decide to split them into the smallest possible deliverable slices, re-estimating each slice. Of course, this is a bit like measuring the coast of Britain. The epic which was 13 points at project inception becomes 16 points of features, and 23 points of stories by the time it’s done in the sprints.

Another often seen estimation and planning problem is similar to Liz’s Target Estimation:

PM: How many story points can you manage this week?
Team: 10.
PM: That’s not good enough. I need you to do 12. How many can you manage this week?
Team: 12.

In some cases, you will see this manifest in “Contract Driven Estimation”, where someone sold the new and not yet developed feature, as well as the date when the feature would be in production. Other projects may also miss deadlines due to the “Shortened Due To X Estimate”. X could be the budget available or some other future constraint. The problem is that the work is not lessened, just the project plan changes.

How Do We Avoid The Need For A Hero?

I have not painted a pretty picture of software development up to this point. Software development is hard, and the accompanying tasks of estimation and planning are just as hard. The reason for this is the complexity involved and the lack of physical rules. For example, in mechanical engineering, there are many physical rules that you must follow, so at the basic task level, a majority of engineers will be able to do the same job in approximately the same time. Those engineers can differentiate themselves through design of complex systems or even just finding a better way to do things.

Software engineering does not have those boundaries. Some people say that it is more art that science. Because of the lack of these restrictions, estimation of tasks is not only inexact, it is almost impossible to consistently do well. Agile methods were born out of the frustration of various software development methodologies. Much of this frustration came from the missed deadlines of longer projects. These agile methods do not improve your estimation abilities at the beginning of a project, but they may use the history of the team to more accurately predict how the project is actually progressing. By using the amount of work completed in the most recent iteration, you can better estimate what will be completed in the current and potentially future iterations. Typical planning processes do not give you this ability. Typical planning assumes that productivity is constant, while we know that a single person is not consistently productive at the same level for a long period of time. A team will have ebbs and flows of productivity among the various iterations.

So, in order to leave you with a few concise bullets that could help improve your development projects, I will try to summarize the ideas:

• Personalities are important, so make sure that your team can work well together.
• Software engineers are not replaceable parts, some people are just that much better than others.
• Estimates of tasks are likely not equal to actual calender hours. Ensure you know how they translate in your company.
• Agile planning uses recent team history, “yesterday’s weather”, to determine how the team will perform in current and future iterations.
• Avoid planning back from the deadline. Do not pick a target date and try to figure out how to make the date, it normally will not work.

If you follow this basic advice, it is likely your project will go smoother than it has before. I can not promise better results for your team, because your team is the one doing the work. Also, I know this is not a thorough investigation of why software development projects fail. This is based on personal experience and various things I have read over my career. Are there some major issues or great advice that I missed? Feel free to add your thoughts in the comments.

http://static.zemanta.com/readside/loader.js