The harsh reality is that in the tech world, companies prefer to hire young, inexperienced, engineers. And engineering is an “up or out” profession: you either move up the ladder or face unemployment. This is not something that tech executives publicly admit, because they fear being sued for age discrimination, but everyone knows that this is the way things are. Why would any company hire a computer programmer with the wrong skills for a salary of $150,000, when it can hire a fresh graduate—with no skills—for around $60,000?  Even if it spends a month training the younger worker, the company is still far ahead. The young understand new technologies better than the old do, and are like a clean slate: they will rapidly learn the latest coding methods and techniques, and they don’t carry any “technology baggage”.  As well, the older worker likely has a family and needs to leave by 6 pm, whereas the young can pull all-nighters.

Initially, I was going to leave this topic alone, but Dave Winer decided to complain as well:

If I can’t get into the game, I can’t imagine there’s much chance for most other people in their 50′s to play a role. Which is really fucked up. It’s probably the reason why we keep going around in the same loops over and over, because we chuck our experience, wholesale, every ten years or so.

The combination of the two struck a nerve. I only partially agree with either quote as they do not look at the real issues here. In particular, both posts talk about ages and startups. Startups are not a huge portion of the software development industry. They may be the beginnings of it, but there are tons of engineers slaving away in a cube for some corporation that has 50,000 employees. However, even in a large corporation, age can be an issue for a few reasons.

Age and Hours Worked

For startups, hours worked are extremely important. Sometimes developers will work 80 hour weeks for months in order to get the product shipped. Many people assume that this is a young person’s game. This is technically true, but not because the people are young. The main reason that older people cannot work long hours is because they typically have a family and kids at home. They may not want to work long hours because they want to see their children at night. Some of these family people will work long hours after their children go back to sleep, but this is not the majority of software developers. If your company requires long hours, and you do not want to work those hours, age is not the issue, the hours are the issue. Maybe you are just not cut out for the startup life.

Age and Cost

For a startup, cost can be a significant issue. Many veteran (10+ years) software developers will have a salary in the top 25% of the industry. This problem is also not specific to the startup world, though startups do feel the cost pinch sooner than a large corporation. Startups have a tendency to pay for significant benefits to offset a lower salary. Sometimes you can get stock options or maybe it is just free food. In either case, your startup salary is likely lower than your salary at an equivalent corporate job. The corporation will eventually cap the experienced developer’s salary as well. Experience is important, but sometimes people just become too expensive. If you are that person, you must realize that you will not always get a raise when you change jobs or get more experience. If you decide to stay in software development, you have decided to make some maximum dollar amount in your city. If you want more money, you need to get into management.

Age and Skills

The TechCrunch post states that “The young understand new technologies better than the old do, and are like a clean slate”. Dave Winer feels that we make the same mistakes over again because we are throwing away or ignoring experience by only hiring younger developers. The truth is somewhere in between. Experience is somewhat helpful when a product is first being developed, but that is not critical at the early stage. Experience is critical when the early startup gains more adoption and needs to become stable. Younger developers do not have the experience to stabilize a product, whether it is by scaling a product for millions of users or just dealing with all of the issues that software has after being used for some time. These skills come with experience. Younger developers can learn these skills, but typically the knowledge is passed on from someone with experience.

Older developers have always had the stigma of staleness attached to them as well. I think this stems from the long tenure at large corporations. The internet changed a lot of that as well. Now, older developers are starting companies of their own. Technology is significantly easier to learn today because of the wealth of information available. 15 years ago, your ability to learn a new technology depended upon how many books you could read and whether your company would send you to training. This would become very expensive, potentially over $5000 per year. Now, I can find better information, quick than before and it is free.

Age Is Just A Number

In reality, age is just a number. There are plenty of younger developers that have no interest in working at a startup, just like those family-oriented older developers. Older developers can learn new technologies without being hampered by whatever “technology baggage” they carry. They just need to decide to do so. There are some younger developers that learned Java in school and have no interest in learning anything else because it is stable and used at many large corporations. There are people of all ages that refuse to work for less than they feel they are worth. Some of these people are misguided, and others may be worth the money. In all of these cases, developers young and old may not be a fit for a startup. To be a fit for a startup, you must have the desire to be in a startup. If that means working long hours at a lower salary while learning all sorts of new technology, then you must be prepared.

Preparation and desire are not age related.

Related posts:

• Finding A Job That Fits
• Tips For Finding Your Next Software Development Job
• Should You Specialize in a Technology?

So, what do the trends from Indeed.com look like?

Obviously, JavaScript continues to grow like a weed, showing a more rapid increase in the last 6 months. Flex seems to have gotten out of the doldrums with a nice upward trend over the past year. PHP is also growing nicely as it continues to be used for major sites like Facebook. In the past six months, Python has started to distance itself from the remaining languages. Ruby and Rails are showing slow growth with Ruby finally separating itself from its Rails legacy. Groovy is still trailing the pack and not growing very rapidly, but it is still having a decent growth period.

Now, let’s look at the trends from SimplyHired.com.

As stated in other job trends posts, because the SimplyHired data is limited to 2 years, we really see the short term bumps. SimplyHired does not show JavaScript to have a dominant position over the other languages like its Indeed counterpart. Javascript is still the leader and has more consistent growth than Flex. Flex had a really good bump in the last quarter, but it seems to be coming down from that temporary high. The trend for PHP looks like it may be stabilizing a bit over the past year with only limited growth. The others, Python, Ruby, Rails and Groovy, show only limited or flat growth. In all of the languages, there was a significant downward trend from January through April probably due to the continued economic issues around the world. Overall, Indeed is showing much more growth than SimplyHired, and this should be concerning for people as you would not expect two job posting aggregators to have such different trends. Hopefully, this is not pointing to a slowdown in web programming jobs overall.

Lastly, lets look at the relative trends for job growth from Indeed.com. This shows an interesting perspective of the job trends, comparing language growth as opposed to percentage of all postings.

Groovy demand has continued its tremendous rise over the past 2 years. Groovy may not have the same number of postings, but this type of growth could mean that it is ready for more mainstream adoption. Ruby and Rails have also shown an increasing growth trend that is much higher than the others. This has not translated to a huge increase in jobs, but even this growth trend shows that Ruby is now able to stand on its own, and not require the Rails framework. Python seems to be pulling away from the other languages in the list in the past year. We could see a rapid rise of demand for Python in the near future if this growth trend continues. PHP, Javascript and Flex do not share the huge growth of the other languages, but they are the current leaders overall and it would be difficult to sustain 1000% growth over a long period.

Based on all of these trends, you can see that web and scripting languages still have excellent demand. The relative growth of Python and Groovy make them “need to know” languages, even if you just start using them for side projects. Javascript is something anyone involved with web applications should already know, and dynamic websites require significant amounts of it. Ruby, Rails and PHP continue to grow and are an excellent addition to any programmer’s toolbox.

Related posts:

• Web 2.0 Programming Language Job Trends – February 2010
• Web 2.0 Programming Language Job Trends
• Traditional Programming Language Job Trends

The philosophy of economics, and psychology, and morality, all overlap in studies that show how people will readily abandon moral responsibilities if they are given ways to avoid the stigma of doing so. This leads me to feel more justified in my belief that programmers do a poorer job of reading, understanding, and implementing a specification when someone else has the responsibility of verification.

To put this more simply, if you do not have the responsibility to test your own code, it will be of lesser quality. I have seen this in practice, and I would have to agree. Code that does not have unit tests almost always has more defects than code that is unit tested. So, why do programmers avoid writing unit tests? There are plenty of excuses, and the second article has a good list of them. From that post comes my favorite excuse:

It’s not my job to test code – I don’t know at what stage a software developer decides that he can just throw code over the wall. If your job title was simple Coder than maybe there’s an excuse. But as your job is to develop working software, you need to be able to say that your code is functional.

Read that last sentence again, “your job is to develop working software, you need to be able to say that your code is functional.” As a software developer, can you really say that you have done your job without providing unit tests?

If you are not writing unit tests, you are not providing any proof that your code actually works. You are doing yourself a disservice and any developer that maintains your code will complain at length about it.

Another classic excuse is that it takes too much time to write unit tests or there is not enough time in the schedule to write the tests. The schedule excuse appears a lot as project scheduling is never as good as we would like. When I first started writing unit tests ages ago, I believed that the schedule excuse was OK. As I continued through my career, I realized that there were more defects in my code that did not have unit tests. I also realized that fixing the defects was harder if there were not unit tests. I know this is anecdotal evidence, but when so many people in our industry say the same things, the anecdotal evidence becomes more than just coincidence.

An interesting side benefit of creating testable code is that the design typically ends up easier to deal with. I think this has to do with the fact that your code becomes more component-oriented. You need to break many of the dependencies that quick-and-dirty coding tends to introduce. If your code is hard to test, it could point to design flaws or at least unnecessary complexities.

So, if you are a software developer, even if you are just learning the trade, test your code. It is your job to show that your software is working. Quit complaining, get over it and start writing unit tests.

Related posts:

• Tools For Unit Testing Java Web Applications
• How Much Testing Is Enough?
• Survey Says: Developers Think Testing Is Failing

1. Caffeine – You need to find your preferred caffeine delivery system. Mine is black coffee, early and often. Others may prefer diet soda throughout the day. There are always the classics as well, Mountain Dew and Jolt.
2. Logic – As a programmer, you deal with logic constantly. Your true mettle will be defined by how you deal with the insanity (real or perceived) of the customer. The customer could be your project manager, the business analyst or the users. For any given project, you can count on someone sounding completely insane.
3. Hours – Expect to be working a lot of them. Programming is one of those jobs that refuses to follow the 9 to 5 path. As you get closer to your deadline, the hours you work will increase quickly. Just hope that you can work from home sometimes. Also, weekends and time off are for management, and you need to be available at all times.
4. Corporate Leash –  You may not have any power to make decisions, but that does not mean the company cannot give you a leash. The ability of constant communication is too much for companies to pass on so you will likely get a crackberry, a pager (people still use them), or some other company provided cell phone. No, your leash will not be an iPhone, this is not meant for fun this is business.
5. Blame – As a programmer, you are the low man (or woman) on the totem poll. Just remember that shit rolls downhill, the buck stops with you and if there is a defect it is obviously the programmer’s fault.
6. Sleep – More importantly it is the lack of sleep that you need to be concerned with. As your hours increase, you will find there is less time to sleep. You will have to call into a conference call at 7AM with your offshore team, or support a 5AM deployment on Saturday.
7. Skills – Your skills typically do not matter in the corporate environment. There are plenty of programmers that do not have jobs and could fill your position. Your expertise in the business domain is not important because there are business analysts that are supposed to be the experts in the business. You are replaceable, you are just a part in the corporate machine.
8. Odors – As the hours get longer and management reads something about the productivity of team rooms, the odors start to get worse. We are not talking about someone heating up their lunch. We are talking about the stereotypical, unkempt programmer sitting in a room full of other unkempt programmers. After some time, the room starts to smell like a camel, and it is not pretty.
9. 24/7 support – Yes, you will be pulled into support at various times, even for something like a customer service site. The application may not be life threatening or revenue threatening, it is just job threatening. So, your corporate leash will ring at the worst possible moment and you will have to get on a conference call with 10 people who have no idea how to fix the problem but need to be involved from the management perspective and one other person that might be able to help you.
10. Stress – Project failure is never an option. You cannot miss the deadline even though it was not the deadline you proposed. No, you cannot remove features from the application in order to make the deadline either. You need to deliver the entire application by the deadline, or the entire team could be fired. Nothing like a little pressure with your morning coffee.
11. Non technical managers – These people are considered the bane of most programmer’s existence. They need to manage a group of programmers, but they do not understand what they do for a living besides “they write code”. They typically overpromise, underdeliver and treat their programmers like cattle or “resources”.
12. Political battles – This is one of those things that programmers are rarely directly involved in. However, your project and even the whole team could become the casualty of some political war. You have no power to prevent any of it, but you are definitely involved. Keep listening to the rumors so that you have some warning that something bad could happen.

Obviously, these ideas will not happen everywhere and are much more of a worst-case scenario. If you see several of these things at your current employer or even a prospective employer, you may want to run the other way. Given that you already carry a cell phone anyway, don’t worry about the corporate leash, they will just want your current cell phone number.

Related posts:

• Are You Flirting With Burnout And Disaster?
• How Do You Hire Programmers?
• How Can Software Development Avoid The Need For A Hero?

First, let’s look at the job trends from Indeed.com:

The first thing you can see in this graph is that there is some growth in the past 6 months! This growth comes after a small downturn after late 2008. Java, C#, Perl and Objective C are all seeing nice gains. Surprisingly, Visual Basic is trending upward slightly for what seems like the largest gain in the past 5 years. C++ is mostly flat as the C# transition continues for Windows developers.

As we have seen in past job trends posts SimplyHired has different short term trends:

This graph shows a decline from a peak in November 2009 through April 2010. However, in the past two months SimplyHired shows an upward trend similar to Indeed. The trend for VisualBasic is a little different as it continues to be flat, and there is very little growth for Objective C.

The last trend report to review is the relative scaling from Indeed. This provides an interesting trend graph based on job growth:

These trends are significantly more interesting because it is based on growth as opposed to absolute number of job postings. Three major points of interest are obvious. Objective C is growing rapidly now that the iPod, iPhone and iPad share a common operating system in iOS. C# is also showing significant growth when compared to the other languages as well. C++ and Visual Basic trail the others as C++ continues a downward trend while Visual Basic has a small uptick in the past few months.

What does all this mean? Well, Java is still a solid choice when looking to get a job and it continues to grow. C# is growing but does not seem to be replacing C++ as it is only slightly decreasing in demand. Perl refuses to go away and even grows a little. I am also curious if the change in trend for Visual Basic is any indication of the future. Lastly, Objective C will continue to be an interesting trend to follow over the next year as development for the various Apple devices converges into a single OS with only differences in presentation.

Related posts:

• Traditional Programming Language Job Trends – February 2010
• Traditional Programming Language Job Trends
• Web 2.0 Programming Language Job Trends

Generally, wholesale re-engineering of a system is a bad idea, because the system still serves a purpose and its’ functionality is a moving target. If the system does not serve a purpose, then what is the point of rewriting it?

The main reason that systems do get replaced regardless of the advice above is corporate politics. What I mean is that in some cases, the politics involved with continuing to maintain a maligned application are costly. The better political decision could be to start over and just replace the old system. This should not be seen as an easy decision, but there is one topic that must be discussed in this situation, sunk costs. Fred Wilson had a fantastic post about sunk costs earlier this week:

Sunk Costs are time and money (and other resources) you have already spent on a project, investment, or some other effort. They have been sunk into the effort and most likely you cannot get them back.

The important thing about sunk costs is when it comes time to make a decision about the project or investment, you should NOT factor in the sunk costs in that decision. You should treat them as gone already and make the decision based on what is in front of you in terms of costs and opportunities.

The moral of the story is, when looking to replace a system, do not consider sunk costs. Many people in management will use sunk costs as part of the argument of build vs. buy. Generally, sunk costs are irrelevant to the discussion. If you are deciding to replace an existing system, mentally you have already written off the costs of the system. So, using the money spent on the existing system as part of the costs of building a new system is disingenuous. To put it in even simpler terms, let’s say you have an old PC that is running slowly but you have not maintained it well. If you decide to buy a new PC, do you include the costs of the old PC? I know it is not a great analogy, but it is the same general idea.

Now, if you can compare the costs of just rebuilding the system vs the cost of buying the system while ignoring sunk costs, you are finally comparing apples to apples. However, given the topic of my recent post, I would like to add a third apple to the comparison. How much would it cost to refactor and enhance the existing system to match the features of the proposed systems? Disappointingly, most management teams will not consider this third option. “We are trying to get rid of the system, why would we throw more money at it?”

For many enterprise applications, the cost of the software and support is in the millions of dollars. So, if a purchased application will cost you $5 million, and you are willing to pay that amount, why not consider talking to the team about modifying the existing system? People will talk about the political problems, but if an application works well people will not care how it got built. The other option, rebuilding the system is almost always a bad choice. There is the thinking that the team will have learned from the effort of building the first system, so the second system will be so much better. What normally happens is that the system gets over-designed, complicated and bloated with tons of features in an attempt to ensure that you do not see the failure of the first system.

What option would you choose?

Related posts:

• Survey Says: Developers Think Testing Is Failing
• Who Did You Learn From Today?
• 10 Questions to Ask When Choosing a Framework

Over the years I’ve read a lot of research papers looking for better ways of doing things. Sometimes I find ideas I can use, but more often than not I come up empty. The problem is there are very few good papers. And by good I mean: can a reasonably intelligent person read a paper and turn it into something useful?

That may not make sense at first glance, because you have not read an academic paper recently. Just head over to the ACM, search for an interesting research topic, and determine if you have an idea of how to use this new technology. Typically, these papers are typically unreadable for the average developer. Again, High Scalability has a nice quote:

There’s a lot out there in the literature that we could be making use of right now, but it’s closed off from the people, i.e., developers, who can turn this research into gold. And it’s largely closed off because researchers don’t consider developers as an audience and they don’t write their papers with the intention of being applied.

The problem here is that academic success is not seen as how many people have adopted and used the technology. Academic success is normally seen in the form of several research grants and tenure. However, academia is not the only group publishing information that is not immediately useful to developers, just look at the average technology specification from W3C. As an example, go to the XQuery Current Status page. Because I am currently working with XQuery, I understand that there are several parts of the technology. If you are a developer that wants to learn about XQuery, the last place you are going to would be the W3C page. Why? Look at that page again. You see that there 6 current standards:

• XQuery 1.0 and XPath 2.0 Data Model (XDM)
• XQuery 1.0 and XPath 2.0 Formal Semantics
• XQuery 1.0 and XPath 2.0 Functions and Operators
• XSLT 2.0 and XQuery 1.0 Serialization
• XQuery 1.0: An XML Query Language
• XML Syntax for XQuery 1.0 (XQueryX)

Try and read the Functions and Operators specification. You may be able to understand some of it, but it is not entirely clear how you are going to use it. Another issue with this is that there are 6 different specifications required to support XQuery 1.0. In addition to XQuery being a more niche technology, the barrier to entry is very high. Thankfully, there are books that help simplify the process of learning these new technologies.

Could we make some of these complex technologies more adoptable? Let’s take Java as an example. I have no idea how many specifications are required to implement the full Java and JEE stack without EJB. However, one of the main reasons Java gained significant adoption is that there is typically a reference implementation available. It may not have been immediately available, but fairly soon after the specification was finalized there was an implementation. In some cases, an implementation from outside of the standards body was chosen as the reference implementation. Tomcat is an example of this when it comes to the JSP/Servlet specifications, even though it was eventually forked for further specifications.

I have talked about the difference between action and ideas before. That is even more apparent in this divide between academia and developers. A book I am reading, Innovation: Need Of the Hour, has an excellent example. Sramana Mitra talks asked a question to the person who “popularized the idea that random accidents and uncertainties … determine the course of history”. Obviously, this is a bit controversial:

“What are you going to do about your thesis?”

His answer: “I don’t do. I just think and write.”

This is a serious issue when it comes to those people that say “It works in theory, but not in practice”. Theory is the basis of much knowledge that you have, but if the theory does not work in practice, then the theory is useless. There are plenty of theories or algorithms that work out fine, but in real world situations they become unusable. When you look at algorithms, some are CPU-bound, some are memory-bound and others may be I/O-bound. Which algorithm you choose is highly dependent upon your situation. In theory, any of these algorithms would apply. However, if there is a reference implementation of these algorithms, you can quickly determine which algorithm will work in your environment. This would increase adoption as more people can easily implement the technology.

Open source reference implementations are even better for developers. Many developers do not like reading books to learn about technology, they learn from reading, writing and executing code. An open source implementation gives the developer some code to read that actually works. The reference implementation does not have to be a reusable library, and it could even execute slowly, but if it exists, developers can learn from it and improve it.

So, the next time you have some interesting new technology in some academic paper, develop a reference implementation that people can use and look at. The adoption of that new technology could come very rapidly.

Related posts:

• What Does Data Portability Mean To You?
• Should You Specialize in a Technology?
• My Favorite Programming And Computer Science Books

Now that the system is in production what do you do? Somebody typically maintains the system, correcting defects and adding features. Slowly, that system continues to grow. However, the programmers and managers realize that the system does not really do what they want it to, or maybe there is a ton of technical debt that should be cleaned up. In many cases, people call for a complete rewrite of the system. If you are an agile development organization, this typically brings additional concerns:

Re-engineering is highly problematic from an Agile point of view. We are not producing functionality. We are not working from user facing stories. There may be business value to what we are doing (such as improving the performance of our application) but it is not directly measurable business value in most cases.

Re-engineering is also too often driven by technical concerns within the team (or its management). We didn’t build it as well as we wish we had. We have learned new things now, and we want to incorporate them.

Generally, wholesale re-engineering of a system is a bad idea, because the system still serves a purpose and its’ functionality is a moving target. If the system does not serve a purpose, then what is the point of rewriting it? So, if we follow the assumption that parts of the system were a good idea and scrapping the system is a bad idea, how do you modify the system? This is highly dependent upon what you are trying to change, and how agile your organization really is.  For our examples, let’s assume the organization is used to Scrum, and uses it in all software projects. How do we deal with the desire to re-engineer a system and resolving technical debt?

High Defect Rates

One way that a technical debt heavy application can present itself is when there is a high defect rate. This may not be a bad thing if there is a desire to rewrite the application. You can use the defect list as the list of user stories, and plan a few sprints to clean up the defects. However, the key is to include some time to analyze the areas or components that have the highest defect density. These areas will be the focus of each sprint. Your defects can become additional unit tests, so that you have your baseline tests as well as proof that the new code should also handle the defect situations. Once you have this safety net of unit tests, you can freely re-engineer and refactor these troublesome areas. If the application was “very buggy”, this type of work can quickly stabilize a system without taking on the pain of trying to rewrite the whole application and introducing different issues.

Replacing Old Technology

Let me start this by saying you should not replace old technology just because you think it is old or because there is something newer and cooler. You need a valid business reason, or management will not agree to the work. One good reason to replace old technology is when support for it is being ended (or planned), and the new version has some features that your application could take advantage of. In particular, you really want to attack this problem when there is a feature to add or there are some defects that need fixing. Hopefully, you can still use the existing unit tests, but if not, the unit tests will need to be modified (or rewritten) to work with the new technology. This is one of the more dangerous tasks because it is such a significant change. Typically, the code will be changing enough for this work to be considered a significant release, and will require all the trappings of a typical longer project. This also requires a significant amount of analysis in order to determine the system impact and the proper user stories. The risk in this type of project is proportional to the size of the technology being replaced. If you are replacing a library that is used for financial calculations, you may be able to organize the sprints in a similar manner to the defect correction example. However, if you are moving from Struts 1.1 to Spring MVC 3.0, you have a significant amount of risk and analysis to do, so plan accordingly.

New Feature Does Not Fit The Design

Sometimes, design decisions made at the beginning of a project may not have taken into consideration some requirement that will appear 6 months later. Because of that problem, designs will need to change over time. The problem here is that sometimes a new feature cannot be implemented with the current design. At one time I worked on an internal content management system. The CMS system used the concept of widgets to define prepackaged html tags with supporting javascript. This was very similar to JSP custom tags, but the original design of the system was completed several years ago, so the widgets were created, parsed and manipulated in the Java code. To add a new widget, which had only happened once after the initial design, you needed to modify the code in a few places as well as creating the new widget classes. Now, a new requirement has come along where a user of the system must be able to define their own widget with html and javascript. In this type of case, a new design was needed as the widget definitions moved from static definition to dynamic definition. As in the previous examples, the unit tests are very important as they provide a minimal set of requirements the new design must satisfy.

As in the old technology replacement example, this work will have risk proportional to the size of the components being replaced. In the CMS example, the design change of a core feature carried significant risk. One benefit was that the widget conversion allowed stories and sprints to be planned nicely. First, the basic dynamic widget design could be completed, and then future sprints would deal with the conversion of each static widget. You could actually run the two widget systems in parallel until the last static widget is converted. Once that is complete, you can freely delete the old static widget classes and marvel at your new implementation.

The key in all of these examples is finding a way to break each task into user stories so that you can plan your sprints. This allows you to get the project on a familiar path that even management can understand.

Related posts:

• Writing Unit Tests Is Your Job, So Quit Making Excuses
• Survey Says: Developers Think Testing Is Failing
• How To Succeed With Scrum When Your Company Is Anti-Agile