Regular Geek » Miscellaneous

Happy 4th Birthday To Regular Geek

Rob Diana — Mon, 12 Dec 2011 13:00:27 +0000

I had no idea when I started this blog that it would last past a few months. Now, four years later, I am still blogging albeit a little slower lately. This year saw a typical blogging funk late in the year, mostly due to a lack of time, but also due to fewer big announcements and more smaller product iterations.

So, the top 11 posts written this year were:

Why 11 posts? Well, the top post was written one day before the anniversary last year and most of the traffic occurred after the anniversary. The job trends posts continue to have a strong showing regardless of when they were written. The social posts still tend to appear but they are more programming related as well. This continues a trend from last year.

Other posts that were very popular from previous years:

My total statistics at the end of year three show that I was already writing long posts:

Months Blogging: 36
Posts Per Month: 11 (400 posts total)
Words Per Post: 851
Total Words In Posts: 339528

I continue to go against common wisdom and write posts that are even longer:

Months Blogging: 48
Posts Per Month: 12.3 (180 posts this year, and 580 posts total)
Words Per Post: 887
Total Words In Posts: 514328 (174,800 words this year)

This year saw the introduction of my list posts (93 so far) in the Geek Reading category, and the subsequent demise of those posts with the changes in Google Reader. I do hope to restart those posts when I can determine the best way to implement them. Ignoring the Geek Reading posts, my blogging was technically slower than last year, with 87 posts or just over 7 posts per month. I continued the trend of writing far too many words per post, but people keep reading.

The other side of the blog statistic picture is the external statistics. This past year, the blog had solid traffic growth and good subscriber numbers. Technorati continues to change how their rankings work where my ranking in Technology and InfoTech seems to range from 500 to 5000 and 100 to 2000 respectively. Google changed how their search engine worked again, and that seems to positively impacted traffic to the blog. Feedburner finally dropped FriendFeed from the subscriber numbers, and it recently peaked at 3010 subscribers.

The social landscape continues to change as much of the social sharing has been wildly different than previous years. I no longer see traffic from Digg or Google Buzz, but I now see traffic from DZone, LinkedIn and Google+. Facebook and Twitter look to be mainstays in social for any blog.

As always, thank you for your continued reading and social sharing. Hopefully, this blog will continue to be useful to you.

Changes Are Coming

Rob Diana — Thu, 28 Jul 2011 13:53:30 +0000

Changes are coming to RegularGeek! These are not really big changes, but they are something. Because some of the changes affect the content on this blog, I wanted to give you notice of what is coming.

First, I hope to finally update the blog with a new logo. I have had the logo for a while, but I need to tweak the design a bit before I update everything. There will probably be category and archive changes soon as well. The categories are not detailed enough, and will move closer to post tags. So, if you have tried to follow only the programming posts or just the social media posts, you may need to take a look at the new categories to see what to follow. The archives have been an annoyance of mine for quite some time. A simple monthly list works fine when you are under 18 months, but after that the list gets too lengthy. So, I will be looking into new ways to navigate the archives. Any recommendations on these items or the navigation are welcome.

Second, there will be a new type of daily post. I have talked about this in the past, but I will be posting a daily links post. These posts will be in a new category called Geek Reading, and will contain my daily Google Reader shares. I have debated doing this for quite some time, but not enough people use Google Reader to get those shares. So, a daily link post will get my “curated and filtered” choices published to all of the blog readers. For those of you not familiar with my sharing, there are about 35 posts per day taken from tech news, application development and a little bit of business. These posts will be published around noon Eastern time.

Lastly, I will be posting more about HTML5. HTML5 is not just HTML, as it also contains CSS3 and a whole bunch of new APIs. Of course, when you are dealing with APIs that means you are writing JavaScript. With JavaScript, I will probably focus on core JavaScript and some JQuery as that is what I normally use, though I could talk about other interesting libraries at times. More importantly, I wanted to ensure that you get content that talks about all aspects of web application development, not just server-side programming. I am more of a generalist when it comes to application development, so it makes sense for the content to deal with more aspects of web development.

So, that is what is changing here. Hopefully, the majority of readers will like the changes. As always, let me know what you think in the comments.

Guest Posts on RegularGeek!

Rob Diana — Sun, 06 Mar 2011 14:00:14 +0000

For quite some time now, I have been getting requests for guest posts on this blog. Well, I finally crafted guest post guidelines, so guest posts are now being accepted! If you are interested in guest posting, please review those guidelines. There are a lot of things that are important with guest posts, like what kind of formatting is expected as well as what you should be posting about. Those are fairly well covered in the guidelines.

There are some things that I did want to highlight here as well. First, I will not accept posts that are really just one big advertisement. I am trying to help people get more content, and advertisements do not really help anyone. In addition, the posts need to be relevant to the same sort of content that I normally post, so it should be about social media, programming or something like that.

Because I still write somewhat frequently, I will only be posting one guest post per week. I need to see how well this process will work, but that is how this will start. I am very particular about what content appears on this blog, so I also reserve the right to stop accepting posts or changing the guidelines at any time. Another type of post I will not accept are news-related posts. Basically, I can not promise to publish the guest post in a timely manner, so the posts can not be time sensitive.

With all of that said, I hope that you will submit a guest post for consideration.

The first post will be appearing on Monday, so keep watch on the blog for some new writers!

Happy 3rd Birthday To Regular Geek

Rob Diana — Sun, 12 Dec 2010 13:51:52 +0000

In this rare double-post weekend, it is time to celebrate the 3rd birthday of this blog. For a short time this year, I wondered whether the blog could continue as work continued to demand my time. There were periods of blogging funk, typically every late summer, when there is little news to talk about and just as little software development changes. Thankfully, I find blogging fun and an outlet for ideas, so it continues into the coming new year. Conveniently enough, this is also post 400, another simple milestone. I would like to thank all of my readers for making it through the year.

So, the top 10 posts written this year were:

The job trends posts had a very strong showing overall, and has become an ongoing series of posts. In particular, I will be looking more at trends in specific areas of technology over the course of the next year. There are some interesting technologies like the NoSQL datastores and HTML5 that look to make 2011 very interesting from the software development perspective.

Other posts that were very popular from previous years:

My total statistics at the end of year two show that I was already writing long posts:

Months Blogging: 24
Posts Per Month: 12.4 (154 posts in 2009)
Words Per Post: 777 (914 words per post in 2009)
Total Words In Posts: 231402 (140,730 words in 2009)

I continue to go against common wisdom and write posts that are even longer:

Months Blogging: 36
Posts Per Month: 11 (100 posts this year)
Words Per Post: 851 (over 1000 words per post! tl;dr)
Total Words In Posts: 339528 (108,126 words this year)

The other side of the blog statistic picture is the external statistics. Last year, the blog had some nice growth and good subscriber numbers. Technorati continues to change how their rankings work, and Google has tried to lessen the importance on PageRank, so I am really only left with Feedburner subscribers which have increased by about 1000.

Obviously, the blog has grown considerably again this year. Much of the growth is due to the various social services that are popular like Google Reader, Google Buzz, Digg, Twitter and Facebook. Even LinkedIn is starting to emphasize the sharing links more.

So, thank you to all of you. I will not make promises about making my posts shorter, I do not seem to be able to make a point in less than 800 words. This will likely get even worse as I get more into analysis of technology trends and job trends. I do hope you stick around as the information will still be as useful as ever, or not but please keep reading.

A Simple Introduction To Playing With Big Data

Rob Diana — Sun, 19 Sep 2010 14:00:13 +0000

With social media, big data has come to the forefront of technology. Whether you want to continuously search Twitter, aggregate the social activity on several sites, or do some mining of people’s activity on Facebook, handling big data is critical. There are two questions you need to answer when looking at a project that will handle big data. First, how is big data defined and when do I know I am dealing with it? The second question is how do I deal with big data?

How big is big data?

Big data thresholds change over time. This is really due to how well traditional storage mechanisms can deal with them. Part of the storage problem is hardware related, eg. can the disk store a file larger than 4GB? That question may not be a big deal now, but 15 years ago it was a major concern. Another question about size is, how well an RDBMS can store the data? Will the database crash if it tries to manage 100GB of data? Yes, 100GB of data in one database was huge before 2000. Technologies like database partitioning, where a large table was physically split and managed by the database engine, were still young. Now, even open source and free databases have partitioning and replication. The size of big data has increased dramatically as well. When people talk of big data, they mean hundreds of millions of rows in one table and a database potentially over 1TB, yes one terabtye. Even though big data is a hot topic, you have few opportunities to really interact with big data. For our purposes, lets assume you are going to aggregate data from social services in some way, otherwise this post would be fairly short and uninteresting.

How do you deal with big data?

One of the first questions when dealing with any database, big data or smaller data, is what are you going to do with it. Is your primary function search of the data? Are you going to try to analyze the data using typical data mining techniques? Are you creating more of a FriendFeed-like reading and browsing service? Knowing your target is very important as it will likely change the way your data is stored as well as affect your choice in technologies. One major assumption that I am making is that you do not want to spend money on expensive tools like Oracle, SAS or Informatica. So, what kind of tools and technologies do you need to look at?

Data Storage

Data storage is possibly the most important decision when dealing with large amounts of data. Traditional RDBMS software can handle huge amounts of data but sometimes require extensive knowledge to manage. MySQL can easily handle many data storage needs and it is well known by many developers. It is the easy choice for many people. However, there are a growing number of NoSQL choices that may also make sense. Some of the NoSQL options have very good text search capabilities, while others have been optimized for speed of reads or writes. Knowing how your application will handle data access helps refine this choice. Also, do not forget about the potential of a mixed environment where some data is in an RDBMS and other data is better suited to a NoSQL datastore. There is a large list of categorized NoSQL options at NoSQL-Databases.org.

Data Caching

No matter how well architected your data storage solution is, sometimes reads are just not fast enough. This will typically happen if you have a highly trafficked site, but maybe there is just some data that does not change too frequently. In order to squeeze as much speed as possible out of your application, you probably want some level of data caching in your application. The basic idea is that your data cache is on big hashmap stored in memory which allows extremely fast reads. This is much faster than traditional database access or basic file I/O. If you have paid any attention to web application development over the past several years, you have heard of memcache. Memcache is a data caching server that you can use with your application. This is one option you can take, but some people like to have more control over how data caching works with their application. In that case, you need to find a data caching library for your language of choice. For Java, there are several available, and some have been integrated into web frameworks like Spring. In particular, ehcache has good integration with Spring so you could quickly include data caching in your application.

Distributed Computing

If you take the NoSQL route, many of those solutions are meant to be deployed in a distributed environment. In many cases, the software will have an agent running on several servers in order to store some of your data on that server. The master or orchestrator (the terminology could be different) will be configured to know which agent to talk to for the requested data. This is a gross simplification of the process, but it should give you an idea of what to expect. Distributed computing has various potential issues as well. If one of your servers crashes, or even if you have to perform some maintenance on a server, how do you continue to retrieve the data stored on that server? Is the data replicated to multiple agents in order to provide simple fail-over capabilities? Do you need to provide your own clustering solution to support the data storage? In some cases, you may even feel that the existing software do not provide you with a good enough solution so you need to build your own. Distributed computing is at the center of many solutions when dealing with big data. Knowing more about how things work will give you a better idea of how to architect your solution as well as what failure points may exist.

Search

Search is a separate field entirely due to its focus on relevance and speed. Speed is critical in search because nobody wants to wait more than a minute for reasonable results. Thanks to Google, the longer a user waits, the higher the expectations will be. For example, if I wait a minute to get results from a search engine, I would expect that they would be highly relevant to my question. Google’s focus on speed with good enough results definitely changed how we interact with search. If your application will have significant search requirements, you need to look at your data storage to determine whether search is core to its function or whether you need an external solution. In years past, search was the domain of the RDBMS vendors, but the rise of the internet and Google has changed things. Search is not about finding the structured data in your database, it now looks at anything on the internet. There are various search projects on Apache that deal with various levels of search. Lucene is the core search engine index software and can be considered a low-level search technology. Solr, using the Lucene libraries, provides search through web services in order to keep search as a distinct application outside of your application. Solr and Lucene are focused on keyword searches just like most search technology. Nutch, also built on Lucene, is Apache’s answer to web crawling, so if you wanted to search the contents of various web pages, this is the solution for you.

Probability, Statistics and Machine Learning

If you decide to do any analysis of your data, there is a host of information you may need to review. If you are planning to graph trends or even simply report on your data, then you need a basic understanding of some simple statistics. You do not need a deep understanding, but even gaining knowledge of standard deviations could prove valuable. If you decide that you want to take your trends a step further and look at expected trends or even simple prediction, probability will rear its ugly head. Just like statistics, some simple concepts in probability will go a long way for many web applications. However, there are times when statistics and probability do not give you the results or the functionality that you desire. At that point you will need to delve into the realm of machine learning. This is not an idea that should just be jumped into as machine learning uses some advanced statistics, probability and mathematics to show how things work. In some cases, you may be able to treat things like a black box and implement an algorithm for simple categorization, like naive bayes, but it may not give you the results you desire. In those cases, you may need to understand more of how these machine learning algorithms work in order to determine what the best approach may be. This may be a difficult area to understand, but you can do some amazing things with machine learning. How cool would it be to personalize your site based on the user’s past behavior without the user needing to explicitly select categories or keywords?

Do I need a PhD to do all this?

Typical databases are easy to work with. You can use a GUI to create a database and some tables. You can write a query to get back information. Big data changes everything and there are a lot of technologies that try to make things easier. Thankfully, you do not need a PhD to work with big data, because many tools and libraries have been created to make these technologies more accessible to the typical developer. Sometimes more advanced knowledge would be helpful, but in many cases you might be able to treat the technologies as a black box, just like your old RDBMS. You might also think that your case is special and nobody has done anything like it before. If you are developing a web application, I highly doubt what you are doing is really unknown. It may not be known to you, but there may be academic papers explaining things or even solutions in an unrelated field. Big data did not start with social media, it really started in financials, pharmaceuticals and health information. So, if you can’t find something specific to fit your needs, broaden your search and the information is probably out there.

Helping With Digital Curation

Rob Diana — Tue, 27 Jul 2010 17:56:49 +0000

I have previously talked about digital curation and that you should follow my shares in Google Reader. The problem is that only some of you use Google Reader, some use Google Buzz and everyone else is using Twitter and Facebook. I do not want to clutter my Twitter stream with my 40 shares per day, so I am thinking of taking a slightly different direction. I also did not want to push daily content down my RSS feed without seeing how the readers of this blog felt about the idea.

So, I am thinking of creating a daily post that is the list of articles that I am sharing from Google Reader. This is a typical links post, but there will be somewhere between 30 and 40 links per day. The post will probably be published in the morning around 9AM US-Eastern time. Please vote in the poll below. If you have other comments, feel free to comment on this as well.

Note: There is a poll embedded within this post, please visit the site to participate in this post's poll.

Regular Geek Gets A Redesign Mostly

Rob Diana — Sat, 10 Jul 2010 12:33:23 +0000

This post is just a quick note to let people know that the blog has visually changed. There is a new layout, not completely different than before, but definitely different.

One thing you will notice on the right side are the new social images. There are convenient buttons you can use to follow me on Twitter, Facebook or Google Buzz. You can still find me on other sites that I am not as active on the About page. There are also big buttons for following the RegularGeek blog on Twitter, via RSS or Email. The RegularGeek Twitter account currently has only posts from the blog, but I am looking for interesting ways to use it if you have any ideas.

Another change I wanted to point out was the social sharing options on each post. Beneath the post title you can see four icons that you can use to share RegularGeek posts with Twitter, Google Buzz, Facebook and Digg. These icons also appear at the bottom of each post as well. Feel free to share the posts on other social sites as well.

Over the rest of the summer, other things will be changing as well. I will eventually have a logo, but that is still being worked on. Some of the navigation will change, and the static pages will be changing as well.

So, if you are reading this post in an RSS reader, please take a look at the site to see the new changes. As always, let me know in the comments if you like it or hate it.

What Interview Questions Are Helpful In Hiring Developers?

Rob Diana — Fri, 07 May 2010 17:46:08 +0000

Interviews are always an interesting topic, and for software engineers the interview process can take many forms. First, there is the standard interview where you are asked questions regarding the technologies that the potential opportunity uses. Those types of questions are always useful because you can find out if someone is lying on their resume or just how much they really know about a given topic. On the other end of the spectrum, there are the puzzle interviews that Microsoft and Google have become famous for, even though Google no longer uses puzzles. Somewhere in between are those interviews where you get a significantly different programming or design question that is meant to take 15 to 30 minutes of the interview.

I have talked about interviews, job searches and hiring before. In once case, I talked about finding a job that fits from the perspective of the job searcher. Those same questions can be asked by the interviewer to determine whether the prospective employee would be a good fit for the organization. As I stated in that post, some people can thrive in the small company atmosphere, while others require the structure of larger companies. However, those organizational questions still do not get to the heart of the problem. Is this person someone we want to hire? Will this person be productive in our environment? One thing I should note, if your company only hires due to specific technology needs, then you do not need to read any further. Those interviews typical focus on specific technical knowledge, and the rest of this post is not about that. This post is more about those weird puzzle questions.

Are puzzle interview questions helpful in hiring developers? This is an interesting question because so many companies still use puzzle questions in their interview process. The idea behind these questions is that you are supposed to get an idea of how the person thinks through a problem. However, many puzzle questions do not require the interviewee to think for very long. They either get the answer right away, or they require a few hints. A good example of a puzzle question is The Rope Bridge. It is one of the more relevant puzzles, but how much does it help you determine whether your interviewee will be a good developer in your organization?

The other side of this are the “fish out of water” questions. What I mean is that you have the interviewee explain a system for something they have not thought about before. These questions are more easily applied to application design. For example, one design question I have seen is the “elevator system”. Unless they worked in embedded systems, it is unlikely that they have run into this type of situation before. This type of question gives you an idea of how they think through an unknown problem domain. The only issue with something like the elevator system is that people may have problems determining the objects and behaviors within the system. Sometimes an unknown domain means that there will be more questions than answers.

Finally, there are the programming questions that people use. The utility of these questions vary greatly as the complexity is typically varied as well. You can start with fairly simple questions like “FizzBuzz“, where you write a function that prints “fizz” when a number is divisible by 3, “buzz” when the number is divisible by 5 and “fizzbuzz” when the number is divisible by both 3 and 5. The solutions is quite simple, but it does tend to stump a lot of people. Personally, I do not like “FizzBuzz” because it does not tell you much about a senior level engineer. The problem with simple questions is that interviewees may think they are harder than they really are. Just a few months ago, I wrote about hiring programmers and one statement is very relevant to this discussion, “Most people tend to get very nervous during an interview which makes even simple programs somewhat difficult to write.”

So, if simple programs do not tell you enough, you need something a little harder. For some companies, if you pass an initial phone screen they may ask you to complete a programming assignment or it could even be the entrance criteria for the interview process. Recently, I saw one of these questions on JavaLobby:

By starting at the top of the triangle and moving to adjacent numbers on the row below, the maximum total from top to bottom is 27.

5
9 6
4 6 8
0 7 1 5

I.e. 5 + 9 + 6 + 7 = 27. Write a program in a language of your choice to find the maximum total from top to bottom in triangle.txt, a text file containing a triangle with 100 rows.

This is an interesting problem although it may not be entirely relevant to your future job. One issue with this type of programming question is that you do not see the thought process of the interviewee, and the problem could be too difficult for a 30-minute segment of an interview. So, you need to find something that could be completed within 30 minutes and give you a solid feeling for how competent a person is. One question I have run into that fits this nicely and does not require any domain knowledge is the following:

You have multiple lists of numbers that may or may not be sorted and need to be sorted into one large list. How do you do this?

In my experience, this problem takes anywhere between 15 and 30 minutes and it requires the person to write some code on a whiteboard. It is an interesting question for a few reasons:

it is definitely solvable within 30 minutes
it does not require advanced algorithmic knowledge like AI or machine learning
it can require code without being too extensive
it can lead to other talking points like performance and memory consumption.

The key thing to remember is that you need to be comfortable working with this person as well as being comfortable with their level of knowledge. Finding the appropriate level of detail and complexity in interview questions is like a dark art. Some people figure it out after years of study and refuse to tell you their secrets.