Given what you normally read on this blog, this may seem like an oddly timed post, but I have run into serialization issues lately in some production code and Mark Needham recently wrote an interesting post about this as well. Coincidentally, Mark is also working with MarkLogic, and there is an interesting item in his post:

The advantage of doing things this way [using lightweight wrappers] is that it means we have less code to write than we would with the serialisation/deserialisation approach although it does mean that we’re strongly coupled to the data format that our storage mechanism uses. However, since this is one bit of the architecture which is not going to change it seems to makes sense to accept the leakage of that layer.

The interesting part of this is that he has accepted using the data format of the storage mechanism, XML in MarkLogic in this case. Why is this interesting? First, it is a move away from the ORM technologies that try to hide the complexities of converting data into objects in the RDBMS world. Also, this is a glimpse into the types of issues that could arise from non-RDBMS storage choices as well as how to persist objects in general.

So, an RDBMS is typically used to map object attributes to a table and columns. The mapping is mostly straightforward with some defined relationship for child objects and collections. This is a well-known area, called Object-Relational Mapping (ORM), and several open source and commercial options exist. In this scenario, object attributes are stored in a similar datatype, meaning a String is stored as a varchar and an int is stored as an integer. But, what happens when you move away from an RDBMS for data persistence?

If you look at Java and its session objects, pure object serialization is used. Assuming that an application session is fairly short-lived, meaning at most a few hours, object serialization is simple, well supported and built into the Java concept of a session. However, when the data persistence is over a longer period of time, possibly days or weeks, and you have to worry about new releases of the application, serialization quickly becomes evil. As any good Java developer knows, if you plan to serialize an object, even in a session, you need a real serialization ID (serialVersionUID), not just a 1L, and you need to implement the Serializable interface. However, most developers do not know the real rules behind the Java deserialization process. If your object has changed, more than just adding simple fields to the object, it is possible that Java cannot deserialize the object correctly even if the serialization ID has not changed. Suddenly, you cannot retrieve your data any longer, which is inherently bad.

Now, may developers reading this may say that they would never write code that would have this problem. That may be true, but what about a library that you use or some other developer no longer employed by your company? Can you guarantee that this problem will never happen? The only way to guarantee that is to use a different serialization method.

What options do we have? Obviously, there are the NoSQL datastores but the actual object format is the relevant question not which solution to choose. Besides the obvious serialized object, some NoSQL datastores use JSON to store objects, MarkLogic uses XML and there are others that store just key-value pairs. Key-value pairs are typically a mapping of a text key to a value that is a serialized object, either a binary or textual format. So, that leaves us with XML, JSON and other textual formats.

One of the benefits of a structured format like XML or JSON is that they can be made searchable and provide some level of context. I have talked about data formats before, so I won’t go into a comparison again. However, do these types of formats avoid the issues that native Java object serialization has? This is really dependent upon what library you are using for serialization. Some libraries will deserialize an object without any issues regardless of whether the object field list has changed. Other libraries could have problems depending upon whether a serialized field exists in the target object, or there might not be solid support for collections (though that is doubtful at this point).

Given that even structured formats could have serialization issues, is the only safe path hand-coded mappings like those used by ORM tools? Some JSON and XML serialization tools use the same mapping methods as the ORM tools in order to avoid these problems. However, once you define these mappings, you are explicitly stating how an object gets translated. This explicit definition will require maintenance, but that is definitely cleaner than trying to trace down a serialization defect in some random stack trace.

So is implicit object serialization really worth the potential headaches? Or should we just consider it evil and never speak of it again?

A few days after Norm’s post, two other posts appeared refuting his stance even though in many comments it is considered a non-debate. First, Manu Sporny talked about the move to JSON being more of a paradigm shift to simpler markup. The complaints about SOAP and XML Schemas are obvious, but he complicates the argument by introducing JSON-LD into the conversation. JSON-LD introduces syntax for JSON to denote LinkedData, and there is notation very similar to XML Namespaces, to which Norm replies “Wow. By the time you start doing that, you’re sure you wouldn’t be better with a richer markup vocabulary?” Lastly, James Clark throws his opinion into the mix. His commentary is more about the fact that XML is losing web developers which could be a bad thing.

Now that you have the background story, I wanted to state that people are missing Norm Walsh’s original point. This problem is about context and it is not really being treated that way. People are using JSON for web development because there is almost zero learning curve. It is used because of the increasing trend of JavaScript heavy sites to drive interactivity and some of the mashup creativity. Because Twitter’s is readily available, people have created widgets to display their tweets on a web page. For many web developers that means grabbing a JSON representation of some tweets and converting it to HTML. This process barely takes longer than trying to find the correct API documentation.

In this same context, if the APIs are XML based you then need something to parse the XML into an appropriate JavaScript object. You can already tell that this is getting more complicated than simple JSON. To make matters worse, past browsers handled XML differently and sometimes very poorly. Because web developers were depending on the XML support in the browser, the problems of cross-browser support arose again. Obviously, developers do not want to go down that path and JSON is easier anyway.

However, what if you are using PHP or Java on the server? PHP has plenty of XML handling libraries, with SimplePie being a hugely popular RSS feed processing library. If you can make the Twitter API call from your Java server code, there are plenty of libraries for handling XML there as well. So, in that context XML may be a better option.

If you look at the type of problem, this could also change the data format. In the Twitter API example, if you have a simple widget that just displays tweets, then a solution based on JavaScript and JSON makes a lot of sense. What if that widget needed to be more dynamic? For example, let’s say that someone registered on your site will see those tweets displayed differently and there are additional links in the tweet for replying or retweeting. You could write some JavaScript code to just check for registered users and generate different HTML, but this gets unwieldy if the number of different displays grows beyond 2 or 3. If the data is in XML, then you can write different XSLT scripts for each display which remains separate from the main widget code. You just need to select the appropriate XSLT based on the user interacting with the site. At this point people are likely going to complain about the use of SOAP for web services and its complexity. Let’s ignore that option as REST has won, and SOAP is overly complex, can we agree on that and move on?

As with any programming problem, different requirements and different contexts may call for different technologies. If you get stuck on saying that JSON is better than XML (or the other way around), you lose another tool in your toolbox.

The other context that people are missing is why Twitter and Foursquare chose to support JSON only. This is likely a question of application complexity and analytics. Like any good API provider, Twitter is probably tracking all calls to the API and this includes the data format requested. It is very possible that the demand for XML was fairly low and it did not warrant separate support. In addition to this, there are plenty of JSON processing libraries available for mainstream languages like Java, so there was little risk in dropping support for XML. If there is no support for XML, then their API becomes simpler to support. That means less code to maintain, simpler maintenance of code because there are not multiple representations of one set of data, and fewer questions about the different formats.

So, quit whining about whose data format is better. Each one is better in a different context, otherwise it is highly unlikely that they would have become so popular. The important thing is to learn both formats, and other popular ones that appear, that way you can make an educated decision on which format to use in your situation.

I have started to see some development trends recently, one that makes complete sense to me and another that took me by surprise even though I should have connected the dots. I am not the only person seeing these trends either. O’Reilly posted a list of the top books from a cyber-monday promotion, and the trends should be fairly obvious:

1. Data Analysis with Open Source Tools
2. Head First Python
3. HTML5 Mobile Web Development (O’Reilly Training and Video)
4. JavaScript Patterns
5. REST in Practice
6. Making Software
7. Cassandra: The Definitive Guide
8. Hadoop: The Definitive Guide
9. HTML5: Up and Running
10. Learning Python

So, we have 3 books about big data, 4 books about client-side web development and 2 about Python. The trend that makes sense to me is big data. With all of the web activity, ecommerce or social media, there is a ton of data floating around. This data is worth billions of dollars to marketing people, and we are already seeing that it can help in other segments like customer service and even charities. Even the known big data problems, like search and web analytics, are still fairly young technologies. For many years it has been difficult to gain experience working with big data, but the rise of social media and the application APIs have given almost any programmer free access to larger amounts of data to play with.

The other trend that I was not paying attention to was client-side development. I had seen all the pieces falling into place but did not put everything together. The pieces are JavaScript frameworks like JQuery and YUI, RESTful web services and HTML5. Taken separately, they are all very useful technologies. If you put them together, you get a very interesting web application platform that is very different than the traditional MVC model. Today, this may seem obvious, but because of the traditional separation between client and server programming, the evolution occurred in parallel. REST and the proliferation of application APIs really made AJAX calls become much more interesting than simple auto-suggest text boxes. The combination RESTful web services and a JavaScript framework allowed web developers to create real dynamic content from various web applications, like Twitter widgets, RSS widgets and others. HTML5 will make these dynamic capabilities even more interesting.

Is this all saying that MVC is dead? Not exactly, MVC is just evolving. First, the model and view are still very much present, as the model is only changing from a Java object to a text-based object in JSON or XML. Obviously the view still exists, it is just being built dynamically, and potentially rebuilt based on user interaction. The controller is the part that is really changing because of the benefits of web services. If you look at Spring MVC, there is still a very strong coupling between the controller and the view. This coupling was due to the page-based model of web interaction. With RESTful web services, the web moved to a data-centric model where pages are optional.

If you review the list of books again, you will see that Python is represented twice. This is not as obvious a trend as big data or client-side development, but it is still surprising for a list of top 10 books. I was surprised until I read why Python is in the list. In the O’Reilly post, there is a link to an explanation of data science (a very good read) and that article talks about the use of Python within the big data community. The reasoning is that data needs to be massaged or “conditioned” as the article states. When dealing with this type of unstructured activity, scripting languages are used widely. Ages ago, Perl was the language of choice for this type of thing, but in the past 10 years Python has grown significantly in popularity. In addition to its usage in big data and as a general glue language, Python is also being used for server code where Java has been the standard language.

If you put all of this together, this list of books is a very good sampling of web development’s future.

Disclosure: All book title links are Amazon affiliate links. If you buy something I will probably make a few pennies.

Java is still a great choice for app dev teams that have developed the architecture and expertise to develop and maintain business applications. It is also an excellent choice (along with C#) for software vendors to develop tools, utilities, and platforms such as BPM, CEP, IaaS, and ECP.

He goes on to state that “Java development is too complex for business application development”. Depending on your point of view, I could agree with this, but some of his points just do not make any sense. “Business requirements have changed”, “Development authoring is limited to programming languages”, “Java frameworks prove complexity” and “Java has never been the only fame in town” really talk more to the state of software than any criticism of Java itself. My favorite point is “Java is a 20 year old language based on C++”, so obviously it is not suited for enterprise applications. That is like saying C++ is a 30 year old language based on C and meaning that it is not suitable for writing an operating system or a game. What does one have to do with the other? There are two points that do make an interesting case for the future of Java:

• Java bungled the presentation layer. Swing is a nightmare and JavaFX is a failure. JSF was designed for pre-Ajax user interfaces even though some implementations such as ICEfaces incorporate Ajax. There is a steady stream of new UI approaches reflecting Java lack of leadership in in the presentation layer.
• Java’s new boss is the same as the old boss. Oracle’s reign is unlikely to transform Java.Oracle’s recent Java announcements were a disappointment. They are focused on more features, more performance, and more partnerships with other vendors. So far, it appears that Oracle is continuing with Sun’s same failed Java policies.

Java and presentation were never a happy couple. Thankfully, AWT was not even mentioned. Oracle’s stewardship of Java has not started well, though I am sure things will get better. The long term question is still there, what innovations can really be added without being obviously bolted onto the language? There is another way to look at this as well. What if we ignore these two issues? If Java did not change and people could only add features if they build there own library, would that make a big difference to people? Java is a fairly mature language, and does not really need additional core features. Cool features could be added through other frameworks, which would mean that the long-term leadership would not really matter.

The presentation layer has long been separated from Java in terms of actual implementation, so that is not a real concern either. Many application user interfaces are being developed within a browser, and this trend will likely continue with HTML5. There is also a trend moving the control of the application into the browser as well. What this means is that the server is just a collection of web services (REST or otherwise) that serve data in some predetermined format. This data is then rendered into the pages based on whatever the action was mean to accomplish. Granted, this trend is a topic for another day, but from the server perspective, and enterprise applications, it has very interesting implications.

So, if the server application is just a collection of web services that serve data in JSON or XML, what language do they need to be written in? In reality it does not matter, and this is a good reason to wonder if Java can maintain its leadership in the enterprise. Ruby and Python have been gaining developer mindshare for years based on their popularity for web development. They are also great glue languages, which means that enterprise developers may be using them outside of the enterprise applications already. C++ and C# are obvious alternatives for Java on the server as well, and Perl is still a very viable option especially when you remove the presentation concerns. Basically, any language that can be executed by a server that responds to HTTP requests is an option in the web services scenario.

Is Java dead for enterprise applications? Obviously not, and until something significant changes, it is still a very good option for people to use. Should Java be concerned about the growth of other languages and the simplicity and platform-independence of using web services? Absolutely.

First, a quick refresher on how garbage collection works in theory. You continuously create objects in Java, but at some point the memory needs to be cleaned up. Generally, if you create objects within a method, those objects may be garbage collected when the method has completed. Another important point is that if you keep passing objects around to various other classes and methods, those objects will only be garbage collected when there are no longer any references to them. Sometimes the garbage collector will clean more aggressively if you start using large amounts of memory as well. Granted, this is a simplification of the way garbage collection really works, but it gives us some basic ideas on how to keep your java application running smoothly without those weird OutOfMemory errors.

Do not create a lot of objects. This sounds silly, but a surprising number of issues can be resolved by limiting the number of objects that get created. Think of it a different way. If we have a reporting application, are you going to create objects for every row of data in a report? If your application has a lot of reports each having thousands of rows, that is a lot of objects to create.

Create objects only when you need them. If you do need to hold onto a lot of data, does it make sense to create thousands of objects? Can you just pass around a list of ids or a small hashmap? This is similar to the concepts of lazy loading data from a database when you are dealing with persistence frameworks. Why load an entire object when just the id will suffice?

Close all files and streams. This is probably one of the most common issues when it comes to memory management in Java. Sometimes you just miss closing the file because you are more concerned with the contents of the file. There are also times when your error handling takes a different path than the normal code and thus the file stays open. Just remember, finally is your friend.

Do you copy lists or pass lists? This is a harder problem to deal with. If your lists are long and filled with large objects, you may want to avoid the overhead of copying the data in the list. However, if you continuously pass lists around, you are maintaining several references to the objects in that list. This could mean that those objects will not be garbage collected.

These are just some simple tips for clean memory management. In some cases, these issues are easy to avoid. In other cases, the amount of cleanup you need to do is rather annoying and easy to skip. A lot of frameworks, like Spring, will do a lot of the basic cleanup for you, especially in the areas of persistence. The real key is to keep this as simple as possible and to clean up as you go. Just because Java has automatic garbage collection does not mean you should make more trash.

Too many times, especially when it comes to social media and technology, businesses flock to the hot new trend or the shiny new object with delusions of grandeur. Two years ago, clients would come to me and say, “I want a blog.” Today, they say, “I want a Facebook page,” or “I want a Twitter account.” My answer has always been the same: Why? Seldom can answer that question.

As a software engineer, you need to ask yourself why almost every day. A software engineer needs to make technical decisions at almost every step in the development process. When the project is first in design, we may determine that the software will be developed in Java and deployed to Apache and Tomcat servers. Why? Can you defend your reasons for picking Java and Tomcat? Does the application need to be deployed into a web infrastructure or does it make more sense as a desktop application? Can you defend your decision against someone that thinks the application should be written in Ruby?

In many cases, engineers may take the lazy route and just make decisions based on what they are comfortable with. In my case, that likely means Java on Tomcat. By taking the comfortable route, you may miss opportunities to make a better choice, or even learn new things.

In addition to the initial platform choices, there are several other things that need to be decided. If we are writing a web application, do we use traditional servlets, Struts, Spring or some other framework? This is when the decisions start to get really interesting. Spring may be the best choice for your organization, but are your developers familiar with Spring? Maintenance is a significant concern for any development organization. If your developers can not maintain an application due to a lack of technical knowledge, you have failed as an organization. Sometimes, you may be able to get training on the newer technology, but other times you may have to choose the older but known technology. These decisions get fairly hairy as well because people tend to feel very strongly about their favorite frameworks.

Once the core framework is chosen, the persistence framework needs a direction. Sometimes this is guided by your core framework, but frameworks like Spring allow you to integrate almost anything. With Spring, you can easily integrate Hibernate, iBatis or standard JDBC. You can also use JPA with any of those technologies as the JPA provider. Why you choose one persistence framework over another is highly dependent upon your organization and what you plan to do with your persistence layer. Hibernate is really good for straightforward data models that can easily be mapped to Java domain objects. iBatis allows you to customize your SQL fairly easily and allows great flexibility in your SQL. JDBC allows you to focus on the actual SQL without any of the quirks of the frameworks. JPA is considered the future of Java persistence, so it is a good technology to start working with.

This is really just the beginning of the decisions made on a project as well. There are various design decisions that are made every day that typically do not get questioned until a problem arises. You may be able to save yourself some time by questioning all of your decisions. Did you make the decision based on sound reasoning or did you make the same decision as you did last time because it is simpler to keep the same direction? Can you answer why? More importantly, can you explain why to your fellow engineers?

By now you have heard that Oracle bought Sun. I did not want to write about it yesterday as I wanted to sit and think about the effect this has on anything Sun related. There is some good high-level business analysis from Reuters as well. There is one important quote from that article that needs sharing:

Sun was more comfortable striking a deal with Oracle because the two operate complementary rather than overlapping businesses, and antitrust roadblocks are less likely.

This is probably the most significant difference between an Oracle and IBM deal for Sun. This deal also gets Oracle into the hardware game without having to buy a lesser player. However, the software side of this deal is much more interesting.

Java

So, what happens to Java? Hopefully nothing. The JCP exists in order to take real ownership out of the hands of one entity anyway. Oracle also has a lot of interest in keeping Java alive and well. We may see a renewed push for all things data related in Java now that Oracle is a much more important player in that space.

NetBeans

Oracle already has a tool called JDeveloper that technically competes with NetBeans. Up until recently, both tools were considered terrible by many people with only Eclipse and IntelliJ considered as decent java development environments. However, the recent versions of NetBeans have been reviewed well, so Oracle could have upgraded their java tools as a bonus.

MySQL

This is the big question in this deal. MySQL is used almost everywhere as a free alternative to the expensive commercial RDBMS. MySQL is used as the backbone of WordPress and many internet sites as well. Is there a danger that Oracle will neglect MySQL? Absolutely. Will that really happen? Absolutely not.

Oracle purchased an interesting business with MySQL. They can keep it open source, add on “premium” features that enterprise customers may want, and eventually push for upgrades to their enterprise offering of Oracle. In addition, they get some nice consulting and services deals out of this as well. If anything, this just makes Oracle’s database position much stronger. They have always had a problem getting smaller businesses to run on Oracle, and now they have something that almost anyone can run.

The main problem with Oracle buying MySQL is that there will definitely be work on a “more open source” version of MySQL. My understanding is that a fork of MySQL was started a while ago, but with Oracle now owning it, I am assuming that people will put more effort in an open source offering. I call this a problem because it just means that there will be yet another choice in the database game, and it is likely to confuse things a bit.

Final Words

Oracle is a big winner with this deal, mainly because of the hardware they have always coveted. However, there are two jewels in this deal, MySQL and the cloud services. I already talked about MySQL, but the cloud services that Sun has mentioned recently will be a big bonus for Oracle. Oracle is already working with cloud providers to have Oracle as a data storage choice, so owning their own cloud offering could be huge for them. They have always wanted an entire stack offering from hardware to database and middleware. Now they can offer that, and soon offer it in the form of cloud services. Oracle has just positioned themselves as a major player for the next few years.

Basic Java Testing – JUnit and TestNG are the most popular tools for basic java unit testing. JUnit typically has a small advantage as it has been around longer and is included with most packages of Eclipse. However, I do not think one is really better than the other, they are just different.

Mock Objects – EasyMock and jMock are popular mock objects frameworks. Sometimes you have to simulate the way another system works, and mock objects are perfect for this. You can even create your own mock object, but the frameworks make it easy to avoid extra code.

Database Interaction – DBUnit is really the only decent framework I have seen. It allows you to insert data before the test is run and cleans up after the test is complete. However, capturing the appropriate data for a test is not simple for anything but the most trivial applications.

User Interface Testing – UI testing is notoriously difficult and many applications skip it entirely. If you are testing your generated html, you will want to look at the following tools and determine which best suits your needs. This is a special case, as there is no “one true way” to test the user interface. These are all free tools as I am a fan of keeping things as cheap as possible if the tools are good enough.

• Selenium – Maybe the most complete package of the bunch and supports testing in IE, FireFox and Safari. Your unit tests are written in java, so you just need to learn the API. They also have interesting server options where you can run your tests across servers. You can even record tests using a FireFox add-on.
• Canoo WebTest – This is another framework that can record tests with a FireFox add-on. The tests for WebTest are typically written in xml files. This gives you the option to build a library of fragments of functionality to include in various tests. You can also write your tests in Groovy or use the Java API directly. These can also be run directly from within Ant.
• Watij – This is a port of the popular Ruby testing framework, Watir. The tests are written in Java using their API. They also include BeanShell so that you can interactively test your code. This is a really interesting option if you are not sure what a unit test should look like initially.
• JWebUnit – This is more of a meta-framework, where it can build upon HtmlUnit or Selenium. HtmlUnit is fairly good by itself, but sometimes a framework is just too low-level for some people or people find that they keep doing the same things and decide to build upon the framework. There is one problem with JWebUnit, the documentation is very sparse, so the learning curve ciould be problematic.
• Cactus – This is another meta-framework, but this is from Apache. It is meant as a wrapper for testing an entire application from basic java testing, servlet testing and down to the user interface testing. Due to this meta concept, the documentation can be a bit vague. However, I have heard good things from those people that spent the time to learn more. Disappointingly, I have not had the time to dive into it much.

There are a lot of other tools out there as well. FitNesse is very popular, but it takes a very distinct approach to how unit tests are built. For Struts applications, there is a framework called StrutsTestCase that is immensely helpful as well. For a big list of testing and related software, take a look at XProgramming.com. They list testing frameworks for almost every language. Lastly, it is always a good idea to know how well your software is tested. This is not a question of whether all of your tests pass, but how much of the code is actually tested. My favorite test code coverage tool is Cobertura. It is fairly simple to use and the reports are generated in a similar manner to how JUnit generates reports.

So, if you know of a good free java testing tool, drop me a note in the comments. Sometimes a good tool may not be getting enough publicity or I may not have had time to look at it. In any case, keep testing until you have good test coverage. And remember the testing mantra, “Red, Green, Refactor”