The good news is that I may have a solution for you!

I was recently testing Portico with a large, widely deployed C++ application. The Portico C++ interface uses JNI to wrap the core Java library. Other applications then link against this interface and use the library to do whatever it is they need to do.

When testing this large application I noticed a lot of crashes in code that was totally unrelated to Portico. For the life of me I couldn’t figure out what the problem was. This was made even more frustrating by the lack of any help from Google. Eventually however, I stumbled across this post, which discusses something that sounded quite familiar.

My current understanding is that the JNI uses various signals to complete its work, and that if you have an application that puts its own signal handlers in place, they will replace the JNI handlers thus causing problems. You can try starting the JVM with -Xrs to make it reduce signal use, but this doesn’t entirely eliminate it, potentially leaving you with the problem.

Fortunately, Sun ship a signal handler chaining library with the JVM. When I linked against that (no code change required!) everything started to work happily! The libjsig.so library is in the same directory as libjvm.so which you need to link against anyway, so the grand solution is to just as a -ljsig to your compile/linking command. Talk about a simple solution to an incredibly frustrating problem. For full details see the article above.

I’ve decided to post this little discovery here in the hope that Google finds it and saves someone else a whole bunch of pain. Never say I don’t do anything for you Internet.

Useful resources:
If JNI based application is crashing, check signal handling!
Revelations on Java signal handling and termination

It feels like there are 400 commands, each of which is basically the same, yet possessing subtle differences; only 4 of which I’m ever going to use. It seems, as an inexperienced Git user, like I’m drowning in a pool of commands that all look the same and give me no indication as to which are more important or more common than the others.

To put it another way, I keep hearing about the “power” but it seems like nobody is paying attention to the “usability”.

I’m sure I’ll get rubbished for that by people who want to tell me it’s much more usable than Subversion because you can do x, y and z (or you can do them easier than you can in Subversion). While that might be true, I don’t define usability as a list of things you can do, but rather how easy it is to figure out what I need to do without having to dive into mountains of man pages or countless tutorials.

For example, consider the typical command used to commit something with Git:

git commit -a

-a? what’s the -a?

A quick look at the man page tells me:

Tell the command to automatically stage files that have been modified and deleted, but new files you have not told git about are not affected.

Great. Everybody get that? No? I didn’t think so. What the hell is the stage and what has it got to do with my commit? I just want to commit my stuff dammit!

If you trawl any number of the Git tutorials Google finds, they’ll either ignore the -a like it’s not even there, or offer some cursory explanation that amounts to “everyone does it that way”. I’m left to wonder, if everyone does it that way, why isn’t that the default?! Sure, I could ignore it, but like a lot of developers I’m an anal-retentive. Leaving it there, not knowing why, just gnaws at me, like that little switch is laughing at me knowing I can’t get rid of him despite not wanting him there. Or maybe I can get rid of him…? who the hell knows. I just want to commit my stuff and move on with life.

Let me give you another example. Consider the situation where I want to revert back to the way the repository was before I started making my most recent changes. Perhaps I deleted a file and now I want it back.

Git documentation and tutorials start throwing all these things at me. There’s git-revert, git-reset, git-reset --soft and git-reset --hard. Even worse, there are all these warnings about being careful when using some of them.

I type “git revert” and it wants more command line arguments. Further, the options it throws at me talk about editing commit messages or not doing a commit. Of course I don’t want to do a commit, I want to get back to a known-good state! I give up on revert and try “git reset”, it tells me that the file I accidently deleted was locally modified, giving me a hint of hope that Git has seen this and reverted my state. A quick check reveals… NOTHING! git-status confirms that the file is still deleted. What happened? I don’t know. I try “git reset –soft” and get no output. I check the status again and again I see that nothing has happened. I finally give up and try for what feels like the a-bomb option, “git reset –hard”. Huzzah! It worked! Why? No ******* idea, but it worked. What is the point of revert, reset or reset –soft? I still have no idea. I’m sure there is a point, but I just wanted to get a file back.

Just telling me that I have to use -a or –hard isn’t a solution. If I HAVE to use a particular switch to complete some common, simple task, I should be able to understand why without having to learn a whole bunch about the way Git works internally.

I think this is what Subversion really gets right. There are a small bunch of core commands that are well defined, well named and “do what they say”. Yes, for some tasks SVN is cumbersome or might not even do what you want, but it gets the fundamental usability ideas right.

Don’t get me wrong, as with Linux, once you climb the learning curve everything might be fine. Before I used a Mac I ran Gentoo. Although I hated it at first, I grew to love it (faults and all) over time, but the learning curve required Crampons, and in reality, the hostility of it is going to turn a lot of people off.

This whole class of problem, this lack of intuitiveness, is why I describe the experience as very “Linux”. It’s all there, it’s all functional, it can do amazing things when utilized properly, but the whole thing can is so terribly unintuitive that I just end up annoyed and going back to a tool that 95% of the time, gets out of my way.

Late last year, Joel talked about the new Fog Creek offices they’d just completed and were moving into. Apart from looking absolutely amazing, one of the main things that stands out is individual offices for programmers. This is one of Joel’s central requirements for office design and he has spoken about it many times before. Apart from generally wanting to create a programmer nirvana, he firmly believes that private offices enhance productivity and ultimately allow his employees to be happier and more effective.

On reflection, I did mostly agree with Joel. Having previously worked from a home office, the quiet environment allowed me to be incredibly productive. My main problem when working from home was that I missed the human interaction that you get working in an office environment. Every now and then I’d fly out to Perth to visit the main office. I always enjoyed these times. The office was much more active and vibrant. It had a strange appeal I couldn’t quite describe.

Last week I noticed the article Open-plan offices are making workers sick. Among the many things it pointed out was a statement that the continual activity of open plan offices is distracting and contributes to lowering productivity of employees.

The argument seemed to make sense, but I didn’t quite realise just how much sense it made until yesterday. At the moment I’m working on implementing a new service in Portico. Having been through this before with the other services in the HLA specification, I know that the early parts of development require a bit of extra thinking as you try and figure out how everything hangs together. Sitting down and designing it all out, as well as implementing the initial stuff is half the battle. After that, the rest of the work is really just repeating the early stages with slight variations and testing it all.

I’m someone who finds noise quite distracting when I’m trying to think. For a lot of tasks I’m OK, but when I’m really trying to think, when I’m really concentrating, I can’t stand noise. Naturally this doesn’t gel well with the open-plan environment. Yesterday, to escape it all, I picked up my laptop and carted it down to the board room.

The effect was amazing. Two hours whisked by like it was 15 minutes. I was able to sit there and concentrate, not having my flow interrupted by the conversations or the comings and goings of others. Not only was I productive during that time, but I felt fresher at a time of the day when I’m normally starting to wind down. I was ready to keep on working.

I also noticed that I spent less time mindlessly wandering the internet. Not once in the entire session did I check my email, wander onto the Portico support forums or flick over to Digg. Sitting in a distraction free environment caused me to spend less time distracting myself. This I found particularly interesting, as if the availability of time-wasters like Digg, Flickr, Twitter and the Internet in general were only crutches I used to reset my system when I lost my flow after an interruption. Taking away the interruption prone environment meant I remained focused on what I was doing longer and less aware of (or in need of) digital diversions of my own.

Programming in the flow is essential to both being productive and feeling productive. Sitting out here in an open-plan office, I can feel my flow being constantly interrupted. In such an environment this kind of distraction is unavoidable. Sometimes it’s nice, sometimes you crave the distraction and it inadvertently gives you the mental break you need. More often then not, it is disruptive. I can feel it slowing me down, and worse still, I can feel it wearing me down, tiring me out.

Open-plan offices do have many advantages. They’re cheaper to fit out, people can bounce off one another, I tend to find them lighter and brighter (they would be much better than a windowless office at the end of a windowless hallway!). Open-plan offices are everywhere and I’m just going to have to find a way to deal with it, but I do miss the productivity I used to have, even if I do now get the social interaction I previously desired.

Unfortunately, getting JGroups to do this isn’t as easy as it really should be. It appears from traffic on their issue tracker that they’ve gone back on forth on the idea of making all threads Daemon by default or not. Suffice to say at this point in time, they default to user threads.

To save anyone else some pain in trying to figure out what is going on, I’ve included the code necessary to get JGroups using Daemon threads. It took me a while to figure it out, but I got there in the end. Further, if it wasn’t for the open source nature of JGroups I would not have been able to figure this out (or perhaps it would have taken much longer). Being able to trawl through source code really is a wonderful thing.

Anyway, here is the code. Remember to set both the ThreadFactory *AND* the ThreadPool. When JGroups creates its internal components on creation of a channel, it creates both the default factory and pool. However, when you set the factory, it doesn’t also set the pool, so if you only change the factory then the pool still has a link to the initial, default factory. I will submit a patch for this as it seems logical that any change to the factory should trigger a change to the pool as well.

JChannel channel = new JChannel(...);

// create a thread factory that will use daemons
DefaultThreadFactory factory =
	new DefaultThreadFactory( Util.getGlobalThreadGroup(),
	                          "Incoming",
	                          true,
	                          true );

// set the thread factory on the transport
channel.getProtocolStack().getTransport().setThreadFactory( factory );

// set the thread pools on the transport
LinkedBlockingQueue<Runnable> queue = new LinkedBlockingQueue<Runnable>();
Executor pool = new ThreadPoolExecutor( 2,
                                        10,
                                        5000,
                                        TimeUnit.MILLISECONDS,
                                        queue,
                                        factory );
channel.getProtocolStack().getTransport().setDefaultThreadPool( pool );

UPDATE: I’ve just checked out the latest code from CVS and this problem appears to be fixed there. I’m running version 2.6.3.GA. So if you are having this problem and using that version or earlier, the above code should help you. If you are using a later version, well, you shouldn’t be having this problem

While I have no doubt of the value that exists in the legacy project, or the quality of its contents, I have found myself cursing it many, many times. Outwardly it would seem that I hold some form of contempt for the project, or the quality of the project, however, that notion this is more assuredly not true. The biggest problem this project has is its build system. While understanding the design and implementation of this large system is difficult enough, it often feels like a developer is actively engaging in a fight with the build system just to compile or package the application or part therein.

This experience has driven home a particular points to me: Although much thought and consideration may be evident in the architecture and design of the product, the same standard hasn’t been applied to the build system itself. A simple build system is fundamental to the experience of developing for a project. The most amazingly elegant, well designed application can be made to feel unstable, unpredictable and generally brittle if the process used to build it is unnecessary complex and convoluted.

Like most commercial environments, this product was developed under real-world pressures and deadlines. Taking time out to focus exclusively on the build system is difficult to justify when the reality is that it adds no direct features of any value for the end user. However, it has been my experience that neglecting these aspects of a system always ends up costing you time and/or money in the end. To go from a clean windows install to a fully functional development environment for this project took 3 entire days. The process involved many, many manual steps, the location, download and installation of countless dependencies, a great many problems with version mismatches for these dependencies and considerable assumptions being made about the level of exposure developers had to particular tools.

In Portico we spent a LOT of time working on the build. When significant changes were made to the project (such as adding a C++ interface that required building C++ code), the effect on the build was considered and a lot of thought, time and effort was expended to make sure we had a process that was both easy to use and required the user have the only the bare minimum environment previously installed (just Java and a C++ compiler). This often involved many iterations and on several occasions we had to throw away significant parts of the existing build, replacing them with something that better suited the problem at hand.

Although we could have spent less time and still come up with something that “worked”, the task was deemed of such high importance that the extra time and effort seemed justifiable. After all, developers would be interacting with this build day-in and day-out. Yes, it cost time, and by extension it cost my employers money. However, the payoff is immediately obvious. The setup time for a new developer is about 15 minutes and involves two steps: 1) checkout from version control, 2) “./ant sandbox”

That’s it. Spending all that time making sure the build was just right translates not only into time saved later on, but to a much more pleasurable development experience. Such a simple build allows a developer to focus on solving problems with the application rather than having to constantly switch mental contexts in order to engage in hand-to-hand combat with the build system. While quality is a function of thought and reflection, the experience of quality extends only as far as the weakest link. If building your software feels like a huge effort that might not even give you what you want at the end, then it doesn’t matter how good the code is, the experience is going to be bad.