Last night I completed most of the benchmarking for fedWS2. However, given that we are talking about web services, the major portion of time the server or client spends doing work will be on the serialization/de-serialization of the messages. Given this, my activities were more of a benchmarking of XFire (which fedWS2 uses to provide a web service interface) than anything else. Still, given that this has the most direct baring on the end-user experience of using fedWS2, it is more than relevant.

Test Results Here

I have completed all the official tests that I was planning to do, however I still want to run them on Mick’s AMD64 running 64-bit Linux to see what kind of difference that makes.

Test Setup

These tests were conducted using three computers, detailed below:

• Computer One (fedWS2 - Windows XP) P4 2.4Ghz (no hyper-threading) 1GB ram
• Computer Two (client - Linux 2.6.12) P4 3.06Ghz (ht) 1GB ram
• Computer Three (client - Windows XP) P4 2.80Ghz (ht) 1GB ram

I ran tests for packet sizes of 4KB, 8KB, 16KB, 32KB and 64KB (packets were captured using tcpmon to verify sizes before running the tests). For each packet size, I ran tests with the following clients:

• 1 Linux
• 2 Linux
• 1 Windows
• 2 Windows
• 1 Linux and 1 Windows
• 2 Linux and 2 Windows

Where more than one client on a particular platform was used, both ran on the same machine under different executions. Also, where more than one client was used in a run, the number of iterations recorded is the average of all clients involved. I kept an eye on the CPU utilization and under the heaviest load it only pushed about 75%, so neither machine should have been too disadvantaged by running more than a single client on a single machine. Just to be sure, I ran a 2 client test with one on each platform to compare the results to the “same machine” 2 client tests. The tests used a legacy simulation from DSTO that the labs just finished creating a suite of specialised web-service clients for, so this was what you would term a “real world test”.

The two client machines are virtually the specs, although the tests show that there is little, if any real difference between Linux and Windows in this case.

Test Client Software

The test client software was a little program I wrote that simply asked for a set of information and then discarded it when it arrived. It looped around doing this as many times as it could in the 60 seconds that a run lasted. The client application used the Apache Axis client side framework and was compiled and run under Sun’s latest JVM (1.5.0_05).

Interesting Points

From anecdotal tests, the original fedWS (which used Axis on the server side) could barely manage 8 calls per second for a single client with virtually no data. However, that was a while ago and differences in hardware should be taken into consideration when throwing that number around. That said, XFire is VERY slick and the numbers shown here are extremely impressive when you consider the amount of information that is being pushed around.

I won’t take a detailed look at these numbers yet; rather, I’ll post the finished paper with a full analysis and pretty graphs when it’s done. For now I just wanted to get the numbers out there as many people on the XFire mailing list seemed interested in this.

This entry was posted on Sunday, November 20th, 2005 at 5:17 pm and is filed under fedWS, hla, java. You can follow any responses to this entry through the RSS 2.0 feed. Responses are currently closed, but you can trackback from your own site.