Ade On Middleware

FUSE Community Day Paris 2010

2010-09-13T10:43:00.000-07:00

I'll be at the FUSE Community Day, Paris event on October 14th! Feel free to sign up and come along - If the London event in June is anything to by I know it's going to be a great day out :) We've got some great speakers lined up (including all of our FUSE rock stars like Claus Ibsen, Rob Davies, and James Strachan.

Rockin'!

Survival of the fittest: the evolution of the SOA registry / repository concept

2010-09-08T01:09:00.000-07:00

I've decided to write a little about SOA registries and repositories, as a few customers have brought up these issues recently and I wanted to clear the air (and, in some sense, clear my own head too). I dragged in some sanity checks from my fellow consultants at FUSE just to get their feeling on the adoption of SOA registry tooling, so hopefully what follows will make some sense.

There was a flurry of activity some years ago (say, around 2007/2008) about SOA (Service-Oriented Architecture) Registries and Repositories. SOA evangelists, architects and "thinkers' waxed lyrical about how real SOA experts (with capital letters) used registries and repositories, and that the 'little-SOA' folk just didn't get the concept, god bless them. Vendors rushed to provide (sometimes expensive) SOA registry / repository tools, a move probably motivated more from the realization that underlying SOA infrastructure and middleware was becoming a commodity, rather than the fact that customers really did need a registry.

Funny that.

Because in many of the SOA customers I've worked with, they simply haven't needed a repository/registry. In fact, the mandate from on top to get a SOA registry or repository has been a distraction! Now, don't get me wrong: I'm not saying the SOA registries are a bad idea. Not at all. I'm just saying that, as a tool, they solve a particular problem. And, if you don't have that problem, then you don't need the tool.

Let's consider what a SOA registry / repository might do for you.

Store contracts, service level agreements, code, deployable artifacts, documentation (this is the 'repository' nature of the tool)
Act as a run-time lookup for physical location of services - thus abstracting a services physical network address from a 'logical' network address. (this is the 'registry' nature of the tool)
Keep a track of who is using what services, and how much (this is the 'governance' nature).
Enforce service level agreements (again, this is more governance: making sure that the services are available and doing what they should do in a timely fashion)

Interestingly, I've found that the 'repository' problem is often solved using readily available file-sharing tools, HTTP servers, or even source-code management tools like SVN, GIT or CVS. Not glittery, sexy, 'made-to-measure' SOA repositories, but pragmatic, workable and relatively simple. One customer I know took a pragmatic approach which I really admire: "We're going to use a Wiki for our first set of services. When our Wiki becomes a pain, then we'll look at getting something more appropriate. But not until then."

The 'registry' problem is sometimes solved in software: for example, the Artix Locator, integrated with Apache CXF, allows services to register themselves automatically with a well-known-registry, which clients can then look up at runtime. Nice! I've seen other CXF users write their own registries to achieve something similar - it's really not rocket science (although, you can make your registry as complicated as you like!). I would like to see an open-source implementation of such a runtime-registry, perhaps making use of recent innovative projects like Apache Zookeeper to keep track of who's up and who's down. As an additional note on registries: one customer noted that the OSGi registry, particular in the context of distributed OSGi (dOSGi), adds another dimension to the idea of dynamic, transparent lookup of services. And, if you're acting smart in a JBI or ServiceMix 4 world, you can achieve location transparency through the NMR or ActiveMQ POQs (plain-old-queues).

Interestingly though, if you have your network folk in the room, they'll argue that they can solve this whole lookup and 'location transparency' problem in a heart-beat in hardware with a network switch: think about it: I can go to www.google.com every day - in fact, so can my mam - and never worry about what machine my search gets directed too. So, often people can achieve the 'registry' functionality easy, sometimes with zero code.

So, while there is certainly some argument for using registries and repositories for storage and retrieval of information about what services you have on the network, there are often many options available to you that don't need a dedicated software tool. And, keep in mind, if you have only a small number of services - say up to 20 - you really don't have a management problem - so ad hoc approaches like Wiki's may be perfectly adequate.

With this in mind, I think that 'SOA registry-repository' offerings begin to offer value when they tackle SOA Governance issues, and I think this is the evolutionary route these products are taking. IT owners want to know that security policies are being enforces - these policies should be configurable, not 'designed into' the fabric and code of a service where they're difficult to change. They want to know if the response times are being sluggish. They want a record of who accessed what, and when. This could be for a legal audit trail, but is probably more useful internally as a 'proof-point' to show that service re-use is happening, and, more importantly, a way for a core SOA Centre of Competence to demonstrate the value that its services are offering the rest of the organization. Now this, for me, is a place where these tools can offer real value. As a Progress employee I am biased, and will of course recommend Actional which does an awesome job in this regard - but, as a fair and open-minded person, I have seen users of CXF propose to use HP SOA Centre to achieve something architecturally similar. Actional acts as a 'gateway'. You register your services with it (hey, a touch of 'registry!'). You apply your security policies. And away you go: all SOA requests get passed through gateway and SLAs are enforced and reported. Governance made easy. The only problem with this architecture, from my perspective, is that it creates a potential 'gateway bottleneck; however, with appropriate load-balancing in place this hasn't yet been a problem in practice.

At the end of the day, the tooling is evolving to the point of value around the concepts of governance - I welcome this. However, I still believe that if you're embarking on creation or adoption of SOA concepts, you shouldn't get stalled in the early stages by selection of SOA registry / repository. First, understand your needs - your real needs - in this area, and then, go look for right tool for the job.

Versioning WSDL interfaces in an OSGi world

2010-09-08T00:53:00.000-07:00

Some time ago (2007, I think) I wrote a short paper with Oliver Wulff on WSDL versioning. Recently, I shared this paper with some smart OSGi-savvy ServiceMix users, and it raised a discussion about how does WSDL interface versioning mix with the versioning concepts in OSGi? Below is an extract of an email response I sent back to them: for me, the bottom line is that WSDL interface versioning (in fact, any interface versioning) is a separate concern to the way we version implementations.

For anyone out there who is implementing Web Services using OSGi runtimes like FUSE ESB (built on Apache ServiceMix and Apache Karaf), you might find the thoughts below of interest.

From my side, I think the most important point is that my paper concerns WSDL versioning, that is, the versioning of an *interface*; the versioning of an OSGi bundle is, from a service consumer's perspective, an *implementation* detail and a totally separate concern. I believe you should consider the versioning of the interface and the implementation as two *separate* evolutionary paths.

So, I could have, say, version 1 of an 'foo' web service, in namespace http://www.myorg.com/services/foo/v1, defined in foo-v1.0.wsdl. We might generate the Java JAX-WS/JAXB-B classes, and then package as a bundle foo-ws-api-1.0.jar. I could implement it today in bundle foo-v1.0-impl-1.0.0.jar, and then fix a bug later in foo-v1.0-impl-1.0.1.jar, add a cool new feature in foo-v1.0-impl-1.1.0.jar, and then do a completely new, high-performance implementation in foo-v1.0-impl-2.0.0.jar. All of these implementation bundles would implement *the same* version of the WSDL!!

Now, imagine we need to do a minor change to the web service interface: say add a new operation, or add a new parameter to a method. We can do this *without* breaking on the wire interoperability by creating the new PortType in foo-v1.1.wsdl, but still in the namespace http://www.myorg.com/services/foo/v1. We would deploy the JAX-WS content for this in foo-ws-api-1.1.jar. We could do a new implementation, and put the implementation of this in foo-v1.0-impl-1.0.0.jar. Now, we can deploy the new version of both the interface and the implementation, and, if we've been careful, the new deployment will handle requests existing consumers of version 1.0 of the interface. KEEP IN MIND THAT WHILE THIS IS POSSIBLE TO DO ... (sorry for shouting!) it's actually quite tricky to understand and I think that the complexity overweighs the benefits for most users. So, it's probably better to try to keep WSDL interface's versioning at major release numbers.

Now, with this in mind, we can consider what happens when we deploy a whole new version of the WSDL interface! We can deliver foo-v2.0.wsdL, with portTypes and all in namespace http://www.myorg.com/services/foo/v2. We can deploy this API as bundle foo-ws-api-2.0.jar, and deliver our first implementation bundle as foo-v2.0-impl-1.0.0.jar. And, later, when we fix some bugs or add some features, we can deploy the new implementation as foo-v2.0-impl-1.0.1.jar.

Get the picture? For me, the rate of change of an interface should be different to that of an implementation - ideally, it should be significantly slower, changing maybe on a yearly base. We need to apply different versioning semantics, and evolve them at different rates to the implementation bundles.

Hope that helps!

Best,
Ade.

Karaf's Fabulous Features, and, what you can do to make them even more fabulous

2010-08-25T10:46:00.000-07:00

I've been working for sometime with the Karaf OSGi Shell, through my exposure to the great ServiceMix 4. The features mechanism provided by Karaf/ServiceMix allows you to leverage the modularity and control of OSGi bundles by grouping sets of bundles - containing your Camel routes, web services, business logic, RESTful services - into easily manageable 'features'. In this post, I'm going to talk a little about what features are, why they're good, and what we could do in Karaf to make their usage even easier. If you like the proposals, please, follow the JIRA links and vote for them so that we can get the community behind it. These enhancements are small, but, based on my experience, they could have a big impact on ease-of-use and adoption of a very powerful deployment mechanism.

Features: a group of bundles by any other name would sell as sweet.

A feature is just a set of bundles, described using a very simple XML file - this file is called a 'feature descriptor', and is also referred to as a 'feature repository'. Here's an example that describes a single feature, 'feature-b', that depends on another feature 'feature-a', which is itself described in another repository. I've thrown in some default configuration as well that will be synched into the OSGi Config Admin service; don't worry about this for now, I'm just showing off.



<features name="feature-b-0.0.1">
 
 <repository>mvn:com.fusesource/common-features/0.0.1/xml/features</repository>
 
 <feature name="feature-b" version="0.0.1>
  <feature version="0.0.1">feature-a</feature>
  <bundle>mvn:com.fusesource/bundle-b/0.0.1</bundle>
  <config name="feature-b">
   a=1
   b=2
  </config>
    </feature>
    
</features>

Features can have sensible names, like 'InsuranceQuoteService' or 'CustomerUpdatesFlow', and they can be versioned so that you can track their evolution and upgrade or rollback with ease. Features can 'depend on' other features, which means that when you install a feature, it and all of its dependent features get installed too. This is very neat: how many times have you realized that, yet again, all your SOA, RESTful services and integration flows all rely on the same common backend code? You can describe these dependencies easily and elegantly using Karaf features. And, there's a set of tools that allow you to suck down all of your feature dependencies from Maven servers onto a local drive in an elegant directory structure - automatically, as part of your build - so that you can tar.gz or .zip it all and and deliver your feature into the production environment. This last point is so important: these techniques allow you to use all of your Maven-style bundle URIs on production machines that don't have Maven installed and don't have access to the external Internet.

Making features really, really easy to use

While all of this goodness is there for the taking in Karaf, there are a number of small improvements that, I think, will go a big way to ease the adoption of the features mechanism. My own usage of 'features' is based on what I've learnt and observed from the Karaf source itself: I want to make it easier for other developers to create, package and deploy features. And so, I've created today a number of issues on the Karaf JIRA to get the ball rolling.

KARAF-165: Create an improved Maven feature-assembly plugin. Right now, to make a feature I've got to add almost a hundred lines of Maven verbage to my pom.xml in order to assemble a feature. I've got to use the attach-artifact goal of the org.codehaus.mojo/build-helper-maven-plugin to deploy my features file into Maven. I've got to use the add-features-to-repo goal from the org.apache.karaf.tooling/features-maven-plugin to suck down all the dependent bundles. I've got to a whole load of other stuff to perform the packaging to .tar.gz and .zip. The problem here is that I'm using a whole load of generic plugins to do a very specific job, and I'm having to tell the plugins what to do instead of telling them what I want done. I'd prefer to have a single more declarative plugin to do this. It might look like this:
```
<plugin>
 <groupId>org.apache.karaf.tooling</groupId>
 <artifactId>feature-assembly-plugin</artifactId>
 <version>2.2.0</version>
 <executions>
  <execution>
   <id>create-repo</id>
   <phase>generate-resources</phase>
   <goals>
    <goal>create-repo</goal>
   </goals>
   <configuration>
    
    <featureFile>file:${basedir}/target/classes/features.xml</featureFile>

     
    <features>
     <feature>feature-a</feature>
    </features>
   </configuration>
  </execution>
 </executions>
</plugin>
```
The plugin should produce a .tar.gz and .zip file, containing the feature descriptor (and all dependent descriptors) and all bundles (and dependent bundles)in a Maven-style directory, similar to the system/ directory currently used in Karaf. Note that this plugin doesn't need you to list out all the feature repositories / descriptors that your feature file may transitively include - it will detect these dependencies at runtime and work out the details.

KARAF-151: We need to add a hot-deploy mechanism, that will detect feature assemblies dropped in the deploy/ directory, and unarchive the file into, say, a contrib/ directory. 'contrib/' would be the application-level equivalent to the current 'system/' directory; it would contain all bundles that are required to run your features. After exploding a feature assembly into the contrib/ directory, Karaf should scan the directory structure for feature repository files, and add these dynamically to the runtime.

The result? If we implement these two enhancements to Karaf, we'll end up with a double-whammy. Developers will find it incredibly easy to create feature assemblies. Administrators and operations folk will be delighted that all the have to do is copy a feature assembly to the deploy directory, and then ssh into the Karaf/ServiceMix runtime and list, install and upgrade or rollback features using the 'feature' commands.

If you like these proposals, please vote for the issues!

Seeing the wood for the trees: a for ActiveMQ?

2010-08-23T04:50:00.000-07:00

I have proposed a treeConnector for ActiveMQ - see the JIRA issue here and please vote if you like it. Am including the text of the proposal below.

The ActiveMQ network connector is excellent at facilitating self-organizing networks of ActiveMQ brokers with dynamically discovered routing, and can be used in situations where network availability is sporadic and not guaranteed. Network connectors can be used along with master-slave pairs to create a distributed ‘messaging fabric’. The network connector can also accommodate the creation of hierarchical networks, however, I believe there is a good argument for treating tree topologies as a special case in their own right. In this note, I’m going to describe why trees should be treated as a special case, and describe what a treeConnector might look like.

Hierarchical (or ‘tree’) networks are a special case of the abstract notion of a network; they have numerous applications in the real world in cross-geography, wide-area deployments. Consider a retail organization with 1,000’s of stores, who wish to send and receive information to and from head-quarters (HQ) and the stores: you can envisage a hierarchy of ActiveMQ brokers: one for HQ, one for each of the regions (e.g. Ireland, UK, France, Germany), and then a broker in each store/outlet in each of the regions. When HQ wants to send a message to a store, it should be able to write a message to the store’s queue and have it dynamically routed to the store via the regional broker. Likewise, if a store wants to send a message to HQ, it should be able to write to a local queue in the store and have the message dynamically routed to HQ.

Right now in ActiveMQ, there are a number of issues with using a network connector to achieve a tree topology:

Spillage of consumer advisories. When a consumer connects to a store, an advisory message is generated and sent to all brokers within the networkTTL range. These advisories don’t just travel up the tree to the top (HQ): they can also spill back down the tree to peers who are simply not interested. We have seen this in a network with 1,000 brokers connected to a regional broker, and a networkTTL of two: the advisory gets sent to the regional broker, HQ, *and* the 999 peers of the broker.

Resource wastage (as a consequence of spillage). When a consumer advisory from a store broker (say, broker ‘A’) reaches another broker within the networkTTL (say, broker ‘B’), then ‘B’ creates a subscription for the destination (to do forwarding) *and* can result in the allocation of a thread for this destination. So, it’s possible that Broker B will allocate a thread for the destination ‘BrokerA.Incoming’, despite the fact that it will never receive or send to this queue. If there are only a small number of peers at this level then this is not a problem; however, if there are many peers (as in the case of 1000 stores per region) then this will be noticeable.

Sensitivity to networkTTL. In order to minimize spillage and resource wastage, you need to se the networkTTL on each broker to the distance between the broker and the root - in this example, 2. However, a later reconfiguration of the network could result in this setting being too low (which means messages won’t go the distance) or too high (which means you get inadvertent spillage)

These issues can be addressed if we create a new tree connector for ActiveMQ:

Brokers would identify themselves as ‘leaf’, ‘branch’ or ‘root’ nodes. There should only ever be one root. If a broker has not been identified as either ‘leaf’ or ‘branch’ then it should consider itself to be the root.

Tree connectors would be configured using a ‘many to one’ approach. Leaf brokers would configure a tree connector to their branch. Branch brokers would configure a tree connector to the root. The root shouldn’t need to configure anything, except of course a transport listener.


  <treeConnector nodetype="branch" uri="tcp://headquarters.myorg.com:61616"/>


 <treeConnector nodetype="leaf" uri="tcp://uk.myorg.com:61616"/>

If a consumer connects to a leaf node, then the consumer advisory travels up the tree, all the way to root: the concept of networkTTL is ignored. If this advisory travels through a ‘branch’ then the branch delegates the advisory upwards. Branches do not pass advisory messages from children to their other child nodes.

If a consumer connects to a branch node, then the consumer advisory travels up the tree. Additionally, the branch broker will send the advisory downwards to its children.

In this way, we reduce spillage of advisory messages, and ensure that trees can self organize without too much intervention or worrying about the network time-to-live.

Thoughts?

An easy, useful, NMR: Monsieur Nodet, vous êtes une légende.

2010-08-13T07:58:00.000-07:00

I was discussing the ServiceMix NMR component this week with a colleague; and my interest was piqued enough to take a fresh look at this little fella. I grew to dislike the JBI NMR some time ago, and have written about these feelings a number of times on this blog. But its time to draw a line under those dark times, and look to the future. And, in the brave new world of ServiceMix 4, the new NMR becomes lightweight, liberating, and startlingly useful.

Rather than explain the technology first, let's talk about the problem that it might solve. In ServiceMix, you can deploy integration or business logic as OSGi bundles - this much we know. Now, say you want to send some information between two bundles: how can you do it? There are a number of options open to you - here's some of the most popular options.

Use OSGi services. I'm a huge fan of the POJO-based approach that comes with OSGi services. It's easy to use. Drawbacks? There are two I can think about - whether they're relevant to your use case will depend on, well, your use-case. First, the call is always going to be executed synchronously on the callers thread. If you've got lots of throughput going through the system, this may not be desirable. Second, in order to invoke on the OSGi service, you will need to marshal your incoming payload into POJO objects: maybe this marshaling is something you simply don't want to do. Maybe you just want to pass data through the system as fast as you can.
Use JMS queues. We love ActiveMQ, and ServiceMix is tightly integrated with ActiveMQ. But it seems that forcing the use of a JMS queue just to send some information from A to B, in the same JVM, feels like overkill - despite all the underlying optimizations available in ActiveMQ for in-memory messaging.

Use the NMR. Ching! The penny drops. The ServiceMix NMR allows you to send anything to another bundle, and allows you to do this either synchronously or asynchronously. So: you get choice on whether you want your code to be handled on a separate thread. And (and this is the best bit) the NMR, unlike the old JBI NMR, does not demand that the payload be XML. You could send your granny through. If your Granny was a Java Object. If it wouldn't cause too much confusion, I'd vote for a name change, and call this the Denormalized Message Router (DMR), as it no longer tries to enforce a canonical format.

Playing with this, I put together a little demo in a few minutes where one bundle did a file pickup (of a non-XML file) via Camel, sent the file to an NMR service in another bundle over the NMR, where the file is processed. Worked a treat. No pain. I was shocked at how easy this worked. The other thing that shocked me was how simply the Camel-NMR component is to configure. I was expecting tricks like 'you must provide an XML QName to describe your endpoint', or any number of other intellectual land-mines. But the short page of documentation on the Camel website was all I needed.

So where do I go from here? I feel really, really positive about this NMR now, in a way that I didn't before. It's become a technology I can use without having to worry about the mechanics underneath, like how I can drive my car without having to know how my engine works.

I must compliment Guillaume Nodet's work in this area. Taking the hard stuff out, and leaving a very useful and very usable NMR core, is smart thinking.

OK OK OK I give up: I’m getting really weary of XML

2010-08-10T23:00:00.000-07:00

At a water-cooler in a global Swiss financial institution I had a great chat with an enterprise architect. How’s it going? he asked. The correct answer to such questions is normally just to politely say ‘grand, thanks’. This time though I opened up and let out on what what has been a burning, smoldering, nagging realization: I’m getting really, really tired of XML. It’s everywhere. It’s on my queues, it’s in my SOAP messages, it’s in my integration flows, it’s in my configuration (thanks Spring!). It’s unwieldy, it’s unkind, it’s finicky, it’s bloated and most of the time it’s plain unreadable. In the last few weeks I’ve been debugging some difficult SOAP integration flows, and despite my ability to parse XML in my head, I’m finding it tiresome.

Did XML really deliver what it said it would? All those great features... Schema Validation: a great feature, but rarely powerful enough, and one most users end up disabling for performance reasons. XSLT? Cryptic. XPath: very cool, and very useful. XQuery: never got to use it in anger. Namespaces? Overly complicated and very messy when you start to use large numbers in the same message. Versioning? You have to do it with namespaces, and even then it’s very tricky to roll it out - do it wrong, and your versioning scheme just amounts to rolling out new schema releases that are incompatible with those of the past. Dynamic resolution of namespaces from the web? How many times have I had something work in development only to have it break in pre-production environments because those machines are fire-walled and can’t reach the outside world?! Partial message encryption and digital signatures? Very cool, but one of those advanced features that many people may never get to use.

Rant over.

But what happened at the water cooler? Our conversation became animated, exciting, and in the end cut far too short by pressing and more immediate work issues. Here’s the gedanken experiment: could we provide a set of enterprise services - providing things like access to data and core business task and transactions - *without* using XML all? If so, then what would it look like? Here’s some of the options we discussed in our brief encounter:

Revitalize IDL and the Common Data Representation of CORBA. Great stuff, CORBA (I salute you). However, CORBA got some things wrong: the representations are binary not human readable without appropriate tools. There’s no technology that allows you to XPATH style querying of data (very nice if you want to access just one part of the payload, instead of unmarshaling the whole kit and kaboodle). Also, IDL mappings to languages like Java, C++ and others tended to be clunky and are certainly dated at this stage.

JSON. Who can argue with simplicity of a data format that can be described in a single HTML page, is simple and fast to parse, and is supported by oodles of languages (see http://json.org). Great stuff indeed, and I think this is an area very much worthy of investigation further. Bringing in schema definitions for JSON allows us to be more specific about the content that can be held in JSON payload - another plus [need citation]. I think JSON would need more though to make it truly ‘enterprisey’: for example, need an XPATH-like way to get into parts of the payload. And, how can you do things like partial message encryption?

CSV. Don’t dismiss this one straight away. Comma separated value format, and it’s close cousins ‘fixed width fields’ and ‘name-value pairs’ are arguably the simplest formats around, providing minimal overhead.

Serialized Java Objects. I shudder at the though of restricting my data to a single programming language. No thanks. Enough said.

The options above are not exhaustive, and I have not included other approaches that may be in the pipeline. I have a prescient CORBA-savvy friend who stood firm on all of this ten years ago and said ‘Screw this XML stuff. It’s rubbish and will all blow over when people realize that’ (I paraphrase for emphasis - I’m sure he used slightly different words; however, his passion on the subject was clear). He’s written his own payload format that quite possibly will destroy XML, if it catches on.

In our water cooler discussion we didn’t come to any conclusions; there is no ‘winner’ here. However, there is a strengthening of something I’ve always believed since I got into middleware: there is no single, perfect, complete solution. Heterogeneity is crucial in operating systems, hardware platforms, programming languages, and frameworks; there is no reason why supporting heterogeneity of middleware transports and payloads should be any less important. Understanding that the conditions whereby one approach is better than another is key, and far more valuable than adopting a ‘one-size-fits-all’ approach. And then adopting open standards and technologies that can support this heterogeneity (and here I can smugly wear my ServiceMix, CXF and Camel hat) is the next most important thing. When I think of how CXF does RESTful content negotiation my mouth waters.

And so reinvigorated, I returned to work, and tamed the outstanding XML issues on my plate. As I lid-down for the night, I fear that some day this blog entry will haunt me, but I shall publish and be damned.

The six thousand topic man: hosting many topics in the same ActiveMQ

2010-06-30T07:54:00.000-07:00

An ActiveMQ user was enquiring about whether ActiveMQ (get it from fusesource.com!) could handle a publish-subscribe messaging architecture with six thousand topics. I've often seen production deployments of FUSE support tens to hundreds of JMS destinations; however, I wasn't quite sure how it would perform with a huge number of topics. Of course, you could reduce your number of topics by introducing message selectors on a smaller number of topics: but that avoids the question rather than answering it up front.

Throwing some questions at the FUSE engineering team got back a lot of confidence that it would indeed work just fine. Still though, I always like to try things and out and see for myself. So, I slapped together a JMS client that wrote 1,000,000 non-persisted messages to 6,000 JMS topics. Then, I put together another JMS client with 6000 consumers, with appropriate session and connection pooling in place. The result? Alarmingly straightforward: it worked just fine! While quietly content with this outcome, it's worth mentioning some background things I did on the Broker...

First, I needed to switch off the default 'thread-per-consumer' model in ActiveMQ. This is done by setting the JVM system variable -Dorg.apache.activemq.UseDedicatedTaskRunner=false - by default, this is set to 'true' in the ./bin/activemq[.bat] startup script. I tested what happens if I leave this to 'true' and indeed I ended up with 6,000 threads in the broker. Ouch. When you disable this setting, the consumers are served from a pool of threads, and total thread count never got above sixty threads.

Next, I configured the Broker's transport connector to use 'nio:' rather than 'tcp:': this means we get a cleaner, more scalable threading model within the broker.

And so, it all works just fine. Dejan Bosonac's article on Python messaging: ActiveMQ and RabbitMQ suggests that you can get up to as much as 32,000 JMS destinations on a single broker; that's good to know, but I can't think of a situation where I'd need that right now.

ActiveMQ pooling: a pool by any other name would smell as sweet

2010-06-29T09:42:00.001-07:00

Working with a FUSE customer today who voiced confusion at all the different ways you can do a pooled connection factory in ActiveMQ. With confusion comes fear, uncertainty and distrust. Should I be using Jencks? Or should I be using the Spring CachedConnectionFactory? Wasn't there something on the activemq-pool documentation that said it didn't actually pool consumers? Help!

I took this confusion directly to the source, and had an enlightening discussion with James Strachan and Gary Tully at FuseSourceabout this. As a nice drop off, we ended up updating the documentation on Camel ActiveMQ, ActiveMQ Spring Support, and the Javadoc for the org.apache.activemq.pool.PooledConnectionFactory.

The bottom line is this: while Jencks was recommended in the past, it's no longer necessary as you can just use the org.apache.activemq.pool.PooledConnectionFactory from activemq-pool project. Alternatively, you can of course use the Spring CachedConnectionFactory, as outlined in this great article.

Here's the real sneaky thing though: the JavaDoc documentation of org.apache.activemq.pool.PooledConnectionFactory suggested that this connection factory doesn't pool consumers. This is in fact not not a drawback or a failing: it simply doesn't make sense to 'pool' JMS consumers. Maintain a collection of them for concurrent consumption in parallel? Sure! But keep a 'pool' of consumers, whereby you return consumers into the pool for reuse later on when you're finished? Don't do it! It simply doesn't make much sense - and, at a technical level - could end up creating havoc as the 'idle' consumers would still get messages delivered to their internal 'prefetch' queues, where they'd dwell until the consumer is activated again. We updated the documentation to better explain what the PooledConnectionFactory does.

We realized that the confusion comes from a number of outdated resources on the web that mentioned a myriad combination of ways to do pooling. That, and, the fact that it's easy to confuse the different concepts of 'pooling' and simply maintaining a collection of resources: the former involves sharing and reuse, while the latter does not.

Bottom line: forget about Jencks. Use activemq-pool's PooledConnection or Springs CachedConnectionFactory to manage your connection, session and producer pools. And don't go talking about 'consumer pools' - it really doesn't make sense - talk about 'collections of consumers'.

Four things you need to know about the new JBI cluster engine in ServiceMix 4

2010-04-23T08:48:00.000-07:00

My thanks to Gert Vanthienen, who helped clear the light today as I dug my way through some odd behaviour from the new JBI clustering mechanism in ServiceMix 4 - odd behaviour that was down to some configuration 'gotchas'. I've written about this clustering mechanism before on an earlier blog; there are, however, a number of tricks you need to know about the way to use this mechanism, and I think it's worth mentioning here.

Make sure you register all JBI endpoints when using the OSGi packaging

Cluster the endpoint the sends to the NMR, not the endpoint that receives (this is counter-intuitive!)

If using a network of brokers, disable conduitSubscriptions on the ActiveMQ network connector.

Give each SMX instance a unique clusterName

.

Note: I still haven't fully made my peace with JBI; however, I must say that I'm mellowing a little now that ServiceMix 4 allows you to do simpler packaging (as bundles rather than as service-assemblies) and provides some optimizations to get rid of excessive NMR traffic. Anyway: on to that list:

First, make sure you register your endpoints as JBI endpoints. If you're using the new OSGi packaging of JBI endpoints (which I recommend!) then make sure you add the appropriate EndpointExport to your Spring context. You would of course be doing this for non-clustered JBI endpoints, but you do also need to be doing this for clustered endpoints as well.


<bean class="org.apache.servicemix.common.osgi.EndpointExporter" />

Second, cluster the producer, not the consumer. You would think that 'clustering' would involve some kind of configuration for each of the replicated endpoints 'listening' on the NMR. However, in the SMX 4 implementation, it goes the other way. You need to cluster the endpoint that's putting the information onto the NMR. For example, if you have a file poller endpoint, and you want to send the incoming file to a cluster of bean consumers, you need to 'cluster config' the file-poller using the OsgiSimpleClusterRegistration bean. This is entirely non obvious; here's how to do it. On the file-poller, do:


<beans xmlns="http://www.springframework.org/schema/beans"
 xmlns:file="http://servicemix.apache.org/file/1.0"
 xmlns:clu="http://fusesource.com/clusterdemo"
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:schemaLocation="http://servicemix.apache.org/file/1.0 http://servicemix.apache.org/schema/servicemix-file-2009.01.xsd
       http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.0.xsd">

 <file:poller id="filePoller" service="clu:file-poller" endpoint="endpoint"
  targetService="clu:payload-receiver" file="/tmp/incomingXML" />

 <bean
  class="org.apache.servicemix.jbi.cluster.engine.OsgiSimpleClusterRegistration">
  <property name="endpoint" ref="filePoller" />
 </bean>

  <bean class="org.apache.servicemix.common.osgi.EndpointExporter" />
  
</beans>

and for the 'clustered' bean endpoint, you can omit the OsgiSimpleClusterRegistration bean:


<beans xmlns="http://www.springframework.org/schema/beans"
 xmlns:bean="http://servicemix.apache.org/bean/1.0" xmlns:clu="http://fusesource.com/clusterdemo"
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:schemaLocation="http://servicemix.apache.org/bean/1.0 http://servicemix.apache.org/schema/servicemix-bean-2010.01.0-fuse-01-00.xsd
       http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.0.xsd">

 <bean:endpoint id="beanConsumer" service="clu:payload-receiver" endpoint="endpoint"
  bean="#myBean" />

 <bean id="myBean" class="org.file.processor.su.MyBean" />

  <bean class="org.apache.servicemix.common.osgi.EndpointExporter" />
 
</beans>

Third, and this is the trickiest, disable conduit subscriptions when using networked brokers. If you are using a network of embedded brokers for your internal cluster queue, you must disable conduit subscriptions to ensure that the use of message selectors is respected across the different consumers listening on the cluster queue. How do do this? Simply set conduitSubscriptions=false on your network connector; something like this:


        <networkConnectors>
            <networkConnector name="brokerA" uri="static://(tcp://localhost:61616)" duplex="true"
            conduitSubscriptions="false"/>

And, lastly, you do need to give each instance of SMX a unique name for clustering purposes. You can do this be dropping a file called 'org.apache.servicemix.jbi.cluster.engine.config.cfg' into the etc/ directory of your ServiceMix 4 instance. In this properties file, you can provide a variable called 'clusterName' to identify the this ServiceMix instance in the cluster.


clusterName = smx1

And that's it!

New book on CXF by Packt

2010-02-10T23:46:00.000-08:00

Great to see that Packt have published a book on CXF. As a project, CXF has been consistently establishing itself over the last few years, with a growing community, and, importantly, an increased footprint in production systems. That a publishing house has finally gotten around to putting a book together on CXF is a great endorsement of the project’s success. A copy of the book is winging it’s way to me as I write, so I look forward to providing a fuller review when I get it.

*Sigh*. This is the book I would have liked to have written if I had had the time ;)

Camel vs. JBI

2009-08-10T06:03:00.000-07:00

People are, in general, blown away by Camel. Hearts melt and grown men weep with joy when they learn how easy it is to write an integration flow, package it as an OSGi bundle, and then drop it into the FUSE ESB 4 container. But when the emotions have subsided and rationality comes again to the fore, I often hear the question ‘if Camel can do so much so easily, then why do I need JBI anymore?’. The answer is, that in many cases, you *don’t*.

This is hard for some ServiceMix users to stomach, and delightful to others. I have written before about my ‘love affair’ with JBI on this blog, a love affair that was soon to be blighted by some hardened home truths about what really goes on on the NMR. With my JBI relationship in almost tatters, I embraced Camel with open arms. Am I on the rebound? Hardly. Camel is simply a better technology choice when implementing ESB solutions than JBI. What makes it better? A great DSL (domain specific language) for implementing integration patterns. Extensible beyond Java into other scripting languages and technologies. No canonical intermediary format (XML). Dynamic routing. On-demand type conversion. Parameterizable routes (do *that* with JBI!). Support for unit testing of routing logic with ‘mock’ endpoints. An extensive list of technology components, with easy-to-use URI configuration. All these things make for a faster way to produce more with less.

With all these things going for it, then why continue to support JBI in ServiceMix/ FUSE ESB 4? There are a number of good reasons, and, I like to think that while my amorous intentions for JBI are firmly in the past, we do now remain “Just Good Friends’. There are many users out there who currently rely on JBI solutions, and there are a certain number of users for whom the JBI architecture is a cornerstone of a larger enterprise architecture. For these users, JBI will always be important: one of the reasons why is the promise of a standards-based modular architecture. This is good in large organizations because the adherence to a standard (i.e., the JBI 1.0 specification), you know that a component written by one of your teams in, say, Texas, will work when deployed in JBI a solution developed by a team in Hyderabad. Or, alternatively, a component written by a third-party for a CRM System will plug right into your JBI framework.

Of course, you can get this kind of plug’n’play with Camel - the Camel architecture and APIs are open source! However, some enterprises are going to be wary of working to an API that could change at the whim of a community, and prefer the solidity and comfort that only a standards-based specification can afford. I can appreciate this argument. However, I must respond that, if I were Chief Architect, I’d rather join in with a thriving open source community than a specification that’s been largely untouched and unrevised for some time, such as JBI 1.0.

All that aside, it’s clear that what’s right for some organizations is typically not right for others. ServiceMix (and FUSE) is all about supporting the integration community and, most importantly, offering choice. The advice I give my customers is to use Camel as the integration technology of choice, with an underpinning of ActiveMQ for reliable, asynchronous messaging and a touch of CXF for providing SOA or RESTful facades - all deployed in FUSE ESB 4 using OSGi. For me, the JBI container that used to be at the centre of ServiceMix / FUSE ESB is now a simply a feature that can be used to integrate with third-party or existing components that conform to the JBI specification. Camel, through the ‘jbi:’ and ‘nmr:’ components, can interoperate with such components as necessary. You get the best of all worlds.

To the JBI warriors out there, I salute you. You were not wrong. It’s just that there’s a new ‘right’ in town.

OSGi and dOSGi - from the source

2009-04-10T04:51:00.001-07:00

Very excited about some upcoming webinars on fusesource.com: two legends of OSGi will be talking about why, what and how of OSGi technology. On April 14th (my birthday :)), Eric Newcomer, co-chair of the OSGi Enterprise Expert Group, is going to give an overview of OSGi for the enterprise. Then, on April 28th, the legendary David Bosschaert, one of our Fellow engineers at Progress, and based here in Dublin, will talk about Disributed OSGi. dOSGi answers the natural question asked by OSGi newcomers when the learn about OSGi services: is there any way we share the service registry across a number of JVMs, so that a consumer in one JVM can lookup and invoke on a service in another JVM transparently? David, a committer on Apache CXF, has implemented early prototypes on how to make this happen under the covers, and, I believe, has set CXF up to be part of the reference implementation for dOSGi.

I'm gonna grab me a sandwich log in to these webinars: they're too good to miss!

I note as well that registration at the webinars entitles you to a 10% discount on our online, live training sessions on Apache Camel & Apache ServiceMix 4. It's all good. We ran our first delivery of this course this week, and I was genuinely surprised with how our students coped with - and enjoyed! - the online learning experience. Two days of webinars, live coding, exercises, demonstration, chat via IRC, and access to our tech support guys. Of course, I was very lucky to have Hadrian Zbarcea, PMC chair of the Apache Camel project, with me for the duration, piping in with some great input where Camel is going. Nice.

But anyway: for now, I'm off for Easter...

Muscle meets Might

2009-03-26T06:01:00.001-07:00

Just saw the article on Red Hat Adds Muscle to Apache CXF. Hurrah! This is a great thumbs up and vote of confidence for the great work that has been done by the Apache CXF community. Bringing in this muscle, and combining it with the might of the many committers of the CXF project, lead by fusesource.com's Dan Kulp, is going to establish CXF further as the de facto implementation for open-source web services.

The new JMS flow in ServiceMix 4

2009-03-26T02:41:00.001-07:00

I wandered down to the hotel bar last night and bumped into Guillaume Nodet; we got chatting about the new support for JMS flows in ServiceMix 4. I've written about JMS / JCA flows in a previous blog entry, arguing that, if not correctly configured, they can lead to far too much distribution and a consequent impact on performance. Back then, I argued that using explicit JMS queues - through ActiveMQ or whatever JMS broker you want - can provide just as effective a mechanism for distribution, message persistence, reliability and location transparency.

The latest stuff in SMX4 introduces some new stuff that makes me feel a lot happier about the concept of implementing the NMR with a JMS flow. First, if you want a JBI endpoint to be clustered, you have to explicitly mark it as clustered. This I like, as it gives the developer/designer back control of where the distribution points are in a flow. Second, the JBI flow now uses a single distributed queue for clustering. Previously in ServiceMix 3 a seperate queue was created for each JBI endpoint using the JMS flow; the new approach means less queues and less threads required to support the JMS flow. Different JBI endpoints will listen off the "NMR" queue using optimized message selectors to ensure that messages get consumed by the correct target services. Third, you can now optimize the JBI messaging layer to omit the use of JBI DONE messages: so, if you're implementing a one-way interaction over the JMS flow, you don't end up having to deal with an DONE that you simply don't need. Finally, Guillaume has made a cleaner separation between the JMS flow and the underlying JMS transport. So, you can use ActiveMQ if you want, or, alternatively, plug in another JMS broker to handle your JBI traffic.

I'm looking forward to playing with this, and perhaps documenting these new features. One clear point is that if you've made use of JMS flows in your architecture for ServiceMix 3, then you'll have that same support in ServiceMix 4, with just a little tweaking of configuration.

Good work, Guillaume! And, thanks again for the sandwich!

HamsterJam, ApacheCon, and building businesses out of Apache Software License

2009-03-25T10:09:00.000-07:00

Just thought I'd check in from ApacheCon EU 2009 in Amsterdam. This definitely one of the most engaging conferences I've had the good fortune to attend. Yesterday I gave a one day workshop on building web- and RESTful services using Apache CXF, showing also how to plug in the JMS transport into CXF so that you can do XML/JMS or SOAP/JMS from JAX-WS code. I was blessed with a really good audience, which made for a fun day. Had an interesting scrape though: throughout the day my MacBookPro kept on 'freezing' out of the blue! Not the kind of thing you want when you're doing live-coding: took me eight hours to figure that this was happening because I'd left my bluetooth mouse on in my bag, and it was periodically taking control of my machine whenever it could make a connection. Ouch!

Today I broke with my usual tradition of doing technical talks to join the business track, where I got to follow up after a great panel discussion with my own talk, Adopting Open Source in the Enterprise (follow the link to pick up PDF of the slides). I also really enjoyed Paul Fremantle's talk on building business out of Apache Software License, which agreed with my own observations on the need for ASL-based business to give real business value in terms of their support, services, training and consulting. In my own talk I had talked about Ricardo's Law of Competitive Advantage, and how organizations will support themselves in their adoption of open-source unless a vendor - like Progress with FUSE - can demonstrate that they can provide that support cheaper, better, faster than the organization can do themselves. For me, a vendor offering services around Apache-licensed community software acts as a 'mountain guide', helping users to get most value out of the commons code while minimizing risk.

Right now I'm in a talk by Andreas Gies (a colleague from Progress) who has such great experience in adoption open source for distributed team development. He's a Maven committer, and, in my eyes, an absolute legend. Go Andreas, Go!

On the question of 'ServiceMix vs Camel'

2009-03-10T02:30:00.000-07:00

There was an interesting discussion on the ServiceMix users list on how to choose between implementing integration solutions using ServiceMix or camel; thought I'd post my contribution to that discussion here. ServiceMix should really be thought of as a container, that facilitates the deployment of JBI-based integration solutions and Camel-based solutions for enterprise integration. And, of course, ServiceMix 4 is an OSGi container, which extends the concept of what can be deployed - JBI service assemblies, spring configuration, OSGi bundles, etc.

The question really becomes "Should I use the ServiceMix JBI components (-eip, -file, -ftp, etc.) or should I just use camel components?" I think the decision around this will depend on whether the JBI architecture forms an important underpinning of your solution: if so, then it makes sense to use the JBI components. If, however, you have no desire to go down the pure JBI route, then you can choose to adopt either a pure Camel or a more blended approach.

As Chris Custine and Jean-Baptiste have both pointed out, Camel integrates very effectively with JBI, so you can get the best of both worlds. You might find that the JBI servicemix-ftp component is more robust than the camel-ftp component, and choose to use the former as part of a Camel route. Or, you might prefer the servicemix-eip Aggregator's persistence of state over that of the aggregator in Camel.

So: my advice is, in a nutshell, move forward embracing a blend of both JBI and camel components, and, consider 'ServiceMix' to be a container for both.

Me and Guillaume, talking our heads off about stuff...

2009-02-27T14:28:00.000-08:00

... on a podcast on ZDNet, done with Dana Gardner... enjoy!

FUSE adoption: is it the economy, or just the sheer love of innovation?

2009-02-25T01:50:00.000-08:00

There's a nice post on the adoption of FUSE ESB on ZDNet; in the last few months I've observed an increase in activity and excitement around FUSE. Why might that be? As the entry's quote from Debbie suggests, economic conditions certainly are prompting users to ask questions of their existing close-source providers, and evaluate in more earnest the open-source options and alternatives that are available.

From my perspective though, it's more than that: at a technical level I think users are seeing that the Apache projects that FUSE is based on - ServiceMix, ActiveMQ, Camel and CXF - offer a new way to rapidly create solutions for integration, SOA, and, REST. The potential for developers to rapidly create integration flows using Camel leaves most battle-hardened integration architects and developers bowled over with excitement.

And users are excited by OSGi too; I remember the day when Eric Newcomer, ex CTO of IONA, who was pivotal in bringing OSGi forward into the Enterprise computing space, presented to a group of us in Boston on how this OSGi could change the way we think about deploying Java solutions. David Bosschaert, a colleague of mine at Progress, has continued the charge of OSGi, and has put some very interesting stuff on OSGi and on distributed OSGi up on his blog, <coderthoughts/>. An of course, the illustrious and prolific Guillaume Nodet, also at Progress on the fusesource team, has done great work on driving ServiceMix 4 to an integration-focused OSGi based runtime.

So whatever the reason - economic pressure or technical advantage - I welcome the surge in FUSE activity :)

Free! Free! Free! Screencasts on how to use FUSE ESB 4 (based on Apache Servicemix 4)

2009-02-18T02:26:00.001-08:00

As part of my bi-weekly "Getting Started with FUSE ESB 4" webinar series, I've put together some screencasts of how to do all the good stuff you want to do with FUSE ESB 4. Check out the links at the bottom of the fusesource.com Training Videos page.

Right now the links will download quicktime movies, which you can then run on your machine. The movies range in size from 90Mb to 180Mb, so kick off those downloads, grab a nice cup of coffee, and the sit back and enjoy the ride as I cover how to build and install FUSE ESB 4, create and deploy an OSGi bundle from scratch, create and deploy a web service using CXF, deploy an EIP using a Camel route, and finally, deploy a JBI-based solution into the FUSE ESB 4 runtime.

If you like the videos, or have feedback, please leave a comment!

The White Source

2009-02-11T07:31:00.000-08:00

Ah lovely: a colleague just pointed out an open letter urging Obama To Consider Open Source. I chuckle with content at the thought of Obama hacking away from the oval office - "Not now Mr. Vice President, I'm in the middle of a maven build. Nearly done. Just one... more... compile..." ;)

While it's great to see so many players in the open source community willing to step up to the plate, the use of open source in government is not news; it's a trend that started some time ago; the FUSE suite, for example, has a number of prominent government users already.

Still though, it's great to see the open source movement move forward in a unified way. Who knows: maybe that letter will make it's way onto Obama's desk in the white house...

The day we brought down the internet :(

2009-02-10T11:38:00.000-08:00

Today we had the first of our FUSE ESB 4 Webinar series; we had a fabulous triple digit attendance at the session. We dived right into some live coding, showing how to deploy bundles and other artifacts into FUSE ESB, and were all set to tear away with web services, REST, and a whole lot more... until, unfortunately, after an hour of noble service, some machine in some datacentre decided to give up the ghost, leading to the termination of the webinar.

While it's fun to think that our community's enthusiasm managed to melt some motherboard somewhere, I'm deeply disappointed by this: not just because were about to get the really cool stuff FUSE can do, but also because we had such enormous participation from the attendees who were all very keen to learn more. We'll be working hard to make sure the outage doesn't happen on the next session on February 23rd, when we'll present at 10am GMT and again at 1pm EST. Additionally, I'l be recording some screen casts on each of the six use-cases and we'll post them on fusesource.com.

Till then, thanks to all our attendees for taking part - we hope you enjoyed our session and look forward to hearing from you again.

JAX-RS in FUSE ESB 4.0.0.3

2009-02-10T02:37:00.000-08:00

There's a line in RadioHead song "Phew, for a minute there, I lost myself..." I nearly did loose myself yesterday as I tried to reproduce some of Roberto Rojas's work on deploying a JAX-RS service in FUSE ESB 4.0.0.3 (based on Apache Servicemix 4). Roberto's blog entry refers to a preview edition of FUSE ESB 4, and things have moved on since then.

Luckily, I was able to ping off some of the good folk on the CXF project - Dan Kulp, Sergey Beryozkin, and David Bosschaert, and they got me going again. There's been some reworking of the JAX-RS support in SMX4, and this you need to update the cxf-bundle in FUSE 4.0.0.3, as per below:

servicemix> osgi list | utils grep CXF
[ 89] [Active ] [ 50] ServiceMix :: CXF Binding Component (2008.01.0.fuse)
[ 105] [Active ] [ 50] Apache CXF Runtime JBI Binding (2.1.3.fuse)
[ 106] [Active ] [ 50] Apache CXF Runtime JBI Transport (2.1.3.fuse)
[ 107] [Active ] [ 50] ServiceMix :: CXF Service Engine (2008.01.0.fuse)
[ 142] [Active ] [ 50] Apache CXF Bundle Jar (2.1.3.fuse)
servicemix> osgi uninstall 142

servicemix> osgi install -s mvn:org.apache.cxf/cxf-bundle/2.1.3.3-fuse

FUSE ESB 4.0.0.4, released just last night (Feb 9th 2009), already uses version 2.1.3.3-fuse of cxf-bundle, so if you're using 4.0.0.4 then you shouldn't need this step.

With this in place, you can now go and deploy your restful service. For example, I threw together a bundle with a JAX-RS 0.8 service: to install it, I need to install a few dependencies into FUSE ESB 4:

First, we need the JAX-RS (JSR 311) api. I'm using version 0.8:

servicemix> osgi install -s wrap:mvn:javax.ws.rs/jsr311-api/0.8

Next, you need to install the specs locator and the servicemix jsr311-api bundle so that FUSE can locate and load up the spec and implementation correctly.

servicemix> osgi install -s mvn:org.apache.servicemix.specs/org.apache.servicemix.specs.locator

servicemix> osgi install -s mvn:org.apache.servicemix.specs/org.apache.servicemix.specs.jsr311-api-0.8

While my example isn't using JSON, there is currently a runtime dependency on Jettison, so you need to install that too. You can use the bundle from SpringSource like this:

servicemix> osgi install -s mvn:org.codehaus.jettison/com.springsource.org.codehaus.jettison/1.0.1

Or, alternatively, you can use the cool wrap: functionality to wrap a plain-old java jar

servicemix> osgi install -s mvn:org.codehaus.jettison/jettison/1.0.1

Hurray! Now with all the prerequisites in place, I can install my JAX-RS service.

servicemix> osgi install -s mvn:com.progress.pso/simple-restful-service

And now my RESTful service works a treat :)

Free live online tutorial on FUSE ESB 4!

2009-02-02T13:58:00.000-08:00

Come see me and my fusesource.com colleague Jack Britton give a live two-hour online tutorial on "getting started with FUSE ESB 4" on February 10th! We'll be covering everything from building the source, to implementing EIPs, to writing and deploying web-services, to deploying JBI artifacts. There'll be lots of live code, probably a few hiccups, and a whole-lot-of-fun.

Register here!

Message Selectors and ActiveMQ

2009-01-09T12:01:00.000-08:00

Having just completed a consultancy on FUSE Message Broker (Enterprise Apache ActiveMQ), I just thought I'd write some notes about the use of message selectors in JMS, and, in particular, how FUSE Message Broker implements them.

Message selectors are easy to use, allowing a consumer to specify a filter, using an SQL-like syntax, describing the messages of interest. This filter, example "customerStatus = 'Gold'" instructs the broker to only send messages to the consumer that have a customerStatus header equal to "Gold".

While message selectors are attractive and conceptually very nice, they do place a burden on the broker: consider the case where you have a large number of consumers, say n, on a queue: for each message the broker must evaluate between 1 and n message selectors before delivering the message. In the case of topic, all n message selectors will have to be evaluated for each message. Ouch: that's a lot of work to be putting on the broker, particularly if you're looking for high-throughput.

But apart from this, ActiveMQ's memory paging architecture has an optimisation that can flummox you when using message selectors. When messages are sent persistently, ActiveMQ will only page a portion of the messages in memory, but leave the rest on disk. This is very reasonable: however, one side effect of this is that selectors are only evaluated on those messages in memory, not those in the store. So, if your page size is 100, and you have 100 messages for which there is no current matching consumer, you end up with a "hung" queue. For example, consider the case where you have two consumers, selecting on "color = 'red'" and "color = 'blue'" respectively - here, "color" is a custom JMS message header. Let's say that the first (red) consumer is currently not in operation, and we receive 100 messages for red. Blue is waiting, waiting, waiting, but does not get any messages as there are no matching ones present. Now, we get message 101 - a nice blue message. However, our selectors are only matching against the first 100 messages (all red!) so blue still hangs, waiting, despite the fact that there's a perfectly matching message.

That example is perhaps contrived, but it does demonstrate a potential "hanging consumer" problem. For me though it's a natural side-effect of wanting to limit the broker-side overhead of having to evaluate message selectors. You can of course modify the page size to reduce the chance of the problem happening, but you're just reducing the probability of the lock happening.

My customer had come up with a conceptually elegant callback design, whereby as messages are forwarded through a server, a consumer is created (for each message) to listen on a queue for a subsequent response, using a unique message identifier within a message selector. While conceptually nice as a design, the overhead (in terms of dynamic consumer creation and message selector processing) just isn't worth it. We proposed an alternative architecture using named queues and a fixed pool of response consumers that's going to work way, way better.

Looking forward to ApacheCon Europe 2009

2009-01-08T07:34:00.001-08:00

This year I'm thrilled to be doing a number of talks at ApacheCon Europe 2009, and a full day-long tutorial on building web and REST services with Apache CXF (FUSE Service Framework). Abstracts below - see you there!

Talk: Tales from the Front Line: how ServiceMix, ActiveMQ & CXF are being used to solve real problems. In this session we discuss how application integration and middleware-focussed Apache projects - like ServiceMix, ActiveMQ, Camel & CXF - are being used to build innovative solutions that add real value to organizations in health-care, government, airline-solutions and finance. For each use-case, we discuss both logical and deployment architecture concerns. We also discuss the motives for adopting open-source in these solutions, and why these Apache projects where chosen above other open source alternatives.

Talk: Adopting Open Source in the Enterprise.
For some organizations, the decision to adopt open source software
is not taken lightly, and, once made, can require significant shifts
in thinking and organization culture. This session discusses the
issues organizations face when strategically adopting open source,
including how to select open source projects, how to match open source
involvement & contribution with corporate strategy, how to select &
manage professional open source software vendors, what needs to be
done in terms of licensing and legal, and what kind of infrastructure
you need to support your open source strategy.

Tutorial (full day): Apache CXF - Developing and Deploying Open Source SOA Endpoints
This one-day tutorial will show how to design, build and deploy RESTful and Web-based web services using Apache CXF. The session will cover code-first and WSDL first services design for JAX-WS, and show how to use JAX-RS for restful services. We'll show how to swap different transports (HTTP, JMS) with different payload types (XML, SOAP, JSON). And, we'll show how CXF's support for JavaScript can be used to create client- and serverside components for AJAX applications. This workshop will combine a mix of lectures, whiteboarding, and hands-on exercise sessions.

Sharing an Oracle JDBC pool in SMX4

2008-12-18T05:00:00.000-08:00

In a previous post, I talked about how to share a JDBC pool using ServiceMix4; in that example, I used a Postgres connection pool. I wanted to the same with Oracle as the base driver and the commons-dbcp pooling library and went on my merry way. But then, disaster struck: I got a dreaded ClassNotFound exception. "Can this be?" I thought... isn't OSGi supposed to shield me from class-loading hell and put some colour back into my graying hair?

Delving in, I figured out what the issue was, and, I'm glad to say, the ServiceMix OSGi kernel was behaving exactly as it should. I had wrapped the commons-dbcp JAR without much thought: it turns out that code in the commons org.apache.commons.dbcp.BasicDataSource class (see below) is doing a Class.forName() call to load the oracle.jdbc.driver.OracleDriver class.

<bean id="oracle-ds" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close">
<property name="driverClassName" value="oracle.jdbc.driver.OracleDriver"/>
<property name="url" value="jdbc:oracle:thin:@localhost:1521:BLAH"/>
<property name="username" value="blah"/>
<property name="password" value="blah"/>
<property name="maxActive" value="10"/>
<property name="poolPreparedStatements" value="true"/>
</bean>

Now, OSGi bundles have to specify the packages they import using the Import-Package tag. That's all well and good if you know what these packages are in advance. Poor commons-dbcp can't know in advance what kind of driver it should load, so Import-Package just doesn't work here. Instead, you've got to allow the commons-dbcp bundle to import anything: you can do this using the DynamicImport-Package tag, setting it to DynamicImport-Package: *.

We're currently patching the commons-dbcp bundle on the ServiceMix bundle repository so that it includes the DynamicImport-Package: * tag by default.

How to share a single JDBC Pool across SMX4 bundles

2008-12-17T13:25:00.001-08:00

In the last week I have been working on a solution involving a number of Camel routes deployed into FUSE ESB 4 (ServiceMix4). The new OSGi archetypes for ServiceMix4 (servicemix-osgi-camel-archetype, servicemix-osgi-cxf-code-first-archetype and servicemix-osgi-cxf-wsdl-first-archetype) are really handy in that regard - nice work Ashwin Karpe! Those new archetypes have been checked in to SVN, but may not be in the release build yet so if you want them you may need to check them out and build them locally.

Because SMX4 is built on OSGi, you can deploy any Java artifact, not just JBI components. I put this to the test: my customer wanted to have a single, shared JDBC pool across all of his bundles. You can do this by creating the pool in a simple Spring file, and registering it as a service for all the other bundles to use. To do this, I installed the driver for PostgreSQL into my SMX container, using the nifty "wrap:" feature that will take a jar and osgi-ify it. Then, I created a Spring context file to intantiate the bean, like this:


    <bean id="postgresPoolingDS" class="org.postgresql.ds.PGPoolingDataSource">
      <property name="serverName" value="localhost"/>
      <property name="databaseName" value="play"/>
      <property name="portNumber" value="0"/>
      <property name="user" value="play"/>
      <property name="password" value="pa55w0rd"/>
      <property name="dataSourceName" value="postgres"/>
      <property name="initialConnections" value="1"/>
      <property name="maxConnections" value="100"/>
    </bean>

    <osgi:service id="postgresPoolingDSService" ref="postgresPoolingDS" 
      auto-export="all-classes"/>

Here's the really neat bit: when you drop this spring file into the deploy directory of ServiceMix 4, the ServiceMix kernel will osgi-ify the file and deploy it as a bundle. Hey-presto: the pool is created and registered as a service in the OSGi service registry.

To access this service from one of my camel routes, I used the <osgi:reference> to get a reference to the dataSource in my bundle's spring file. You can then inject the reference into your route and away you go.


    <osgi:reference id="dataSource" interface="javax.sql.DataSource" />

The "look-up" is done by interface type: in this case I'm OK as there's only one javax.sql.DataSource registered. I should really do a more explicit query, but I guess I'll leave that as an exercise for the interested reader.

For me, this kind of thing is OSGi at its best: allowing sharing and reuse of Java classes and Java objects in a really nice, modular way.

Take care when propagating transport headers through Camel!

2008-12-01T11:55:00.000-08:00

Last week I worked with a fellow FUSE enthusiast on a very cool proof-of-concept showing how to route XML messages containing partially encrypted message payload. We used FUSE Services Framework (Apache CXF) to create a SOAP service and consumer, and used the nifty WSS4J interceptors (thanks to Glen Mazza's excellent blog entry on the subject) to encrypt selected elements of the SOAP message. CXF is such a great project - we got our service running really quickly without any trouble at all.

Feeling boisterous and buoyant with the taste of success, we forged ahead to put an intermediary Camel route between the service and the consumer, to do some content-based routing based on the non-encrypted part of the payload. This kind of X-Path routing is really easy to do with FUSE Medation Router (Apache Camel). Thinking that our coffee break was only minutes away we ran a quick test only to witness an exception on the SOAP consumer, deep in some HTTP-commons code: "bad chunk charachter '60'". Very strange, we thought: we knew that the unencrypted version worked fine. And, we know that it worked fine when the consumer talked directly with the service.

We investigated. A cursory look at the ASCII table shows that '60' is the code for '<'. Hmmm. Instantly this stank of something going on with the XML payload. But what? Our hopes for fresh coffee were now overrun by a burning desire to figure this out. We looked at all the angles, and eventually after an hour or so of scatching our heads, saw the light. The CXF consumer (the client!) was transmitting the payload using HTTP chunking: this has two effects at a payload level. First, a transport attribute, Transfer-Encoding, is set as "chunked"; second, the payload is then transmitted "chunked" by breaking it up into smaller chunks and transmitting each chunk preceded with a line containing a number indicating the size of the chunk.

CXF provides support for chunking, as do the Camel HTTP and Jetty components; so, the problem was not the chunking per se. Here's what was going on: the Camel route was receiving the payload and reassembling (or "unchunking") the content; however, our route was passing on the Transfer-Encoding header intact. So, when the payload arrived at the target service, the payload was unchunked, but the header suggested the opposite. The server read the first line of the payload to get the chunk size, got a '<', and correctly argued that this is an invalid character to describe the chunk size.

We disabled chunking on the client side in CXF, and then everything worked fine - effectively this just removes the problem by cutting it off at source. In general though, I think it raises a word of caution: be mindful of the headers you propagate through integration flows.

Does Unique Selling Point have any meaning in for Open Source professional services firms?

2008-11-10T06:21:00.000-08:00

On my way home back from a gig in Kiev last week, I read a truly enlightening, fun and excellent white paper from the smart folk of Wellesley Hills Group, posted at QuickArrow. The paper - or rather, collection of articles - entitled "Truth and Lies in Professional Services Marketing", argues that the dynamics of services business are different from those of a product business: consequently, the rules and mantra's of marketing need to be tweaked (or thrown out) when applied to professional services businesses.

I was particularly interested in because, in the last year, I have studied a Diploma in Business Development at the Irish Management Institute; I graduate next week (with distinction: hurrah!). The Dip Bus. Dev. is a great program, particularly for myself as a person of technical origin. Regularly I would meet with my assigned business mentor, who would administer a ritual and deserved mental beating as she questioned everything I had ever held dear. One of the most difficult things for me on this course was to let go of my need for scientific, clear reasoning, and embrace fuzzier, more intuitive business concepts.

But back to that paper! Let's say you're running a professional services firm that specializes in open source. What can you say is your Unique Selling Point? Hmmm. That you know the source? Everyone knows the source dude, that's the whole point! That you have committers on your team? Better... however, anyone with smarts can become a committer in an open an vibrant community, so if this is your USP and your open source projects are successful, then your competitors will quickly become committers: there goes your USP, gently flying out the window. The open nature of open source means that the barriers to entry are low; sure, becoming an expert on an open-source project takes time, but anyone with smarts and the willingness to invest the time can eventually break into your space. Back to the USP: you could talk about experience, customer-focus, reliability, ... but then, so will all your competitors, so they're not good unique selling point candidates either.

So: forget about "uniqueness" for now. The nature of open source services is that, if your project is successful, then your firm will become one of many others - an ecosystem, if you will - offering similar services around the code. Being a PS firm in open source is very much like being a law firm, or a chartered accountancy firm, or a doctor: all rely on an open body of knowledge at which they have become experts, and they sell that expertise for a price. It's not about "uniqueness": when you want a lawyer, accountant or doctor you do not care if they're unique or different: you care that you can trust these guys to solve your problem and make things right. A strong reputation, with credible proof of their expertise, is more important than a unique selling point.

Successfully marketing your services, then, is not about USPs; it's about having a genuine, credible, reputation. So how can you build this? By taking part in the community. By contributing fixes, patches, and documentation. By helping newbie users into the community. By speaking, writing and blogging about the knowledge for which you are expert, at community events (with your peers) and at industry events. By being clear about how you your services can help others, who don't have your level of experience or expertise. By being reliably, trustworthy, and professional with your customers - so that they're happy to endorse you.

Perhaps, it can be argued, this "reputation" becomes your unique selling point. However, it's a reputation built on real evidence, not on slick marketing material or finely-tuned elevator pitches.

Give that paper a read - it's well worth it!

Explicitly creating dead letter queues in ActiveMQ using individualDeadLetterStrategy

2008-11-05T14:09:00.000-08:00

There are lots of great features in ActiveMQ that are documented only in the code, which is a shame, as it means that it sometimes means developers are "six degrees of google searchin'" away from finding out how to enable the feature.

I came across this one while working for a customer: they're happy with the concept of the default dead letter queue in ActiveMQ, but need to know how configure an individual DLQ for each of their queues. The answer is to configure an 'individualDeadLetterStrategy', and I've updated the ActiveMQ wiki pages to show how to configure this. This change should make it's way through to the Apache ActiveMQ web site overnight.

JMS/JCA Flows in ServiceMix: the wrong level of abstraction for distributed ESB

2008-11-03T10:09:00.001-08:00

The ongoing argument over REST versus RPC has created useful debate around programming abstractions, particularly with respect to the use of abstractions in distributed computing. Abstractions are useful in that they remove underlying complexities and allow us to focus on the task at hand. However astractions fail if they hide away important detail that could influence design or implementation decisions. For example, the problem with RPC, some RESTful folks would argue, is that in making remote invocations look like local invocations, RPC facilitates poor design decisions that neglect caching, error handling, or optimization of network traffic. And, they may have a good point here; however, in this blog entry I don't wish to go down the well-beaten REST/RPC track, instead though I want to discuss the use of a particular abstraction in implementing integration solutions with FUSE ESB (Enterprise Apache ServiceMix). I've found that while this abstraction - the JMS/JCA flows - offers exciting possibilities, it is the wrong abstraction, and should be avoided.

I had been pondering for some time about the use of abstractions within the ServiceMix implementation of the JBI Normalized Message Router (NMR). The NMR is used to send messages from one component to another; in ServiceMix, this is implemented using a number of different "flows". Depending on the kind of message exchange (syncronous or asynchronous), quality of service (reliable, transacted), and whether the target service is clustered, the NMR will choose a "flow" to match.

Four flows are provided in ServiceMix - single-threaded (ST) , SEDA, JMS & JCA. Of these, the latter two have interesting and exotic qualities. First, because they're both based on JMS, they are reliable: messages can be persisted as they travel between components. Second, and again, because they are implemented using ActiveMQ JMS queues, this means that if the ServiceMix container is deployed in a cluster, then messages can transparently be delivered across JVM containers. This gives us a truly "distributed ESB".

Many are drawn to this visionary usage of the NMR; however, I strongly believe that clustering & persistence of the NMR is the wrong level of abstraction. First, you are unwittingly exposing your solution to potentially unacceptable performance hit. With a little knowledge of the NMR's message passing semantics, you'll see that even a simple integration solution (like my File -> Pipeline EIP -> transfomer -> JMS) takes more NMR messages than you think: in this case, seven messages are used. Wow, a simple little integration solution using the JMS-flow takes a total of three persisted JMS queues and four temporary queues! And each of those persistant queues, in ActiveMQ, will need it's own thread! In fact, if you've left the default flow settings enabled in ServiceMix, you'll see that we create two queues (and hence, two threads) for each service on the NMR (one for JMS-flow, and another for JCA-flow!). I was curious about the impact of these flows on ServiceMix threading, so played around with the configuration. I found that by simply disabling the JMS & JCA flows in ServiceMix3, I was able to reduce the number of threads on startup by about 50%. Yup, that's right. I halved the number of threads lurking in the JVM.

So, a deeper look shows that this "flow" abstraction is going to consume resources and impact on your latency & throughput. It gets even hairier though: with a flow like the one I detailed above, it's possible that in a clustered environment your message exchange could start out on the file component of machine A, get pipelined by machine B, get transformed back in machine A, and then get send to the JMS provider on machine B. All that remote network traffic when all I ever wanted was to do a little transformation on a file and drop it into a queue.

I had the good fortune to be in town with Guillaume Nodet, Godfather of Servicemix, last week, and talked to him about this; Guillaume is an intelligent and fun guy and if you get a chance to meet him then take it! It turns out that the JMS/JCA flow feature of ServiceMix was there before the NMR was refactored to become JBI-compliant. So, back when it was invented, it wasn't burdened by the additional NMR traffic mandated by the JBI spec. So, these flows made more sense back then, but really don't make so much sense now. You can get everything you want in terms of reliability by using explicit queues as checkpoints in a transactional SEDA architecture: just use the SEDA flow, and, where transaction propagation is required, set the message exchange to be synchronous.

Rather than engage in feature debridement in ServiceMix3, Guillaume has made the right move in choosing to leave these flows in ServiceMix3 but omit them from ServiceMix4. There is talk about reimplementing the JMS flow for ServiceMix4 for "backwards compatability", but I for one would rather see Guillaume invest his and the community's efforts on other more pressing stuff. A really nice feature of ServiceMix4 is that you will be able to transactional propagation on the SEDA flow with asynchronous as well as synchronous messages, so those who invest in the SEDA appraoch now can avail of better thread-performance at the flick of a switch when they move to ServiceMix4.

Both Guillaume and I were agreed in that if you want to introduce clustering, reliability or transactionality into your integration solution, then do so using explicitly named JMS queues. That way, you have full control over when and where your message exchanges are persisted, transacted or clustered. It's easy, and more importantly, it's the Right Level of Abstraction for integration solutions.

AMQP back in the radar...

2008-10-24T13:00:00.000-07:00

Just learned that Microsoft has joined up with AMQP. This is great news - I was playing with AMQP some time ago but in recent months I got a feeling that there was a lot of sitting on the fence about whether people are going to take up on it or not. There's nothing so sad as taking a great idea with a well thought out standard and seeing it get a lack-lustre take-up. This is just the kind of injection AMQP needed.

As I recall, Apache Camel already has support for AMQP, so that'll be a very nice bridging tool for organizations who want to migrate piecemeal towards an AMQP-based infrastructure.

It'll be nice to see what the full extend of Microsoft's participation will be: are they going to buy in wholesale or are they going to just look nice in the background of wedding photographs, enjoying the party? Will they be actively implementing AMQP brokers, or will they just add some AMQP connectivity options to their existing integration offerings?

Things are getting interesting again for AMQP :)

Sneak Preview: my FUSE ESB / ServiceMix screencasts...

2008-10-22T23:40:00.000-07:00

Over the last few weeks I've worked on putting some screen casts together on how to get "over the hump" with ServiceMix: you've read all about it, you've downloaded it, and then you wonder: where do I start?

In the screen casts, I show:

Three things to make you more productive with ServceMix - see how to set up your development environment to get stuff done faster.

How to create a simple flow - from file pickup to JMS queue.

How to XSLT transform file data and place on a JMS queue using a pipeline EIP.

The screencasts aren't long, 8, 25 & 19 minutes respectively. Use the streamed video below to get an idea of what's going on in the screen casts: if you like them, then let me know and I'll send an FTP link to where you can download the full resolution 800x600 quick-time movies.

Three things to make you more productive with ServceMix

How to create a simple flow - from file pickup to JMS queue

How to XSLT transform file data and place on a JMS queue using a pipeline EIP

FUSE TV: Andy Warhol may have been right...

2008-10-22T10:25:00.000-07:00

Didn't Warhol say something about in the future everyone will have 15 minutes of fame? I, along with some of my Progress colleagues, have been recorded for FUSE TV. My own slot is 13m 13s, which means that I may have only 1m 47s left. Hmmmm... better use those precious seconds sparingly...

m2eclipse: developing with ServiceMix / FUSE archetypes has never been so easy

2008-10-08T13:31:00.001-07:00

Hurrah for Cool Tooling!

Maven archetypes are great: anyone developing with ServiceMix will know and appreciate good archetypes, but will probably grumble disapprovingly at the long command lines they entail. In ServiceMix, the smx-arch command-line short-cut made things easier, but I was always uncomfortable with having to swich from Eclipse to the command-line to create my new projects.

In the last few days, Guillaume Nodet put in a tiny little script onto the FUSE repository (http://repo.open.iona.com/maven2) that generates an archetypes-catalog.xml file containing all the archetypes. You can import this URL into m2eclipse, and then use the really neat m2eclipse dialogs to create your maven/eclipse projects directly from the catalog.

I love it so much I added an entry on the FUSE wiki; go there for more details!

Carpe springem - community to the rescue?

2008-10-06T13:24:00.000-07:00

I was following a trail on the CXF users list, when Dan Kulp introduced "FreeSpring" (http://www.freespring.org), a community-based effort that has come about to alleviate the effects of the new SpringSource licensing system. Their aim is to solve three key problems for Spring users who do not want to pay SpringSource subscriptions for fixes after the three-month post-release dust-settling period: creating versioned binary distributions, applying community fixes, and, importantly, providing distributions through maven.

I admire the initiative, but was of course keen to find out how freespring.org plan to keep the lights on... their intent is to get some corporate backing from benevolent organisations (perhaps SpringSource competitors?) and also to sell advertising space. I think the latter may be a go-er: freespring.org has the opportunity to become a hub for open-minded Java developers.

Using AspectJ to diagnose ServiceMix component performance

2008-09-25T09:12:00.000-07:00

An inventive FUSE customer showed me an approach to diagnosing his ServiceMix integration flow by injecting some aspect-oriented code, and it's so cool that I thought it would be worth sharing. The problem is this: given that you have an integration flow, how can you diagnose how long each endpoint in the flow is taking to do its business? The customer in particular was experiencing exceptionally high latency on his integration flow (in the order of 900ms) that just seemed plain wrong: the question was, where in the flow was he incurring the hit? Was it in transformation, EIP, resequencing, or something else?

I used AspectJ to reproduce my customer's approach, weaving in some diagnosis code into the ServiceMix call stack. You can do this by creating a simple aspect like the one below, which is called whenever a call is made to Flow.send(). The measurePeformance() method does some logging, but also does a simple timer around the message invocation. The result will only be accurate to the millisecond; however, if you're trying to isolate big-elephant-in-the-room bottlenecks then this should be sufficient.


@Aspect
public class PerformanceAdvice {

   @Around("execution(* org.apache.servicemix.jbi.nmr.flow.Flow.send(..))")
   public Object measurePerformance(ProceedingJoinPoint thisJoinPoint) {  
       Object ret = null;
       try {
           MessageExchange me = (MessageExchange) thisJoinPoint.getArgs()[0];
               System.out.println("send() to " + me.getEndpoint().getServiceName() + "... (status: " + me.getStatus() + ", role: "
                       + (me.getRole() == MessageExchange.Role.PROVIDER ? "PROVIDER" : "CONSUMER")
                       + ")");
               long startTime = System.currentTimeMillis();
                ret = thisJoinPoint.proceed();
               long endTime = System.currentTimeMillis();
               System.out.println(me.getEndpoint().getServiceName() + " done; elapsed time = " + (endTime - startTime) + " ms.");


       } catch (Throwable e) {
           System.out.println(e.getMessage());
       }
       return ret;
   }
}

To weave this code into ServiceMix (I used version 3.3.1.6-fuse from http://open.iona.com) was surprising easily: I added the JARs for aspectj (aspectjlib.jar, aspectjtools.jar, aspectjrt.jar, aspectjweaver.jar) into ServiceMix's lib directory. I also jarred up my PerformanceAdvice class and dropped it into the lib directory too. Finally, I modified the ServiceMix configuration file - conf/servicemix.xml - to create my advice and turn on auto-proxying for AspectJ. First, I added the Spring AOP namespace to the tag:

xmlns:aop="http://www.springframework.org/schema/aop"

Then, I added the following elements within the element:


<aop:aspectj-autoproxy/>

<bean id="performanceAdvice" class="ps.progress.com.PerformanceAdvice">

And that's it! To test, I ran a simple integration flow that reads from a JMS queue and uses a pipeline to transform the message and send the result to another queue. Here's the output:


 Invoking on {http://progress.com/ps/smx/demo/transformer}Pipeline... (status: Active, role: PROVIDER)
 {http://progress.com/ps/smx/demo/transformer}Pipeline done; elapsed time = 0 ms.
 Invoking on {http://progress.com/ps/smx/demo/transformer}XsltTransformer... (status: Active, role: PROVIDER)
 {http://progress.com/ps/smx/demo/transformer}XsltTransformer done; elapsed time = 1 ms.
 Invoking on {http://progress.com/ps/smx/demo/transformer}XsltTransformer... (status: Active, role: CONSUMER)
 {http://progress.com/ps/smx/demo/transformer}XsltTransformer done; elapsed time = 0 ms.
 Invoking on {http://progress.com/ps/smx/demo/transformer}JmsOutput... (status: Active, role: PROVIDER)
 {http://progress.com/ps/smx/demo/transformer}JmsOutput done; elapsed time = 0 ms.
 Invoking on {http://progress.com/ps/smx/demo/transformer}JmsOutput... (status: Done, role: CONSUMER)
 {http://progress.com/ps/smx/demo/transformer}JmsOutput done; elapsed time = 1 ms.
 Invoking on {http://progress.com/ps/smx/demo/transformer}XsltTransformer... (status: Done, role: PROVIDER)
 {http://progress.com/ps/smx/demo/transformer}XsltTransformer done; elapsed time = 0 ms.
 Invoking on {http://progress.com/ps/smx/demo/transformer}Pipeline... (status: Done, role: CONSUMER)
 {http://progress.com/ps/smx/demo/transformer}Pipeline done; elapsed time = 1 ms.

You can see that there's lots of messages being sent out - we're most interested in those where the status is Active and the role is PROVIDER (you can modify the PerformanceAdvice class to just print those out if you like). From this, I can see that the call to the XsltTransformer endpoint is taking just 1ms; a nice verification that my XSLT transform is doing anything crazy.

Using an approach similar to this, my customer was able to show that his bottleneck was actually in one of his own handmade components that invoked on a back-end server using RMI - the hit was in the order of 850ms, using the lion's share of the latency. I'm not sure what he's done since then to minimize this hit, but at least we know where the problem is.

In summary: Aspect-Oriented Programming is great for weaving in cross-cutting concerns such as logging, security or transactionality into your application code - it's also a nice tool to have for performance diagnosis.

CXF, JMS & the risk of acknowledged-but-not-processed messages

2008-09-19T03:06:00.000-07:00

May God speed Christian Schneider and his refactoring of the CXF JMS endpoints to be more configurable. I was looking into the reliability of the JMS implementation in CXF, and (from the code), I see that the JMS acknowledgement mode used by CXF is hard-coded to be Session.AUTO_ACKNOWLEDGE. So, I thought, it's not transactional - but that's ok, right? As long as the message is acknowledged after my CXF implementation code then I'm good... right?

I wanted to verify that this was the case; so, I modified the jms_queue demo to exit unmercifully in the middle CXF impl code for the greetMe() and greetMeOneWay() methods. I had hoped that this would mean that the incoming message, having been unacknowledged, would be redelivered - however, it appears that the message is not redelivered. It looks like CXF is acknowledging the message before we actually process it. Ouch.

The implications of this are serious: if you're implementing a listener for one-way (in-only) messages with CXF then there is a possibility that messages delivered just before an emergency server shutdown will have been acknowledged but not processed: lost forever. To get around this for now, I'd recommend using Camel to listen transactionally from the JMS queue and send the message to CXF for marshalling (as per Christian's article "Better JMS Transport for CXF"). That way, if the server goes down then message will be redelivered at a later stage. Of course, you should in your code check to see if the message is a redelivery, and take appropriate action to ensure that your application remains consistent.

The impact of this "early acknowledgement" on request-response services over JMS is not so fatal, as failure of the sever will result in no reply message getting to the client - a timeout will alert the client that something has gone wrong and they can resend.

Setting up a PostgreSQL database for ActiveMQ

2008-09-09T22:08:00.000-07:00

The default ActiveMQ configuration file (in conf/activemq.xml) has a sample fragment of Spring configuration that shows how to use PostgreSQL as a data source. So, pointing ActiveMQ to PostgreSQL is easy; the problem you may run into is how to configure PostgreSQL in the first place with an appropriate account for ActiveMQ.

The instructions below assume that you've installed postgres in the directory /usr/local/pgsql, and that postgres is being run by the user 'postgresql'. It also assumes that you're using an bash or sh shell; however, I"m sure it's not rocket science to map what follows to DOS shell for Windows.

You should include the PostgreSQL bin directory in your path:


 $ export PATH=/usr/local/pgsql/bin:$PATH

Decide on a file system location for the postgresql data files, say: /Users/postgresql/data. Set the PGDATA variable to point to this directory.


 $ export PGDATA=/Users/postgresql/data

The commands that follow will use $PGDATA to determine the location of the database files.

Initialize the database data directory:


 $ initdb

... This will initialize the database in the $PGDATA directory.

Now start the database controller:


 $ pg_ctl -l logfile start

You should see a message 'server starting'

Now create a database called activemq-db


 $ createdb activemq-db

To log into the database, use the psql command:


 $ psql activemq-db

This gets you in as the current user ('postgresql') - the default security is very lax: let's enforce password protection for the database. To do this, first create a password for the postgresql user:


 activemq-db=> alter user postgresql with password 'foo';

Create a username for activemq with appropriate password:


 activemq-db=> create user activemq with password 'pa55w0rd';

Exit the psql shell using the \q command:

\q

Edit the $PGDATA/conf/pg_hba.conf file, to specify that the activemq user can access the activemq-db.


 # TYPE  DATABASE    USER        CIDR-ADDRESS          METHOD
 host    all          postgresql    127.0.0.1         255.255.255.255   md5
 local   all          postgresql                                        md5
 host    activemq-db  activemq      127.0.0.1         255.255.255.255   md5
 local   activemq-db  activemq                                        md5

Now, bounce the database.


 $ pg_ctl reload

You should now be prompted for a password when you log into to the database as the activemq user.


 $ psql activemq-db activemq
 Password for user activemq: 
 Welcome to psql 8.3.3, the PostgreSQL interactive terminal.
 
 Type:  \copyright for distribution terms
        \h for help with SQL commands
        \? for help with psql commands
        \g or terminate with semicolon to execute query
        \q to quit
 
 activemq-db=>

In activemq, configure your postgres-ds data source to use the database 'activemq-db', with user 'activemq' and password 'pa55w0rd'. Now, when you start up, activemq will create the necessary tables to persist messages. You can validate that it has created these tables by using the \d command in psql:


 activemq-db=> \d
              List of relations
  Schema |     Name      | Type  |  Owner   
 --------+---------------+-------+----------
  public | activemq_acks | table | activemq
  public | activemq_lock | table | activemq
  public | activemq_msgs | table | activemq
 (3 rows)

And... you're done!

JAX-RS (JSR-311) & Apache CXF

2008-07-16T09:40:00.000-07:00

Yesterday I did my first Webinar on open.iona.com (see "Open Source in the Enterprise, Webinar III" in http://open.iona.com/resources/video-archived-webinars/). Dana Gardner covered big trends in the middleware and infrastructure sector, I talked about our support in FUSE for building RESTful services using JAX-RS, and my fellow IONA-ian Roland Tritsch did a live code demo showing, among other things, how to access both RESTful and SOAP services using browser-based javascript connected to a FUSE (Enterprise Apache CXF) service. Nice!

I must echo Dan Kulp's thanks to Sergey Beryozkin for putting together some really nice work on CXF's JAX-RS implementation: I've been using the original JRA annotations with CXF previously, and the new JAX-RS stuff is simpler to use: no need to worry about wrapping parameter payload, and much easier to specify content types. Thanks Sergey!

There's some good documentation on the JAX-RS CXF implementation on the CXF site. There's also a list of todo's there as well; one thing I'd love to see is support for marking responses as cacheable so that we really cash-in (errr, exuse the pun) on REST.

Reflections on Progress and IONA from Reykjavik

2008-06-25T12:50:00.000-07:00

I have spent the day traveling on airplanes, and finally arrived in beautiful Reykjavik at about 1600 without baggage but very glad to be in Iceland again. It's a beautiful, awe-inspiring landscape and the people are lovely. Is this a thing were folk from island nations naturally resonate with each other? Anyway: I'm here doing some work on Apache CXF with a big player in the telco market here. I look forward to getting stuck in tomorrow.

I've been out of touch for most of the day, gleaning information on a drip about today's big news that Progress Software has agreed to purchased the company I work for, IONA Technologies, later this year. This is a really great move for Progress; IONA has strong technology offerings, an innovative engineering department and a smart services organization hell-bent on customer success. It's also a fantastic opportunity for the great people I work with in IONA to play an even bigger role in the SOA market. Today's statement also indicated continued investment in IONA's Orbix, Artix and open-source FUSE offerings; I'm looking forward to seeing how Progress will approach a blended, hybrid business model. And, I can't wait to get my hands on Progress's technology :)

IONA has been such a great place to work, since my first incarnation here in 1995 as an engineer. I was reminiscing with myself over dinner on all the places I've traveled to while working as a consultant in the "old IONA", and got so excited that I mapped them on Google Maps. Check it out! Now, where is the new Progress going to take us?

Tales from the Serverside Prague - Day III

2008-06-20T07:14:00.000-07:00

Hurrah for Scala! Ted Neward showed us all the sheer joy of Scala this morning. I think I might be hooked. Yet now I am truly torn: to which language do I dedicate my spare cycles - jRuby, Groovy or Scala? The answer I think is all of them, for the sheer fun of it. The exotic part of me says jRuby first, but then the pragmatic part of me says Scala might be the way to go.

I've been dipping in and out of the day since then: I sat in on John Davies replay of his extreme transaction processing material. After his talk the feedback forms were glowing with positives like "the best material in the whole conference" Nice work John - just what we'd expect from an IONA employee ;)

Ola Bini from ThoughtWorks showed off couple of jRuby test frameworks; one of the nice side applications of this, apart from testing your jRuby code, is that you can use jRuby to test your Java code. As it turns out I spent most his talk down the back doing some Junit test cases for a project I'm working on at the moment; there is not doubt that Ola's tests in jRuby looked a hell-of-a-lot less verbose than my Java-bound test framework.

I attended the fireside chat on Architectural Patterns in Mule by Antoine Borg. He's doing a great job; that said, it's all the same as the stuff I covered in my own fireside chat yesterday! The EIPs supported by Mule directory are implemented in XML; while functionaly equivalent it's no where near as ergonomic as the Camel DSL, in my humble opinion. I've done XML-based EIP before, and you can get bogged down in XML. It is nice to see that Mule 2.0 is using Spring Schemas to simplify the XML configuration.

Off home in a few hours to Dublin. This year's TSSJS in Prague was a really high quality event; hats off to the speakers, and a big thank you to the tech target team!

Tales from the Serverside Prague - Day II

2008-06-20T01:01:00.001-07:00

Neal Ford gave a cracking keynote this morning on the evolution of programming abstractions towards domain specific languages. He talked about internal DSL's (like Apache Camel) using fluent interfaces in languages like Java, and proposed that internal DSL's work better in languagues like ruby and groovy. Ultimately though, internal DSL's are limited by the language they're embedded in, forcing you to match the parent language's syntax . In an external DSL, you design your language from scratch; this year a number of products (like mps from JetBrains) will address the problem of how to easily create domain specific languages.

There was a whole part of Neal's talk about considering XML to be "deprecated". My friend Ciaran McHale has believed this since day one ;) Neal did concede that XML is "marginally OK as a data format", but in terms of configuration or communicating intent it's overkill.

Eugene Ciurana's presentation on real-world SOA & event-driven architecture showed a really nice case study of a SOA system. One of the biggest challenges on this is how to get information from millions of network-enabled children's toys. Interesting stuff, but also interesting in terms of the price tag; he saved quite a lot of money by using open source infrastructure along with best-of-breed commercial products.

I hooked up with Ted Neward and the new editor of TSS, Peter Varhol on the way to John Davies's talk on extreme transaction processing. Ted is in great form; we had a chat about DSLs and what it might take to get the JVM to make the most of multi-core CPUs. Dammit we should have had a tape recorder; who knows though, maybe there was some secret agents knocking around. That said, many of the things we had chatted about came up again in the expert panel on next generation languages, with Ted Neward, Ola Bini and Guillaume LaForge. Ola's jRuby talk later on has shown me the light; I am now torn between going headlong into either a scala or jRuby devlopement binge.

Kirk Pepperdine blew everyone away with a presentation on Java performance and concurrency. The rise of the multi-core CPU means that now, more than ever, sloppy multi-threaded code is going to begin to throw up problems that never arose in our single CPU machines. Hmmm. Maybe I'll hang on to this old single-CPU Dell laptop for a bit longer...

I gave a fireside chat on implementing enterprise integration patterns using Apache software. I've been working with the FUSE distribution of ServiceMix, Active MQ, Camel and CXF for some time now and it was nice to get a chance to share the experience. I showed off the Camel DSL, and we went on to talk about innovation adoption (getting around the "let's just code it up in Java" objection), open source licensing (is there a best license / business model?), and whether standards really matter. On that last point I think we reached some agreement: standards matter at the edges of your system where you're working with other parties. Internally, however, you're free to do whatever you want.

Tales from the Serverside Prague - Day I

2008-06-18T08:01:00.000-07:00

Day one at the server-side Java Symposium in Prague; but the fun started last night when Kirk Pepperdine smacked me for suggesting that the agenda was very like last year. Last year in Barcleona was all about caching technologies; this year the focus is on dynamic languages: jRuby, Groovy and Scala are all on the agenda.

Stephan Janssen kicked off the day's proceedings with a really good keynote on the landscape of technologies for developing rich internet applications, including DHTML, Google Web Toolkit, Flex/Air & JavaFX. I think what was most impressive is that he showed how the parleys.com web-site (an educational web-site for posting up tech talks including audio, video and slides) has been built using all four (five, if you include this Silverlight version). Kudos to Stephan: I've always felt that when you apply a number of techhnologies to the same problem you get a really good picture of which works best. The message? DHTML was difficult. GWT was great, but needed one or two tickles to support Internet Explorer. Flex/Air was great, but you have two different development streams for your online & offline application code. Finally, JavaFX is very promising, but adoption may be hindered by the fact that the nice guys at Sun haven't got a version of JavaFX for Apple Mac yet.

Costin Leau from SpringSource gave an in-depth view of how Spring Dynamic Modules supports OSGi development and deployment. Benefits? Better modulatory, versioning of artifacts and operations control. I look forward to seeing how the new SpringSources Application Platform (based on Tomcat, OSGi and Spring) gets on. Hmmm. Am I the only on that's shocked though that after more than 12 years of Java, we're still coming up with new ways to prevent class loading problems?

GigaSpaces Nati Shalom's session was notable in that he discussed that business drivers behind the need of scalable systems: the number of financial transactions, data, and users is constantly growing. Further, the number of traffice spikes (overload situations that can bring down your business is growing). The challenge is to scale up cost-effectively while not sacrificing reliability and performance. Over-provisioning is one expensive way of catering for peak traffic, but has lead to average industry server utilization rates of 15-20%.

Michael Keith from Oracle presented on SOA using Service Component Architecture (SCA). SCA is all very reasonable and sensible, but I'm not convinced that it's something that developers can get really excited about.

Mike Aizatsky from Google's presentation on synchronization models for multi-threaded applications was a gem, where he described nine models including the unexpected "No Multithreading" threading model. His advice: pick the simplest model, don't over-engineer your solution. I really enjoyed this talk and got a lot out of it. On the way out of Mike's talk I caught the end of Holly Cummin's presentation on java performance tuning & contented(!) locks; it's a pity these two sessions were on the same time, as they would have really complemented each other.

Making powerpoint movies: RESTful services with FUSE

2008-04-09T03:43:00.000-07:00

I made my first powerpoint movie last week, on building RESTful services with FUSE. It's part of the FUSE Master Class, and has been posted on open.iona.com wiki. I've presented that material at a number of places now, but there really is something very challenging about presenting to nothing but Camtasia on a dusty laptop. Not having an audience to feed off is unnerving: I found when I did a playback that there was an irritating amount of "ommm" and "ehhh", and I ended up doing a lot re-takes.

In the end, I settled on writing a script for my narration. Reading from a script does take a bit of magic out of the delivery, but you can still improvise around it and it saves a whole lot of time. Also, I filmed segments of no more than 4 or 5 slides at a time. This keeps you on track and allows you to redo stuff quickly. I have a hunch it also saves on edit time trying to edit large tracts of movie.

Camtasia's an OK tool - this being my first time doing screen capture I don't have any thing else to compare with. I do drool with envy at my colleague's MacBook Pro: Apple's "Keynote" provides built-in support for "filming" your powerpoint, and then you can use i-Movie to edit in talking heads to minimise the "death-by-powerpoint" effect and add a little character. Gotta get me one of those machines :)

That not found in this context JAX-B exception

2008-04-07T02:51:00.000-07:00

I was throwing together a Java code-first example for a CXF web service this morning, and I encountered a scary gotcha. I want to create a simple customer information service, with an interface like this:

public interface CustomerService {
Customer findCustomerByName(String firstName, String lastName);
}

... The Customer type is a straightforward bean, as per below:

public class Customer {
private String firstName;
private String lastName;
private String skypeId;
private String telephoneNumber;
private int age;

public String getFirstName() {
return firstName;
}

public void setFirstName(String firstName) {
this.firstName = firstName;
}

public String getLastName() {
return lastName;
}

public void setLastName(String lastName) {
this.lastName = lastName;
}

public String getSkypeId() {
return skypeId;
}

public void setSkypeId(String skypeId) {
this.skypeId = skypeId;
}

public String getTelephoneNumber() {
return telephoneNumber;
}

public void setTelephoneNumber(String telephoneNumber) {
this.telephoneNumber = telephoneNumber;
}

public int getAge() {
return age;
}

public void setAge(int age) {
this.age = age;
}

public Customer(String firstName, String lastName,
String skypeId,
String telephoneNumber, int age) {
super();
this.firstName = firstName;
this.lastName = lastName;
this.skypeId = skypeId;
this.telephoneNumber = telephoneNumber;
this.age = age;
}
}

When my service implementation returns a Customer, I get an exception:

org.apache.cxf.interceptor.Fault: Marshalling Error: com.iona.ps.codefirst.Customer is not known to this context

Looks quite scary. I lost a lot of time trying to figure out what was going on, thinking that maybe I needed to add JAXB annotations to my Customer class. It turns out however that the problem is that my Bean class doesn't have a default constructor. Ouch. Without a default constructor in place, the JAXB runtime cannot create a default instance of the class. Adding the default constructor to the bean makes everything better.

I've seen through Google that lots of people have encountered this problem before - feels like a usability issue to me. While it may be the case that you need a default constructor in JAXB, CXF should give a more helpful message than "not known to this context".

Little fluffy cloudsmith

2008-03-06T12:59:00.000-08:00

I built my first public software distribution over the last week using Cloudsmith. Cloudsmith is such a cool idea: you can assemble your own favourite software distributions and share them with like-minded people who can use the Cloudsmith "materializer" to download the lot in one go. Check out my FUSE Big Bang Distro, containing the latest (at time of writing) set of IONA FUSE components for Windows. I've also created a distribution I call Joolbox for all those bits and pieces you need in Java development. Right now it just contains Maven, Ant and a few other bits and pieces... watch that space.

I found the current beta UI (based on Seam) to be a little clunky at times; for example, there's no hour-glass when you you're waiting, which can be a little disconcerting when you don't know if you've really pressed something or not. However this clunkiness was more than made up for by the responsiveness of the Cloudsmith team who were able to iron things out; they assure me that a new update of the UI is underway for their big rollout at Eclipsecon. In retrospect, it would have gone a lot easier if I'd actually Followed the Instructions on each dialogue rather than just racing ahead three sheets to the wind ;)

What might Cloudsmith lead to? Thinking with a big hat on, Cloudsmith could lead to the absolute democratization of software distribution: anyone anywhere can rapidly assemble a full software stack from freely available components, and then provide services, support and training on that distribution. And just think of the potential value of having all the worlds developers huddle together as a community in the cloud. In the meantime, Cloudsmith has the potential to solve some of my problems in the small like how to rapidly pull down all my develop tools the next time my hard drive fails, or how to share my development environment with customers and colleagues.

Confused with JBI exchanges? Use the spec, Luke.

2008-02-21T04:21:00.000-08:00

In a previous entry I mentioned my new found love of JBI. After my initial flirtations, my love has grown deeper: I believe that JBI and I are now something of an item. Hurrah!

I'd like to share something about my new relationship. At first, I had admired JBI from afar, and had hoped that I would be able to master JBI without having to go off and read the specification. In some respects this is true - you can build a Fuse ESB JBI-compliant solution entirely through configuration without any code, or without having to dig deep into JBI. This is good: after all, that Fuse ESB uses JBI internally should not be a concern to most users.

However, if you want to code up your own POJO then you need to get an understanding of how your code should interact with JBI's Normalised Message Router (NMR) for different message exchange paradigms (e.g. in-only, on-only-robust, in-out). Implementing an in-only provider? Your code should set the status of the exchange to DONE, and then send the exchange back to the NMR. Doing in-out? Your code should set a response, leave the status alone, send the exchange back to the NMR, and then expect to receive a DONE exchange later on.

For new users accustomed to simple RPC approaches this feels a little strange, confusing, and perhaps a little frustrating. I empathise: if it's an "in-only" message exchange, then why does my code have to send back the exchange? If it's an "in-out" message exchange, and my code has returned a response, then why should I subsequently receive a notification that the exchange is done?

I urge you though to resolve the awkward silence by looking to the JBI 1.0 specification - it's remarkably readable, and the sections on the NMR provide some real insight on what the JBI container expects of you. With these insights in place you can do some great stuff. The answers, by the way to the questions above concern the decoupling in time of the consumer from the provider, and providing support for reliability in the underlying message exchange.

Fuse ESB and JBI, my new-found loves...

2008-02-14T04:59:00.000-08:00

I've having the pleasure of working with Ashwin Karpe on producing some training material on JBI, in particular the Fuse ESB implementation of JBI based on Apache ServiceMix. I first looked into JBI a few years ago back when Apache CXF was Celtix and we were all a lot younger. Back then, I just couldn't see the point of JBI: whenever someone would try and explain it to me they would attempt to explain something I didn't know (JBI) in terms of something I still didn't know (Binding Components (BCs), Service Engines (SEs), Normalized Message Router (NMR) etc.). My eyes would gloss over, and I'd go back to what I was doing previously.

My eyes where opened when I got a play with the great set of JBI components in FUSE ESB. Using FUSE, I can create listeners (for example HTTP, JMS, FTP, FILE) and wire them using enterprise integration patterns (like filters, transformers and content-based routers) into some sweet integration flows, sending data on to other systems (again using transport components like HTTP, JMS, FTP and FILE). Now, that's something tangible that I know I can use when I'm out on the road. Just recently, a guy I know asked me about creating an FTP plugin for CXF to accept CSV (comma separated value) payload: maybe a year or two ago I would have tried to find a way to do that, now, I'd just do it using Fuse ESB.

Most of what you do with FUSE ESB/ServiceMix can be done by simply customising (via configuration) the transport components and patterns that are provided in the box. Occasionally though you may want to write a POJO that sits in your integration flow: I'm currently looking into writing a chapter in our training material on how to wire POJO's (i.e. JBI's MessageExchangeListeners) into a flow.

In the meantime, for anyone interested in playing with JBI, take a look at the documentation on open.iona.com and Tom Purcell's blog, which has some interesting entries on JBI from a beginner's perspective.

OOP 2008

2008-01-28T07:00:00.000-08:00

Just back from OOP 2008 in Munich - this is a fantastic conference, with a great programme and a really smart bunch of attendees. My thanks to Wolfgang Reuter and all the OOP staff for putting together such as great event.

I gave two presentations at the event, a 45 minute presentation on using CXF and GWT to create and consume RESTful services, and then a full-day workshop on building services with CXF - I really enjoyed both; in particular, I was chuffed with the attendance and enthusiasm for CXF at the workshop. One of the attendees has already written up some thoughts on the workshop - thanks James!

I'll happily send on the PDF of my presentation slides for the RESTful services session - just drop me a line in the comments below.

The Dark Side of the Source

2008-01-14T06:40:00.000-08:00

One of the great benefits of using open-source, or any source-available software in general, is that, well, you have the source. If you run into a problem then you can debug right into the source-code of your third-party libraries and figure out what's going on. Fantastic. As a consultant on the road I welcome the ability to dive deep.

However, there is a dark side to source code availability: you can get sucked in very easily debugging code that isn't yours. Think about that. You encounter a bug, and, instead of having X lines of code to debug (where X is the number of lines of application code) you've got X + Y lines of code to debug (where Y is the number of of open-source lines of code). Now, the extra lines of code (Y) may give you greater visibility in solving your problem. Or, you may end up loosing time trying to figure out somebody elses code: I call this "The Dark Side of the Source". Recently, I found myself deep in code involving Jetty Continuations, and lost a lot of time trying to figure out what was going on. In the end we had to get directly in touch with the code author to get a fix: if only we'd known to go their first!

So: having the source is great. But don't assume that just because you have the source you can solve all the problems all the time: the source will help you, but you have to be mindful of the impact on your productivity. Go with an open source community that can help you if you get into trouble, and one that gives you a professional support offering for when you get into deeper trouble. The guys at Redmonk (Michael Cote and Stephen O'Grady) have written a nice paper on this - "Working with Open-Source Companies". I got the paper for free from the open.iona.com communities site; I recommend the read.

Thoughts on JiBX

2007-09-03T08:07:00.000-07:00

I've written before with Ted Neward on code-first vs WSDL-first contract design for services. We had a lot of fun on the Server Side with that - thanks Ted (and the men in black).

Since then, I've mellowed a lot; I'm quite happy to use JAX-B and JAX-WS annotations where appropriate to generate my WSDL and XSD SOA artifacts. Maybe Ted's mantra of "Code first, contract aware" has rubbed off on me.

Recently thought I've noticed another approach, JiBX, that offers a middle ground. Instead of embedding annotations inside your Java classes (code-first), or writing your own schema (contract first), you put XML binding information into a separate "binding" file. You then run a tool against the Java classes and binding file and out drops the schema.

I look forward to playing with JiBX, but I can't help but feel cautious. One of the compelling reasons to use a code first approach is the idea that it's somewhat simpler than writing XSD or WSDL by hand. JiBX just doesn't look "easy" to me; it seems that if you want to you use JiBX then you need to know about your Java interfaces, your desired schema/XML, and, how to use JIB-X binding files. So, we end up having to learn new stuff (JiBX) to do things we can do already (XML Schema).

I think what I'm lacking is the presence of a "JiBX in Anger" article (remember Ant in Anger?) that shows how JiBX solves the Java/XML problem way better than the approaches I know already. From my own recent experience with the CXF java2wsdl tool (see the previous entry on this blog), I know that JiBX would give me explicit control over how XSD artifacts get generated from pre-JDK 1.5 annotation-free classes. Very nice - but worth the overhead of learning JiBX?

Code first mayhem: generating WSDL from EJB interfaces

2007-08-31T09:38:00.000-07:00

I'm working on a project that involves writing stubs for EJB services. Our approach is to write a simple EJB that delegates to a web service (stub "intelligence" lies in the web service).

Obviously, we want the web service to conform to the same API as the original EJB interface: same methods, same parameter lists, etc. So, I'm using the Artix java2wsdl tool to generate corresponding WSDL artifacts (and associated XSD) from the EJB interfaces. This tool is based on Apache CXF java2wsdl, which in turn is based on the reference implementation of JAX-B.

Now, you'd think that this should be a cinch: however, I ran into a number of difficulties and had to some up with some crafty workarounds. The bottom line is that generating contracts from raw code is trickier that you might expect. Read on for your pleasure...

First Problem: Naming Conflicts. The EJB interface uses types that create naming conflicts with wrapped-doc-literal artifacts.

Who would have thought it? The EJB interface used a wrapping style that conflicted with the naming conventions for wrapped-doc-literal. The EJBs had APIs that looked like this:

void sayHello(String s, SayHello params),

java2wsdl will try and create a schema definition for SayHello (the type in the parameter list). The tool will then try and create a wrapper XSD element for the sayHello message, also called SayHello. Ouch... naming collision here we come.

To get around this, I took advantage of the fact that SayHello (the parameter type) was actually in a different Java package: I just needed to map the Java package to a different schema namespace. This is tricky though: the only way to do this in JAX-B is to annotate the source: I only had access to the class files. I posted the problem to the cxf-dev mailing list, you can read more on the problem there. The solution was to artificially recreate the EJB package structure, inserting package-info.java classes to control the mapping. Then, by placing this hierarchy in front of the EJB jars in the classpath, we were able to take control of the mapping as if the annotations had been their all along. Tricky, but fun: in the end we decided this was "Artful Hackery".

Second problem: public fields clash with set/get methods.
Some of the data types used by EJB had a public member (say foo) along with a getFoo() method:

class MyDataType {
public int foo;
public int getFoo();
}

Alarmingly, the java2wsdl compiler complained that there were "two members with the same name." Double ouch. It turns out that to fix this problem, I had to add some smarts to the package-info.java file I created to front the package containing the data types, setting the XmlAccessorType to be FIELD.

Something like this:

@javax.xml.bind.annotation.XmlAccessorType(
javax.xml.bind.annotation.XmlAccessType.FIELD
)

Third problem: beans need to have a no-argument constructor. One of the data-types used in an interface didn't have a default no-argument constructor. This was enough to stop the java2wsdl compiler in its tracks. I couldn't find a JAX-B workaround for this; also I don't know of a way to inject a no-argument constructor into an existing class.

So: hurrah for Java decompilers: I decompiled the offending class, added a constructor, recompiled it and placed it in the CLASSPATH ahead of the Jar so that it would be picked up first.

Phew. "Apart from that, it all went swimmingly".

And who are you, exactly?

2007-08-31T05:17:00.000-07:00

My name is Ade, and I work for IONA Technologies as a Principal Consultant in their Services Organisation. As part of my role I get to travel a lot, meet customers, and design and build software solutions using a pretty wide range of middleware technologies: Web Services, CORBA, J2EE, JMS, open-source, closed source - all the usual suspects are there. I love my job - it's a great way to meet people and learn a whole lot of stuff.

I've been blogging for a while on the IONA SOS blog (see sidebar for a link); I've also been getting lots of great opportunities to speak around Europe on SOA and on middleware.

In this blog I'm going to share my middleware experiences - from the sublime to the ridiculous - and talk about my perspectives on architecting and implementing distributed systems.

While I do have an academic background, it's not actually in distributed systems (although it was a whole lot of fun building a networked supercomputer out of the Maynooth software labs to run my Ph.D. experiments in genetic programming)... It turns out there's a lot your can do with 200 PCs and a shared file system ;)

Anyway - welcome. I hope you find the articles on this blog interesting and helpful.