OK OK OK I give up: I’m getting really weary of XML

At a water-cooler in a global Swiss financial institution I had a great chat with an enterprise architect. How’s it going? he asked. The correct answer to such questions is normally just to politely say ‘grand, thanks’. This time though I opened up and let out on what what has been a burning, smoldering, nagging realization: I’m getting really, really tired of XML. It’s everywhere. It’s on my queues, it’s in my SOAP messages, it’s in my integration flows, it’s in my configuration (thanks Spring!). It’s unwieldy, it’s unkind, it’s finicky, it’s bloated and most of the time it’s plain unreadable. In the last few weeks I’ve been debugging some difficult SOAP integration flows, and despite my ability to parse XML in my head, I’m finding it tiresome.

Did XML really deliver what it said it would? All those great features... Schema Validation: a great feature, but rarely powerful enough, and one most users end up disabling for performance reasons. XSLT? Cryptic. XPath: very cool, and very useful. XQuery: never got to use it in anger. Namespaces? Overly complicated and very messy when you start to use large numbers in the same message. Versioning? You have to do it with namespaces, and even then it’s very tricky to roll it out - do it wrong, and your versioning scheme just amounts to rolling out new schema releases that are incompatible with those of the past. Dynamic resolution of namespaces from the web? How many times have I had something work in development only to have it break in pre-production environments because those machines are fire-walled and can’t reach the outside world?! Partial message encryption and digital signatures? Very cool, but one of those advanced features that many people may never get to use.

Rant over.

But what happened at the water cooler? Our conversation became animated, exciting, and in the end cut far too short by pressing and more immediate work issues. Here’s the gedanken experiment: could we provide a set of enterprise services - providing things like access to data and core business task and transactions - *without* using XML all? If so, then what would it look like? Here’s some of the options we discussed in our brief encounter:

• Revitalize IDL and the Common Data Representation of CORBA. Great stuff, CORBA (I salute you). However, CORBA got some things wrong: the representations are binary not human readable without appropriate tools. There’s no technology that allows you to XPATH style querying of data (very nice if you want to access just one part of the payload, instead of unmarshaling the whole kit and kaboodle). Also, IDL mappings to languages like Java, C++ and others tended to be clunky and are certainly dated at this stage.

• JSON. Who can argue with simplicity of a data format that can be described in a single HTML page, is simple and fast to parse, and is supported by oodles of languages (see http://json.org). Great stuff indeed, and I think this is an area very much worthy of investigation further. Bringing in schema definitions for JSON allows us to be more specific about the content that can be held in JSON payload - another plus [need citation]. I think JSON would need more though to make it truly ‘enterprisey’: for example, need an XPATH-like way to get into parts of the payload. And, how can you do things like partial message encryption?

• CSV. Don’t dismiss this one straight away. Comma separated value format, and it’s close cousins ‘fixed width fields’ and ‘name-value pairs’ are arguably the simplest formats around, providing minimal overhead.

• Serialized Java Objects. I shudder at the though of restricting my data to a single programming language. No thanks. Enough said.

The options above are not exhaustive, and I have not included other approaches that may be in the pipeline. I have a prescient CORBA-savvy friend who stood firm on all of this ten years ago and said ‘Screw this XML stuff. It’s rubbish and will all blow over when people realize that’ (I paraphrase for emphasis - I’m sure he used slightly different words; however, his passion on the subject was clear). He’s written his own payload format that quite possibly will destroy XML, if it catches on.

In our water cooler discussion we didn’t come to any conclusions; there is no ‘winner’ here. However, there is a strengthening of something I’ve always believed since I got into middleware: there is no single, perfect, complete solution. Heterogeneity is crucial in operating systems, hardware platforms, programming languages, and frameworks; there is no reason why supporting heterogeneity of middleware transports and payloads should be any less important. Understanding that the conditions whereby one approach is better than another is key, and far more valuable than adopting a ‘one-size-fits-all’ approach. And then adopting open standards and technologies that can support this heterogeneity (and here I can smugly wear my ServiceMix, CXF and Camel hat) is the next most important thing. When I think of how CXF does RESTful content negotiation my mouth waters.

And so reinvigorated, I returned to work, and tamed the outstanding XML issues on my plate. As I lid-down for the night, I fear that some day this blog entry will haunt me, but I shall publish and be damned.

The six thousand topic man: hosting many topics in the same ActiveMQ

An ActiveMQ user was enquiring about whether ActiveMQ (get it from fusesource.com!) could handle a publish-subscribe messaging architecture with six thousand topics. I've often seen production deployments of FUSE support tens to hundreds of JMS destinations; however, I wasn't quite sure how it would perform with a huge number of topics. Of course, you could reduce your number of topics by introducing message selectors on a smaller number of topics: but that avoids the question rather than answering it up front.

Throwing some questions at the FUSE engineering team got back a lot of confidence that it would indeed work just fine. Still though, I always like to try things and out and see for myself. So, I slapped together a JMS client that wrote 1,000,000 non-persisted messages to 6,000 JMS topics. Then, I put together another JMS client with 6000 consumers, with appropriate session and connection pooling in place. The result? Alarmingly straightforward: it worked just fine! While quietly content with this outcome, it's worth mentioning some background things I did on the Broker...

First, I needed to switch off the default 'thread-per-consumer' model in ActiveMQ. This is done by setting the JVM system variable -Dorg.apache.activemq.UseDedicatedTaskRunner=false - by default, this is set to 'true' in the ./bin/activemq[.bat] startup script. I tested what happens if I leave this to 'true' and indeed I ended up with 6,000 threads in the broker. Ouch. When you disable this setting, the consumers are served from a pool of threads, and total thread count never got above sixty threads.

Next, I configured the Broker's transport connector to use 'nio:' rather than 'tcp:': this means we get a cleaner, more scalable threading model within the broker.

And so, it all works just fine. Dejan Bosonac's article on Python messaging: ActiveMQ and RabbitMQ suggests that you can get up to as much as 32,000 JMS destinations on a single broker; that's good to know, but I can't think of a situation where I'd need that right now.

ActiveMQ pooling: a pool by any other name would smell as sweet

Working with a FUSE customer today who voiced confusion at all the different ways you can do a pooled connection factory in ActiveMQ. With confusion comes fear, uncertainty and distrust. Should I be using Jencks? Or should I be using the Spring CachedConnectionFactory? Wasn't there something on the activemq-pool documentation that said it didn't actually pool consumers? Help!

I took this confusion directly to the source, and had an enlightening discussion with James Strachan and Gary Tully at FuseSourceabout this. As a nice drop off, we ended up updating the documentation on Camel ActiveMQ, ActiveMQ Spring Support, and the Javadoc for the org.apache.activemq.pool.PooledConnectionFactory.

The bottom line is this: while Jencks was recommended in the past, it's no longer necessary as you can just use the org.apache.activemq.pool.PooledConnectionFactory from activemq-pool project. Alternatively, you can of course use the Spring CachedConnectionFactory, as outlined in this great article.

Here's the real sneaky thing though: the JavaDoc documentation of org.apache.activemq.pool.PooledConnectionFactory suggested that this connection factory doesn't pool consumers. This is in fact not not a drawback or a failing: it simply doesn't make sense to 'pool' JMS consumers. Maintain a collection of them for concurrent consumption in parallel? Sure! But keep a 'pool' of consumers, whereby you return consumers into the pool for reuse later on when you're finished? Don't do it! It simply doesn't make much sense - and, at a technical level - could end up creating havoc as the 'idle' consumers would still get messages delivered to their internal 'prefetch' queues, where they'd dwell until the consumer is activated again. We updated the documentation to better explain what the PooledConnectionFactory does.

We realized that the confusion comes from a number of outdated resources on the web that mentioned a myriad combination of ways to do pooling. That, and, the fact that it's easy to confuse the different concepts of 'pooling' and simply maintaining a collection of resources: the former involves sharing and reuse, while the latter does not.

Bottom line: forget about Jencks. Use activemq-pool's PooledConnection or Springs CachedConnectionFactory to manage your connection, session and producer pools. And don't go talking about 'consumer pools' - it really doesn't make sense - talk about 'collections of consumers'.

Four things you need to know about the new JBI cluster engine in ServiceMix 4

My thanks to Gert Vanthienen, who helped clear the light today as I dug my way through some odd behaviour from the new JBI clustering mechanism in ServiceMix 4 - odd behaviour that was down to some configuration 'gotchas'. I've written about this clustering mechanism before on an earlier blog; there are, however, a number of tricks you need to know about the way to use this mechanism, and I think it's worth mentioning here.

Make sure you register all JBI endpoints when using the OSGi packaging

.

Cluster the endpoint the sends to the NMR, not the endpoint that receives (this is counter-intuitive!)

.

If using a network of brokers, disable conduitSubscriptions on the ActiveMQ network connector.

.

Give each SMX instance a unique clusterName

.


Note: I still haven't fully made my peace with JBI; however, I must say that I'm mellowing a little now that ServiceMix 4 allows you to do simpler packaging (as bundles rather than as service-assemblies) and provides some optimizations to get rid of excessive NMR traffic. Anyway: on to that list:

First, make sure you register your endpoints as JBI endpoints. If you're using the new OSGi packaging of JBI endpoints (which I recommend!) then make sure you add the appropriate EndpointExport to your Spring context. You would of course be doing this for non-clustered JBI endpoints, but you do also need to be doing this for clustered endpoints as well.

<bean class="org.apache.servicemix.common.osgi.EndpointExporter" />

Second, cluster the producer, not the consumer. You would think that 'clustering' would involve some kind of configuration for each of the replicated endpoints 'listening' on the NMR. However, in the SMX 4 implementation, it goes the other way. You need to cluster the endpoint that's putting the information onto the NMR. For example, if you have a file poller endpoint, and you want to send the incoming file to a cluster of bean consumers, you need to 'cluster config' the file-poller using the OsgiSimpleClusterRegistration bean. This is entirely non obvious; here's how to do it. On the file-poller, do:

<beans xmlns="http://www.springframework.org/schema/beans"
 xmlns:file="http://servicemix.apache.org/file/1.0"
 xmlns:clu="http://fusesource.com/clusterdemo"
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:schemaLocation="http://servicemix.apache.org/file/1.0 http://servicemix.apache.org/schema/servicemix-file-2009.01.xsd
       http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.0.xsd">

 <file:poller id="filePoller" service="clu:file-poller" endpoint="endpoint"
  targetService="clu:payload-receiver" file="/tmp/incomingXML" />

 <bean
  class="org.apache.servicemix.jbi.cluster.engine.OsgiSimpleClusterRegistration">
  <property name="endpoint" ref="filePoller" />
 </bean>

  <bean class="org.apache.servicemix.common.osgi.EndpointExporter" />
  
</beans>

and for the 'clustered' bean endpoint, you can omit the OsgiSimpleClusterRegistration bean:

<beans xmlns="http://www.springframework.org/schema/beans"
 xmlns:bean="http://servicemix.apache.org/bean/1.0" xmlns:clu="http://fusesource.com/clusterdemo"
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:schemaLocation="http://servicemix.apache.org/bean/1.0 http://servicemix.apache.org/schema/servicemix-bean-2010.01.0-fuse-01-00.xsd
       http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.0.xsd">

 <bean:endpoint id="beanConsumer" service="clu:payload-receiver" endpoint="endpoint"
  bean="#myBean" />

 <bean id="myBean" class="org.file.processor.su.MyBean" />

  <bean class="org.apache.servicemix.common.osgi.EndpointExporter" />
 
</beans>

Third, and this is the trickiest, disable conduit subscriptions when using networked brokers. If you are using a network of embedded brokers for your internal cluster queue, you must disable conduit subscriptions to ensure that the use of message selectors is respected across the different consumers listening on the cluster queue. How do do this? Simply set conduitSubscriptions=false on your network connector; something like this:

        <networkConnectors>
            <networkConnector name="brokerA" uri="static://(tcp://localhost:61616)" duplex="true"
            conduitSubscriptions="false"/>

And, lastly, you do need to give each instance of SMX a unique name for clustering purposes. You can do this be dropping a file called 'org.apache.servicemix.jbi.cluster.engine.config.cfg' into the etc/ directory of your ServiceMix 4 instance. In this properties file, you can provide a variable called 'clusterName' to identify the this ServiceMix instance in the cluster.

clusterName = smx1

And that's it!

New book on CXF by Packt

Great to see that Packt have published a book on CXF. As a project, CXF has been consistently establishing itself over the last few years, with a growing community, and, importantly, an increased footprint in production systems. That a publishing house has finally gotten around to putting a book together on CXF is a great endorsement of the project’s success. A copy of the book is winging it’s way to me as I write, so I look forward to providing a fuller review when I get it.

*Sigh*. This is the book I would have liked to have written if I had had the time ;)

Camel vs. JBI

People are, in general, blown away by Camel. Hearts melt and grown men weep with joy when they learn how easy it is to write an integration flow, package it as an OSGi bundle, and then drop it into the FUSE ESB 4 container. But when the emotions have subsided and rationality comes again to the fore, I often hear the question ‘if Camel can do so much so easily, then why do I need JBI anymore?’. The answer is, that in many cases, you *don’t*.

This is hard for some ServiceMix users to stomach, and delightful to others. I have written before about my ‘love affair’ with JBI on this blog, a love affair that was soon to be blighted by some hardened home truths about what really goes on on the NMR. With my JBI relationship in almost tatters, I embraced Camel with open arms. Am I on the rebound? Hardly. Camel is simply a better technology choice when implementing ESB solutions than JBI. What makes it better? A great DSL (domain specific language) for implementing integration patterns. Extensible beyond Java into other scripting languages and technologies. No canonical intermediary format (XML). Dynamic routing. On-demand type conversion. Parameterizable routes (do *that* with JBI!). Support for unit testing of routing logic with ‘mock’ endpoints. An extensive list of technology components, with easy-to-use URI configuration. All these things make for a faster way to produce more with less.

With all these things going for it, then why continue to support JBI in ServiceMix/ FUSE ESB 4? There are a number of good reasons, and, I like to think that while my amorous intentions for JBI are firmly in the past, we do now remain “Just Good Friends’. There are many users out there who currently rely on JBI solutions, and there are a certain number of users for whom the JBI architecture is a cornerstone of a larger enterprise architecture. For these users, JBI will always be important: one of the reasons why is the promise of a standards-based modular architecture. This is good in large organizations because the adherence to a standard (i.e., the JBI 1.0 specification), you know that a component written by one of your teams in, say, Texas, will work when deployed in JBI a solution developed by a team in Hyderabad. Or, alternatively, a component written by a third-party for a CRM System will plug right into your JBI framework.

Of course, you can get this kind of plug’n’play with Camel - the Camel architecture and APIs are open source! However, some enterprises are going to be wary of working to an API that could change at the whim of a community, and prefer the solidity and comfort that only a standards-based specification can afford. I can appreciate this argument. However, I must respond that, if I were Chief Architect, I’d rather join in with a thriving open source community than a specification that’s been largely untouched and unrevised for some time, such as JBI 1.0.

All that aside, it’s clear that what’s right for some organizations is typically not right for others. ServiceMix (and FUSE) is all about supporting the integration community and, most importantly, offering choice. The advice I give my customers is to use Camel as the integration technology of choice, with an underpinning of ActiveMQ for reliable, asynchronous messaging and a touch of CXF for providing SOA or RESTful facades - all deployed in FUSE ESB 4 using OSGi. For me, the JBI container that used to be at the centre of ServiceMix / FUSE ESB is now a simply a feature that can be used to integrate with third-party or existing components that conform to the JBI specification. Camel, through the ‘jbi:’ and ‘nmr:’ components, can interoperate with such components as necessary. You get the best of all worlds.

To the JBI warriors out there, I salute you. You were not wrong. It’s just that there’s a new ‘right’ in town.

OSGi and dOSGi - from the source

Very excited about some upcoming webinars on fusesource.com: two legends of OSGi will be talking about why, what and how of OSGi technology. On April 14th (my birthday :)), Eric Newcomer, co-chair of the OSGi Enterprise Expert Group, is going to give an overview of OSGi for the enterprise. Then, on April 28th, the legendary David Bosschaert, one of our Fellow engineers at Progress, and based here in Dublin, will talk about Disributed OSGi. dOSGi answers the natural question asked by OSGi newcomers when the learn about OSGi services: is there any way we share the service registry across a number of JVMs, so that a consumer in one JVM can lookup and invoke on a service in another JVM transparently? David, a committer on Apache CXF, has implemented early prototypes on how to make this happen under the covers, and, I believe, has set CXF up to be part of the reference implementation for dOSGi.

I'm gonna grab me a sandwich log in to these webinars: they're too good to miss!

I note as well that registration at the webinars entitles you to a 10% discount on our online, live training sessions on Apache Camel & Apache ServiceMix 4. It's all good. We ran our first delivery of this course this week, and I was genuinely surprised with how our students coped with - and enjoyed! - the online learning experience. Two days of webinars, live coding, exercises, demonstration, chat via IRC, and access to our tech support guys. Of course, I was very lucky to have Hadrian Zbarcea, PMC chair of the Apache Camel project, with me for the duration, piping in with some great input where Camel is going. Nice.

But anyway: for now, I'm off for Easter...