Survival of the fittest: the evolution of the SOA registry / repository concept

I've decided to write a little about SOA registries and repositories, as a few customers have brought up these issues recently and I wanted to clear the air (and, in some sense, clear my own head too). I dragged in some sanity checks from my fellow consultants at FUSE just to get their feeling on the adoption of SOA registry tooling, so hopefully what follows will make some sense.

There was a flurry of activity some years ago (say, around 2007/2008) about SOA (Service-Oriented Architecture) Registries and Repositories. SOA evangelists, architects and "thinkers' waxed lyrical about how real SOA experts (with capital letters) used registries and repositories, and that the 'little-SOA' folk just didn't get the concept, god bless them. Vendors rushed to provide (sometimes expensive) SOA registry / repository tools, a move probably motivated more from the realization that underlying SOA infrastructure and middleware was becoming a commodity, rather than the fact that customers really did need a registry.

Funny that.

Because in many of the SOA customers I've worked with, they simply haven't needed a repository/registry. In fact, the mandate from on top to get a SOA registry or repository has been a distraction! Now, don't get me wrong: I'm not saying the SOA registries are a bad idea. Not at all. I'm just saying that, as a tool, they solve a particular problem. And, if you don't have that problem, then you don't need the tool.

Let's consider what a SOA registry / repository might do for you.

1. Store contracts, service level agreements, code, deployable artifacts, documentation (this is the 'repository' nature of the tool)
   
2. Act as a run-time lookup for physical location of services - thus abstracting a services physical network address from a 'logical' network address. (this is the 'registry' nature of the tool)
   
3. Keep a track of who is using what services, and how much (this is the 'governance' nature).
4. Enforce service level agreements (again, this is more governance: making sure that the services are available and doing what they should do in a timely fashion)

Interestingly, I've found that the 'repository' problem is often solved using readily available file-sharing tools, HTTP servers, or even source-code management tools like SVN, GIT or CVS. Not glittery, sexy, 'made-to-measure' SOA repositories, but pragmatic, workable and relatively simple. One customer I know took a pragmatic approach which I really admire: "We're going to use a Wiki for our first set of services. When our Wiki becomes a pain, then we'll look at getting something more appropriate. But not until then."

The 'registry' problem is sometimes solved in software: for example, the Artix Locator, integrated with Apache CXF, allows services to register themselves automatically with a well-known-registry, which clients can then look up at runtime. Nice! I've seen other CXF users write their own registries to achieve something similar - it's really not rocket science (although, you can make your registry as complicated as you like!). I would like to see an open-source implementation of such a runtime-registry, perhaps making use of recent innovative projects like Apache Zookeeper to keep track of who's up and who's down. As an additional note on registries: one customer noted that the OSGi registry, particular in the context of distributed OSGi (dOSGi), adds another dimension to the idea of dynamic, transparent lookup of services. And, if you're acting smart in a JBI or ServiceMix 4 world, you can achieve location transparency through the NMR or ActiveMQ POQs (plain-old-queues).

Interestingly though, if you have your network folk in the room, they'll argue that they can solve this whole lookup and 'location transparency' problem in a heart-beat in hardware with a network switch: think about it: I can go to www.google.com every day - in fact, so can my mam - and never worry about what machine my search gets directed too. So, often people can achieve the 'registry' functionality easy, sometimes with zero code.

So, while there is certainly some argument for using registries and repositories for storage and retrieval of information about what services you have on the network, there are often many options available to you that don't need a dedicated software tool. And, keep in mind, if you have only a small number of services - say up to 20 - you really don't have a management problem - so ad hoc approaches like Wiki's may be perfectly adequate.

With this in mind, I think that 'SOA registry-repository' offerings begin to offer value when they tackle SOA Governance issues, and I think this is the evolutionary route these products are taking. IT owners want to know that security policies are being enforces - these policies should be configurable, not 'designed into' the fabric and code of a service where they're difficult to change. They want to know if the response times are being sluggish. They want a record of who accessed what, and when. This could be for a legal audit trail, but is probably more useful internally as a 'proof-point' to show that service re-use is happening, and, more importantly, a way for a core SOA Centre of Competence to demonstrate the value that its services are offering the rest of the organization. Now this, for me, is a place where these tools can offer real value. As a Progress employee I am biased, and will of course recommend Actional which does an awesome job in this regard - but, as a fair and open-minded person, I have seen users of CXF propose to use HP SOA Centre to achieve something architecturally similar. Actional acts as a 'gateway'. You register your services with it (hey, a touch of 'registry!'). You apply your security policies. And away you go: all SOA requests get passed through gateway and SLAs are enforced and reported. Governance made easy. The only problem with this architecture, from my perspective, is that it creates a potential 'gateway bottleneck; however, with appropriate load-balancing in place this hasn't yet been a problem in practice.

At the end of the day, the tooling is evolving to the point of value around the concepts of governance - I welcome this. However, I still believe that if you're embarking on creation or adoption of SOA concepts, you shouldn't get stalled in the early stages by selection of SOA registry / repository. First, understand your needs - your real needs - in this area, and then, go look for right tool for the job.