Tag Archives: Integration

webMethods Integration Server: How to Structure Packages

For the webMethods Integration Server (IS) packages are “containers” that bundle together various assets and get deployed as sort-of atomic units; this makes them more or less comparable to web apps in a servlet container.  So if you are given a certain task to solve, you need to figure out a clever way to organize the various parts of the overall implementation. Unless we talk about something trivial and short-lived (ever seen?), this is more complicated than it might appear at a first glance.

I did my first project on Integration Server 4.6 in early 2002 and can now openly confess, that the result, while working fine from a business perspective, was not something I would still be proud of today. To a small extent, the hardware equipment can be blamed. I was working with a notebook that had 256 MB of RAM, a single-core 800 MHz mobile Pentium III, a 20 GB hard disk with a whopping 4200 RPM, and a 14 inch display with a 1024×768 resolution.

You may now ask, how this can affect code quality. Most important in my case was performance. If pressing the save button literally sends the machine into a 2+ minutes period of frantic disk activity, you think more than twice before doing this. (Eventually, I got a memory upgrade to 512 MB.) And this is only the tip of the iceberg. In summary, refactoring is very hard if your concentration is constantly drawn away, because you have to wait for your machine to be ready again. Luckily we are past those issues today.

For the sake of this article let’s work with the following scenario, which is close to something I worked on about two years ago:

The business requirement is to automatically perform modifications to certain sales opportunities in the CRM system and report the result to the account manager.

Of course you can put everything in one package. There will be no need to think about dependency management and much less work for setting up the DEV environment, the CI server, deployment scripts (Chef in my case), etc. And in fact this is how I started. Pretty soon I moved some really generic utility services like data type conversion or date calculation into a separate utilities package. But other than that things stayed quite monolithic.

It didn’t last long, though. When I was half-way through with the initial implementation, the next, more or less separate business scenario that involved the CRM system showed up. Obviously, there was no point in having connection details, services, and other configurations related to the latter multiple times. So I moved all CRM-related stuff to yet another package.

To be clear: I was not taken by surprise when this happened. Instead it was a deliberate decision to be very careful with upfront framework building. This has been one of my personal rules for a while and it has the great advantage of being efficient as well as effective. The efficiency comes from the fact that no time is spent on things that later turn out not to be needed a second time. More important, however, is the effectiveness. It means that the outcome is useful. If you ever came across an API that is cumbersome to use, you know what I’m talking about.

Coming to an end, here is the abstract description of how I structure my IS packages. There are certainly other ways, but this works really well for me and not once in many years has caused issues or created that gut-feeling, which tells you that something is not right, although you cannot say what it is.

  • Business logic: Here is where specifics of the business side are handled. What are the criteria to select an opportunity for processing? What changes shall be applied? Technical details of the CRM system are not in scope here.
  • System details: How to perform a query against the CRM system using the criteria coming from the business logic level above? What are the connection details for a given environment (DEV, TEST, PROD)? Not in scope is e.g. the implementation of auditing.
  • Common topics: How to send notifications (e.g. where is your mail server)? Auditing, dealing with data structures, date manipulation, logging, etc.

In terms of naming conventions I suggest the following

<COMPANY_INITIALS>_<TYPE>_<DESCRIPTION>

Here are a few examples for the famous Acme Corp.

  • AC_APP_CrmOppCheck   : Application that performs checks on sales opportunities
  • AC_CMN_Audit                     : Auditing
  • AC_CMN_Util                         : Utilities
  • AC_SYS_CRM                         : Connectivity and services for CRM system
  • AC_SYS_ERP                          : Connectivity and services for ERP system

If you have questions, please do not hesitate to ask in the comments.

Open Services for Lifecycle Collaboration

I can honestly say that I when I wrote my post about ALM and middleware, I hadn’t heard about the Open Services for Lifecycle Collaboration initiative. But it is exactly the kind of thing I had in mind. These guys are working on the definition of a minimum (but expandable) set of features and functions that allow easy integration between the various tools, which can usually be found in an organization. To my knowledge no products exist yet, but I really like the idea and approach.

Version Control Systems and other Repositories

Recently, a few colleagues and I had a very interesting discussion about what should go into a Version Control System (VCS) and what should not. In particular we were arguing as to whether things like documents or project plans should go in. Here are a few things that I came up with in that context.

I guess the usage of VCS (and other repositories) somehow comes down to a few general desires (aka use-cases):

  • Single source of truth
  • History/time machine
  • Traceability
  • Collaboration
  • Automation of builds etc.

In today’s world with its many different repositories you can either go for a mix (best-of-breed) or the lowest common denominator which is usually the VCS. So what’s stopping people from doing it properly (=best of breed)?

  • Lack of conceptual understanding:
    • Most people involved in those kinds of discussion usually come from a (Java) development background. So there is a “natural” tendency to think VCS. What this leaves out is that other repositories, which are often DB-based, offer additional capabilities. In particular there are all sorts of cross checks and other constraints which are being enforced. Also, given their underlying architecture, they are usually easier to integrate with in therms of process-driven approaches.
    • Non-technical folks are mostly used to do versioning-by-filename and require education to see the need for more.
  • Lack of repository integration: Interdependent artefacts spread over multiple repositories require interaction, esp. synchronisation. Unless some kind of standard has emerged, it is a tedious task to do custom development for these kinds of interfaces. Interestingly, this goes back to my post about ALM needing middleware.
  • Different repositories have clients working fundamentally differently, both in terms of UI and underlying workflow (the latter is less obvious but far-reaching in consequence). Trying to understand all this is really hard. BTW: This already starts with different VCS! As an example just compare SVN, TFS and Git (complexity increasing in that order, too) and have “fun”.
  • Lack of process: Multiple repositories asking for interaction between themselves also means that there is, at least implicitly, a process behind all this. Admittedly, there is also a process behind a VCS-only approach, but it’s less obvious and its evolvement often ad-hoc in nature. With multiple repositories a more coordinated approach is required to the process development, also because often this means crossing organisational boundaries.

Overall, this means that there is considerable work to be done in this area. I will continue to post my ideas here and look forward to your comments!

ALM and ERP Software: Middleware needed for both

The idea to write this post was triggered when I read an article called “Choosing Agile-true application lifecycle management (ALM)” and in particular by it saying that many ALM tools come as a package that covers multiple processes in the lifecycle of an application. Although strictly speaking this is not a “we-cover-everything” approach, it still strongly reminds me of the take that ERP software has made initially. Its promise, put simply, was that an entire organization with all its activities (to avoid the term process here) could be represented without the need to develop custom software. This was a huge step forward and some companies made and still make a lot of money with it.

Of course, the reality is a bit more complex and so organizations that embrace ERP software have to choose between two options: either change the software to fit the organization, or change the organization to fit the software. This is not meant to be ERP bashing, it simply underlines the point that the one-size-fits-all approach has limits. And a direct consequence of this is that although the implementation effort can be reduced considerably, there is still a lot of work to be done. This is usually referred to as customizing. The more effort needs to go there, the more the ERP software is changing into something individual. So the distinction between a COTS (commercial off-the-shelf) software, the ERP, and something developed individually gets blurred. This can reduce the advantages of ERP, and especially the cost advantage, to an extent.

And another aspect is crucial here, too. An ERP system, pretty much by definition, is a commodity in the sense that the activity it supports is nothing that gives the organization a competitive advantage. In today’s times some of the key influencing factors for the latter are time-to-market and, related to that, agility and flexibility. ERP systems usually have multiple, tightly integrated components and a complex data model to support all this. So every change needs careful analysis so that it doesn’t break something else. No agility, no flexibility, no short time-to-market. And in addition all organizations I have come across so far, need things that their ERP does not provide. So there is always a strong requirement to integrate the ERP world with the rest, be it other systems (incl .mainframe) or trading partners. Middleware vendors have addressed this need for many years.

And now I am finally coming back to my initial point. In my view ALM tools do usually cover one or several aspects of the whole thing but never everything. And if they do, nobody these days starts on a green field. So also here we need to embrace reality and accept that something like ALM middleware is required.