Tag Archives: webMethods

webMethods Integration Server: How to Structure Packages

For the webMethods Integration Server (IS) packages are “containers” that bundle together various assets and get deployed as sort-of atomic units; this makes them more or less comparable to web apps in a servlet container.  So if you are given a certain task to solve, you need to figure out a clever way to organize the various parts of the overall implementation. Unless we talk about something trivial and short-lived (ever seen?), this is more complicated than it might appear at a first glance.

I did my first project on Integration Server 4.6 in early 2002 and can now openly confess, that the result, while working fine from a business perspective, was not something I would still be proud of today. To a small extent, the hardware equipment can be blamed. I was working with a notebook that had 256 MB of RAM, a single-core 800 MHz mobile Pentium III, a 20 GB hard disk with a whopping 4200 RPM, and a 14 inch display with a 1024×768 resolution.

You may now ask, how this can affect code quality. Most important in my case was performance. If pressing the save button literally sends the machine into a 2+ minutes period of frantic disk activity, you think more than twice before doing this. (Eventually, I got a memory upgrade to 512 MB.) And this is only the tip of the iceberg. In summary, refactoring is very hard if your concentration is constantly drawn away, because you have to wait for your machine to be ready again. Luckily we are past those issues today.

For the sake of this article let’s work with the following scenario, which is close to something I worked on about two years ago:

The business requirement is to automatically perform modifications to certain sales opportunities in the CRM system and report the result to the account manager.

Of course you can put everything in one package. There will be no need to think about dependency management and much less work for setting up the DEV environment, the CI server, deployment scripts (Chef in my case), etc. And in fact this is how I started. Pretty soon I moved some really generic utility services like data type conversion or date calculation into a separate utilities package. But other than that things stayed quite monolithic.

It didn’t last long, though. When I was half-way through with the initial implementation, the next, more or less separate business scenario that involved the CRM system showed up. Obviously, there was no point in having connection details, services, and other configurations related to the latter multiple times. So I moved all CRM-related stuff to yet another package.

To be clear: I was not taken by surprise when this happened. Instead it was a deliberate decision to be very careful with upfront framework building. This has been one of my personal rules for a while and it has the great advantage of being efficient as well as effective. The efficiency comes from the fact that no time is spent on things that later turn out not to be needed a second time. More important, however, is the effectiveness. It means that the outcome is useful. If you ever came across an API that is cumbersome to use, you know what I’m talking about.

Coming to an end, here is the abstract description of how I structure my IS packages. There are certainly other ways, but this works really well for me and not once in many years has caused issues or created that gut-feeling, which tells you that something is not right, although you cannot say what it is.

  • Business logic: Here is where specifics of the business side are handled. What are the criteria to select an opportunity for processing? What changes shall be applied? Technical details of the CRM system are not in scope here.
  • System details: How to perform a query against the CRM system using the criteria coming from the business logic level above? What are the connection details for a given environment (DEV, TEST, PROD)? Not in scope is e.g. the implementation of auditing.
  • Common topics: How to send notifications (e.g. where is your mail server)? Auditing, dealing with data structures, date manipulation, logging, etc.

In terms of naming conventions I suggest the following

<COMPANY_INITIALS>_<TYPE>_<DESCRIPTION>

Here are a few examples for the famous Acme Corp.

  • AC_APP_CrmOppCheck   : Application that performs checks on sales opportunities
  • AC_CMN_Audit                     : Auditing
  • AC_CMN_Util                         : Utilities
  • AC_SYS_CRM                         : Connectivity and services for CRM system
  • AC_SYS_ERP                          : Connectivity and services for ERP system

If you have questions, please do not hesitate to ask in the comments.

webMethods Integration Server: Create non-printable characters

It is a common requirement (e.g. for search-replace operations on strings) to create characters that are non-printable, which also means there is no obvious way to type them in. In my case the situation was that I had to convert end-of-line characters  created on a Windows system (CR+LF) to Linux (LF only) for some unit tests to pass.

On webMethods Integration Server you have two options to do this. The first is to create a small Java service, which has the down-side that some people shy away from it. (I have even seen organizations that pretty much forbid them, which is probably a bit extreme.) The second way is to have a Flow service (Integration Server’s graphical programming language), which is what I demonstrate here.

So how do I create a LF character? For this you can use the pub.string:URLDecode service with “%0a” (without double-quotes) as input, which is simply the character with ASCII number 10 (0x0a).

And then you simply perform a replace operation using the pub.string:replace service and tell it to replace all CR+LF occurrences with LF. The search part is done best with a regular expression, in this case “\r\n” (again without the double-quotes). And don’t forget to set useRegex to true.

That’s it.

webMethods Integration Server: Continuous Deployment

For more than nine years I have been working on a package for webMethods Integration Server. With the experience gained there, I want to discuss a number of aspects about Continuous Deployment.

Versioning

I recommend the use of semantic versioning, which at its core is about the following (for a lot of additional details, just follow the link):

  • The version number consists of three parts: Major, minor, and patch (example v1.4.2).
  • An increase in the major version indicates a non-backwards-compatible change.
  • An increase in the minor version indicates a backwards-compatible change.
  • A increase in the patch version indicates bug fixes only, no functional changes at all.

It is a well-known approach and makes it very easy for everyone to derive the relevant aspects from just looking at the version number. If the release in question contains bug fixes for something you have in use, it is probably a good idea to have a close look and check if a bug relevant to you was fixed. If it is a minor update and thus contains improvements while being backwards compatible, you may want to start thinking about a good time to make the switch. And if it is major update that (potentially) breaks things, a deeper look is needed.

Each Integration Server package has two attributes for holding information about its “version”. One is indeed the version itself and the other is the build. The latter is by default populated with in auto-increase number, which I find not very helpful. Yes, it gives me a unique identifier, but one that does not hold any context. What I put into this field instead is a combination of a date-time-stamp and the change set identifier from the Version Control System (VCS). This allows me to see at a glance when this package was built and what it contains.

Build

Conceptually the work on a single package, as opposed to a set of multiple ones that comprise one application, is a bit different in that you simply deal with only one artifact that gets released. In many projects I have seen, people take a slightly different approach and see the entirety of the project as their to-be-released artifact. This approach is supported by how the related tools (Asset Build Environment and Deployer) work: You simply throw in the source code for several packages, create an archive with metadata (esp. dependencies), and deploy it. Of course you could do this on a per-package basis. But it is easier to have just one big project for all of them.

Like almost always in live, nothing comes for free. What it practically means is that for every change in only one of the potentially many packages of the application, all of them need a re-deployment. Suppose you have an update in a maintenance module that is somewhat unrelated to normal daily operation. If you deploy everything in one big archive, this will effectively cause an outage for your application. So you just introduced a massive hindrance for Continuous Deployment. Of course this can be mitigated with blue-green deployments and you are well advised to have that in place anyway. But in reality few customers are there. What I recommend instead is an approach where you “cut” your packages in such a way that they each of them performs a clearly defined job. And then you have discrete CI job for each of them, of course with the dependencies taken into account.

Artifact Storage

Once your build has been created, it must reside somewhere. In a plain webMethods environment this is normally the file system, where the build was performed by Asset Build Environment (ABE). From there it would be picked up by Deployer and moved to the defined target environment(s). While this has the advantage of being a quite simple setup, it also has the downside that you loose the history of your builds. What you should do instead, is follow the same approach that has been hugely successful for Maven: use a binary repository like Artifactory, Nexus, or one of the others (a good comparison can be found here). I create a ZIP archive of the ABE result and have Jenkins upload it to Artifactory using the respective plugin.

To have the full history and at the same time a fixed download location, I perform this upload twice. The first contains a date-time-stamp and the change set identifier from the Version Control System (VCS), exactly like for the package’s build information. This is used for audit purposes only and gives me the full history of everything that has ever been built. But it is never used for actually performing the deployment. For that purpose I upload the ZIP archive a second time, but in this case without any changing parts in the URL. So it effectively makes it behave a bit like a permalink and I have a nice source for download. And since the packages themselves also contain the date-time stamp and change set identifier, I still know where they came from.

Deployment

Depending on your overall IT landscape there are two possible approaches for handling deployments. The recommended way is to use a general-purpose configuration management tool like Chef, Puppet, Ansible, Salt, etc. This should then also be the master of your webMethods deployments. Just point your script to your “permalink” in the binary reposiory and take it from there. I use Chef and its remote file mechanism. The latter nicely detects if the archive has changed on Artifactory and only then executes the download and deployment.

You can also develop your own scripts to do the download etc., and it may appear to be easier at a first glance. But there is a reason that configuration management tools like Chef et al. have had such success over the last couple years, compared to home-grown scripts. In my opinion it simply does not make sense to spend the time to re-invent the wheel here. So you should invest some time (there are many good videos on YouTube about this topic) and figure out which system is best for your needs. And if you still think that you will be faster with your own script, chances are that you overlooked some requirements. The usual suspects are logging, error handling, security, user management, documentation, etc.

With either approach this makes deployment a completely local operation and that has a number of benefits. In particular you can easily perform any preparatory work like e.g. adjusting content of files, create needed directories, etc.

Summary

All in all, this approach has worked extremely well for me. While it was first developed for an “isolated” utility package, it has proven to be even more useful for entire applications, comprised of multiple packages; in other words, it scales well.

Another big advantages is separation of concerns. It is always clear which activity is done by what component. The CI server performs the checkout from VCS and orchestrates the build and upload to the binary repository. The binary repository holds the deployable artifact and also maintains an audit trail of everything that has ever been built. The general-purpose configuration management tool orchestrates the download from the binary repository and the actual deployment.

With this split of the overall process into discrete steps, it is easier to extend and especially to debug. You can “inject” additional logic (think user exits) and especially implement things like blue-green deployments for a zero-downtime architecture. The latter will require some upfront thinking about shared state, but this is a conceptual problem and not specific to Integration Server.

One more word about scalability. If you have a big-enough farm of Integration Servers running (and some customers have hundreds of them), the local execution of deployments also is much faster than doing it from a central place.

I hope you find this information useful and would love to get your thoughts on it.