Tag Archives: Ant

Configuration Management – Part 5: The External World

A standard requirement in configuration management is using values that already exist somewhere else. The most obvious places are environment variables (defined by the operating system) and Java system properties. Other are existing (!) databases, e.g. from an ERP system, or files.

The reason for referencing values directly from their original sources instead of duplicating them in a copy-paste fashion is the Don’t-Repeat-Yourself (DRY) principle. Although the latter is typically discussed in the context of code, it applies to configuration at least as well. We all know those “great” applications, which require manual updates of the same value in different places. And if you miss only one, all hell breaks loose.

For re-use of values within a file, the standard approach is variable interpolation. A well-known syntax for that comes from the build tool Apache Ant, where ${variableName} can be placed anywhere into a value assignment. Apache Commons Configuration, and therefore also WxConfig, support this syntax. In WxConfig you can even reference values from other files that belong to the same Integration Server package.

But what to do for other sources? Well, for the aforementioned environment variables and Java system properties, the respective interpolators from Apache Commons Configuration can also be used in WxConfig. But the latter also defines several interpolators of its own.

  • Cross package: Assuming you have an Integration Server package that holds some general values (often referred to as “global”), those can be referenced. (There are multiple ways to deal with truly global values and the specifics really determine which one is used best.)
  • Current date/time: Gets the current date and/or time, which is admittedly a bit of an edge-case. One could argue that this is not really a configuration value, and that is correct. However, there are scenarios when the ability to quite easily have such a value comes in very handy. Think about files that get created during processing of data. Instead of manually concatenating the path, the base filename, the date/time stamp and the file suffix in the code, you could just have something like this in your configuration file: workerFile=${tmpDir}/appFile_${date:yyyyMMdd-HHmmssSSS}.dat Doesn’t that make things a lot cleaner to read?
  • File content: Similar to date/time this was added primarily for convenience and clarity reasons. The typical use-case are templates, where a file (possibly maintained by another application) is just read directly into a configuration value.
  • Code invocation: When all else fails, you can have an arbitrary piece of logic be executed and the result be placed into the configuration value.

It is worth noting that all interpolators resolve at invocation. So you always get current results. In the case of the file content interpolator, this has the downside of file I/O; but if that really becomes a bottleneck, you are still not worse off than when having the read somewhere else in the code. And the file system caching is also still there …

With this set of options, pretty much all requirements for a redundancy-free configuration management can be met.

Ant Filter to Include Files with Configurable Names

I have been using Ant for a lot of things in the past. Just recently, I decided to also leverage it for an important aspect of my approach to shell script programming. This had been accomplished by a combination of Perl and Bash in the past. The “old” approach is ok, except that it doesn’t integrate with Maven as well as Ant does. Also, I have the requirement to run the build process on various operating systems, incl. Windows and Unix flavours that don’t necessarily have Bash installed.

So what needs to happen during the build? A lot of things in shell script programming come up time and again. So I want to encapsulate them as functions and build a library. Sourcing libary files in at run-time is really hard to achieve, if you want to be portable (your favourite search engine has loads of information on the topic). So instead of throwing a lot of time on these sometimes very subtle problems, I decided to have include statements in the source code and pre-process the scripts during the build.

There is include as well as replace functionality in Ant in Maven. But nothing I found gave me the capability to easily retrieve the file name from the currently processed source file. Yes, it would have been possible to come up with something based on normal Ant tasks (perhaps with some support from ant-contrib), but the result would have been quite ugly. So, after some investigation, I decided to write my own Ant filter reader for that. To speed things up, I borrowed from two of the filters that come with Ant, ExpandProperties and ReplaceTokens. (Needless to say my stuff comes with the same licence terms as the rest of Ant.)

Here is how it works:

  • Check the source content (in most cases coming from a file) for a regular expression. This has to match the format in which you have the include statements in your source code. So while I have set the default regex to fit my needs, you can configure it to whatever you need; just remember that Java regex handling is a bit different from e.g. Perl. This also means, that the thing can not only be used for shell scripting but any purpose that requires parametrised includes.
  • Retrieve the file name from the regex match. This is done by using a regex group. By default the first group is taken, but you can also override this if needed.
  • Read the include file line by line and insert them into the original content.
  • Optionally, you can add prefixes and suffixes to every line. By default both are set to an empty string, which effectively switches things off. My personal use case was that I want to be able to include instructions from a file and have them marked as a comment (prefix = “# “). Probably this could have been achieved differently as well. But it was easy to implement on top of the rest and makes things more comfortable for me.

Now, let’s look at an example Ant build file:

<?xml version="1.0"?>
<project name="TestAntTools" basedir="." default="" >

<property name="base.dir" value="../../.."/>
<property name="lib.dir" value="${base.dir}/target"/>

<property name="test.file"
   value="${base.dir}/src/test/resources/main_file.txt"/>

<typedef resource="antlib.xml">
  <classpath>
    <pathelement location="${lib.dir}/ant-0.0.1.jar" />
  </classpath>
</typedef>

<!-- =================================
target: TestProcessIncludes
================================= -->
<target name="TestProcessIncludes" depends=""
    description="Test the filter processIncludes">
  <loadfile srcfile="${test.file}" property="test.file.processed">
    <filterchain>
      <processIncludes>
        <setting key="prefix" value=""/>
        <setting key="suffix" value=""/>
        <searchdir value="."/>
        <searchdir value="../resources"/>
      </processIncludes>
    </filterchain>
  </loadfile>
  <echo>${test.file.processed}</echo>
</target>

</project>

The main file has this content:

# !/bin.bash

# This is before include

##include "incl_file.txt"

# This is after include

And the included file looks like this:
# ===========================
#   This was included
# ===========================

The result from the Ant script above:
Buildfile: /home/XXXXX/workspace/AntTools/src/test/ant/build.xml
TestProcessIncludes:
[echo] # !/bin.bash
[echo]
[echo] # This is before include
[echo]
[echo] # ===========================
[echo] #   This was included
[echo] # ===========================
[echo]
[echo] # This is after include
BUILD SUCCESSFUL
Total time: 142 milliseconds

As you can see, nothing special here. To download the jar containing the filter, please click here. You will need to rename the file upon saving to end up with a proper extension (WordPress does not allow to save attachments with .jar as extension). And for those intersted, here is the Eclipse project with source code etc.