Category Archives: BPM and SOA

Lifecycle Management with SOA and BPM: 1+1> 2

For some time the topics of SOA Governance and BPM have been looked at as if they were two relatively unrelated things. And this perception is correct in the sense that you don’t have to have them together. However, more and more people realize what huge additional benefits are in for them if they combine the two things. In many cases the idea is that you need some logic to govern the actual work (design, development, testing etc.) for a process that has been modeled in a nice fancy tool.

But you can also do it the other way around: Think about what you would get if you could govern your whole IT lifecycle management from one tool. The idea goes like this: You store all relevant information about “objects” that are relevant for your organization in a central repository, and the different attributes that describes those objects (aka assets) are completely freely configurable. You probably need to attach additional information to them, like existing documentation etc. So in a way you can think of these information in the repository as a way to store the knowledge about all relevant aspects of the organization and then leverage this knowledge.

Now based on that groundwork, whenever a request for a new a feature in the IT landscape comes in, you can have it go through a “workflow”. The first steps would probably be about an approval chain. So people from various functions (e.g. product management, operations, security, marketing etc.) would need to either approve or reject this. How the final outcome is determined can be a bit tricky (and is much more a political topic than a technical one).

Then come steps like gathering requirements, signing them off, doing the development etc. You probably also want to integrate this whole thing with your development chain (automated testing, continuous integration etc.). At any given point in time you know where your development stands in terms of the project plan.

So if you step back a bit and look at what you get, we are not talking about development tools any more. Instead this is true, real-time end-to-end visibility. There are clear responsibilities for assigning tasks (a human being has to decide on something) and you no longer need to fear emails that are lost in the inbox of someone’s email program. Instead you get a view into the currently open tasks, their due dates etc. Other advantages existing but for now those are the critical ones. The reason for this is that these functionalities allow you to have automatically generated documentation that satisfies your compliance requirements. In most organizations these things eat up enormous amounts of resources and affect processes that should deliver value to the organization.

Let’s leave it here for now. I am quite interested in your comments on this, so please let me know.

David A. Fisher “An Emergent Perspective on Interoperation in Systems of Systems”

When David A. Fisher wrote this paper in 2006, the hype around Web Services and SOA had just begun. The point that struck me most when reading the executive summary, was that Fisher does not limit his thoughts to technical systems. Instead he accepts the fact that the involved people are also an important part of the equation. This aspect seems to be ignored by most authors and is -in my view- a major reason why many theories seem to be so far from reality.

The paper is more than 60 pages long, so nothing for a quick lunch break reading. However, I recommend reading at least the executive summary. It made me curious enough to schedule special time for the rest of the document.

Asynchronous Communication

In the context of SOA (Service-Oriented Architecture) there has been a revival of the asynchronous communication pattern. And that is really what it is: a pattern. First and foremost we are not talking about a specific product, protocol or API but simply a way how to design systems and applications. After this has (hopefully) been clarified between us, let’s look at some of the in my opinion important aspects. I start off with a (certainly non-academic) definition:

Asynchronous communication simply means that when making a call into an “external system”, my program/component/service etc. does not expect an immediate response with the actual result (this would be synchronous communication). I am only interested in getting a positive acknowledgement that my request has been received (but not processed) correctly. I don’t care about any further logic that is going to be executed in some other program. Instead things continue on my side and at some other place the result of my request may be received and processed further. (There are quite a few use-cases when I don’t expect a result at all.)

At a first glance this may seem way more complicated than simply making a call, wait for the result and then continue. And to a degree this perception is correct, although I must say that in my view people exaggerate greatly here. In all the cases I have seen so far the really bad problems were not caused by the pattern itself. Rather it was a cumbersome integration on the tool-level. If you have to bother with e.g. complicated mappings before you can send data over to your message broker from the “regular” code, this will certainly inhibit the use of the pattern (and rightly so). But you should blame your tool set and not the pattern in this case!

What is indeed a bit challenging when you first start using this pattern, is that it requires a fundamental shift of your mindset for designing your system or application. What you need is a loose coupling of your components. Technically speaking this means that there must be another component that is ready to receive a response matching your original request and then continues working on it.

(If you followed the discussion around SOA lately, you may have come across that term “loose coupling” more than once already. So you can think of asynchronous communication as a means for reaching this design goal. Bear in mind though, that we only look at the communication layer here! Loose coupling should also be concerned about the semantics, so in technical terms the data model needs to support this as well. However, I wanted to discuss asynchronous communication here, so let’s leave it with loose coupling for now.)

With one component sending out a request and another one handling the response, we have a solid foundation for a highly distributed system . Yes, you can also design a distributed system with synchronous communication but this is probably more difficult. What comes to mind when talking about distributed systems is scalability. More specifically, we are talking about horizontal scalability (scale out), which is spreading the load over more machines instead of putting more resources into a single machine. Having many relatively small systems work on things typically has two main advantages compared to going for bigger machines: Firstly the approach scales further and secondly it is cheaper because you can use standard hardware.

Another big plus of asynchronous communication is robustness, provided you are using a message broker that offers guaranteed delivery. Once your message broker has confirmed the receipt of the request you can be sure that it will be delivered to the receiver. (You can use the publish-subscribe pattern to allow the sender not having to know about receivers. More on that in a separate post later.) And it is certainly easier to only have to make the message broker highly available than to do the same for each and every piece of code. So by having the message broker as a central piece of infrastructure you can reach a high level of high availability relatively easily.

That’s it for now, there will be more posts on related topics.

Business Activity Monitoring (BAM) and Related Areas

When you work in the software industry you are probably used to seeing things change rapidly. One of the disadvantages that come with this is a “heterogeneous” use of terms. It is often due to the fact that we do not want to bother ourselves with precise definitions. A very sloppy use of language is the natural consequence.

As some people have noticed in the past I tend to speak rather neatly. I can attribute this to my parents who were very much concerned with a correct usage of language (my father used to work as a German teacher after all). So I may be a bit more aware of the implications of an imprecise use of language. Therefore, today I want to look at some terms that are often used either without much differentiation or without a precise notion of the relationships between them. They are

  • Monitoring,
  • Business Activity Monitoring (BAM) and
  • Reporting

Monitoring is mostly used to describe an operational activity that looks after currently ongoing things at an individual level. It typically deals with questions like “What activities in my organization are currently causing problems?”. In a Business Process Management (BPM) context this means checking that all process instances are running fine. An operator will query the BPM system and ask for process instances that failed entirely, exceeded timeout limits for single process steps etc. Also automated activities fall into this category. So if you have set up an alert that alarms a human being or another system if some threshold has been exceeded, I would also consider this as monitoring.

However, we are gradually moving into a grey area here. Depending on the type of the rule that is associated with this alert, the focus moves more towards the Business Activity Monitoring (BAM) area. So imagine a threshold that is not concerned about a single incident any more. Rather it can be of a “composite” nature and look at an average for a certain time frame (e.g. average order amount per day of the week). Such figures are usually called Key Performance Indicators (KPIs). Managers tend to be much more interested in KPIs than in the characteristics of a single, although very important, activity in the organization. So instead of asking “How are we doing with order 5297523?” they would like to know “How is our on-time order fulfillment rate today?” You can think of this as monitoring for operations management. If you want to know how you are doing NOW from an overall perspective, BAM is what you probably need.

Take away the “now” attribute from this and you pretty much end up with Reporting. This answers question like “How was our on-time order fulfillment rate this month?”. In many cases this will be closely linked to Business Intelligence (BI) and Data Warehousing (DWH) topics. Some of the BI/DWH providers have also realized that their customers do not only want to know how they did in the last month but how things are going at the moment. The simple reason is that looking into the past is good to identify structural issues that can then be tackled. However, it does not help to identify a current or upcoming problem and avoid its worst consequences by taking instant counter measures.

Being able to react in almost real-time to what’s going on, is the real value of BAM. The later you realize that something is going wrong the fewer options you have and the more likely it becomes that you can only try to repair damage already done. Although implicitly mentioned in the above paragraph, I want to highlight that BAM is not only concerned with existing but also with upcoming problems. You should expect a BAM solution to be able to identify trends and alert you based on that.

So there are distinct, yet related use-cases for all these functions and you definitely want to combine them to get most from them. I can well imagine that over time these things will, at least to a degree, merge into broader solutions. One last word here: The statements given here can not reflect all possible variations and in your particular case things may seem to be different. Don’t be disturbed by that feeling. Rather analyze the situation and you will probably discover that the general lines are still the same, although the details may vary.

A Comparison of the Top Four Enterprise Architecture Methodologies

Microsoft has published a comparison of the four most well-known frameworks / methodologies / processes that aim to help you with Enterprise Architecture. These are

There is a whole bunch of other interesting articles on this page, so you might want to browse around there a bit.

Business Analyst and Enterprise Architect

In a recent edition of their ZapFlash newsletter the guys from ZapThink (I could not reach their site when writing this, therefore no link is put in here) bring up a pretty interesting topic. They discuss the relationship between the role of a {en:business analyst} (BA) and that of an enterprise architect (EA). They put them much closer to one another than I would have done prior to reading this, which triggered some thinking on my side that I would like to share with you.

The core point, I think, is that both roles were “invented” in the attempt to link the strategic layer of the organization with the operational one. (Hm, can you think of this as an incarnation of the famous knowing-doing-gap?) This view leaves out completely the notion of business and IT, which would delude things for now. Both sides have realized (or were probably forced to) that in many cases there are significant gaps between what is needed and what exists in reality. So they have both moved from their home territory into “enemy’s land” in order to grab their share of the pie. This has narrowed the gap but not closed it.

What is needed is a role that coordinates and brings together the BA and the EA. It is in my view not realistic (and probably also not desirable) to merge EA and BA into one physical person. But the main problem, from an overall organizational point of view, is that they serve two different masters. They therefore have goals that conflict on a practical level, although the intentions point (hopefully) into the same direction. The result of this combination is that a lot of discussion, negotiations etc. takes place. And at the end of the day a compromise is reached, that nobody is really happy with.

Instead it might be worth thinking about a new organizational layer sitting between business and IT. BAs and EAs would be reporting into the same person and work on the same set of goals. The challenging thing would obviously be to identify those goals. But hey, this is true for all parts of the organization. On a very high level I could think of revenue (coming from the BA side, mirroring how well the needs of business are served) and profit (coming from the EA side, representing how efficient things are). This is certainly just {en: brainstorming}, but it illustrates the point.

An even better approach would probably be to take the line of thinking from the above paragraph and implement it into the existing orgchart. Hm, looks a bit like a matrix organization (anybody out there who really likes them?). Still, saving an organizational layer is certainly worth a reasonable thinking effort.

That’s it for now. I am looking forward to comments on this.

Nicolai M. Josuttis “SOA in Practice”

[amtap amazon:asin=0596529554]

A colleague recommended this book (link to amazon.com and amazon.de) to me and I was not disappointed. On the contrary, this was one of the most interesting IT books I have come across so far. The author is pretty well known in the SOA space and a regular speaker on conferences. He has a lot of real world experience and this shines through. What made the book particularly valuable to me, was that Josuttis points out when something is not black or white but gray and discusses the relevant aspects.

This book is probably not so easy to read for a beginner, but certainly of great value to the more experienced reader. It does not provide checklists or vendor recommendations but focuses on patterns and a good conceptual understanding. It will therefore not become outdated as quickly as many other publications but probably be relevant for a number of years to come.

What is an Enterprise Service Bus (ESB)?

If you all too often hear the term “Enterprise Service Bus” or ESB (we all love those acronyms, right?) but never really knew what it’s all about, you should check out this recording by Marc Richards. He gives a great overview what an ESB is at its core. Rather long but worth every minute!

What I particularly like about this presentation is that it’s completely neutral with regard to technology or vendor. Instead Marc explains what an ESB is supposed to do, but not how this is implemented. The term ESB (at its very core) is a concept and neither a technology nor a product can claim that they are the only rightful incarnation. So statements like “An ESB must be BPEL-based” only demonstrate a fundamental lack of understanding. You could also have an ESB based on CORBA running on a mainframe. Well, this will probably not happen too often but it underlines the point.