Thoughts on Application Logging

For a while now I have been contemplating various aspects on logging and related areas.  Some of them have found their way into this post. I look purely at application logging, leaving out the underlying layers. In particular those comprise the operating system and other software that from the application’s perspective can be considered infrastructure (e.g. databases, middleware).

Stake Holders

There is a bunch of different groups that are affected by the logging aspects of an application. They all have specific requirements and some of those aspects will be looked at now.

  • Developer: Yes, also the developer has some demands towards logging. For me two things are particularly relevant: During the creation of code I don’t want to be distracted from writing the actual logic. And later I don’t want to wade through lots of boiler-plate code.
  • Application administrator: This person knows the internals of the application quite well and helps users when they run into problems with the application; often he or she also serves as a liaison with the system’s administrator. They must be able to quickly find out if a quirk comes from a real problem within the application, is a result of some external problem, or perhaps stems from “wrong” usage.
  • Operations: These guys have to ensure that the entire IT landscape is running smoothly. All too often stuff is thrown upon them that has been designed with very little thought on daily usage. They must be given the possibility to quickly see whether everything is ok or things need to be escalated with application support. In particular this requires integration into system management tools. Those are usually working with JMX, SNMP and log file monitoring.

There is certainly a lot more to say here, but this should give a first overview and make clear that a variety of requirements needs to be fulfilled.

Logging vs. Monitoring vs. Management

One way how to look at the above line is that it describes a hierarchy of aspects that built upon each other: A set of log entries allows me to monitor my application; and monitoring is the basis for management. So logging is indeed a very important step for a smooth, efficient and compliant operation of my organization. The more you move towards the higher-level facets, the more important it is to abstract from the single, raw “event” and see the bigger picture. What also becomes important is correlation of events. Perhaps my application becomes less stable whenever database response times exceed a certain threshold. And Cloud Computing will certainly add something here, too.

Logging Dislike

Most developers think of logging as an unloved necessity. But why is that? In my post about Asynchronous Communication I made the point that poor tooling does not make a pattern bad, it is just poor tooling. Likewise, I think many people simply don’t have the proper tools for logging. Last year, while developing a small system management component, I conducted a small experiment on myself. Instead of hard-coding log messages I went the extra mile and wrote my own message catalog. The result is that my (implicit) workflow has changed. Whenever an additional log statement is needed, I now only have to do the following:

  • Check the message catalog for a message I can re-use; if yes I’m done already (and perhaps have to wire in some parameters).
  • If a new message is needed I need to decide on a new key. This should be done with some confidence that I don’t need to change it later (although this would already be much easier than plain text).
  • Also a message and log level need to be chosen for the key. Those can both be done in a pretty quick-and-dirty approach, since changing them later in the catalog file is easy.

At first this may sound more complex than just putting in a plain text statement. But when putting everything directly into the code, all these things must be chosen with careful consideration because changing them later is much more effort. This distracts me a great deal, since coming up with a good log message and the appropriate log level is often far from trivial. While my initial rational for the message catalog had mainly been automated log file watching, this ease of development proved to be the real “killer” for me.

 

2 thoughts on “Thoughts on Application Logging

  1. The notion of externalizing the log message text got me thinking about localization. Localization would normally be used for providing log messages in different languages – but why not use the same technique to cater log output to the interests of the various stakeholders? Detailed technical output for developers, business-relevant messages for business support users? Maybe overkill, but just a thought worth mentioning…

    Also, externalization of log message text lends itself to continuous improvement over time. If a log message isn’t clear or comprehensive enough for someone, it can be improved or refined over time, without touching the code.

    Anyways – on the importance of logging — for business processes that span a number of different applications (and even different enterprises), log files frequently seem like the common denominator for a place to record logging events (for eventual consumption by another tool, perhaps even without that being the original intent). Certain log messages may demarcate important steps in a business process. So a wise business process monitoring tool should be capable of consuming these events, from disparate systems, and interpreting them as evidence that certain process steps, in a particular process instance have occurred.

  2. Thanks, Kevin. I really like the idea of different log messages per stakeholder. And combining that with your last paragraph, a log management system (or central analyzer for that purpose) could also be one of these stakeholders. The consequence would probably be that not only the content (easily parsable, less text) but also the output channel changes.

    I am currently working on a prototype that already supports various outputs, with log4j being the obvious default. But, and this is already planned for in the overall design, other output channels are possible, too. And when we think about automated processing of log messages, which of course must not be lost, those messages should probably not go through log4j, which does not guarantee delivery, but directly into JMS.

    And so we have arrived at Event-Driven Architecture :-). The next steps would then be to think about in-memory data grids as a channel. Some of those already come with a JMS interface.

Leave a Reply

Your email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.