If you want to understand how graph databases might be able to help you…
Although it is not a focal point of my professional live any more, Linux and I share a long common history. My first encounter was with “S.u.S.E. Linux August 1995″ (kernel 1.1.12) in October or November of 1995. Until then I had only played around a bit with MINIX 1.5 on a 80286-based PC. But it had been a bit of a disappointment for me and I never really got into things.
This changed dramatically with Linux. I spent many hours trying to get a setup with X11 (only FVWM2 was available as window manager then) to work. This was not as easy as today. There was no, or very little, support from YaST, the setup tool of SuSE. And one had to be careful with the monitor (CRT of course) configuration. Because too high frequencies could physically damage your monitor. I never used the system to actually perform any work, it was solely for learning.
This changed with SuSE 4.4 and even more so with SuSE 6.1. I do not remember the exact dates, but SuSE 6.1 was installed in late 1997 when I ran my own small company as a “side-project” to my university studies. But since the company was highly successful, I soon used things like HylaFax on Linux and of course also ran my local mail server (sendmail can be a challenge, especially when Google does not exist yet). There even was a dial-in modem for terminal connections.
What I always disliked about SuSE, was that the updates from one version to another never really worked for me. So in about 2002 I finally decided to switch to Debian Woody (3.0). That was again a learning curve, but the
apt packaging system was vastly superior to what I had seen on SuSE before and I never had problems with upgrades. The experience gained then is also quite helpful these days for the Raspberry Pi.
Things continued with CentOS, Fedora, and Ubuntu – with the last two powering my company notebook for a short while back in 2008. But the experience was mixed and I never liked Linux as a general-purpose desktop operating system. I had used it extensively for writing LaTeX documents (using Xfig a lot for diagrams) during my university time. But other than that, for too many Windows (or later Mac) applications I never found a proper replacement.
But for servers Linux is definitely my preferred OS. And especially so because most of the innovations of the last years, like configuration management systems (e.g. Chef, Puppet, Ansible) and containers (e.g. Docker), came to live on Linux.
I know that many people think along the lines of ignoring warnings in log files. But I am responsible for an environment where we run applications that are sort-of business critical. So I need to take warnings seriously, especially if their wording is not really clear. The consequence is that any unnecessary warning causes operational problems. Because who can tell me, kind-of “written in blood”, that I can absolutely always and forever ignore this warning? Only in that case could I consider adding an exception to the log monitoring system and of course to the system documentation, the operations manual, etc. So for me, and from my consulting past I know many customer think the same, this is not just a small nuisance but a real issue.
On the other hand I have had many discussions with people who told me that I could just ignore this or that entry. In many cases it turned out after some discussion, that the log level was actually chosen badly and
INFO would have been more appropriate. In that respect the semantics of the commonly used log levels deserve a closer look. Here are two good links (link 1, link2) for definitions. When I first read them, my initial thought was that I might have overreached with the first paragraph of this post. But looking at the example from link 1 about
WARN a bit closer, I think my concerns are still valid.
So what can be done? Reality is that you rarely have the ability to get a log statement changed. So you do need a scalable approach to deal with log messages that you consciously choose to ignore. It involves primarily two things: Firstly, you need to have documentation why the decision was made that a given log message is not critical. Secondly, there should be an automated link with your log file monitoring system, that configures an exception in it. Depending on your business this whole area might also be regulated, so the legal side may very well play a role as well. But that is outside the scope of this post.
I know this post is not really actionable, but still wanted to share my thoughts.
Test-Driven Development (TDD) is something I have long had difficulties with. Not because I consider it a bad concept, but found it very difficult to start doing. In hindsight it appears that the advice given in the respective books and online articles was not suitable. So here is the approach that finally worked for me.
It boils down to deviating from the pure doctrine. Instead of writing a test before starting on a new piece of code, I start with the actual code right away. Yes, that violates the core principle, although only for a while. But I have found that in most cases my understanding of the problem is still somewhat vague when I start working on it. So for my brain it is better if I do not have to split its capacity between solving the actual problem and thinking about how to devise a proper test and what all that means for the structure of the future code.
Once the initial version of the working code is there and manually validated, I do add the test. From then on I am in a position to refactor the code without the risk of breaking something. And of course this refactoring is needed because the first version of any code is never really good. While you could write “better” initial code, this would require spending more time upfront than you otherwise need for refactoring later. And it also ignores the fact that you only really understand the problem, when you have finished implementing the solution.
What I later realized was that my approach also helped me to write more testable code. But instead of consciously having to work on it, this sneaked in as a by-product of my modified way of doing TDD. For me this is a more natural way of learning and the results are typically better than following some formal approach.
Interesting presentation, especially the demo at the end. And I also like the part where he explains scheduling via Tetris.
My main programming hobby project will soon celebrate its tenth birthday, so I thought a few notes on how I structure my VCS repository might be of interest. The VCS I have been using since the beginning is Subversion. (When I started, Git had already been released, but was really not that popular yet.) So while some details of this article will be specific to Subversion, the general concepts should be applicable elsewhere as well.
When it comes to the structure of a repository, Subversion does not impose anything from a technical point of view. All it sees is a kind-of file system, with all the pros and cons that come with the simplicity of this approach. It makes it easy for people to start using it, which is really good. But it also does not offer help for more advanced use-cases, so that people need to find a way how to map certain requirements onto that file system concept.
As a result, a convention has emerged and been there for many years now. It says that at the top-level of the project there should be only the following folders:
trunk: Home of the latest version (sometimes called the HEAD revision)
branches: Development sidelines where work happens in isolation from
tags: Snapshots that give meaningful names to a certain revision
You will find plenty of additional information on the subject when searching the Internet. I can also recommend the book “Pragmatic Version Control: Using Subversion“, although it seems to be out of print now.
With these general points out of the way, let me start with how I work on my project. There are only a few core rules and despite their simplicity I can handle all situations.
- The most important aspect for the structure of my SVN repository is that all active development on the coming version happens at
trunk. See this article for all the important details, why you really want to follow that approach in almost all cases.
- Once a new version is about to be released, I need a place where bug-fixes can be developed. So I create a release branch (e.g.
/branches/releases/v1.3) with major and minor version number but not the patch version (I use semantic versioning). From this release branch I then cut the release (v1.3.0 in this case) by pointing the release job of my CI server to the release branch.
- Once the release is done, I create a tag that also includes the patch version. In this example the tag will be from
- Now I return to working on the next release (v1.4) by switching back to
- Bug fixing happens primarily on
trunkwith fixes being back-ported to released versions. There are cases when this is not practical, of course. The two main reasons are that significant structural changes were done on
trunk(you do refactor, don’t you?) or another change has implicitly removed the bug there already. But that is the exception.
- When a bug-fix is needed on a released version, I temporarily switch my working copy to the release branch and do the respective work there. Unless the bug is critical I do not release a new version immediately after that, though. So this may repeat a few times, before the maintenance release.
- The maintenance release is then cut, again, from
/branches/releases/v1.3. And after that a new tag is created to
Those rules have proven to be working perfectly and I hope they will continue to do so for the next ten years. I have been quite lucky in that, although for me this is still a hobby project, the result is used by many global companies and organizations in a business-critical context. There are at least 11.000 installations in production that I am aware about, so I cannot be casual about reliability of the delivery process.
For a long time I had wanted my own Raspberry Pi cluster and I finally managed to get started. There will be five nodes, which will give me 20 cores to play with. The overall plan is to manage the nodes with Ansible and run a Kubernetes cluster on them.
Let’s start with putting these boards together. I quickly settled for using spacer bolts. There are special ones for Raspberry Pi available, but I just got some cheap ones from AliExpress, as shown below. They have two disadvantages, though. Firstly, their diameter is a little bit too big for the wholes in the PCB (printed circuit board). So I used a wood drill size 3 to extend the wholes by just a fraction of a millimeter. You can somewhat see the white left-overs from the drilling at the lower left whole in the picture below.
As power supply I chose a 5-port 50 watts USB model from Aukey, which I got a while ago from Amazon when they were on special offer. Aukey does such promotions quite frequently, so if you can wait a bit, you will be able to safe a few bucks.
And this is what things look like connected (excluding network of course).
That’s it for today. Stay tuned for the next part in this series.
As mentioned in My WiFi Setup with Ubiquiti Networks UAP-AC-PRO I run the Unify Controller software on a Raspberry Pi 3. There is a ready-made package available for Debian and Ubuntu Linux, that can easily be used for this and I have been doing so for more than a year.
Just a yesterday, though, I broke things by overdoing it a bit with the removal of unneeded software from the Raspberry Pi. Through some “chain” the Unifi Controller had been removed and after re-installation it did not work anymore. Instead I saw a constant CPU utilization of an entire core by Java and also errors in
[2017-12-26 13:07:04,783] <launcher> INFO system - *** Running for the first time, creating identity *** [2017-12-26 13:07:04,791] <launcher> INFO system - UUID: yyyyyyy-yyyy-yyyy-yyyyyy-yyyyyyy [2017-12-26 13:07:04,817] <launcher> INFO system - ====================================================================== [2017-12-26 13:07:04,819] <launcher> INFO system - UniFi 5.6.26 (build atag_5.6.26_10236 - release) is started [2017-12-26 13:07:04,819] <launcher> INFO system - ====================================================================== [2017-12-26 13:07:04,867] <launcher> INFO system - BASE dir:/usr/lib/unifi [2017-12-26 13:07:05,057] <launcher> INFO system - Current System IP: xxx.xxx.xxx.xxx [2017-12-26 13:07:05,059] <launcher> INFO system - Hostname: zzzz [2017-12-26 13:07:05,071] <launcher> INFO system - Valid keystore is missing. Generating one ... [2017-12-26 13:07:05,072] <launcher> INFO system - Generating Certificate[UniFi]... please wait... [2017-12-26 13:08:33,574] <launcher> INFO system - Certificate[UniFi] generated! [2017-12-26 13:08:53,004] <UniFi> ERROR system - [exec] error, rc=1
The last couple of lines were showing up repeatedly, so obviously the system tried to restart over and over again. When you search the Internet for this problem, you will find out that you are not alone. Most solutions address available memory and not all people succeed with the various approaches to increase it (typically by removing memory from graphics and increasing swap space).
What I realized was that most discussions were for older versions and a recurring theme was that things changed between minor versions. So something that had worked for v5.6.19 did not necessarily work for v5.6.22 and vice versa. Also, changes to how Java was dealt with were mentioned quite often. Running Java-based applications on Linux can be somewhat delicate, so I do not blame the folks at Ubiquity Networks for that.
This was when I realized that the JVM on my system had changed. Before the accidental cleanup I had used the Oracle 8 JVM that gets installed via the Debian package
oracle-java8-jdk. So I re-installed the latter and configured it as the default JVM via
sudo apt-get install oracle-java8-jdk sudo update-alternatives --config java
This solved my problems instantly and things are up and running again.
In my view a very good (technical) start to using Docker.
While playing around with an ESXi 6.5 test system, I accidentally killed all network connectivity by setting the NICs to pass-through. This post gives a bit of background and the solution that worked for me.
The system is home-built with a Fujitsu D3410-B2 motherboard and an Intel dual-port Gigabit NIC (HP OEM). The motherboard has a Realtek RTL8111G chip for its NIC, which does allegedly work with community drivers, but not out-of-the-box. One of the things I want to run on this box is a pfSense router. So, when I discovered, that the Realtek NIC was available for pass-through, I enabled this. I also enabled one(!) of the two ports of my Intel dual-port NIC. At least, that is what I had intended to do.
Because what really happened was that all three NICs were set to pass-through, which of course meant that ESXi itself had no NIC available to itself any more. This issue showed after the next reboot, when the console told me that no supported NICs had been found in the system. Perhaps not wrong in strict terms, but certainly a bit misleading, when you are not very experienced with ESXi.
Searching the net did not provide a real answer. But after a couple of minutes I realized that perhaps my change about pass-through might be the culprit. The relevant file where these settings are stored is
/etc/vmware/esx.conf. I searched for lines looking like this
/device/000:01.0/owner = "passthru"
and replaced them with
/device/000:01.0/owner = "vmkernel"
After that I just had to reboot and things were fine again.