Tag Archives: asynchronous

FreeNAS 11.1 U7: Install Syncthing in Jail

As part of moving to a new FreeNAS box, I want to replicate data from the old (nas2, running FreeNAS 11.1 U7) to the new (nas3, running FreeNAS 11.3 U5) machine. During the initial phase nas2 will still be my primary storage location. Think of this as something like a burn-in to ensure that there are no dead-on-arrival components in the new box, esp. hard disks of course. This is planned to last for at least two months and I want all my data synchronized constantly.

The solution I laid my eyes on is Syncthing and I want to run it in a FreeNAS jail on both systems. On the new system the installation was smooth, but on nas2 it was not possible to even create a jail. It turned out to be a setting that had not been migrated from the original FreeNAS 9.3 installation, which had been the initial version of FreeNAS on nas2.

All that had to be done was fix the “Collection URL” setting in the jails configuration as shown below.

  1. Go to “Jails / Configuration”
  2. Switch to “Advanced Mode”
  3. Make sure that the URL contains “11.1” (was “9.3” before on my system)

The next step was to install Syncthing with pkg. The problem with FreeNAS 11.1 is that the underlying FreeBSD is no longer maintained (EOL) and therefore no package repository exists for this version. The workaround is to forcibly switch to an existing repository, even if it does not match the FreeBSD version. I am ok with that, as long as only applications and not OS tools are installed (you should carefully think, whether this is also ok for you!). To do this, issue the following command:

# pkg bootstrap -f

You will get a warning about different OS versions and need to confirm that you want to continue. Once this is complete, install Syncthing with

pkg install syncthing

You get the same warning as just before and need to confirm the installation.

[..]
[syncthing] [1/1] Fetching syncthing-1.10.0.txz:  99%   16 MiB   1.0MB/s    00:0
[syncthing] [1/1] Fetching syncthing-1.10.0.txz: 100%   16 MiB   1.0MB/s    00:1
6                                                                               
Checking integrity... done (0 conflicting)                                      
[syncthing] [1/1] Installing syncthing-1.10.0...                                
===> Creating groups.                                                           
Creating group 'syncthing' with gid '983'.                                      
===> Creating users                                                             
Creating user 'syncthing' with uid '983'.                                       
===> Creating homedir(s)                                                        
[syncthing] [1/1] Extracting syncthing-1.10.0: 100%                             
root@syncthing:/ # 

From here, you can just continue with the normal process of setting thins up. A good starting point might be the following YouTube video.

Asynchronous Communication

In the context of SOA (Service-Oriented Architecture) there has been a revival of the asynchronous communication pattern. And that is really what it is: a pattern. First and foremost we are not talking about a specific product, protocol or API but simply a way how to design systems and applications. After this has (hopefully) been clarified between us, let’s look at some of the in my opinion important aspects. I start off with a (certainly non-academic) definition:

Asynchronous communication simply means that when making a call into an “external system”, my program/component/service etc. does not expect an immediate response with the actual result (this would be synchronous communication). I am only interested in getting a positive acknowledgement that my request has been received (but not processed) correctly. I don’t care about any further logic that is going to be executed in some other program. Instead things continue on my side and at some other place the result of my request may be received and processed further. (There are quite a few use-cases when I don’t expect a result at all.)

At a first glance this may seem way more complicated than simply making a call, wait for the result and then continue. And to a degree this perception is correct, although I must say that in my view people exaggerate greatly here. In all the cases I have seen so far the really bad problems were not caused by the pattern itself. Rather it was a cumbersome integration on the tool-level. If you have to bother with e.g. complicated mappings before you can send data over to your message broker from the “regular” code, this will certainly inhibit the use of the pattern (and rightly so). But you should blame your tool set and not the pattern in this case!

What is indeed a bit challenging when you first start using this pattern, is that it requires a fundamental shift of your mindset for designing your system or application. What you need is a loose coupling of your components. Technically speaking this means that there must be another component that is ready to receive a response matching your original request and then continues working on it.

(If you followed the discussion around SOA lately, you may have come across that term “loose coupling” more than once already. So you can think of asynchronous communication as a means for reaching this design goal. Bear in mind though, that we only look at the communication layer here! Loose coupling should also be concerned about the semantics, so in technical terms the data model needs to support this as well. However, I wanted to discuss asynchronous communication here, so let’s leave it with loose coupling for now.)

With one component sending out a request and another one handling the response, we have a solid foundation for a highly distributed system . Yes, you can also design a distributed system with synchronous communication but this is probably more difficult. What comes to mind when talking about distributed systems is scalability. More specifically, we are talking about horizontal scalability (scale out), which is spreading the load over more machines instead of putting more resources into a single machine. Having many relatively small systems work on things typically has two main advantages compared to going for bigger machines: Firstly the approach scales further and secondly it is cheaper because you can use standard hardware.

Another big plus of asynchronous communication is robustness, provided you are using a message broker that offers guaranteed delivery. Once your message broker has confirmed the receipt of the request you can be sure that it will be delivered to the receiver. (You can use the publish-subscribe pattern to allow the sender not having to know about receivers. More on that in a separate post later.) And it is certainly easier to only have to make the message broker highly available than to do the same for each and every piece of code. So by having the message broker as a central piece of infrastructure you can reach a high level of high availability relatively easily.

That’s it for now, there will be more posts on related topics.