Encrypted offsite backups on auto-mounted media with Bacula & vchanger

Since backups are the most important process regardless of what industry you are in, it pays to put a lot of thought into creating a reliable, easy-to-use, scalable, and secure backup solution.

Why reliable?
This should be pretty obvious. Without a reliable backup solution that "just works" you can not be 100% sure that you will be able to restore any data at any time, and this clearly a bad thing. Files are lost or accidentally deleted by users and there is no excuse for not being able to restore this data.

Why easy-to-use?
If your backup process is not simple and painless for the end users (eg: the person or persons responsible for rotating the backup media) then sooner or later shortcuts will be taken, steps will be skipped, errors or warnings will be ignored, and you will not have the data you need when disaster strikes - and it will strike, it's just a matter of time.

Why Scalable?
How much data are you backing up today? Tomorrow? Next week, month, year? If you build a backup system to only handle your current needs, it will surely need to be replaced sooner than you'd think. You want to make sure that the backup solution you build will take care of your needs now and will continue to work into the future with minor adjustments rather than needing to be completely replaced in a year or two

Why Secure?
The only thing more important than having good offsite backups of your data is making sure that someone else does not have access to your data. This is where encryption comes in. If your backups are written to an encrypted device (hard drive, tape drive, CDROM, etc) then you can safely transport your backup media on a regular schedule to an off-site location without the fear of your data being compromised.

The Concept:

The end result we are working towards is an inexpensive, reliable, open-source backup system with multiple, removable, encrypted, inexpensive SATA hard drives that may be removed and changed by an end user without the need to do more than unplug a drive and plug another one in.

Each encrypted drive will contain many 10GB files which Bacula treats as file volumes. The 10GB value for each volume was chosen since it is a reasonable filesize if/when a volume or multiple volumes need to be moved from one drive or system to another.

To reach our goal of creating a reliable, easy-to-use, scalable, and secure backup solution, we will be making use of the following tools/technologies:

  • Linux Operating system - Gentoo is the distribution used, but these instructions will work with any Linux distribution.
  • Bacula - An open-source, client-server based, scalable, enterprise-ready backup solution. This tutorial assumes that you have a working Bacula configuration
  • vchanger - vchanger was designed to be used with Bacula to utilize multiple, removable disk drives as backup media.
  • cryptsetyup - Cryptsetup is used to conveniently setup dm-crypt managed device-mapper mappings. We will be using LUKS, the Linux Unified Key Setup, the standard for hard disk encryption.
  • udev - udev provides a dynamic device directory on a Linux system containing only the files for actually present devices. It creates or removes device node files in the /dev directory as they are added or removed.
  • autofs - autofs is used to automatically mount a device or partition when access to a directory is attempted. The auto-mounting is performed based on user-defined rules.
  • SATA Drives - In this tutorial, we are using several 750GB standard internal SATA hard drives. They will be connected to the system via an eSATA dock.

Preparing The Hard Drives:

Overwriting the drives with random data
In order to be sure that your data is securely protected, the first step in preparing each drive requires that we overwrite the entire drive with random data. This will ensure that it will be (nearly) impossible for someone to determine where your encrypted data begins and where it ends, making it much more difficult for them to mount a cryptographic attack on your drives to gain access to your data.

On Linux (or any *nix system) there exists a simple but useful tool called dd. dd has several uses, but we will be using it to overwrite each of the removable hard drives with random data from /dev/urandom.

The first step is to plug in a new drive and then determine what block device node your system assigns to it. To find out, run dmesg after plugging the drive in. You should then see something similar to this at the end of the dmesg output:

root@host: # dmesg
--[snip]--
ata7: exception Emask 0x10 SAct 0x0 SErr 0x0 action 0xe frozen
ata7: irq_stat 0x02400000, PHY RDY changed
ata7: hard resetting link
ata7: link is slow to respond, please be patient (ready=0)
ata7: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
ata7.00: ATA-8: ST3750528AS, CC38, max UDMA/133
ata7.00: 1465149168 sectors, multi 0: LBA48 NCQ (depth 0/32)
ata7.00: configured for UDMA/133
ata7: EH complete
scsi 6:0:0:0: Direct-Access     ATA      ST3750528AS      CC38 PQ: 0 ANSI: 5
sd 6:0:0:0: Attached scsi generic sg2 type 0
sd 6:0:0:0: [sde] 1465149168 512-byte logical blocks: (750 GB/698 GiB)
sd 6:0:0:0: [sde] Write Protect is off
sd 6:0:0:0: [sde] Mode Sense: 00 3a 00 00
sd 6:0:0:0: [sde] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
 sde: unknown partition table
sd 6:0:0:0: [sde] Attached SCSI disk

The bolded text in the dmesg output above shows that when the Seagate ST3750528AS 750GB SATA drive was plugged into an eSATA dock, udev assigned it a block device node of /dev/sde. Your drive may be assigned a different node and you need to be 100% sure that you identify the correct drive because the next command will completely overwrite the entire contents of the drive, including the partition table.

As root, be sure to substitute sde with the correct device indicated by dmesg on your system and run the following command:

root@host: # dd if=/dev/urandom of=/dev/sde bs=1M

Repeat these two steps (dmesg & dd) for each of the remaining drives.

*IMPORTANT* Depending on the size of the hard drive and the speed of the system, this operation will take hours or even days. Be patient. To ensure that your encrypted hard drives are as secure as possible against a cryptographic attack, you must allow this process to finish on each drive.

If you are impatient, or just want to know how far along the process is, you can keep track of the process in another shell by typing the following command:

root@host: # while true; do killall -USR1 dd; sleep 1; done

This will send a USR1 signal to the running dd command every second, causing the running dd command to display it's progress like so:

415308037312 bytes (415 GB) copied, 64613.7 s, 6.4 MB/s
390926+9047 records in
390926+9046 records out
415322717376 bytes (415 GB) copied, 64615.7 s, 6.4 MB/s
390940+9047 records in
390939+9047 records out
415337101874 bytes (415 GB) copied, 64617.7 s, 6.4 MB/s
390954+9047 records in
390953+9047 records out
415351781938 bytes (415 GB) copied, 64619.8 s, 6.4 MB/s

Preparing The Hard Drives:

Encrypting The Drives

Once each of the drives has been completely overwritten with random data, it is time to create the encrypted container that will hold our Bacula backup data. I say container rather than partition because I prefer to use the entire drive since creating one or more partitions is not required, and would serve us no purpose.

Generate a key file to unlock the encrypted containers

We are going to be using a randomly generated key file to open and unlock our encrypted containers. This key file will be created using random data taken from /dev/urandom.

Create a secure 4096 Byte key file from /dev/urandom:

root@host: # dd if=/dev/urandom of=/etc/bacula/include/Bacula_Key_File bs=4096 count=1
1+0 records in
1+0 records out
4096 bytes (4.1 kB) copied, 0.000635056 s, 6.4 MB/s

Set the ownership and permissions of the key file to make it readable only by root:

root@host: # chown root:root /etc/bacula/include/Bacula_Key_File
root@host: # chmod 400 /etc/bacula/include/Bacula_Key_File

Keep this key file in a safe place (or multiple places), accessible only by you. Encrypt a copy of it with GPG and store it on a thumb drive if you like, but what ever you do, don't lose it. If you lose this key file, you will not be able to access the data on your drives. You have been warned.

Create the encrypted container on each drive

We will use a "less-than-secure", simple, temporary passphrase that will be only used for three things:

  • Create the encrypted LUKS container on each drive
  • Store itself to Key Slot 0 of the LUKS container
  • Add the secure 4096 Byte key file to Key Slot 1 of the LUKS container

Once the 4096 Byte key file has been added to key slot 1 of the LUKS container, it is immediately used to remove the temporary passphrase from key slot 0 of the LUKS container. The 4096Byte key file will then be the only way to unlock the encrypted container on the hard drive to access the data or to add/remove other keys.

Perform these next three steps with each new drive, be sure to check the dmesg output after plugging in each new drive to verify that you are operating on the correct device and substitute it for /dev/sde in the example below:

root@host: # echo "tempvolpassphrase" | cryptsetup -v luksFormat /dev/sde
Command successful.

Now add the 4096 Byte key file to key slot 1 of the LUKS container:

root@host: # cryptsetup -v luksAddKey /dev/sde /etc/bacula/include/Bacula_Key_File
Enter any LUKS passphrase: tempvolpassphrase entered, no keystrokes will be echoed
key slot 0 unlocked.
Command successful.

Remove the simple temporary passphrase in key slot 0 of the LUKS container using the secure 4096 Byte key file that we just stored in key slot 1:

root@host: # cryptsetup -v luksKillSlot /dev/sde 0 --key-file /etc/bacula/include/Bacula_Key_File
key slot 0 verified.
Command successful.

Now, each hard drive has an encrypted LUKS container that can only be unlocked with the secure 4096 Byte key file stored in key slot 1 of the LUKS encrypted container.


Preparing The Hard Drives:

Creating The Filesystem In The Encrypted Container

Before the filesystem may be created on the drive, the encrypted container must first be unlocked and opened with the secure 4096 Byte key file:

Unlock the encrypted container with the following command:

root@host: # cryptsetup -v luksOpen --key-file=/etc/bacula/include/Bacula_Key_File /dev/sde tempcontainer
key slot 1 unlocked.
Command successful.

If successful (as indicated above) there will now be a new block device node in /dev/mapper called tempcontainer:

root@host: # ls -la /dev/mapper
total 0
drwxr-xr-x  2 root root     220 Jan 23 14:49 .
drwxr-xr-x 15 root root   14920 Jan 23 14:49 ..
crw-rw----  1 root root  10, 62 Jan 22 15:34 control
brw-rw----  1 root disk 254,  1 Jan 22 15:34 tempcontainer

Next a filesystem is created inside this unlocked, encrypted container block device node. You may use any filesystem you prefer (ext2, ext3, ext4, reiserfs, xfs, jfs, etc). We will be using reiserfs in all of our examples.

As root, run the following command on each drive and answer yes to the "...is entire device, not just one partition!" warning:

root@host: # mkreiserfs /dev/mapper/tempcontainer
mkreiserfs /dev/mapper/tempcontainer
mkreiserfs 3.6.19 (2003 www.namesys.com)
--[snip credits]--
/dev/mapper/tempcontainer is entire device, not just one partition!
Continue (y/n):y
Guessing about desired format.. Kernel 2.6.34-gentoo-r2-SMP is running.
Format 3.6 with standard journal
Count of blocks on the device: 247712
Number of blocks consumed by mkreiserfs formatting process: 8219
Blocksize: 4096
Hash function used to sort names: "r5"
Journal Size 8193 blocks (first block 18)
Journal Max transaction length 1024
inode generation number: 0
UUID: 6c5b725d-50c4-4605-b750-4f23575b9b5f
Initializing journal - 0%....20%....40%....60%....80%....100%
Syncing..ok
--[snip credits]--

ReiserFS is successfully created on /dev/mapper/tempcontainer.

Close the encrypted container and remove the device-mapper block device node in /dev/mapper:

root@host: # cryptsetup -v luksClose tempcontainer
Command successful

Preparing The Hard Drives:

Testing Each Drive

Now we will unlock the encrypted container, mount the reiserfs filesystem, set the permissions so that Bacula can write to it, and then unmount the filesystem in the encrypted container and close the container. This will verify that everything is working correctly before we move on to automating the whole process.

Create a temporary location to mount our encrypted filesystems. (Perform this first step only once):

root@host: # mkdir /mnt/temp

Perform the following six steps for each drive.

Unlock the encrypted container:

root@host: # cryptsetup -v luksOpen --key-file=/etc/bacula/include/Bacula_Key_File /dev/sde tempcontainer
key slot 1 unlocked.
Command successful.

Mount the filesystem in the encrypted container:

root@host: # mount -t reiserfs /dev/mapper/tempcontainer /mnt/temp

Verify that it was mounted OK:

root@host: # mount -t resiserfs
/dev/mapper/tempcontainer on /mnt/temp type reiserfs (rw)

Set the ownership and permissions so that only the Bacula user may write to this filesystem, then list the mounted directory to verify proper ownership:

root@host: # chown -R bacula:bacula /mnt/temp
root@host: # chmod 750 /mnt/temp
root@host: # ls -la /mnt/temp
total 699644065
drwxr-x--- 4 bacula bacula        2352 Jan 24 13:41 .
drwxr-xr-x 3 root   root             0 Jan 23 15:37 ..

Unmount the encrypted partition:

root@host: # umount /mnt/temp

And finally, close the encrypted container and remove the device-mapper block device node in /dev/mapper:

root@host: # cryptsetup -v luksClose tempcontainer
Command successful.

Automating The Process:

OK! All of your new hard drives are encrypted, and each has a filesystem inside the encrypted container which spans the entire drive. The next steps begin to pull all of this together so that creating the device-mapper block device node, unlocking the encrypted container, and mounting the filesystem in the encrypted container is all done automatically.


Configuring udev:

If the drives were not encrypted, nothing else would need to be done with regards to configuring udev.

This is true because on most current Linux systems, when a drive is plugged into the system, udev creates a block device node in /dev/ (eg: /dev/sde, /dev/sdd, etc) and also creates the appropriate symlinks in the /dev/disk/by-id, /dev/disk/by-label, /dev/disk/by-path, and /dev/disk/by-uuid - with each symlink pointing to the device's block node (eg: /dev/sde).

The drive can then be referenced by its block device node, (/dev/sde), or by any one of the symlinks created under the /dev/disk directory structure.

We could then move on to configure autofs where we are going to make use of the device's UUID symlink in the /dev/disk/by-uuid directory.

However, since our drives are encrypted, we need to address this additional issue by making some configuration additions to udev's rules so that udev can unlock the encrypted partitions on our drives and then create the required symlinks in the /dev/disk directory structure (including the /dev/disk/by-uuid directory) pointing to the device's block node of the unlocked container in the /dev/mapper directory.

Using a filesystem's UUID for referencing a removable drive is handy since the UUID will always be the same*, whereas a device's kernel block device node (/dev/sda, /dev/sdb, etc) may be different depending on the order in which it is added to the system and the number of drives presently connected etc.

*Note: A filesystem's UUID can be changed, but we are not going to do that, so for all intents and purposes the UUID will always be the same. :)


udev's Rule Files
udev's custom rules (on Gentoo) are located in the /etc/udev/rules.d directory. The filenames of the files in this directory begin with a number and the ones ending with ".rules" are processed by udevd in numeric order.

We are going to create a file called /etc/udev/rules.d/55-eSATA-LUKS.rules containing the following block of text:

/etc/udev/rules.d/55-eSATA-LUKS.rules:

KERNEL!="sd[a-z]*", GOTO="end"
ACTION=="add", PROGRAM!="/sbin/blkid -p %N", GOTO="end"
#
# Open luks partition if necessary
PROGRAM=="/sbin/blkid -o value -p -s TYPE %N", RESULT=="crypto_LUKS", ENV{crypto}="mapper/", ENV{device}="/dev/mapper/%k"
ENV{crypto}!="?*", ENV{device}="%N"
ACTION=="add", ENV{crypto}=="?*", RUN+="/sbin/cryptsetup luksOpen --key-file=/etc/bacula/include/Bacula_Key_File %N %k"
ACTION=="add", ENV{crypto}=="?*", TEST!="/dev/mapper/%k", GOTO="end"
ACTION=="remove", ENV{crypto}=="?*", RUN+="/sbin/cryptsetup luksClose %k"
LABEL="end"

Without going into too much detail about writing udev rules (others have done a great job already - See credits on the last page of this post), the udev rules file listed above basically says the following:

  • When a device is detected, check to see if it is a hard drive (eg: sda, sdb, sdc, and so on). If it is not, we're done, and go to the end of this rule file.
  • If it is a hard drive, check it using the blkid program to see if it is of type "crypto_LUKS"
  • It is is a "crypto_LUKS" container, unlock the encrypted container using the secure 4096 Byte key file. The cryptsetup command is passed the temporary device node %N (a temporary node which udev creates to reference the device before the device's final device node is created) and the kernel's device node %k (eg: sde). This command is responsible for unlocking the encrypted container and creating the block device node in /dev/mapper as seen previously.
  • Finally, if the udev rule was called because the device has been removed, then a command is run to close the encrypted container and remove the device-mapper block node in /dev/mapper.

Now tell udevd to reload its rule files:

root@host: # udevadm control --reload-rules

If you plug in one of your newly configured drives you should see in a new device-mapper block node created in /dev/mapper with the same name as the kernel device of the drive:

root@host: # ls -la /dev/mapper
total 0
drwxr-xr-x  2 root root     180 Jan 24 16:37 .
drwxr-xr-x 15 root root   14980 Jan 24 16:32 ..
crw-rw----  1 root root  10, 62 Jan 22 15:34 control
brw-rw----  1 root disk 254,  4 Jan 22 15:34 sde

The device-mapper block device name sde was assigned by the %k in the cryptsetup line in our udev rules file above.

If you unplug the drive and wait a few seconds, the /dev/mapper/sde block device node should be automatically removed. If this device node is not automatically created or deleted, go no further. Something is not working correctly with your udev rules and you have some troubleshooting to do. If this is all working as expected, then you are ready to configure autofs in the next section

*NOTE* - On my test system, there is a 180 second (3 minute) delay from the time I plug a drive in and the /dev/mapper block device node is created. During this delay there is a udevadm settle process running. I understand that the udevadm settle command has a 180 second default timeout, but I'd really like to understand what is causing this unneccessary delay. If someone can help me clear this one issue up I'd really appreciate it! Please leave a comment below.


Automating The Process:

Configuring autofs:

As mentioned in the previous section, udev is responsible for creating the appropriate symlinks in the /dev/disk/by-id, /dev/disk/by-label, /dev/disk/by-path, and /dev/disk/by-uuid which point to the kernel block device node (eg: /dev/sde, or /dev/mapper/sde, etc) when a new device or filesystem is detected.

This is important to know because we are planning on using Bacula with vchanger, and vchanger allows its removable magazines to be defined and referenced by their filesystem's UUID.

A filesystem's UUID may be determined by several methods. One way is by using the blkid program. First make sure that you have an encrypted container unlocked:

root@host: # cryptsetup -v luksOpen --key-file=/etc/bacula/include/Bacula_Key_File /dev/sde tempcontainer
key slot 1 unlocked.
Command successful.

Run the blkid program to identifty the TYPE and UUID of the device-mapper block device node "/dev/mapper/tempcontainer" created by the previous command:

root@host: # blkid /dev/mapper/tempcontainer
/dev/mapper/tempcontainer: UUID="6c5b725d-50c4-4605-b750-4f23575b9b5f" TYPE="reiserfs"

The blkid output above shows the UUID of the reiserfs filesystem in the encrypted container. As expected the UUID shown here matches the UUID displayed when the reiserfs filesystem was created in the "Creating The Filesystem In The Encrypted Container step previously. Save each UUID to a text file, as they will be needed later when vchanger is configured.

List the contents of /dev/disk/by-uuid to confirm the uuid symlink points to the tempcontainer block device node in the /dev/mapper directory:

root@host: # ls -la /dev/disk/by-uuid
drwxr-xr-x 2 root root 260 Jan 23 15:48 .
drwxr-xr-x 6 root root 120 Jan 21 19:29 ..
lrwxrwxrwx 1 root root  16 Jan 23 15:48 6c5b725d-50c4-4605-b750-4f23575b9b5f -> ../../mapper/tempcontainer

The /dev/mapper/tempcontainer block device node is exactly what the UUID symlink should be pointing to. Now close the encrypted container and remove the device-mapper block device node in /dev/mapper:

root@host: # cryptsetup -v luksClose tempcontainer
Command successful.

Editing the autofs map files:
Now we are going to configure autofs automatically mount filesystems based on their UUID.

autofs's main configuration file is /etc/autofs/auto.master. We are going to add one line to it:

/etc/autofs/auto.master

--[snip]--
/mnt/eSATA-1_Backups    /etc/autofs/auto.vchanger       --timeout=30
--[snip]--

This tells autofs to consult the /etc/auto/auto.vchanger map file whenever access to a file or directory below /mnt/eSATA-1_Backups is attempted.

Next, create the file /etc/autofs/auto.vchanger which will contain only the following one line:

/etc/autofs/auto.vchanger:

*       -fstype=auto,rw,sync    :/dev/disk/by-uuid/&

Restart autofs to activate the changes we just made. On Gentoo Linux run the following as root:

root@host: # /etc/init.d/autofs restart

Verify that autofs is now watching the /mnt/eSATA-1_Backups directory:

root@host: # mount -t autofs
automount(pid29068) on /mnt/eSATA-1_Backups type autofs (rw,fd=4,pgrp=29068,minproto=2,maxproto=4)

This confirms that automount is watching the /mnt/eSATA-1_Backups directory for access requests.

Now, whenever access to a file or directory below /mnt/eSATA-1_Backups is attempted, autofs runs*, or references the /etc/autofs/auto.vchanger map file to determine what steps to take.

In this case, /etc/autofs/auto.vchanger instructs autofs to to scan the /dev/disk/by-uuid directory to try to match the directory being accessed in /mnt/eSATA-1_Backups with a symlink having the same name. If a match is found, autofs creates a directory with the same name under /mnt/eSATA-1_Backups and mounts it on /dev/disk/by-uuid/<matching uuid symlink> - Which of course points to the device-mapper's block device node for the encrypted partition on the external disk.

For example, if a drive is inserted, and our udev rules have located and unlocked the encrypted partition (as above), and the following command is run:

root@host: # ls -la /mnt/eSATA-1_Backups/6c5b725d-50c4-4605-b750-4f23575b9b5f

autofs scans the /dev/disk/by-uuid directory and finds the 6c5b725d-50c4-4605-b750-4f23575b9b5f symlink and will create a directory /mnt/eSATA-1_Backups/6c5b725d-50c4-4605-b750-4f23575b9b5f, then mount /dev/disk/by-uuid/6c5b725d-50c4-4605-b750-4f23575b9b5f on it with read, write and sync options, and will unmount it after 30 seconds of inactivity.

The sync option will slow writes to the hard drive but is an added safety measure used on removable media to ensure that data is written immediately to help prevent data loss or curruption if the drive is removed before it is properly unmounted.

Verify that autofs has mounted the encrypted partition functionality by running:

root@host: # mount
/dev/sda3 on / type ext3 (rw,noatime)
/dev/sda1 on /boot type ext2 (rw,noatime)
--[snip]--
automount(pid29068) on /mnt/eSATA-1_Backups type autofs (rw,fd=4,pgrp=29068,minproto=2,maxproto=4)
/dev/mapper/6c5b725d-5....9b5f on /mnt/eSATA-1_Backups/6c5b725d-5....9b5f type reiserfs (rw,sync)

The bolded "/dev/mapper" line above confirms that autofs has created the directory in /mnt/eSATA-1_Backups, and has mounted the encrypted filesystem. This means that udev and autofs are working as expected and we can move on to configure vchanger.

If this mount command does not show that the encrypted partition has been automatically mounted, it is possible that more than 30 seconds have passed since the previous ls command was issued and autofs has automatically unmounted the partition. Run that command one more time, then run the mount command immediately after.

* NOTE: I found out the hard way that if the autofs map file (eg: /etc/autofs/auto.vchanger) is executable, then autofs will attempt to run it to generate its mapping rules, rather than reference it to read its mapping rules. Make sure your /etc/autofs/auto.vchanger file is chmod -x to save yourself future headaches.


Configuring vchanger:

Josh Fisher has not only done an excellent job on his vchanger program, but his installation and configuration documentation is some of the most detailed and clear documentation I have seen. That being the case, I will not go into too much detail on installing and configuring vchanger - there is no reason to re-invent the wheel. Instead, I will direct you to the vchanger Project Page on Sourceforge where you may download the vchanger source code and installation documentation.

Take the time to go through the vchanger documentation paying careful attention to the sections titled: 3. Overview and 4. Virtual Autochanger Implementation. These two sections will give you a clear understanding of some vchanger definitions, and the general concept of vchanger's functionality and its interaction with Bacula.

Make sure that you also go through and understand sections 8. Configuring vchanger, 9. Initializing New Magazines and 10. Testing vchanger. These sections are the core of getting vchanger to work with Bacula.

Our system consists of:

  • An external eSATA dock.
  • Six (6) encrypted 750GB SATA drives.
  • Each drive contains sixty-nine (69) 10GB Bacula "file" volumes.
  • All volumes are in one pool called "Offsite-eSATA"

For reference, our vchanger.conf file is listed below with all comments and blank lines removed. The vchanger documentation and the comments in the default vchanger.conf file are very detailed and helpful and many of the settings are self-explanatory, but one setting to note is the "automount dir" line.

This line tells vchnager under which directory that autofs has been configured to automount the filesystems on the encrypted partitions and therefore where to expect to find its magazines, drives, and volumes. Since we are using autofs to automount our drives, we need to uncomment the "automount dir" line and define this setting as "/mnt/eSATA-1_Backups", the same directory configured in our /etc/autofs/auto.master file.

The UUIDs in the lines defining our magazines were the UUIDs reported by the blockid program earlier during testing, as well as by the mkreiserfs program when the filesystems were created.

/etc/bacula/include/vchanger.conf

changer_name = "c0"
work_dir = /var/lib/bacula/vchanger/c0
logfile = /var/lib/bacula/vchanger/c0.log
log_level = LOG_ERR
Virtual_Drives = 1
slots_per_magazine = 69
magazine_bays = 1
automount_dir = /mnt/eSATA-1_Backups

magazine = "UUID:6c5b725d-50c4-4605-b750-4f23575b9b5f"
magazine = "UUID:e6ffe50b-bbe7-854f-fea8-766ae1265eba"
magazine = "UUID:f73dedc6-e570-e605-02d3-5ad3a07519b9"
magazine = "UUID:267e5d28-8c20-f5ed-fca6-ae051b01c9f3"
magazine = "UUID:d9e60274-a828-e2ea-c07f-1a11ca7141a2"
magazine = "UUID:8e30757a-2f10-cde3-29e8-ca4a36d18134"

Configuring Bacula:

In the Bacula configuration files listed below, only the parts relevent to this tutorial are included. On Gentoo, the default location for Bacula's configuration files is /etc/bacula, but we keep ours one directory below in /etc/bacula/include so that we can allow Gentoo's dispatch-conf (configuration file updating tool) to keep the default Bacula config files up-to-date for reference while not touching our modified files.



Configuring Bacula's Director Daemon
To keep our configuration managable, each of the configuration sections have been pulled out of bacula-dir.conf and are stored in separate files which are pulled into the bacula-dir.conf file via the file include option:

/etc/bacula/include/bacula-dir.conf

Director {
  Name = backup-dir
  DIR port = 9101
  QueryFile = /usr/libexec/bacula/query.sql
  WorkingDirectory = /var/lib/bacula
  PidDirectory = /var/run
  Password = "verysecurepassword"
  Maximum Concurrent Jobs = 6
  Messages = Daemon
}

--[snip]--

# Include Storage
@/etc/bacula/include/Storage

# Include Pools
@/etc/bacula/include/Pools

# Include Jobs
@/etc/bacula/include/Jobs

--[snip]--

In the Storage file, a Storage definition block is configured. The Name = c0 is the same as the changer_name in our vchanger configuration.

The Device = eSATA-Changer is our changer device that will be configured later in the Autochanger definition block of the Bacula storage daemon confgiuration file.

The Autochanger = yes line must be present for this Storage resource, and the Media Type = Offsite-File needs to match the Pool defined next.

/etc/bacula/include/Storage

Storage {
  Name = c0
  Address = storagedaemon.example.com
  Password = "verysecurepassword"
  Device = eSATA-Changer
  Autochanger = yes
  Media Type = Offsite-File
  }

In the Pools file a Pool resource named Offsite-eSATA is created with pretty basic options. The Storage = c0 line defines that this pool will use the c0 Storage resource defined above.

/etc/bacula/include/Pools

Pool {
  Name = Offsite-eSATA
  Storage = c0
  Pool Type = Backup
  Recycle = yes
  AutoPrune = yes
  Volume Retention = 4 weeks
  Recycle Oldest Volume = yes
  Volume Use Duration = 2 hours
  Maximum Volume Bytes = 10737418240  # Small (10GB) easy to move/transfer volume sizes
}

NOTE: The Volume Use Duration = 2 hours line should be enough to prevent Bacula from trying to use an already-written-to volume that has a status of "append" when/if a magazine is changed, while allowing enough time for consecutively run jobs to complete writing to the relatively small 10GB volume. For more details on this issue, see the vchanger docs, section 12.3, paragraph 3.

Since the vchanger "magazines" (the encrypted drives with multiple 10GB Bacula volumes) will be swapped in and out, we need a way for Bacula to know which volumes are available for use. That is, which 10GB volumes are on the drive(s) currently plugged into the system.

In the bconsole command line interface to Bacula, the command update slots instructs bacula to poll a specified tape library to see which volumes are available and which slots they are in. Bacula then updates the InChanger and Slot columns of the Media table in the database with this information. Now Bacula knows which volumes are currently available to it and in which slot of the library each of them is.

To automate this process so that no one needs to manually load bconsole and run the "update slots" command each time a magazine is plugged in or unplugged, a custom script that uses bconsole will be used. It will be scheduled to run via a special Bacula job type of Admin.

In the Jobs file, a job called UpdateSlots with a type of Admin is created. This purpose of this job is to run a script via Bacula's RunsScript definition. The script will call the bconsole program and run the "update slots" command. This job will use the same schedule as the other nightly jobs, but will have a higher priority (lower number) so that it will be the first job to run every night. This way, the magazine(s) may be swapped out each morning when all the backup jobs have finished. No other manual steps are required.

/etc/bacula/include/Jobs

Job {
  Name = UpDateSlots
  Client = None
  Type = Admin
  FileSet = None
  Storage = c0
  Schedule = WeeklyToOffsiteDisk
  Messages = Standard
  Priority = 8   # Other nightly jobs have a priority of 10
  SpoolData = no
  Pool = Default

  RunScript {
    RunsWhen = Before
    RunsOnClient = no
    Fail Job On Error = no
    Command = "/etc/bacula/include/runbefore-updateslots.sh"
  }
}

Here is the simple bash shell script that uses bconsole to run the "update slots" command:

/etc/bacula/include/runbefore-updateslots.sh

#!/bin/bash
BC="/usr/sbin/bconsole"
CONF="/etc/bacula/include/bconsole.conf"
STORAGE="c0"
DRIVE="0"   # If multiple drives are defined in the Changer
            # device bconsole will ask for a drive number

$BC -c $CONF << WAAEOF
update slots storage=$STORAGE drive=$DRIVE
quit
WAAEOF



Configuring Bacula's Storage Daemon

In the Bacula storage daemon configuration file an Autochanger called eSATA-Changer is first defined. Per the vchanger documentation, the Changer Command points to the vchanger binary. vchanger is configured to run as the user bacula and group of bacula. Your configuration may differ, but the user configured here must have read/write access to the vchnager directories created during the vchanger installation as well as read/write access to the Bacula "file" volumes on the encrypted drives.

The rest of the parameters are the substitution variables that will be passed to the vchanger program and are defined in Bacula's documentation as follows:

  • %c - Changer Device Name
  • %o - Command (Loaded, Load or Unload)
  • %S - Autochanger slot to act on (base 1)
  • %a - Archive Device Name
  • %d - Changer drive index (base 0)
/etc/bacula/include/bacula-sd.conf

Autochanger {
  Name = eSATA-Changer
  Device = eSATA-1
  Changer Command = "/usr/local/bin/vchanger -u bacula -g bacula %c %o %S %a %d"
  Changer Device = /etc/bacula/include/vchanger.conf
}

Per the vchanger documentation, the Changer Device points to /etc/bacula/include/vchanger.conf, the vchanger configuration file.

The Device in the Autochanger definition block points to the name of a tape drive, or a filetape drive which needs to be defined in the badcula-sd.conf file as well:

Next in the Bacula storage daemon configuration file the eSATA-1 referenced in the eSATA-Changer Autochanger definition is configured:

/etc/bacula/include/bacula-sd.conf(cont)

Device {
  Name = eSATA-1
  DriveIndex = 0
  Autochanger = yes
  Device Type = File
  Media Type = Offsite-File
  Label Media = no
  Random Access = yes
  Removable Media = yes
  Automatic Mount = yes
  Archive Device = /var/lib/bacula/vchanger/c0/0/drive0
}

Credits:

  • I have to thank a guy that I only saw referenced as "dave" whose website seems to have gone missing. Dave's page titled: "Privacy tools and discussion for Debian GNU/Linux" was one of the first clear resources I had found several years back when I was learning about Cryptsetup and LUKS and I still have my printed hard-copy of his (now extinct) page.
  • Uwe Hermann whose web page titled: "HOWTO: Disk encryption with dm-crypt / LUKS and Debian" was also a big help in getting clear information on Cryptsetup and LUKS.
  • Josh Fisher who's excellent vchanger program extends Bacula's disk-to-disk capabilities in a very flexible and scalable manner.
  • I'd also like to thank someone who goes by the nick "jodel" on the gentoo forums for this post which was instrumental in helping me to understand the power of udev.
  • Darrik Mazey and Matt Tidd for listening to me drone on about this project, and assisting in proof reading this very long document.
  • And finally, Kern Sibbald and the rest of the Bacula team at Bacula.org (Open-Source Bacula) and Bacula Systems (Enterprise Bacula Support) for writing, releasing and maintaining Bacula, an open-source, modular, flexible, scalable, enterprise-ready client-server bacukup solution.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Few Modifications

A few things I ran into running this on current versions of cryptsetup.

1. You can create the encrypted drive WITH key in one command now;
cryptsetup -v luksFormat /dev/sdb --key-file /etc/bacula/include/Bacula_Key_File

2. There is a new format for the arguments? for key-file. For example;
cryptsetup -v luksOpen --key-file /etc/bacula/include/Bacula_Key_File /dev/sdb tempcontainer

3. I had to install some requirements in my ubuntu server 12.04 x64.
sudo apt-get install libblkid-dev
and
sudo apt-get install uuid-dev

4. I had a lot of trouble with the Client = None and Fileset = None. I thought they were built in keywords, wasn't until I read http://blog.serverfault.com/2011/01/10/some-notes-on-setting-up-backups-... that I realized they were just dummy ones created.

Very informative ,well written.

Thank you, this tutorial helped a huge amount.I've been struggling to automate the decryption and mounting/unmounting. This tutorial enabled me to accomplish exactly what we needed.

Great job!

Hi! Great job with this howto!

I'm using Bacula since 2.4 releases and it's the first time I found a solution to encrypt all the Bacula volumes and get the 'perfect' OUT-OF-OFFICE solution.

Thanks!

Thanks so much for this!
Incredibly thorough. As a recent Bacula convert I've found it really useful.

Post new comment

  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <a> <em> <b> <i> <u> <strong> <cite> <code> <pre> <ul> <ol> <li> <dl> <dt> <dd>
  • Lines and paragraphs break automatically.

More information about formatting options

CAPTCHA
This question is for testing whether you are a human visitor and to prevent automated spam submissions.
F
@
p
%
}
t
Enter the code without spaces and pay attention to upper/lower case.