Testing simple ZFS failure on FreeBSD

ZFS is one of the most robust and fault tolerant filesystems around. It can also be fairly tricky to set up and maintain so I’m writing at least one blog post based on my experiences setting up my NAS. I definitely did make some mistakes when I configured my storage but it’s served me well for a while now. Unfortunately, other parts of the motherboard I selected have been a bit temperamental. See https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=213877 for details. By writing this I’m hoping I feel a bit more confident with my upcoming drive replacement – one of the issues I have is only 4 storage drives, so increasing the total storage available means replacing all of the drives, one by one.

An aside: VirtualBox is frustrating as hell

While writing this blog post, I added and removed a fair few drives, as well as scrapping the VM configuration completely and starting again. For the benefit of future searchers, if you hit an error like:

VirtualBox error message

UUID {c56e7d80-8abc-44db-a27c-669a8c98162c} of the medium 'F:\FreeBSD Test\rootfs.vdi' does not match the value {bad50d40-beeb-40a4-9e03-3a98bb85fe5e} stored in the media registry ('C:\Users\live\.VirtualBox\VirtualBox.xml').

when starting your virtual machine, have a read of https://forums.virtualbox.org/viewtopic.php?p=368077&sid=6ea45b253eb27bb8cf202f1857f3a2ce#p368077. Most of the rest of the thread is irrelevant.

The “bad” UUID wasn’t actually stored in the path shown, but in the .vbox file – the definition of the VM. Close VirtualBox’s interface, update the UUID and away we go again. Of course, if you’ve got important data in your VM, you may want to clone and reattach instead of editing files or modifying the drive UUID itself.

Virtual Machine Setup

I’m using VirtualBox 5.1.8 on a Windows 10 host. Newer versions should be fine, and the host OS shouldn’t matter.
To install FreeBSD, I’m using FreeBSD-11.0-RELEASE-amd64-disc1.iso, although most other recent releases should work.

Create a VM that looks something like this, but don’t add any disks yet
One of the initial VM creation screens

To speed things up, I’m creating disks using the vbox-img command instead of via the GUI

PS F:\FreeBSD Test> vbox-img.exe createbase --filename rootfs.vdi --size 10240000000
PS F:\FreeBSD Test> vbox-img.exe createbase --filename disk1.vdi --size 10240000000
PS F:\FreeBSD Test> vbox-img.exe createbase --filename disk2.vdi --size 10240000000
PS F:\FreeBSD Test> vbox-img.exe createbase --filename disk3.vdi --size 10240000000
PS F:\FreeBSD Test> vbox-img.exe createbase --filename disk4.vdi --size 10240000000
PS F:\FreeBSD Test> vbox-img.exe createbase --filename disk5.vdi --size 10240000000

You will actually need the full 50GB free by the end of this post.

Disk setup in VirtualBox

Set up the disks so they look like this in VirtualBox, with hotplugging enabled for every drive except the root FS.
The only other thing I needed to change was adding an IDE controller as VirtualBox didn’t want to boot from a SATA attached CD drive for unknown reasons

Adding an IDE controller so VirtualBox will boot

For posterity, here’s the other VM settings I used, although I’m not sure if they’re needed.

4-other-settings-1

5-other-settings-2

FreeBSD Setup

I won’t go through installing FreeBSD here, except for the following:
* Install the system to the first drive shown, ada0. The defaults should be fine
6-freebsd-install
* Install an SSH server
* You may want to change to Bridged Networking in VirtualBox – I couldn’t SSH in using the VirtualBox network
* Log in as root on first boot and add a user through the console as FreeBSD sets PermitRootLogin no for sshd by default
* I install sudo and nano to make my life easier

At this point you should be able to sudo su to root, and see the following setup

root@vm-freebsd:/usr/home/voltagex # camcontrol devlist
<VBOX CD-ROM 1.0>                  at scbus0 target 0 lun 0 (cd0,pass0)
<VBOX HARDDISK 1.0>                at scbus2 target 0 lun 0 (ada0,pass1)
<VBOX HARDDISK 1.0>                at scbus3 target 0 lun 0 (ada1,pass2)
<VBOX HARDDISK 1.0>                at scbus4 target 0 lun 0 (ada2,pass3)
<VBOX HARDDISK 1.0>                at scbus5 target 0 lun 0 (ada3,pass4)
<VBOX HARDDISK 1.0>                at scbus6 target 0 lun 0 (ada4,pass5)

I have based my ZFS configuration on http://jrs-s.net/2015/02/06/zfs-you-should-use-mirror-vdevs-not-raidz/ – my NAS only has 4+1 disk bays, so I’ve matched that configuration here. Apparently it’s better to have 6 devices, but I can’t yet bend space and time, nor afford to import another SuperMicro chassis to Australia.

For this test, we’re going to create two mirrors and ignore the cache.

root@vm-freebsd::~ # zpool create tank mirror ada1 ada2 mirror ada3 ada4

 root@vm-freebsd:~ # zpool status
  pool: tank
  state: ONLINE
  scan: none requested
config:

    NAME        STATE     READ WRITE CKSUM
    tank        ONLINE       0     0     0
      mirror-0  ONLINE       0     0     0
        ada1    ONLINE       0     0     0
        ada2    ONLINE       0     0     0
      mirror-1  ONLINE       0     0     0
        ada3    ONLINE       0     0     0
        ada4    ONLINE       0     0     0

errors: No known data errors
root@vm-freebsd:~ # zfs list
NAME   USED  AVAIL  REFER  MOUNTPOINT
tank    70K  18.4G    19K  /tank

Good enough for our testing purposes

Now, create a dataset.

root@vm-freebsd:~ # zfs create tank/data
root@vm-freebsd:~ # cd /tank/data/

Let’s create a 4GB file of random data to represent the important data installed on our NAS – the hardest part of this was working out what syntax to use for arithmetic. This is csh, we’re not in ~~Kansas~~ bash any more.

root@vm-freebsd:/tank/data # dd if=/dev/random of=file1.dat bs=`expr 1024 * 1024` count=4000
4000+0 records in
4000+0 records out
4194304000 bytes transferred in 93.065192 secs (45068451 bytes/sec)

After all of that, it's time to get to the point of this article – what happens when things go wrong?

In the virtual machine settings, go to Storage and remove the fourth disk. Annoyingly you can remove a disk (because we turned on hot-plugging) but not add one while the VM is turned on.

Most server/NAS boards are going to support hot plugging, even if the idea of ripping out a spinning drive worries me a little bit.

Now that we've removed a drive, what does our pool look like?

root@vm-freebsd:/tank/data # zpool status
  pool: tank
 state: DEGRADED
status: One or more devices has been removed by the administrator.
        Sufficient replicas exist for the pool to continue functioning in a
        degraded state.
action: Online the device using 'zpool online' or replace the device with
        'zpool replace'.
  scan: none requested
config:

NAME                      STATE     READ WRITE CKSUM
    tank                      DEGRADED     0     0     0
      mirror-0                ONLINE       0     0     0
        ada1                  ONLINE       0     0     0
        ada2                  ONLINE       0     0     0
      mirror-1                DEGRADED     0     0     0
        ada3                  ONLINE       0     0     0
        12153649206614195490  REMOVED      0     0     0  was /dev/ada4

errors: No known data errors

Oh no, drive failure! Lucky we have smartmontools running and smtpd set up to email a real email address*, so I was able to catch the problem (that I just caused)

  • I wonder how many users actually have this set up. I still log on and check mutt when I remember.

If you “plug” the disk back in via the same Storage settings, you’ll need to run zpool online tank ada4 and you’ll be able to see

root@vm-freebsd:/tank/data # zpool status
  pool: tank
 state: ONLINE
  scan: resilvered 19.5K in 0h0m with 0 errors on Sun Nov  6 22:54:01 2016
config:

        NAME        STATE     READ WRITE CKSUM
        tank        ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            ada1    ONLINE       0     0     0
            ada2    ONLINE       0     0     0
          mirror-1  ONLINE       0     0     0
            ada3    ONLINE       0     0     0
            ada4    ONLINE       0     0     0

In a real server, it’d take a lot longer to “resilver” and replace the disk in the array. In this configuration, if you lose a second disk while the resilver is happening, you lose everything.

Let’s try a slightly more realistic situation. If a drive has failed, you’d plug in a new one (disk5.vdi in my example) and replace old with new. I thought FreeBSD would assign a new device name but it took the old ada4, leading to this slightly confusing command

<br />root@vm-freebsd:/tank/data # zpool status
  pool: tank
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Sun Nov  6 23:00:53 2016
        2.23G scanned out of 3.91G at 127M/s, 0h0m to go
        1.11G resilvered, 57.16% done
config:

        NAME                        STATE     READ WRITE CKSUM
        tank                        DEGRADED     0     0     0
          mirror-0                  ONLINE       0     0     0
            ada1                    ONLINE       0     0     0
            ada2                    ONLINE       0     0     0
          mirror-1                  DEGRADED     0     0     0
            ada3                    ONLINE       0     0     0
            replacing-1             OFFLINE      0     0     0
              12153649206614195490  OFFLINE      0     0     0  was /dev/ada4/old
              ada4                  ONLINE       0     0     0  (resilvering)

To make the resilver take a bit longer for my next test, so I’ve created a couple more 4GB files. Then I’m going to cause a kernel panic.

root@vm-freebsd:/tank/data # sysctl debug.kdb.panic=1

Yes, if you run this as root it will actually take your system down. Do(n’t) try this at home.

After rebooting and logging in again via SSH, has my pool survived? Aside from one slightly scary message when I first ran zpool status (1 byte a second?!), it seems to be running fine.

root@vm-freebsd:~ # zpool status
  pool: tank
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Sun Nov  6 23:13:54 2016
        1.60G scanned out of 11.7G at 1/s, (scan is slow, no estimated time)
        818M resilvered, 13.64% done
config:

        NAME                       STATE     READ WRITE CKSUM
        tank                       DEGRADED     0     0     0
          mirror-0                 ONLINE       0     0     0
            ada1                   ONLINE       0     0     0
            ada2                   ONLINE       0     0     0
          mirror-1                 DEGRADED     0     0     0
            ada3                   ONLINE       0     0     0
            replacing-1            DEGRADED     0     0     0
              6482039156816553123  OFFLINE      0     0     0  was /dev/ada4/old
              ada4                 ONLINE       0     0     0

errors: No known data errors
root@vm-freebsd:/usr/home/voltagex # zpool status
  pool: tank
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Sun Nov  6 23:13:54 2016
        2.70G scanned out of 11.7G at 93.5M/s, 0h1m to go
        1.34G resilvered, 22.99% done
config:

        NAME                       STATE     READ WRITE CKSUM
        tank                       DEGRADED     0     0     0
          mirror-0                 ONLINE       0     0     0
            ada1                   ONLINE       0     0     0
            ada2                   ONLINE       0     0     0
          mirror-1                 DEGRADED     0     0     0
            ada3                   ONLINE       0     0     0
            replacing-1            DEGRADED     0     0     0
              6482039156816553123  OFFLINE      0     0     0  was /dev/ada4/old
              ada4                 ONLINE       0     0     0  (resilvering)

Okay, what about my data?

root@vm-freebsd:~ # ls /tank/data
ls: /tank/data: No such file or directory

Uhh…

root@vm-freebsd:~ # zfs mount -a
root@vm-freebsd:~ # ls /tank/data/
file1.dat file2.dat file3.dat

Phew, looks good.

I’d like to find out if there’s any other failure modes I should know about (could I “remove” the cache from my NAS by re-purposing it as the boot drive?) but at least I know I’m not going to lose my data if my system decides to reboot halfway through resilvering, and that removing and replacing a drive isn’t really that scary.

Benchmarking compression on FreeBSD

After I commented on HackerNews that I was having performance issues with compressed ZFS, I started looking into benchmarking the performance of the Atom C2758 CPU that’s in my home NAS.

I found the fsbench project via a reply to my comment and hit my first challenge. The code I needed to run on my FreeBSD system was hosted in a VCS called Fossil.

[voltagex@beastie ~/src]$ sudo pkg install fossil
Password:
Updating FreeBSD repository catalogue...
Fetching meta.txz: 100% 944 B 0.9kB/s 00:01
Fetching packagesite.txz: 100% 6 MiB 824.2kB/s 00:07
Processing entries: 100%
FreeBSD repository update completed. 25371 packages processed.
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
fossil: 1.35_1,2

Number of packages to be installed: 1

The process will require 3 MiB more space.
1 MiB to be downloaded.

Proceed with this action? [y/N]: y
Fetching fossil-1.35_1,2.txz: 100% 1 MiB 371.8kB/s 00:03
Checking integrity... done (0 conflicting)
[1/1] Installing fossil-1.35_1,2...
[1/1] Extracting fossil-1.35_1,2: 100%
Message from fossil-1.35_1,2:
After each upgrade do not forget to run the following command:

fossil all rebuild

[voltagex@beastie ~/src]$ fossil clone https://chiselapp.com/user/Justin_be_my_guide/repository/fsbench
Usage: fossil clone ?OPTIONS? FILE-OR-URL NEW-REPOSITORY
[voltagex@beastie ~/src]$ fossil clone https://chiselapp.com/user/Justin_be_my_guide/repository/fsbench fsbench
#nothing happens for a while
Round-trips: 2 Artifacts sent: 0 received: 3383
Clone done, sent: 564 received: 6760744 ip: 216.250.117.7
Rebuilding repository meta-data...
100.0% complete...
Extra delta compression...
Vacuuming the database...
project-id: c2da1bb713a9809141b9c12d0019c6d73c005f0b
server-id: bd54879d0e061500b44f7ece394c4accab5fc7dd
admin-user: voltagex (password is &quot;3b518f&quot;)

Well okay, apparently I'm an admin now. All I really wanted to do was get the code.

[voltagex@beastie ~/src]$ cd fsbench
-bash: cd: fsbench: Not a directory

Wat?

https://www.fossil-scm.org/xfer/doc/tip/www/quickstart.wiki (no IDs or anchors so I can’t link directly to the section) says that I need to run fossil open on the file that was created.

[voltagex@beastie ~/src]$ file fsbench
fsbench: Fossil repository - SQLite3 database

Oh hey, that's kinda neat.

fossil open fsbench
#snipped file listing
project-name: fsbench
repository: /usr/home/voltagex/src/fsbench
local-root: /usr/home/voltagex/src/
config-db: /home/voltagex/.fossil
project-code: c2da1bb713a9809141b9c12d0019c6d73c005f0b
checkout: bf701e2f58e59f50dc8bfacb4e7ba916b43931fc 2015-07-11 07:55:41 UTC
parent: aa43cf887068b73178d3fcaa432f17a9d3585e63 2015-04-04 18:07:33 UTC
tags: trunk
comment: Clarifications in Results.ods (user: m)
check-ins: 107

Bugger. Files all over my src/ path. fossil close doesn’t clean up the mess it made, either.

[voltagex@beastie ~/src]$ mkdir fsbench &amp;&amp; cd fsbench &amp;&amp; fossil open ../fsbench
mkdir: fsbench: File exists
mkdir fsbench-checkout &amp;&amp; cd fsbench-checkout &amp;&amp; fossil open ../fsbench
[voltagex@beastie ~/src/fsbench-checkout/]$ cd src
[voltagex@beastie ~/src/fsbench-checkout/src]$ cmake -DCMAKE_BUILD_TYPE=Release .
-bash: cmake: command not found

Damnit. I’ll be right back. (FreeBSD really really needs an Australian package mirror)

A short time and a few package installs later, I have a working fsbench executable

Now, let’s get some test data

[voltagex@beastie ~/src/fsbench-checkout/src]$ curl https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.7.2.tar.xz | xz -d - &gt; linux-4.7.2.tar
[voltagex@beastie ~/src/fsbench-checkout/src]$ ls -lah linux-4.7.2.tar
-rw-r--r-- 1 voltagex voltagex 646M Aug 26 21:33 linux-4.7.2.tar

[voltagex@beastie ~/src/fsbench-checkout/src]$ ./fsbench LZ4 linux-4.7.2.tar
Codec version args
C.Size (C.Ratio) E.Speed D.Speed E.Eff. D.Eff.
LZ4 r127
231195303 (x 2.928) 173 MB/s 750 MB/s 114e6 493e6
Codec version args
C.Size (C.Ratio) E.Speed D.Speed E.Eff. D.Eff.
done... (4*X*1) iteration(s)).

173 MB/s – that doesn't quite match the 10 megabytes a second I was getting transferring data to that dataset a couple of weeks ago.

Here's where things get a bit more interesting (for me, at least). The data I was trying to move to the NAS a little while ago was a set of tiles from a LiDAR survey done in nearby Canberra. During <em>GovHack 2016 </em>I'd investigated the LAZ format, which took a 135GB dataset down to a more managable 18.7GB. When I copied the files to my NAS, it's the compressed form that I'd used. From this it seems like FreeBSD's LZ4 code will try very hard to recompress already-compressed data, whereas fsbench doesn't do this (LAZ is compressed, LAS isn't):

[voltagex@beastie /BoxODisks/paperweight/Data/Compressible/ACT_8ppm_Final_Delivery]$ ~/src/fsbench-checkout/src/fsbench LZ4 ACT2015-C3-ORT_6966100_55_0002_0002.las
Codec version args
C.Size (C.Ratio) E.Speed D.Speed E.Eff. D.Eff.
LZ4 r127
873970150 (x 1.526) 89.3 MB/s 710 MB/s 30e6 244e6

[voltagex@beastie /BoxODisks/paperweight/Data/Compressible/ACT_8ppm_Final_Delivery]$ ~/src/fsbench-checkout/src/fsbench LZ4 Compressed/ACT2015-C3-ORT_6966100_55_0002_0002.laz
Codec version args
C.Size (C.Ratio) E.Speed D.Speed E.Eff. D.Eff.
LZ4 r127
145529952 (x 1.000) 10.7 GB/s - 0e0 0e0
Codec version args
C.Size (C.Ratio) E.Speed D.Speed E.Eff. D.Eff.
done... (4*X*1) iteration(s)).
[voltagex@beastie /BoxODisks/paperweight/Data/Compressible/ACT_8ppm_Final_Delivery]$

So yes, there’s definitely a difference in compression speeds there but it looks like the little Atom C2758 is capable enough to cope with a few compressed datasets.

I don’t think it’s a Samba issue either. Windows is reporting well over 100 megabytes per second copying over the network to an SSD. I’m having difficulty reproducing the original problem, possibly because of caching on the ZFS pool.

Enforce changing of passwords in a Debian preseed automated install

If you’re setting up an automated Debian install, your users and passwords will be set up something like

 passwd passwd/root-login boolean false
 passwd passwd/make-user boolean true
 passwd passwd/user-fullname string Adam Baxter
 passwd passwd/username string voltagex
 passwd passwd/user-password string changeme
 passwd passwd/user-password-again string changeme
#you can also use user-password-crypted and supply a hash of the password, 
#see this StackExchange question for details

in your preseed.cfg. This example would set up a user with a username of voltagex and a not-very-secure password of changeme, as well as sudo permission.

The trick with this is you can actually enforce password policies at the same time with a late-command that runs just before the system reboots:

preseed         preseed/late_command    string  in-target passwd --expire voltagex

This will set the password to “expired” and result in the following when the user first logs in:

A screenshot showing that the password for the new user has expired

Resistance is futile: WhatsApp for Android hangs on “looking for backups”

TLDR: WhatsApp for Android 2.12.453 won’t really function unless it has access to read your contacts. Scroll down if you like story with your solutions.

I was finally pestered into installing WhatsApp after resisting for ages. Life was much simpler when I only needed one application to contact everyone, whether that was IRC, email, Pidgin (when it was Gaim) or even Trillian at the start of The Great Instant Messaging Fragmentation. Now I need multiple battery draining applications to get in touch with various people. It starts to make SMS look attractive.

Firstly, I got this encouraging message:
xuhvein

Anyway, once WhatsApp had “verified” my phone number, it stopped at a screen like this

wp-1457321129729.png
A loading screen that says “looking for backups”

I left it for a few minutes but it didn’t seem to help. Restarting the app / forcing it to close didn’t seem to help either. Some Googling revealed that it was “looking for backups” in Google Drive. I didn’t have Google Drive installed on my device so I grabbed it just in case. That didn’t help so I begrudgingly fired up adb logcat and got to work.

The first problem with logcat is that without any arguments, log messages will scroll past far too quickly. I was on a borrowed laptop so I didn’t want to install the full Android suite to get the GUI log viewer.

The documentation doesn’t say that you can filter by package (app) name, only by a “tag”. Bugger it, grep will do. After a while I narrowed it down to the following messages:

03-07 13:12:03.212 798 1493 I ActivityManager: START u0 {act=action_show_restore_one_time_setup cmp=com.whatsapp/.gdrive.GoogleDriveActivity} from uid 10215 on display 0
03-07 13:12:03.270 798 1485 W ActivityManager: Permission Denial: opening provider com.android.providers.contacts.ContactsProvider2 from ProcessRecord{b2ede58 6856:com.whatsapp/u0a215} (pid=6856, uid=10215) requires android.permission.READ_CONTACTS or android.permission.WRITE_CONTACTS
03-07 13:12:04.012 798 820 I ActivityManager: Displayed com.whatsapp/.gdrive.GoogleDriveActivity: +704ms (total +3s737ms)

No exceptions there but it definitely didn’t display anything to do with Google Drive. Fine, I’ll let you scan my contacts, WhatsApp (sorry to the two people I know who will be angry at me for their phone numbers being scanned).

wp-1457320243228.png

After swiping away / force stopping the app, restart it. You can skip the permission prompt here, it’ll work fine. You can even go back and deny permissions to the contacts again, but WhatsApp is very insistent that it will read (and write!) to your contact list. Denying the permission after the fact will give you a wobbly app and the following in your logs:

03-07 13:54:25.602 798 1015 W ActivityManager: Permission Denial: opening provider com.android.providers.contacts.ContactsProvider2 from ProcessRecord{f618cd3 14391:com.whatsapp/u0a217} (pid=14391, uid=10217) requires android.permission.READ_CONTACTS or android.permission.WRITE_CONTACTS
03-07 13:59:35.874 798 1939 I ActivityManager: START u0 {cmp=com.whatsapp/.RequestPermissionActivity (has extras)} from uid 10217 on display 0
03-07 13:59:36.161 798 820 I ActivityManager: Displayed com.whatsapp/.RequestPermissionActivity: +249ms
03-07 13:59:54.549 14391 14441 E WhatsApp: at com.whatsapp.util.Log.a(Log.java:244)
03-07 13:59:54.549 14391 14441 E WhatsApp: at com.whatsapp.util.Log.a(Log.java:240)
03-07 13:59:54.549 14391 14441 E WhatsApp: at com.whatsapp.util.Log.i(Log.java:56)
03-07 13:59:54.549 14391 14441 E WhatsApp: at com.whatsapp.ig.a(ig.java:847)
03-07 13:59:54.549 14391 14441 E WhatsApp: at com.whatsapp.akz.a(akz.java:307)
03-07 13:59:54.549 14391 14441 E WhatsApp: at com.whatsapp.apu.a(apu.java:28)
03-07 13:59:54.549 14391 14441 E WhatsApp: at com.whatsapp.apu.doInBackground(apu.java:29)
03-07 13:59:54.549 14391 14441 E WhatsApp: at com.whatsapp.util.cq.run(cq.java:2)

Enjoy the future of messaging! I’ll be over hanging out with the few remaining IRC users on Freenode if you need me.

FreeNAS Jail: “pkg: Unable to find catalogs”

When creating a new jail in FreeNAS 9.3, you might get an error after running pkg install (and the catalogues don’t seem to be saved at all

# pkg install cfdisk gdisk
Updating repository catalogue
digests.txz                         100% 2080KB 130.0KB/s 170.4KB/s   00:16
packagesite.txz                     100% 5375KB 141.5KB/s 235.1KB/s   00:38
pkg: package field incomplete: comment
Incremental update completed, 24491 packages processed:
0 packages updated, 0 removed and 24491 added.
pkg: Unable to find catalogs

WTF?

Turns out the jail config is wrong, at least it is if you built your jail using http://download.freenas.org/jails/9.2/x64/freenas-standard-9.2-RELEASE.tgz (I’d found a reference to this and not realised the format had changed between releases. Oops.)

You should be using http://download.freenas.org/jails/9.3/x64/freenas-standard-9.3-RELEASE.tgz and the associated mtree file but the fix if you’ve used the 9.2 config is to

 cat > /usr/local/etc/pkg/repos/FreeBSD.conf
FreeBSD: {
url: "pkg+http://pkg.FreeBSD.org/${ABI}/latest",
mirror_type: "srv",
enabled: yes
} #ctrl+d to end

and then run pkg update again.

It looks like FreeNAS caches and protects the jail templates, so I recommend creating a new template with the correct URL. You should probably remove and recreate the template in the UI.

FreeNAS 9.3 on the SuperMicro 5028A-TN4 / A1SRi-2758F

Warning: work in progress article

If you have this board you probably have IPMI. Use the “Virtual Storage” option to mount the FreeNAS ISO:

ipmi-virtual-storage.png

You may have to screw around in the BIOS (actually UEFI, but they do a great job of emulating the terrible UX from 20 years ago) to get the “ATEN Virtual CDROM” to show up as a boot option. You may also be able to boot the installer from one of the rear USB ports but this was an exercise in frustration.

Once you get into the FreeNAS installer, drop to a shell and run

kldload xhci.ko

You should get output similar to what’s shown below.

success

Type exit to get back to the installer and continue onwards. When you hit install, you should get to choose your USB3 device – it’s probably right down the bottom so scroll down.

success2

Here’s where https://forums.freenas.org/index.php?threads/problems-installing-9-3.25516/ is unfortunately wrong. I’m not sure whether adding

set kFreeBSD.xhci_load=YES

to GRUB’s options ever worked, but it certainly didn’t help me. So I tried something else.

maybesuccess3

At the FreeNAS/GRUB boot menu, hit ‘e’ to edit the FreeNAS entry, then add

kfreebsd_module_elf /ROOT/default/@/boot/kernel/xhci.ko

. At the moment I’m not sure whether position is important, but I added it after the line that loads ispfw.ko.

Unfortunately until we can get to grub.cfg or loader.conf we’ll have to re-enter that line every time we want to boot from USB3.

2015, the year of the Never Ending Open Source To Do List

This year was the year I took on way too much. It’s so easy to hit Fork on a GitHub repository, or to raise an issue, to try to help out on someone else’s project but eventually it all catches up with you.

I feel most guilty about bailing on WheelMapespecially as building accessibility is something that affects me personally. I was supposed to help implement a new search API but I got distracted building a Dockerfile to make my development setup easier. This eventually ended up sapping my enthusiasm for the project, then (paid) work got crazy and I had to step away.

I feel like I have open-source-ADD. There are so many cool projects to help out on, so many bugs that could be fixed if I could just spend 30 minutes on them… then another 30 minutes, and another until it’s Sunday at 3am and I really should sleep, lest my nighttime exploits affect my ability to pay for my Internet connection.

I didn’t really have a term for the way I was approaching open source / volunteer work until I replied to someone on Hacker News. The Never Ending Open Source To Do List. It’s like making the minimum repayments on a credit card while continuing to buy every game in a Steam Sale. It’s not going to end well – eventually you have to start paying interest.

in an article titled A Lot Happens, Jesse writes

You can’t be emotionally all in on everything. You can’t make another 24 hours appear to be “present” for everything. Instead, I stole time and ran my emotional credit card like it was limitless.

I guess reading about what Jesse went through stuck with me. I am NOT comparing our two experiences, but I can definitely relate to some of the patterns of behaviour (and I feel a bit guilty because I know I’ve asked favours of Jesse in the past).

You should go and read that one, and the followup. This article will still be here when you get back. It should serve as a warning for everyone who has stayed up late at night, somewhere near the Ballmer Peak, trying to get one more patch in or one more mailing-list discussion sorted.

In my perfect world, someone would pay for me to be Open Source Batman, swooping in on every call for contributors to help get projects going again. I mean, I was able to help PureDarwin by sending a few well-placed emails, wasn’t I?

In reality, I’m trading in my free time for a bit of an ego boost when issues get closed or patches get merged. My free time is a limited resource and sometimes it’d probably be better used getting fit or learning TypeScript.

I’m lucky I’ve had a bit of spare cash this year so I’ve been able to just spin up another instance to run another build for whatever project I’m working on this second, without putting too much thought into it. Unfortunately, the Australian dollar took a dive this year and most of the services I rely on are billed in US dollars. Renewals for things like my password manager, my VPS and my development tools came up and I found myself facing a bit of a shortfall.

Next year I’ll look into setting up a Patreon or something to cover the cost of the AWS instances I’ll be using for a lot of development as I’ve just turned off my DigitalOcean instance due to the falling Australian dollar.

But first I’ve got to do something worthy of being paid for – thus the cycle of the Never Ending Open Source To Do List starts again.

I hope that whatever job I find next year (hire me!) allows me to contribute to open source. I really do derive a lot of happiness from contributing to open source, but it’s a delicate balance between extracting a bit of a buzz from hitting the Fork button and getting enough sleep.

I don’t claim to have all the answers, and you should go read Ashe Dryden’s post on OSS and unpaid labor if you are also struggling with your open source contributions.

Here’s to knocking off a few more items on the List in 2016.