Recently, I upgraded my home NAS from a DS212J to something a little more modern and powerful. The DS212J is quite old, and Synology is very likely to drop software support for it very soon - in fact, they’ve already stopped providing major firmware updates. I figured it’d be fun to see how difficult it would be to boot Debian, or some other Linux distro that supports ARM processors.

In summary: it’s not too difficult! Thankfully, Synology seem to be relatively open, and it’s not too difficult to reverse engineer the boot process. The board also exposes a serial console via UART (pinouts are relatively easy to find on the internet). In addition to that, the source code for various GPL’d bits of the firmware is available for download.

I’ve documented some of my notes below, in case someone would like to follow along. This won’t be a tutorial, so I’m going to omit a lot of the low-level details.

Boot process overview

The bootloader is U-Boot.

It boots a kernel image and a small initrd stored in the built-in flash storage. This kernel and initrd image can be upgraded during a DSM firmware upgrade. It’s possible to inspect the kernel and initrd image, by simply downloading the .pat firmware file from Synology’s website, and extracting it as a tarball.

zImage, and rd.bin is the kernel and initrd image respectively. They’re in the legacy U-Boot image format. hda1.tgz contains the root filesystem for the DSM operating system.

The operating system/root filesystem itself is stored, and replicated, on each hard-drive. The init process will try to mount the root filesystem on boot-up and then perform some soundness checks to ensure that it does indeed contain the DSM operating system. Part of this check involves reading version information at /etc.defaults and comparing that to the version file in the initrd. The full boot process can be found by extracting the initrd from the firmware file, and inspecting the linuxrc.syno init script.

The init process will also look for any pending upgrade operations and execute them, but I did not look too deeply into this process.

If at any point in the boot process, something isn’t quite right with the root filesystem (e.g wrong partition/filesystem layout, or a magic file that requests a factory reset was found in the rootfs), the initrd drops into a setup mode. It then waits for commands from the Synology Assistant software to wipe, and install the DSM software on the hard drives.

Partition format

Here’s the fdisk output of a test drive that I installed DSM on:

Disk /dev/sdb: 74.53 GiB, 80026361856 bytes, 156301488 sectors
Disk model: USB3.0
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x1e954ebc

Device     Boot   Start       End   Sectors  Size Id Type
/dev/sdb1          2048   4982527   4980480  2.4G fd Linux raid autodetect
/dev/sdb2       4982528   9176831   4194304    2G fd Linux raid autodetect
/dev/sdb3       9437184 156287327 146850144   70G  f W95 Ext'd (LBA)
/dev/sdb5       9453280 156094559 146641280 69.9G fd Linux raid autodetect

The first partition is part of a software RAID 1 configuration, and contains the DSM operating system. It can be mounted for inspection and modification by using mdadm:

# mdadm --create --verbose /dev/md0 --metadata=0.90 --level 1 --raid-devices 2 /dev/sdb1 missing
mdadm: /dev/sdb1 appears to contain an ext2fs file system
       size=2490176K  mtime=Thu Nov 18 23:38:57 2021
mdadm: size set to 2490176K
Continue creating array? y
mdadm: array /dev/md0 started.
# mount /dev/md0 ./some-folder

The second partition also appears to be part of a software RAID 1 configuration, and is used as swap.

The remaining partition/s are used to store data.

Booting Debian

My first thought was to clear out the DSM operating system, and use debootstrap to install Debian in its place. Synology hadn’t appeared to implement any kind of secure boot mechanism, so I was hopeful that this would work.

Unfortunately, there’s a few more things that needs to be done to pretend to be Synology firmware. Otherwise, the boot process will fail, and enter setup mode, as detailed above.

The first complaint from the Synology boot process is about a missing init in the target filesystem. This didn’t make any sense, since I could clearly see that the boot process was checking for the presence of /sbin/init on the target root filesystem, and that it clearly exists when I inspected it on my computer.

The problem was that /sbin/init on modern Debian distros is actually a symlink to /lib/systemd/systemd. Synology’s boot script mounts the target filesystem at /tmpRoot so it can perform its soundness checks (it pivot_roots to the target filesystem at the end of its boot process, if successful). The script is actually checking for the existence of /tmpRoot/sbin/init, which points to /lib/systemd/systemd, and that doesn’t exist in the boot environment. This is easily worked around by making it a hard link instead.

The second complaint from the Synology boot process is that my target root filesystem is missing some files that the process expects to exist:

  • /etc.defaults/synoinfo.conf
  • /etc.defaults/VERSION
  • /.syno/patch

These are used to ensure that the firmware installed on the drives match the kernel and initrd image stored in flash. I just copied those files from the firmware update image into my root filesystem. It’s worth mentioning that the boot process will also write diagnostic logs to /.log.junior, so it should be possible to diagnose any issues without serial console access.

The final complaint from the boot process:

FATAL: kernel too old
[   22.610000] Kernel panic - not syncing: Attempted to kill init!

Hm?

Uncompressing Linux... done, booting the kernel.
[    0.000000] Linux version 2.6.32.12 (root@build4) (gcc version 4.6.4 (Marvell GCC release 20150204-c4af733b 64K MAXPAGESIZE ALIGN CVE-2015-0235) ) #25556 Thu Mar
4 17:56:48 CST 2021

Oh. I guess 2.6.32.12 is a bit outdated… Looks like systemd/glibc doesn’t like that.

Booting Debian Part 2

Obviously, if you’re booting a distro that doesn’t require a modern kernel, you can probably stop here and just boot by modifying the root filesystem as described above. Up until this point, there’s no need to actually modify the NAS itself, or require any special hardware.

I was set on booting a modern Debian installation though, so I needed to boot a more recent kernel. Since I was going to boot my own kernel, with my own initrd, I could also abandon Synology’s boot process altogether. The upside is that we no longer need to pretend to be Synology firmware. The downside is that, in order to do this, it’s necessary to have serial console access.

The U-Boot firmware on the NAS supports downloading images over TFTP, so we can use it to boot a different kernel, and load a different initrd image.

I was surprised to learn that standard Debian kernel for ARM platforms actually includes support for the DS212J. I used QEMU user mode emulation to chroot into my ARM Debian installation to install the linux-image-marvell package. This generates an initrd image, and gives me a kernel image plus a bunch of device tree blobs to use.

The version of U-Boot on the DS212J is too old to support DTBs unfortunately. However, Debian ARM kernels supports a workaround for this issue: it’s possible to simply append the DTB to the kernel image. When the kernel boots, it will look at that location for a DTB, if one isn’t passed to it by the bootloader.

Here’s what I did to produce something that U-Boot could boot:

# cat vmlinuz-5.10.0-9-marvell ../usr/lib/linux-image-5.10.0-9-marvell/kirkwood-ds212j.dtb > zImage
# mkimage -A arm -O linux -T kernel -C none -a 0x00008000 -e 0x00008000 -n "vmlinuz-5.10.0-9-marvell" -d zImage uImage
# mkimage -A arm -T ramdisk -C none -a 0x00800000 -e 0x00800000 -n "initrd.img-5.10.0-9-marvell" -d initrd.img-5.10.0-9-marvell uInitrd

To boot over TFTP, I interrupted the boot process to get to a U-Boot prompt, and used the following commands:

tftpboot 8000000 uImage
tftpboot 8300000 uInitrd
setenv bootargs console=ttyS0,115200 root=/dev/sda1 rw
bootm 8000000 8300000

The address that the TFTP server needs to respond to is the one stored in the serverip U-Boot environment. On my NAS, it’s 192.168.101.111:

Marvell>> printenv
[...snip...]
netmask=255.255.0.0
ipaddr=192.168.101.224
serverip=192.168.101.111

It’s also worth mentioning that the Synology kernel did not support kexec, otherwise I’d have considered chainloading a more modern kernel.

No SATA

The first thing I realised after booting was that the new kernel wasn’t detecting any drives connected via the SATA ports. In fact, it didn’t even sound like they were getting power.

I initially suspected that this was some sort of driver issue, but it turned out that the device tree for the DS212J failed to include some power regulator drivers. The SATA driver initialised correctly, but nothing was instructing the hardware to actually start delivering power to the drives.

Since the source code for the Synology kernel was available, I was able to determine that GPIOs 29 and 31 controlled power delivery to the drives. I was able to confirm that, by manually toggling them in software via /sys/class/gpio/, and hearing the drives spin up.

Once I added the right entries to the device tree, I was able to boot straight from SATA.

High Idle CPU usage

I quickly noticed that the idle CPU usage would sit at around 19%, mostly taken up by the systemd-udevd process. This was very unusual, since there shouldn’t be any hardware changes once booted. Clearly, systemd-udevd was waking up far more often than it should.

I tracked down the source of the events that were causing systemd-udevd to wake up. It was a fan alarm event, from the fan controller driver. Somehow, Linux was getting hundreds of these events every second, and the fan was definitely not failing.

The device tree configures GPIO 35 as the fan alarm indicator, and I wanted to inspect what was going on. Indeed, looking at the pin transitions, it was constantly changing state. On a hunch, I decided to stop the fan from spinning to see what would happen. The pin would hold its value when the fan was stopped, then start changing its value when I let go.

Clearly, that GPIO is a tachometer input, not a fan alarm input. Thus, every time the fan blades turned, it’d generate a udev event, causing systemd-udevd to waste cycles handling the interrupts.

Once I fixed the device tree configuration, CPU usage at idle returned back to a respectable figure.

Exercises left to the reader

That was as far as I’ve gotten. I was able to boot into a modern Debian installation.

There are some ideas, and refinements that I’ve yet to work on:

  • Use tiny-initramfs to generate a smaller initrd.
    • The default Debian-generated one is 15MB, and will not fit in the 4MB onboard flash.
  • Actually flash the modified kernel and initrd to the onboard flash to make the install more permanent.
    • Right now, booting requires the serial console, and a TFTP server to serve the images.
  • Make Debian do something useful.
    • Serve files or something, I dunno.
  • Work on building a Synology firmware downgrader.
    • By default, Synology does not allow downgrading firmware.
    • Some users prefer running earlier versions of DSM.
    • Since we know how to boot into our own code by pretending to be Synology firmware, we can make a downgrader that flashes old Synology kernel and initrd images.
    • This would let users downgrade to arbitrary versions of DSM.
  • Upstream my device tree changes to the mainline kernel sources.