It took me 2 months to boot CM4 from NVMe

I was very excited when I heard, that Raspberry Pi Compute Module 4 supports NVMe via PCIe. No more USB->SATA3 bottlenecks (even if they don’t affect things much) on Raspberry Pi 4 boards. I was genuinely convinced that CM4 boot from NVMe will be as simple as Booting Raspberry Pi 4 from SSD! After weeks of trials and errors…

Booting CM4 from NVMe took weeks

Don’t worry, it will take you about 30 min to get everything sorted. I spent weeks battling issues caused by Integral NVMe drives. After excellent support from DeskPi (thanks Mandy and Jackie!) several CM4 boards and two Super6C to test, I eliminated everything else that could cause issues. My first piece of advice is – to get compatible NVMe drives.

I reached out to Integral asking for assistance as 3 of their M.2 drives gave me the same boot issues which made troubleshooting hard. I hope to learn more about why these NVMe are causing issues at boot while others don’t, especially since they meet the specification of Super6C.

Integral Update:

The Integral support desk was very helpful, promptly answered to my ticked and suggested that the issues with the NVMe in questions were related to the ones I have being Gen3 while the Kingston one I purchased, Gen4. In both cases we ended up puzzled over the issue as Super6C board supports Gen2+.
The long story short, new drives are in the post to test with the board thanks to the Integral support and despite the troubles, I’m pleased with the help received in this matter.

If you are getting a 60-second delay on each boot or you are stuck on this screen after reboot, chances are that your NVMe drive is not a good match for your setup. I should have tried other drives sooner, but at least you have this guide as a result of my misery.

Compute Module 4 – level hard

A case could be made that CM4 boards are not made for average tinkerers, they are much harder to work with than regular Raspberry Pi 4 boards. The fact that you need extra hardware (CM4-specific carrier board, or in my case DeskPi Super6C Cluster Board) is a good indication of what lies ahead. The same theme continues with the way of doing things, documentation and lack of tutorials that would appeal to someone less experienced.

If you like the board I’m using in this guide, you can take a look at the links below to get one for yourself. The guide is valid for any CM4 carrier boards that feature boot tools.

• AmazonUK: Super6C Board, Super6C Enclosure
• AmazonUS: Super6C Board, Super6C Enclosure
• DeskPi Store: Super6C Board, Super6C Enclosure

I have some plans for my cluster, so keep an eye on my articles if you want to know what my cluster will end up doing.

32 flavours of Pi

To make things even more fun, CM4 comes in 2 flavours. The regular Compute Module 4 has eMMC storage (which acts in the same way as a microSD card slot) and Compute Module 4 Lite is a version without that onboard storage. Each module is also available with WiFi and without and in various RAM configurations for a total of 32 SKUs available for sale.

To pick the one you want, you have to “decode” the model number. Here is a handy guide:

SKU: CM4001000

CM4 0 01 000
Compute Module 4
WiFi (1|0 = yes|no)
RAM (01|02|04|08  = 1GB|2GB|4GB|8GB)
Storage (000|008|016| = SD Card|8GB eMMC|16GB eMMC|32GB eMMC)

But you already have one in your hand, so let’s talk about the CM4 boot from NVMe.

CM4 EEPROM & Boot

Unlike Raspberry Pi 4, the boot order (and EEPROM firmware) cannot be updated from the same board. CM4 modules can either operated in Operating System mode, as the mass storage devices (CM4 versions) and EEPROM programming mode. You’ll need another computer to enable CM4 boot from NVMe. In this guide, I’ll cover Windows and Linux approaches. Let’s start with common preparation steps

First, check if you are lucky

If you are lucky, your CM4 board could come with firmware that already supports NVMe boot. If that’s the case, Compute Module 4 without eMMC storage would boot Raspberry Pi OS from NVMe without any changes.

The biggest obstacle here is to image your NVMe drive, as Raspberry Pi Imager will only recognise it as a bootable USB option when a USB-NVMe adapter is used. These are not terribly expensive and you can get one for £17 on Amazon. It’s good to have one of these handy for other tasks too. If that’s the case, flash Raspberry Pi OS on your NVMe drive via a USB adapter, plug it into your carrier board and remove the micro SD card (otherwise it will take the boot priority).

If your CM4 board has older firmware (or|and) and comes with eMMC storage it will always boot from the eMMC unless the boot order is changed.

1. CM4 NVMe boot using Windows
2. CM4 NVMe boot using Linux
3. Cloning OS to NVMe

CM4 NVMe boot using Windows

We need a working OS on your CM4 board. For CM4 Lite versions the process is simple: flash Raspberry Pi OS on a microSD card (use advanced options to set the default user, hostname and enable SSH).

If you have CM4 boards with eMMC you’ll need a rpiboot utility (direct link) and a carrier board that allows you to put the CM4 module into USB storage mode.

To do this, locate the J2 header on your board. Super6C one is also labelled as RPI BOOT. You will need to put the jumper on the pins (or toggle a switch depending on your board) to put the CM4 into mass storage mode.

Start rpiboot on Windows and reboot your CM4 board. After a couple of seconds, you should see eMMC as a USB drive in Raspberry Pi Imager. Flash Rasberry Pi OS as normal, remove the jumper and power cycle your board to boot the OS.

If your CM4 board doesn’t show up as Mass Storage Device, make sure it’s the only USB thing plugged into your computer. It tried to flash my mouse at first until I unplugged it.

Updating EEPROM firmware

You can use this download to “cheat” and update the firmware to a stable release from 04.11.22. Simply unpack the files into "C:\Program Files (x86)\Raspberry Pi\recovery" replacing existing files inside that folder.

It’s time to talk about the boot.conf. It’s a file that contains the boot configuration of your file. The default boot order is displayed like this

BOOT_ORDER=0xf25641 <- reads from this side

The numbers represent the boot medium and 0xf is a reboot. CM4 module will try every medium from the right until it finds the one that boots. On CM4 Lite modules, this shouldn’t need modification if your firmware already supports NVMe (boot medium#6) but you can also trim it to size. The regular CM4 boards with eMMC storage (boot medium #1) require an edit since eMMC storage is treated in the same way as micro SD cards and you can’t remove it to bypass the boot option.

For CM4 boot you can go with:

BOOT_ORDER=0xf25416

#above boot configuration
NVMe, SD card, USB, USB CM4, Network, Reboot

Then go up one folder and in the same directory as rpiboot.exe press Shift and right-click on the empty space inside the folder (switch to Large Icons view) to open PowerShell/Command Line in that folder. Then run:

./rpiboot -d recovery

Use the J2 jumper to put your CM4 board into RPI BOOT mode and power cycle the board. The flash will take a couple of seconds and once done, you’ll see a confirmation in the terminal (if you have a screen connected to your Raspberry Pi, it will turn green).

The output should look like this:

RPIBOOT: build-date Jul 18 2022 version 20220718~085937 5a25e04b
Loading: recovery/bootcode4.bin
Waiting for BCM2835/6/7/2711…
Loading: recovery/bootcode4.bin
Sending bootcode.bin
Successful read 4 bytes
Waiting for BCM2835/6/7/2711…
Loading: recovery/bootcode4.bin
Second stage boot server
Loading: recovery/config.txt
File read: config.txt
Loading: recovery/pieeprom.bin
Loading: recovery/pieeprom.bin
Loading: recovery/pieeprom.sig
File read: pieeprom.sig
Loading: recovery/pieeprom.bin
File read: pieeprom.bin
Second stage boot server done

If you have any problems at this stage, make sure no other USB devices are connected to your computer. Once booted, you will be able to verify this with:

sudo CM4_ENABLE_RPI_EEPROM_UPDATE=1 rpi-eeprom-update

#output
BOOTLOADER: up to date
   CURRENT: Fri  4 Nov 15:19:15 UTC 2022 (1667575155)
    LATEST: Tue 26 Apr 10:24:28 UTC 2022 (1650968668)
   RELEASE: default (/lib/firmware/raspberrypi/bootloader/default)
            Use raspi-config to change the release.

  VL805_FW: Using bootloader EEPROM
     VL805: up to date
   CURRENT:
    LATEST:

But if you want to do this properly

Access the terminal and SSH into your CM4 module (make sure it’s not in Boot mode). Then update and download the following dependencies:

sudo apt install git libusb-1.0-0-dev build-essential -y

#then clone usb boot
git clone --depth=1 https://github.com/raspberrypi/usbboot
cd usbboot
make

Note that make will enable the usage from the installation folder (it’s fine for now unless you want it to be a user command). Access the recovery directory and:

cd usbboot/recovery

#remove the existing firmware
rm -f pieeprom.original.bin

#download the latest stable firmware & update
curl -L -o pieeprom.original.bin https://github.com/raspberrypi/rpi-eeprom/raw/master/firmware/stable/pieeprom-2022-11-04.bin

./update-pieeprom.sh

Chances are, there is new firmware available by the time you are reading this. You can simply go to this page and click on the firmware of your choice (look at the dates), open it and right-click on the Download button to copy the direct link to the firmware. Once you update the EEPROM file you should see:

/home/pi/usbboot/tools/rpi-eeprom-config --config boot.conf --out pieeprom.bin pieeprom.original.bin set +x 
new-image: pieeprom.bin 
source-image: pieeprom.original.bin 
config: boot.conf

At this stage, you can make the necessary changes to your boot.conf file:

sudo nano boot.conf

#and run for good measure
./update-pieeprom.sh

It’s time to get these files back onto your Windows machine. The easiest way is to use Notepadd++ and SFTP plugin. It will let you explore files and folders on your Raspberry Pi board, and easily upload, save and download files to your computer too.

Zip the archive to save yourself time:

cd
zip recovery.zip -r usbboot/recovery 

And use Notepad++ explorer to save the file to your computer (via right click “Save As” menu). It’s a really comfortable way of interfacing with files on Raspberry Pi. Unpack zipped files into recovery_nvme folder. On Windows, open C:\Program Files (x86)\Raspberry Pi and paste the folder inside it. You should see a default recovery folder inside that directory. Back it up, as should things go wrong, you’ll need it.

CM4 NVMe boot using Linux

If you just jumped to this paragraph, without reading previous chapters, please note, that you will need another Raspberry Pi (or Linux) machine connected to your CM4 board. You cannot update the firmware of the CM4 from the same Compute Module board.

The same rpiboot utility you have downloaded on your computer is available as usbboot on Linux. It can be used to push EEPROM firmware to your CM4 board. First, you’ll have to get the latest boot firmware and modify the boot order of your module.

On your CM4 board download the following packages & usbboot

sudo apt install git libusb-1.0-0-dev build-essential -y
git clone --depth=1 https://github.com/raspberrypi/usbboot
cd usbboot
make

Note that make will enable the usage from the installation folder (it’s fine for now unless you want it to be a user command). Access the recovery directory and:

cd usbboot/recovery

#remove the existing firmware
rm -f pieeprom.original.bin

#download the latest stable firmware & update
curl -L -o pieeprom.original.bin https://github.com/raspberrypi/rpi-eeprom/raw/master/firmware/stable/pieeprom-2022-11-04.bin

./update-pieeprom.sh

Chances are, there is new firmware available by the time you are reading this. You can simply go to this page and click on the firmware of your choice (look at the dates), open it and right-click on the Download button to copy the direct link to the firmware. Once you update the EEPROM file you should see:

/home/pi/usbboot/tools/rpi-eeprom-config --config boot.conf --out pieeprom.bin pieeprom.original.bin
set +x
new-image: pieeprom.bin
source-image: pieeprom.original.bin
config: boot.conf

It’s time to talk about the boot.conf. It’s a file that contains the boot configuration of your file. The default boot order is displayed like this

BOOT_ORDER=0xf25641 <- reads from this side

The numbers represent the boot medium and 0xf is a reboot. CM4 module will try every medium from the right until it finds the one that boots. On CM4 Lite modules, this shouldn’t need modification if your firmware already supports NVMe (boot medium#6) but you can also trim it to size. The regular CM4 boards with eMMC storage (boot medium #1) require an edit since eMMC storage is treated in the same way as micro SD cards and you can’t remove it to bypass the boot option.

For CM4 boot you can go with:

BOOT_ORDER=0xf25416

#above boot configuration
NVMe, SD card, USB, USB CM4, Network, Reboot

For PowerShell use rpiboot -d recovery to start the process. The window will wait until you short the J2 header and connect the micro USB cable and reboot your CM4 module.

The output should look like this:

RPIBOOT: build-date Jul 18 2022 version 20220718~085937 5a25e04b
Loading: recovery/bootcode4.bin
Waiting for BCM2835/6/7/2711…
Loading: recovery/bootcode4.bin
Sending bootcode.bin
Successful read 4 bytes
Waiting for BCM2835/6/7/2711…
Loading: recovery/bootcode4.bin
Second stage boot server
Loading: recovery/config.txt
File read: config.txt
Loading: recovery/pieeprom.bin
Loading: recovery/pieeprom.bin
Loading: recovery/pieeprom.sig
File read: pieeprom.sig
Loading: recovery/pieeprom.bin
File read: pieeprom.bin
Second stage boot server done

If you have any problems at this stage, make sure no other USB devices are connected to your computer. Once booted, you will be able to verify this with:

sudo CM4_ENABLE_RPI_EEPROM_UPDATE=1 rpi-eeprom-update

#output
BOOTLOADER: up to date
   CURRENT: Fri  4 Nov 15:19:15 UTC 2022 (1667575155)
    LATEST: Tue 26 Apr 10:24:28 UTC 2022 (1650968668)
   RELEASE: default (/lib/firmware/raspberrypi/bootloader/default)
            Use raspi-config to change the release.

  VL805_FW: Using bootloader EEPROM
     VL805: up to date
   CURRENT:
    LATEST:

Cloning Raspberry PI OS to NVMe

Our next step is to clone the os from a micro SD card or eMMC storage to NVMe. If you have the NVMe/SATA>USB adapter I mentioned earlier – it’s the easiest and quickest way as I discovered over time.

Broken cloning with rpi-clone

At first, I opted for rpi-clone. Probably the easiest way to clone partitions. I have used that countless times when booting Raspberry Pi from USB, – it seemed like the right fit for CM4 boot from NVMe. It was not. Not without fixes.

rpi-clone has not been updated in about 4 years. Still works great for RPI4, however, there was no support for NVMe at all. Thankfully, it was “kind of fixed” thanks to a PR. This pull request contains changes to rpi-clone bash file. This was my 1st stumbling block.

sudo apt-get install git -y

#or even better install these as we will use them later
sudo apt install git libusb-1.0-0-dev build-essential -y

#get rpi-clone utility
git clone https://github.com/billw2/rpi-clone.git

cd rpi-clone/

#remove rpi-clone file and replace it with modified version
sudo rm rpi-clone
curl -L -o rpi-clone https://raw.githubusercontent.com/geerlingguy/rpi-clone/123-nvme/rpi-clone

#optional
sudo cp rpi-clone rpi-clone-setup /usr/local/sbin

#create mount directories
cd
mkdir boot root

#mount partitions
sudo mount /dev/nvme0n1p1 boot
sudo mount /dev/nvme0n1p2 root

#change UUID in boot partition
cd boot/etc
sudo nano fstab

#change UUID in root partition
cd boot/
sudo nano cmdline.txt

Save yourself time and use dd instead to clone the eMMC/micro SD bit by bit onto NVMe. That worked well. It worked surprisingly well although it takes about 10 min to complete (depending on your eMMC size).

#if you want to check the drive names
lsblk

sudo dd if=/dev/mmcblk0 of=/dev/nvme0n1 bs=4MB status=progress

#Successfull clone ends with
7817+1 records in
7817+1 records out
31268536320 bytes (31 GB, 29 GiB) copied, 420.186 s, 74.4 MB/s

At this point, if you can also wipe the eMMC with fdisk and use that storage for something.

Reboot and pray

CM4 boot from NVMe is… not easy. If you followed these steps and have one of the checked by me NVMe drives, you should be good to go. I’m hoping to hear from Integral about the NVMe issue and find out why I had so many issues with these drives. I hope the whole process took you 30 min at best and it was easy to follow regardless of the platform you used.

Final thoughts

I’ve learned a bit more about Linux. It’s not knowledge I wanted, I spent more time on this than I care to admit. I’d rather spend that time on actual projects, but if this article saves you days of research and compresses your struggle to about 30 min of reading time and following instructions – it was worth it. Want to know more or leave a helpful comment? See this Reddit thread. Otherwise, buy me a coffee using the options below as it’s late and I’m still reading things up.