Support Wissensdatenbank

Run Proxmox with NVME RAID

Unfortunately it is not possible to run Proxmox with VROC until today 15.1.2020. The support by Proxmox and Intel is insufficient, there is a lack of usable drivers and support. We deliberately do not want to use a software RAID. Nevertheless, we do not want to do without a fast NVME RAID. Note that we do not guarantee this manual and that this configuration is not officially supported by Proxmox. Use it at your own risk!

To use NVME RAID you need the following:

• Fast server with PCI-E x16 GEN3
• Highpoint SSD7101A-1 RAID controller with NVME disks
• Normal hard disk or SSD drives
• The two installation files of Highpoint
• The proxmox_kernel_rebuild.sh script

1. Now start installing Proxmox on your hard drives. After you have installed and configured Proxmox you now need to compile the NVME drivers for Proxmox.
2. Use the script below, save it as proxmox_kernel_rebuild.sh and run it. Attention this process can take several hours! 
3. Now install the just created /pve-kernel/pve-headers-5.3.13-1*.deb and the /pve-kernel/pve-kernel-5.3.13-1*.deb Attention this process can take several hours! 
4. If the kernel could not be installed even after 4h waiting time, install GRUB and abort after 4h with “CTRL + C” and start point 3 again.
   apt-get install grub
5. Restart the server
6. Now block further kernel updates (IMPORTANT):
   apt-mark hold pve-headers-5.3.13-1
   apt-mark hold pve-kernel-5.3.13-1
7. Now install the two Linux drivers from Highpoint:
   ./rsnvme_linux_src_v1.2.18_19_12_11
   ./RAID_Manage_Linux_v2.3.14_17_07_26

INFO: If you ever want to update your kernel, just start at point 2 again and let the new kernel compile.

Script: proxmox_kernel_rebuild.sh

#!/bin/bash

# chmod +x proxmox_kernel_rebuild.sh
# ./proxmox_kernel_rebuild.sh

if ! pveversion; then
echo "Not a proxmox machine."
return
fi

id=`id -u`
if test $id -ne 0; then
echo "Please run the script with root privilege"
return
fi

# NOTE 30GB or more free space is needed
avail=`df . --output=avail | grep -v Avail`
let avail=$avail/1024/1024
if test $avail -lt 30; then
echo "In order to build new kernel package, 30GB or more freespace is needed."
df .
echo "Quiting now."
exit 1
fi

# Install toolchain
apt-get --assume-yes install patch diffutils git make flex bison dpkg-dev debhelper dh-python sphinx-common
apt-get --assume-yes install asciidoc-base libssl-dev libdw-dev libelf-dev libiberty-dev libnuma-dev libslang2-dev
apt-get --assume-yes install lz4 xmlto zlib1g-dev lintian

# clone pve-kernel git
echo "Clone pve-kernel from upstream git repository, it may take hours."
rm -rf pve-kernel
git clone git://git.proxmox.com/git/pve-kernel.git pve-kernel

cd pve-kernel/ || exit 1

# patch kernel to do not builtin nvme driver

diff="ZGlmZiAtLWdpdCBhL2RlYmlhbi9ydWxlcyBiL2RlYmlhbi9ydWxlcwppbmRleCBkMzQ0ZDRjLi5k
Yzc5N2JiIDEwMDc1NQotLS0gYS9kZWJpYW4vcnVsZXMKKysrIGIvZGViaWFuL3J1bGVzCkBAIC0z
Nyw3ICszNyw2IEBAIFBWRV9DT05GSUdfT1BUUz0gXAogLWUgQ09ORklHX0JMS19ERVZfU0QgXAog
LWUgQ09ORklHX0JMS19ERVZfU1IgXAogLWUgQ09ORklHX0JMS19ERVZfRE0gXAotLWUgQ09ORklH
X0JMS19ERVZfTlZNRSBcCiAtZSBDT05GSUdfTkxTX0lTTzg4NTlfMSBcCiAtZCBDT05GSUdfSU5Q
VVRfRVZCVUcgXAogLWQgQ09ORklHX0NQVV9GUkVRX0RFRkFVTFRfR09WX09OREVNQU5EIFwK"

echo "$diff" | base64 -d | patch -p1 || exit 1

echo "Download kernel sources and build pve-kernel, it may take hours, please be patient."

make || exit 1

pvek=`ls pve-kernel*.deb`

if [ "$pvek" == "" ]; then
echo "Something is wrong, failed to generate pve-kernel package."
exit 1
else
echo "Now everything is ready, you can install the built $pvek package"
fi