Help Request - Proxmox Partitions/Boot - CloverBootLoader

Ac5000 · 9 months ago

Help Request - Proxmox Partitions/Boot - CloverBootLoader

scrapeus@feddit.de · 9 months ago

Hey, i have had the same trouble on an DL380 G9. Those bioses don’t support booting from PCIe at all. My server can’t even boot from drives from the Raid controller in IT-Mode.

I would suggest, by proxmox being a hypervisor, to just install proxmox on a single SATA disk and try to boot from there. This is what I have done in the end.

You can then use your NVMes as storage pool. Also you bifurcation can always also be a problem when trying to boot from those devices.

I would also as a last call try to disable bifurcation and see if one drive will show up. Maybe then you could use 2 real PCIe slots with cheap m2 to PCIe adapters.

BlueÆther@no.lastname.nz · 9 months ago

a ssd sata drive in a cd-drive caddy converter is a good way to get a boot drive as well

Ac5000 · 9 months ago

The server has 24x 2.5" bays. I have an old SSD drive that I figure I could use as a last resort to be the Proxmox boot drive and then just use the NVMe’s as storage.

I was just hoping to have the Proxmox install/configuration in the NVMe RAID1 just for some minor safety in case a drive dies. From what I’ve read, this should be possible. I’m just lacking the knowledge to know what I’ve done wrong. (Mostly my lack of understanding the blkid results.)

30021190@lemmy.cloud.aboutcher.co.uk · 9 months ago

DL380 G9. Those bioses don’t support booting from PCIe at all.

They actually do but it can only be a HPE supported BootROM… anything non-HPE is ignored (weirdly, some Intel and Broadcom cards PXE boot without the HPE firmware but not all).

Most of these boards have internal USB and internal SD slots which you can boot from with any media, intact HPE sell a USB SD card raid adaptor for the usb slot. So I would recommend using SD card for this…

scrapeus@feddit.de · 9 months ago

I wouldn’t suggest usb or sd-cards with proxmox due to its constant logging. You will fry them really quick unfortunately. Had that problem with NVMes.

For litterly anything else I would also suggest SD-Cards.

Ac5000 · 9 months ago

The original plan is to use an SD card with Clover in read-only mode to bootload Proxmox running on the NVMe drives. (Read-only to prevent frying the SD card) This server has a built in SD Card slot Dell calls “vFlash” that you can actually remotely partition and configure. That’s where I was going to put the final configuration of Clover.

How fast/often is Proxmox writing logging? It’s concerning that you say you had this fry some NVMes since that’s what I’m trying to do here. Is this a setting that you can adjust?

scrapeus@feddit.de · 9 months ago

The problem is more with zfs on consumer grade NVMes. I have/had problems in that configuration due to the bigger sector sizes. Proxmox itself does do frequent writes, but I don’t know how often exactly. I know that my problems went away with not using zfs.

https://www.reddit.com/r/Proxmox/comments/idlqh3/zfs_extremely_high_ssd_wearout_seemingly_random/

Ac5000 · 9 months ago

Thank you for the details and link.

I looked around a little and seems like there are settings to help avoid this problem. Letting me know about this problem makes sure I catch it early. Unlike some of the people I’ve found that didn’t see the problem until it was already pretty bad…

I’ll keep this in mind if I can ever get this to work.

Ac5000 · 9 months ago

From what I’ve read online, Dell does something similar. There’s some sort of card/add-on that can enable directly seeing and booting from PCIe but they are costly.

This server has the internal USB and a build in SD slot accessible from the rear. (There’s also a dual card option like you mention for redundancy.)

My plan was to get Clover working with USB, then use the vFlash SD slot to hold the Clover bootloader in read-only mode. This would hopefully prevent the SD card from dying quickly.

fuggadihere@lemmy.world · 9 months ago

I was about to suggest scrapeus’ solution but only try boomg off a single nvme, since the zfs mirror adds another layer of complexity. At least you will be able to rule the zfs mirror root out of the equation.

Ac5000 · 9 months ago

Prior to doing the Proxmox install, and prior to the PCIe bifurcation, I still was unable to see the drives directly in iDRAC/bios. What I’ve read online is Dell does this for “reasons” and they happen to sell an add-on card to let you directly access NVMe from PCIe.

While I’m not ruling out the ZFS mirror issue, I don’t think it’s the cause of my problem considering both Clover and the Proxmox install debug can see the drives/partitions. I just don’t understand partition/device/boot structures and processes enough to make sense of what I’m seeing in the blkid/preboot results.

Trying to find information about it online just gets me bad guides about making partitions. The Linux docs for blkid and fdisk also don’t seem to have the explanation of the results, just the arguments for the commands.

Ac5000 · 9 months ago

I don’t think the bifurcation is causing me issues. Before I enabled it, I wasn’t able to see the drives from iDRAC/Bios. From what I’ve been able to research, this is expected and Dell sells the “solution” to booting directly from them. (Add-in card that’s pretty pricey…)

I do have an old SATA SSD that I’m considering slotting into one of the bays and using to boot. But I see that as a “last resort” option. I was hoping to have a bit of redundancy with the Proxmox install/configuration itself.

I feel that there’s a solution to the current setup and I just lack the knowledge to fix it. Everything I’ve been able to find points to my current setup being able to work. I’m just being hindered by not understanding partition/device/boot structure.

From what I understand, and what I saw during the Proxmox installation, if I can get past whatever part of the POST/boot process is preventing seeing the drives directly, I can use Clover to bootload from there. I’ve been able to boot into Clover just fine, and it was able to “see” the drives and partitions. I just don’t know which one should hold the Proxmox boot and if I’ve configured the Clover config correct.

BlueÆther@no.lastname.nz · 9 months ago

I had the same issue on an IBM m4 (with out bifurcation), I have tried with Clover and had limited success. What I’m doing at the moment is I have proxmox installed to a single nvme and the same iso installed to a thumb drive.

I then, on first boot, remapped the boot pool to point to the nvme drive. Down side is I have to update the kernel on both the thumdrive and nvme drive if I need to update the kernel version

Ac5000 · 9 months ago

What were the issues you had with Clover in particular? I’d be interested to hear since I’m trying to head down that path myself.

For your “remap” can you explain what you did/have an example? I think this might give me the knowledge I’m lacking since I think part of my problem is not understanding which partition/PARTUUID is the Proxmox boot/what I should point Clover at.

BlueÆther@no.lastname.nz · edit-2 9 months ago

I cant remember what the issue was sorry

If you go down the route of booting off a usb key to chain load proxmox there will be no real ware to the usb as it’s only used for booting.

On thye first boot it proxmox will complain that there are two zvol with the same name, and you just rename the one on the USB to ***_USB or the like and then continue the boot - I think I did a post on reddit or proxmox forum on how to do this

Edit, found it https://www.reddit.com/r/Proxmox/comments/ybxcxx/using_clover_to_boot_of_my_nvme_drive/

I have a M4 that I did this for:

Install proxmox to NVME as ZFS

Install proxmox to USB as ZFS

On first boot change the USB ZFS rootfs to a new name (this is from the top of my head but is backed up by thegeekdiary.com):

zpool import // find the ID of the NVME pool

zpool import -f NNNNNNNNNNNN

zpool import rootfs rootfsusb

zpool export rootfsusb

reboot

Ac5000 · 9 months ago

Thank you for responding and providing the link and info. The top comment in that reddit post has the same link I posted above.

For the

zpool import // find the ID of the NVME pool

How did you find the ID of the NVME pool? I think is part of the problem I have where I see multiple partitions and not entirely sure which is the “boot” partition I should be pointing to. I think in your case, you’re pointing to the “data” partition, but this might help me eliminate one of my options.

I’m also not sure how the raid1 plays into things since it seems like both physical drives seem to have the same partitions. Not sure if I can just point to one of the “boot” partitions on one of the drives and it’ll find it’s partner when it starts booting?

BlueÆther@no.lastname.nz · 9 months ago

the command zpool import will list all the pools and IDs

Decronym@lemmy.decronym.xyz · edit-2 9 months ago

Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I’ve seen in this thread:

Fewer Letters	More Letters
NVMe	Non-Volatile Memory Express interface for mass storage
PCIe	Peripheral Component Interconnect Express
SATA	Serial AT Attachment interface for mass storage
SSD	Solid State Drive mass storage

4 acronyms in this thread; the most compressed thread commented on today has 15 acronyms.

[Thread #167 for this sub, first seen 27th Sep 2023, 09:05] [FAQ] [Full list] [Contact] [Source code]

Help Request - Proxmox Partitions/Boot - CloverBootLoader

Help Request - Proxmox Partitions/Boot - CloverBootLoader

Background

Installation

Problem and Possible Solution

Troubleshooting and Where I Need Help

The Request(s)

fdisk

blkid

preboot.log

config.plist