Grub install error unable to identify a filesystem

I was having the same issue and came to an understanding about the situation. In my setup I have two USB drives in RAID 1 using mdadm with metadata v1.2 (the default), over the entire drives (not partitions on them).

The basic idea is that GRUB needs to embed itself into your storage device to boot load the operating system. As I understand it, there are three locations in which this data is embedded:

1. The first block of your device, typically (always?) 512 bytes. This is what the BIOS runs at startup, leaving the responsibility of the entire startup sequence to this 512 byte program. GRUB embeds its first stage boot loader to this location, the sole purpose of which is to find and load GRUB stage 2, which is stored elsewhere (and thus can be larger than 512 bytes).
2. The partition table. Since the 512 byte block is usually insufficient to boot the main OS, the partition table leaves space at its start so that first stage boot loaders can embed additional data (specifically the GRUB stage 2 boot loader). The exact amount of space available depends on the partition table being used, which means that during installation GRUB attempts to identify the partition table to verify that there is enough space.
3. The filesystem. In the same way that the partition table leaves space for the second stage boot loader, so does the filesystem. In this case GRUB needs to identify the filesystem to determine how much space is available to embed.

The second stage GRUB boot loader that is embedded directly into the free space allowed by the partition table and filesystem is /boot/grub/i386-pc/core.img. In my case this file is 25KB in size.

The error here is because GRUB was unable to detect a filesystem on the device, so it cannot know that the boot loader is safe. One workaround for this the —skip-fs-probe, -s option for GRUB which disables this check, making GRUB assume that there is enough space. However if you do not have a partition table on the device, you will simply get an error indicating that it cannot find the partition table (and thus cannot find where to store the second stage boot loader).

At this point the only option left is using blocklists instead of embedding core.img directly. Since GRUB can’t find a spot to put it, it uses the obsolete technique of leaving core.img on the filesystem (in my case, on my /boot partition at /grub/i386-pc/core.img) and embedding the location within the 512 byte first stage boot loader, instead of the file itself. Since the first stage boot loader doesn’t have enough space to safely access the filesystem, it bypasses it and simply accesses where core.img was last. This is unsafe because the filesystem containing core.img may reorganize files, or if for example you reinstall grub (thus deleting and recreating core.img), it will likely be at a new location. Again to work around this uncertainty it is possible to set the immutable attribute on the file to prevent the filesystem from moving or modifying it in any way. See the wiki entry for more information.

In my case, I was receiving the error because mdadm metadata version 1.2 stores the persistent superblock information (containing identifying information of each RAID array member) 4KB from the start of the device. This prevented GRUB from detecting the partition table and filesystem that was stored on the whole device RAID array. To solve this issue I recreated the RAID array using metadata version 1.0, where the superblock is stored at the end of the device (otherwise it is the same). Since it was simply RAID 1, GRUB was able to natively detect my partition table and filesystem and embed core.img without error.

Sources:

• GRUB Technical Info: A low level overview of the boot process and design (paraphrased above).

• $grub2_src/util/setup.c lines 359 to 594: The source code responsible for embedding GRUB into BIOS systems.

• mdadm(8) —metadata, -e: a description of different mdadm metadata formats and their locations on the device.

Jeremy Davis — Tue, 2021/03/16 — 08:41 —
8 comments

Issue:

Automatic security update of `grub-pc` package fails.

Affects:

All AMI (AWS EC2), OVA & VMDK v16.x appliances released to date. ISO & LXC/Proxmox builds are NOT affected.

Severity:

PITA — This issue means that the recent `grub-pc` package update isn’t installed (and thus remains vulnerable) on TurnKey v16.x systems. On face value that doesn’t sound good. But it’s not as bad as it sounds… Of the 7 CVEs patched by the `grub-pc` security update, only CVE-2021-20233 appears to be relevant to TurnKey users. And that one relates to USB… (For full details; please see Debian Security Advisory DSA-4867-1).

I will provide further details about the issue below (scroll down to «What the issue looks like»), but first I’ll post what to do:

To resolve — or check if you’re ok

Log into your server as `root` (or `admin` for AWSMP users). Then manually ensure that there are no broken packages:

apt install --fix-broken

(AWSMP users, will need to pre-fix `sudo`).

If it responds like this:

Reading package lists... Done
Building dependency tree       
Reading state information... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

Then you are NOT AFFECTED and you can safely ignore the rest of this post.

If you have been hit with this issue, then it will interactively ask you where to install `grub` (the default bootloader). First you should see this screen:

As that text notes, there is no harm in installing it places it doesn’t need to be. But to ensure that this (and any future grub updates) are installed to the correct place it is important that it is installed to where it needs to be.

As part of the build process, we always install grub to the primary (and only) disk image that contains TurnKey Linux. In the case of OVA/VMDK builds that should be `/dev/sda`; in the case of our AMI (AWS EC2 instance) that should be `/dev/xvda`.

The next screen will ask you to select where to install (OVA/VMDK):

Assuming that you haven’t added any additional volumes, then you only need to install to `/dev/sda` in OVA/VMDK; or `/dev/xvda` AMI (AWS EC2). If you have additional permanent volumes in use on your server, then unless you are 100% sure which is which, please don’t hesitate to install to all disks. It’s important to note, that if you have ANY DOUBT at all, please install it everywhere you can!

To select the relevant places to install grub, please use the arrow keys to move up & down the list, space to select/deselect the individual options and tab to move between the list and the «Ok». Here’s is what OVA users might expect after selecting `/dev/sda`:

Once you click Ok, it will go about installing grub to the relevant place. Please note that any of the following warnings/errors can _safely be ignored_:

• File descriptor 3 (pipe:[xxxxxxx]) leaked on vgs invocation. Parent PID xxxxx: grub-install

• grub-install: error: unable to identify a filesystem in hostdisk//dev/sda; safety check can't be performed.

• (or `hostdisk//dev/xvda` for AWS users).

• grub-install: warning: File system 'ext2' doesn't support embedding.

• grub-install: warning: Embedding is not possible.  GRUB can only be installed in this setup by using blocklists. However, blocklists are UNRELIABLE and their use is discouraged.

• grub-install: error: diskfilter writes are not supported.

---

What the issue looks like

It can be confirmed to exist if either you have been getting emails that look like this:

CRON-APT RUN [/etc/cron-apt/config]: Tue Mar 9 20:50:01 UTC 2021
CRON-APT SLEEP: 2699, Tue Mar 9 21:35:00 UTC 2021
CRON-APT ACTION: 5-install
CRON-APT LINE: /usr/bin/apt-get -o quiet=1 dist-upgrade -q -y -o APT::Get::Show-Upgraded=true -o Dir::Etc::sourcelist=/etc/apt/sources.list.d/security.sources.list -o Dir::Etc::sourceparts=nonexistent -o DPkg::Options::=--force-confdef -o DPkg::Options::=--force-confold
Setting up grub-pc (2.02+dfsg1-20+deb10u4) ...
You must correct your GRUB install devices before proceeding:
 DEBIAN_FRONTEND=dialog dpkg --configure grub-pc
 dpkg --configure -a
dpkg: error processing package grub-pc (--configure):
installed grub-pc package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
grub-pc
E: Sub-process /usr/bin/dpkg returned an error code (1)

Or perhaps if you’re not getting the emails, when you log in via SSH, you will see a message at the bottom of the MOTD (message of the day — the message you see when you first log in) saying `You have mail`. If you check your mail (e.g. for the `root` user: `cat /var/mail/root`) then you will see the above message.

If you didn’t get the above email, then that’s a separate issue. Please feel free to ask about that below or open a new thread in the support section of the forums and we can discuss that further…

This content is also available as issue #1579 on our Issue Tracker.

if you have any problems, questions or other feedback, please feel free to comment below, or open a new thread in the support section of the forums.

Good luck with it all and I look forward to hearing from you.