Error read only file system during write on dev disks

Recently I wanted to install ESXi 6.5 on a USB disk and also use that disk as a datastore to store VM on. I couldn’t get any VMs to run off the USB disk but I spent some time getting the USB disk presented as a datastore so wanted to post that here.

Installing ESXi 6.5 to a USB is straight-forward.

And this blog post is a good reference on what to do so that a USB disk is visible as a datastore. This blog post is about presenting a USB disk without ESXi installed on it – i.e. you use the USB disk entirely as a datastore. In my case the disk already had partitions on it so I had to make some changes to the instructions in that blog post. This meant a bit of mucking about with partedUtil, which is the ESXi command line way of fiddling with partition tables. (fdisk while present is no longer supported as it doesn’t do GPT).

1. First, connect to the ESXi host via SSH.

2. Shutdown the USB arbitrator service (this is used to present a USB disk to a VM): /etc/init.d/usbarbitrator stop (Update: one of the readers emailed me to say this step and the next are not needed and in fact it stops the USB disk from being made visible. So skip them, but if skipping them doesn’t help then try again without skipping them.

3. Permanently disable this service too: chkconfig usbarbitrator off

4. Now find the USB disk device from /dev/disks. This can be done via an ls -al. In my case the device was called /dev/disks/t10.SanDisk00Cruzer_Switch0000004C531001441121115514.

So far so good?

To find the partitions on this device use the partedUtil getptbl command. Example output from my case:

The “gpt” indicates this is a GPT partition table. The four numbers after that give the number of cylinders (7625), heads (255), sectors per track (63), as well as the total number of sectors (122508544). Multiplying the cylinders x heads x sectors per head should give a similar figure too (122495625).

An entry such as 9 1843200 7086079 9D27538040AD11DBBF97000C2911D1B8 vmkDiagnostic 0 means the following:

• partition number 9
• starting at sector 1843200
• ending at sector 7086079
• of GUID 7086079 9D27538040AD11DBBF97000C2911D1B8, type vmkDiagnostic (you can get a list of all known GUIDs and type via the partedUtil showGuids command)
• attribute 0

In my case since the total number of sectors is 122495625 (am taking the product of the CHS figures) and the last partition ends at sector 7086079 I have free space where I can create a new partition. This is what I’d like to expose to the ESX host.

There seems to be gap of 33 sectors between partitions (at least between 8 and 7, and 7 and 6 – I didn’t check them all :)). So my new partition should start at sector 7086112 (7086079 + 33) and end at 122495624 (122495625 -1) (we leave one sector in the end). The VMFS partition GUID is AA31E02A400F11DB9590000C2911D1B8, thus my entry would look something like this: 10 7086112 122495624 AA31E02A400F11DB9590000C2911D1B8 0.

But we can’t do that at the moment as the disk is read-only. If I try making any changes to the disk it will throw an error like this:

From a VMware forum post I learnt that this is because the disk has a coredump partition (the vmkDiagnostic partitions we saw above). We need to disable that first.

5. Disable the coredump partition: esxcli system coredump partition set --enable false

6. Delete the coredump partitions:

7. Output the partition table again:

So what I want to add above is partition 9. An entry such as 9 1843232 122495624 AA31E02A400F11DB9590000C2911D1B8 0.

8. Set the partition table. Take note to include the existing partitions as well as the command replaces everything.

That’s it. Now partition 9 will be created.

All the partitions also have direct entries under /dev/disks. Here’s the entries in my case after the above changes:

Not sure what the “vml” entries are.

9. Next step is to create the datastore.

That’s it! Now ESXi will see a datastore called “USB-Datastore” formatted with VMFS6.

VMware always recommends formatting and creating a new VMFS volume using the vSphere Client as it automatically aligns your VMFS volume. However, if you do not have access to the vSphere Client or you wanted to format additional VMFS volumes via a kickstart, you can do so using the CLI and the partedUtil under /sbin.

~ # /sbin/partedUtil
Not enough arguments
Usage:
Get Partitions : get
Set Partitions : set [«partNum startSector endSector type attr»]*
Delete Partition : delete Resize Partition : resize
Get Partitions : getptbl
Set Partitions : setptbl [«partNum startSector endSector type/guid attr»]*
Fix Partition Table : fix
Create New Label (all existing data will be lost): mklabel
Show commonly used partition type guids : showGuids

With ESXi 5, an MBR (Master Boot Record) partition table is no longer used and has been replaced with a GPT (GUID Partition Table) partition table. There is also only one block size of 1MB versus the 2,4 and 8 that was available in ESX(i) 4.x

We can view the partitions of a device by using the «getptbl» option and ensure we don’t have an existing VMFS volume:

~ # /sbin/partedUtil «getptbl» «/vmfs/devices/disks/mpx.vmhba1:C0:T2:L0»
gpt
652 255 63 10485760

Next we will need to create a partition by using the «setptbl» option:

/sbin/partedUtil «setptbl» «/vmfs/devices/disks/mpx.vmhba1:C0:T2:L0» «gpt» «1 2048 10474379 AA31E02A400F11DB9590000C2911D1B8 0»

The «setptbl» accepts 3 arguments:

• diskName
• label
• partitionNumber startSector endSector type/GUID attribute

The diskName in this example is the full path to the device which is /vmfs/devices/disks/mpx.vmhba1:C0:T2:L0

The label will be gpt

The last argument is actually a string comprised of 5 individual parameters:

• partitionNumber — Pretty straight forward
• startSector — This will always be 2048 for 1MB alignment for VMFS5
• endSector — This will need to be calculated based on size of your device
• type/GUID — This is the GUID key for a particular partition type, for VMFS it will always be AA31E02A400F11DB9590000C2911D1B8

To view all GUID types, you can use the «showGuids» option:

~ # /sbin/partedUtil showGuids
Partition Type       GUID
vmfs                 AA31E02A400F11DB9590000C2911D1B8
vmkDiagnostic        9D27538040AD11DBBF97000C2911D1B8
VMware Reserved      9198EFFC31C011DB8F78000C2911D1B8
Basic Data           EBD0A0A2B9E5443387C068B6B72699C7
Linux Swap           0657FD6DA4AB43C484E50933C84B4F4F
Linux Lvm            E6D6D379F50744C2A23C238F2A3DF928
Linux Raid           A19D880F05FC4D3BA006743F0F84911E
Efi System           C12A7328F81F11D2BA4B00A0C93EC93B
Microsoft Reserved   E3C9E3160B5C4DB8817DF92DF00215AE
Unused Entry         00000000000000000000000000000000

Once you have the 3 arguments specified, we can now create the partition:

~ # /sbin/partedUtil «setptbl» «/vmfs/devices/disks/mpx.vmhba1:C0:T2:L0» «gpt» «1 2048 10474379 AA31E02A400F11DB9590000C2911D1B8 0»
gpt
0 0 0 0
1 2048 10474379 AA31E02A400F11DB9590000C2911D1B8 0

UPDATE (01/15) — Here is a quick shell snippet that you can use to automatically calculate End Sector as well as creating the VMFS5 volume:

partedUtil mklabel ${DEVICE} msdos
END_SECTOR=$(eval expr $(partedUtil getptbl ${DEVICE} | tail -1 | awk ‘{print $1 » \* » $2 » \* » $3}’) — 1)
/sbin/partedUtil «setptbl» «${DEVICE}» «gpt» «1 2048 ${END_SECTOR} AA31E02A400F11DB9590000C2911D1B8 0»
/sbin/vmkfstools -C vmfs5 -b 1m -S $(hostname -s)-local-datastore ${DEVICE}:1

Note: You can also use the above to create a VMFS-based datastore running on a USB device, however that is not officially supported by VMware and performance with USB-based device will vary depending on the hardware and the speed of the USB connection. 

We can verify by running the «getptbl» option on the device that we formatted:

~ # /sbin/partedUtil «getptbl» «/vmfs/devices/disks/mpx.vmhba1:C0:T2:L0»
gpt
652 255 63 10485760
1 2048 10474379 AA31E02A400F11DB9590000C2911D1B8 vmfs 0

Finally we will now create the VMFS volume using our favorite vmkfstools, the syntax is the same as previous release of ESX(i):

~ # /sbin/vmkfstools -C vmfs5 -b 1m -S himalaya-SSD-storage-3 /vmfs/devices/disks/mpx.vmhba1:C0:T2:L0:1
Checking if remote hosts are using this device as a valid file system. This may take a few seconds…
Creating vmfs5 file system on «mpx.vmhba1:C0:T2:L0:1» with blockSize 1048576 and volume label «himalaya-SSD-storage-3».
Successfully created new volume: 4dfdb7b0-8c0dcdb5-e574-0050568f0111

Now you can refresh the vSphere Client or run vim-cmd hostsvc/datastore/refresh to view the new datastore that was created.