EDIT: Substantially updated 2019-02-16 to include additional troubleshooting information.
I’ve had iPXE and iSCSI environments in place for years now, but for the first time I’m attempting to do an iSCSI boot and the iPXE is having a problem with the conversation with the iSCSI target.
The storage server
Code:
CentOS Linux release 7.6.1810 (Core)
Linux san1srvp01.********.net 3.10.0-957.5.1.el7.x86_64 #1 SMP Fri Feb 1 14:54:57 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
zfs-0.7.12-1.el7_6.x86_64
The backing block instance
Code:
Disk /dev/zpool1/jane: 8422 MB, 8422687232 bytes, 16450561 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0xa9a554b4
Device Boot Start End Blocks Id System
/dev/zpool1/jane1 63 80324 40131 12 Compaq diagnostics
/dev/zpool1/jane2 * 80325 16434494 8177085 7 HPFS/NTFS/exFAT
The backing fileio instance
Code:
Disk /zpool1/nas1/Media/c0d0.img: 8422 MB, 8422686720 bytes, 16450560 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0xa9a554b4
Device Boot Start End Blocks Id System
/zpool1/nas1/Media/c0d0.img1 63 80324 40131 12 Compaq diagnostics
/zpool1/nas1/Media/c0d0.img2 * 80325 16434494 8177085 7 HPFS/NTFS/exFAT
Simplified iSCSI Target configuration. This is an example of block using ZFS zvol, and I’ve also tried fileio which behaves no differently.
Code:
{
"fabric_modules": [
{
"discovery_enable_auth": true,
"discovery_password": "********************************",
"discovery_userid": "san1srvp01",
"name": "iscsi"
}
],
"storage_objects": [
{
"alua_tpgs": [
{
"alua_access_state": 0,
"alua_access_status": 0,
"alua_access_type": 3,
"alua_support_active_nonoptimized": 1,
"alua_support_active_optimized": 1,
"alua_support_offline": 1,
"alua_support_standby": 1,
"alua_support_transitioning": 1,
"alua_support_unavailable": 1,
"alua_write_metadata": 0,
"implicit_trans_secs": 0,
"name": "default_tg_pt_gp",
"nonop_delay_msecs": 100,
"preferred": 0,
"tg_pt_gp_id": 0,
"trans_delay_msecs": 0
}
],
"attributes": {
"block_size": 512,
"emulate_3pc": 1,
"emulate_caw": 1,
"emulate_dpo": 1,
"emulate_fua_read": 1,
"emulate_fua_write": 1,
"emulate_model_alias": 1,
"emulate_rest_reord": 0,
"emulate_tas": 1,
"emulate_tpu": 0,
"emulate_tpws": 0,
"emulate_ua_intlck_ctrl": 0,
"emulate_write_cache": 0,
"enforce_pr_isids": 1,
"force_pr_aptpl": 0,
"is_nonrot": 1,
"max_unmap_block_desc_count": 1,
"max_unmap_lba_count": 262144,
"max_write_same_len": 65535,
"optimal_sectors": 32768,
"pi_prot_format": 0,
"pi_prot_type": 0,
"queue_depth": 128,
"unmap_granularity": 16,
"unmap_granularity_alignment": 0,
"unmap_zeroes_data": 0
},
"dev": "/dev/zpool1/jane",
"name": "jane",
"plugin": "block",
"readonly": false,
"write_back": false,
"wwn": "8688850f-7200-48a0-ad32-0f4f9397a836"
}
],
"targets": [
{
"fabric": "iscsi",
"tpgs": [
{
"attributes": {
"authentication": 0,
"cache_dynamic_acls": 0,
"default_cmdsn_depth": 64,
"default_erl": 0,
"demo_mode_discovery": 1,
"demo_mode_write_protect": 1,
"fabric_prot_type": 0,
"generate_node_acls": 0,
"login_timeout": 15,
"netif_timeout": 2,
"prod_mode_write_protect": 0,
"t10_pi": 0,
"tpg_enabled_sendtargets": 1
},
"enable": true,
"luns": [
{
"alias": "414d07d6b4",
"alua_tg_pt_gp_name": "default_tg_pt_gp",
"index": 2,
"storage_object": "/backstores/block/jane"
}
],
"node_acls": [
{
"attributes": {
"dataout_timeout": 3,
"dataout_timeout_retries": 5,
"default_erl": 0,
"nopin_response_timeout": 30,
"nopin_timeout": 15,
"random_datain_pdu_offsets": 0,
"random_datain_seq_offsets": 0,
"random_r2t_offsets": 0
},
"chap_mutual_password": "****************",
"chap_mutual_userid": "san1srvp01",
"chap_password": "****************",
"chap_userid": "jane",
"mapped_luns": [
{
"alias": "c8ce872be3",
"index": 2,
"tpg_lun": 2,
"write_protect": false
}
],
"node_wwn": "iqn.1999-10.net.********:jane"
}
],
"parameters": {
"AuthMethod": "CHAP,None",
"DataDigest": "CRC32C,None",
"DataPDUInOrder": "Yes",
"DataSequenceInOrder": "Yes",
"DefaultTime2Retain": "20",
"DefaultTime2Wait": "2",
"ErrorRecoveryLevel": "0",
"FirstBurstLength": "65536",
"HeaderDigest": "CRC32C,None",
"IFMarkInt": "2048~65535",
"IFMarker": "No",
"ImmediateData": "Yes",
"InitialR2T": "Yes",
"MaxBurstLength": "262144",
"MaxConnections": "1",
"MaxOutstandingR2T": "1",
"MaxRecvDataSegmentLength": "8192",
"MaxXmitDataSegmentLength": "262144",
"OFMarkInt": "2048~65535",
"OFMarker": "No",
"TargetAlias": "LIO Target"
},
"portals": [
{
"ip_address": "192.168.40.1",
"iser": false,
"offload": false,
"port": 3260
}
],
"tag": 1
}
],
"wwn": "iqn.1999-10.net.********:san1srvp01"
}
]
}
The PXE/iPXE/TFTP/HTTP server
Code:
CentOS release 6.10 (Final)
Linux sy1srvp01.********.net 2.6.32-754.10.1.el6.i686 #1 SMP Tue Jan 15 17:33:10 UTC 2019 i686 i686 i386 GNU/Linux
tftp-0.49-8.el6.i686
The iPXE implementation will first hand off to scripts matching host name, uuid, or mac in that order. This is the individual iPXE boot script for this mac mac-0007e90feaf5.ipxe
Code:
set username jane
set password ****************
set reverse-username san1srvp01
set reverse-password ****************
set initiator-iqn iqn.1999-10.net.********:jane
sanboot iscsi:192.168.40.1::::iqn.1999-10.net.********:san1srvp01
The initiator
Code:
Compaq ML370 (Generation 0)
BIOS P17 (12/18/2002)
Processor 866/133 Mhz with 256k Cache
RAM 1 GB
Intel Boot Agent GE v1.2.22
The PXE -> iPXE chain load
Code:
PXE 2.1 Build 084 (WfM 2.0), RPL V1.25
PX->EB: PXE! at 9CC2:0070, entry point at 9CC2:0106
UNDI code segment 9CC2:0000, data segment 969B:0000 (602-628kB)
UNDI device is PCI 00:06.0, type DIX+802.3
602kB free base memory after PXE unload
iPXE 1.0.0+ -- Open Source Network Boot Firmware -- http://ipxe.org
Features: DNS HTTP iSCSI TFTP AoE ELF MBOOT PXE bzImage Menu PXEXT
I’ve used a network trace to follow the iPXE sanboot iSCSI boot process. From a high level it is:
- Login Command (CHAP)
- Test Unit Ready
- Read Capacity(10)
- Read(10) <- Failure
First, the Read Capacity(10) return an unexpected value of 63 for LBA, instead of 16450560. It then attempts a Read(10) at LBA 64 which predictably fails with Logical Block Address Out Of Range. Testing indicates this is caused by specific interaction between iPXE and LIO, but the exact cause is unknown.
Read Capacity(10) — Request
Code:
Frame 27: 114 bytes on wire (912 bits), 114 bytes captured (912 bits)
Ethernet II, Src: Intel_0f:ea:f5 (00:07:e9:0f:ea:f5), Dst: SuperMic_6c:a9:82 (00:25:90:6c:a9:82)
Internet Protocol Version 4, Src: 192.168.4.13, Dst: 192.168.40.1
Transmission Control Protocol, Src Port: cifs (3020), Dst Port: iscsi-target (3260), Seq: 773, Ack: 637, Len: 48
iSCSI (SCSI Command)
Flags: 0xc1, F, R, Attr: Simple
SCSI CDB Read Capacity(10)
[LUN: 0x0000]
[Command Set:Direct Access Device (0x00) (Using default commandset)]
[Response in: 29]
Opcode: Read Capacity(10) (0x25)
Control: 0x00
Read Capacity(10) — Response
Code:
Frame 29: 74 bytes on wire (592 bits), 74 bytes captured (592 bits)
Ethernet II, Src: SuperMic_6c:a9:82 (00:25:90:6c:a9:82), Dst: Intel_0f:ea:f5 (00:07:e9:0f:ea:f5)
Internet Protocol Version 4, Src: 192.168.40.1, Dst: 192.168.4.13
Transmission Control Protocol, Src Port: iscsi-target (3260), Dst Port: cifs (3020), Seq: 685, Ack: 821, Len: 8
[2 Reassembled TCP Segments (56 bytes): #28(48), #29(8)]
iSCSI (SCSI Data In)
SCSI Payload (Read Capacity(10) Response Data)
[LUN: 0x0000]
[Command Set:Direct Access Device (0x00) (Using default commandset)]
[SBC Opcode: Read Capacity(10) (0x25)]
[Request in: 27]
[Response in: 29]
LBA: 63 (0 MB)
Block size in bytes: 512
SCSI Response (Read Capacity(10))
[LUN: 0x0000]
[Command Set:Direct Access Device (0x00) (Using default commandset)]
[SBC Opcode: Read Capacity(10) (0x25)]
[Request in: 27]
[Time from request: 0.000252000 seconds]
[Status: Good (0x00)]
Read(10) — Request
Code:
Frame 32: 114 bytes on wire (912 bits), 114 bytes captured (912 bits)
Ethernet II, Src: Intel_0f:ea:f5 (00:07:e9:0f:ea:f5), Dst: SuperMic_6c:a9:82 (00:25:90:6c:a9:82)
Internet Protocol Version 4, Src: 192.168.4.13, Dst: 192.168.40.1
Transmission Control Protocol, Src Port: cifs (3020), Dst Port: iscsi-target (3260), Seq: 821, Ack: 693, Len: 48
iSCSI (SCSI Command)
Flags: 0xc1, F, R, Attr: Simple
SCSI CDB Read(10)
[LUN: 0x0000]
[Command Set:Direct Access Device (0x00) (Using default commandset)]
[Response in: 33]
Opcode: Read(10) (0x28)
Flags: 0x00
Logical Block Address (LBA): 64
...0 0000 = Group: 0x00
Transfer Length: 4
Control: 0x00
Read(10) — Response
Code:
Frame 33: 214 bytes on wire (1712 bits), 214 bytes captured (1712 bits)
Ethernet II, Src: SuperMic_6c:a9:82 (00:25:90:6c:a9:82), Dst: Intel_0f:ea:f5 (00:07:e9:0f:ea:f5)
Internet Protocol Version 4, Src: 192.168.40.1, Dst: 192.168.4.13
Transmission Control Protocol, Src Port: iscsi-target (3260), Dst Port: cifs (3020), Seq: 693, Ack: 869, Len: 148
iSCSI (SCSI Response)
Flags: 0x80
SCSI: SNS Info
[LUN: 0x0000]
.111 0000 = SNS Error Type: Current Error (0x70)
Valid: 112
0... .... = Filemark: False
.0.. .... = EOM: False
..0. .... = ILI: False
.... 0101 = Sense Key: Illegal Request (0x5)
Sense Info: 0x00000000
Additional Sense Length: 10
Command-Specific Information: 00000000
Additional Sense Code+Qualifier: Logical Block Address Out Of Range (0x2100)
Field Replaceable Unit Code: 0x00
0... .... = SKSV: False
.000 0000 0000 0000 0000 0000 = Sense Key Specific: 0x000000
The iPXE response to the console
Code:
Could not open SAN device: Input/output error (http://ipxe.org/1d704039
Could not boot image: Input/output error (http://ipxe.org/1d704039
The message logged on the iSCSI target
Code:
Feb 13 09:17:41 san1srvp01 kernel: cmd exceeds last lba 64 (lba 64, sectors 4)
Testing and Troubleshooting
- Identical behavior is observed when attempting to boot a VM using this iSCSI LUN, ruling out the physical machine and its network card.
- The iSCSI device behaves correctly when mounted using the native Linux initiator, and the device was initially populated over an iSCSI mount using dd to copy the image file to it.
- I created a custom build of iPXE that forces Read Capacity(16) and Read(16), to no avail.
- I have found one documented instance of similar behavior which was determined to be caused by the operational parameters supplied (or not) during the Login stage, in that case. In response, I created a custom build with the same parameters and parameter values as those observed during a working mount using the native Linux initiator, to no avail.
- I have tried moving the fileio backing store image from ZFS to an xfs volume, to no avail.
- I have tried the iPXE sanboot with the block device initialized with zeros, to no avail. This suggests the problem is not related to partitions or block device content.
- Can anyone attest to having a working setup equivalent to this?
- Does anyone know of definite problems with this setup?
- Does anyone know what would cause LIO to behave in this way?
- And the hardest question of all, does anyone know how to fix it?
-TIA
EDIT: Substantially updated 2019-02-16 to include additional troubleshooting information.
I’ve had iPXE and iSCSI environments in place for years now, but for the first time I’m attempting to do an iSCSI boot and the iPXE is having a problem with the conversation with the iSCSI target.
The storage server
CentOS Linux release 7.6.1810 (Core)
Linux san1srvp01.********.net 3.10.0-957.5.1.el7.x86_64 #1 SMP Fri Feb 1 14:54:57 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
zfs-0.7.12-1.el7_6.x86_64
The backing block
instance
Disk /dev/zpool1/jane: 8422 MB, 8422687232 bytes, 16450561 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0xa9a554b4
Device Boot Start End Blocks Id System
/dev/zpool1/jane1 63 80324 40131 12 Compaq diagnostics
/dev/zpool1/jane2 * 80325 16434494 8177085 7 HPFS/NTFS/exFAT
The backing fileio
instance
Disk /zpool1/nas1/Media/c0d0.img: 8422 MB, 8422686720 bytes, 16450560 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0xa9a554b4
Device Boot Start End Blocks Id System
/zpool1/nas1/Media/c0d0.img1 63 80324 40131 12 Compaq diagnostics
/zpool1/nas1/Media/c0d0.img2 * 80325 16434494 8177085 7 HPFS/NTFS/exFAT
Simplified iSCSI Target configuration. This is an example of block using ZFS zvol, and I’ve also tried fileio which behaves no differently.
{
"fabric_modules": [
{
"discovery_enable_auth": true,
"discovery_password": "********************************",
"discovery_userid": "san1srvp01",
"name": "iscsi"
}
],
"storage_objects": [
{
"alua_tpgs": [
{
"alua_access_state": 0,
"alua_access_status": 0,
"alua_access_type": 3,
"alua_support_active_nonoptimized": 1,
"alua_support_active_optimized": 1,
"alua_support_offline": 1,
"alua_support_standby": 1,
"alua_support_transitioning": 1,
"alua_support_unavailable": 1,
"alua_write_metadata": 0,
"implicit_trans_secs": 0,
"name": "default_tg_pt_gp",
"nonop_delay_msecs": 100,
"preferred": 0,
"tg_pt_gp_id": 0,
"trans_delay_msecs": 0
}
],
"attributes": {
"block_size": 512,
"emulate_3pc": 1,
"emulate_caw": 1,
"emulate_dpo": 1,
"emulate_fua_read": 1,
"emulate_fua_write": 1,
"emulate_model_alias": 1,
"emulate_rest_reord": 0,
"emulate_tas": 1,
"emulate_tpu": 0,
"emulate_tpws": 0,
"emulate_ua_intlck_ctrl": 0,
"emulate_write_cache": 0,
"enforce_pr_isids": 1,
"force_pr_aptpl": 0,
"is_nonrot": 1,
"max_unmap_block_desc_count": 1,
"max_unmap_lba_count": 262144,
"max_write_same_len": 65535,
"optimal_sectors": 32768,
"pi_prot_format": 0,
"pi_prot_type": 0,
"queue_depth": 128,
"unmap_granularity": 16,
"unmap_granularity_alignment": 0,
"unmap_zeroes_data": 0
},
"dev": "/dev/zpool1/jane",
"name": "jane",
"plugin": "block",
"readonly": false,
"write_back": false,
"wwn": "8688850f-7200-48a0-ad32-0f4f9397a836"
}
],
"targets": [
{
"fabric": "iscsi",
"tpgs": [
{
"attributes": {
"authentication": 0,
"cache_dynamic_acls": 0,
"default_cmdsn_depth": 64,
"default_erl": 0,
"demo_mode_discovery": 1,
"demo_mode_write_protect": 1,
"fabric_prot_type": 0,
"generate_node_acls": 0,
"login_timeout": 15,
"netif_timeout": 2,
"prod_mode_write_protect": 0,
"t10_pi": 0,
"tpg_enabled_sendtargets": 1
},
"enable": true,
"luns": [
{
"alias": "414d07d6b4",
"alua_tg_pt_gp_name": "default_tg_pt_gp",
"index": 2,
"storage_object": "/backstores/block/jane"
}
],
"node_acls": [
{
"attributes": {
"dataout_timeout": 3,
"dataout_timeout_retries": 5,
"default_erl": 0,
"nopin_response_timeout": 30,
"nopin_timeout": 15,
"random_datain_pdu_offsets": 0,
"random_datain_seq_offsets": 0,
"random_r2t_offsets": 0
},
"chap_mutual_password": "****************",
"chap_mutual_userid": "san1srvp01",
"chap_password": "****************",
"chap_userid": "jane",
"mapped_luns": [
{
"alias": "c8ce872be3",
"index": 2,
"tpg_lun": 2,
"write_protect": false
}
],
"node_wwn": "iqn.1999-10.net.********:jane"
}
],
"parameters": {
"AuthMethod": "CHAP,None",
"DataDigest": "CRC32C,None",
"DataPDUInOrder": "Yes",
"DataSequenceInOrder": "Yes",
"DefaultTime2Retain": "20",
"DefaultTime2Wait": "2",
"ErrorRecoveryLevel": "0",
"FirstBurstLength": "65536",
"HeaderDigest": "CRC32C,None",
"IFMarkInt": "2048~65535",
"IFMarker": "No",
"ImmediateData": "Yes",
"InitialR2T": "Yes",
"MaxBurstLength": "262144",
"MaxConnections": "1",
"MaxOutstandingR2T": "1",
"MaxRecvDataSegmentLength": "8192",
"MaxXmitDataSegmentLength": "262144",
"OFMarkInt": "2048~65535",
"OFMarker": "No",
"TargetAlias": "LIO Target"
},
"portals": [
{
"ip_address": "192.168.40.1",
"iser": false,
"offload": false,
"port": 3260
}
],
"tag": 1
}
],
"wwn": "iqn.1999-10.net.********:san1srvp01"
}
]
}
The PXE/iPXE/TFTP/HTTP server
CentOS release 6.10 (Final)
Linux sy1srvp01.********.net 2.6.32-754.10.1.el6.i686 #1 SMP Tue Jan 15 17:33:10 UTC 2019 i686 i686 i386 GNU/Linux
tftp-0.49-8.el6.i686
The iPXE implementation will first hand off to scripts matching host name, uuid, or mac in that order. This is the individual iPXE boot script for this mac mac-0007e90feaf5.ipxe
set username jane
set password ****************
set reverse-username san1srvp01
set reverse-password ****************
set initiator-iqn iqn.1999-10.net.********:jane
sanboot iscsi:192.168.40.1::::iqn.1999-10.net.********:san1srvp01
The initiator
Compaq ML370 (Generation 0)
BIOS P17 (12/18/2002)
Processor 866/133 Mhz with 256k Cache
RAM 1 GB
Intel Boot Agent GE v1.2.22
The PXE -> iPXE chain load
PXE 2.1 Build 084 (WfM 2.0), RPL V1.25
PX->EB: PXE! at 9CC2:0070, entry point at 9CC2:0106
UNDI code segment 9CC2:0000, data segment 969B:0000 (602-628kB)
UNDI device is PCI 00:06.0, type DIX+802.3
602kB free base memory after PXE unload
iPXE 1.0.0+ -- Open Source Network Boot Firmware -- http://ipxe.org
Features: DNS HTTP iSCSI TFTP AoE ELF MBOOT PXE bzImage Menu PXEXT
I’ve used a network trace to follow the iPXE sanboot
iSCSI boot process. From a high level it is:
- Login Command (CHAP)
- Test Unit Ready
- Read Capacity(10)
- Read(10) <- Failure
First, the Read Capacity(10)
return an unexpected value of 63 for LBA, instead of 16450560. It then attempts a Read(10) at LBA 64 which predictably fails with Logical Block Address Out Of Range
. Testing indicates this is caused by specific interaction between iPXE and LIO, but the exact cause is unknown.
Read Capacity(10) — Request
Frame 27: 114 bytes on wire (912 bits), 114 bytes captured (912 bits)
Ethernet II, Src: Intel_0f:ea:f5 (00:07:e9:0f:ea:f5), Dst: SuperMic_6c:a9:82 (00:25:90:6c:a9:82)
Internet Protocol Version 4, Src: 192.168.4.13, Dst: 192.168.40.1
Transmission Control Protocol, Src Port: cifs (3020), Dst Port: iscsi-target (3260), Seq: 773, Ack: 637, Len: 48
iSCSI (SCSI Command)
Flags: 0xc1, F, R, Attr: Simple
SCSI CDB Read Capacity(10)
[LUN: 0x0000]
[Command Set:Direct Access Device (0x00) (Using default commandset)]
[Response in: 29]
Opcode: Read Capacity(10) (0x25)
Control: 0x00
Read Capacity(10) — Response
Frame 29: 74 bytes on wire (592 bits), 74 bytes captured (592 bits)
Ethernet II, Src: SuperMic_6c:a9:82 (00:25:90:6c:a9:82), Dst: Intel_0f:ea:f5 (00:07:e9:0f:ea:f5)
Internet Protocol Version 4, Src: 192.168.40.1, Dst: 192.168.4.13
Transmission Control Protocol, Src Port: iscsi-target (3260), Dst Port: cifs (3020), Seq: 685, Ack: 821, Len: 8
[2 Reassembled TCP Segments (56 bytes): #28(48), #29(8)]
iSCSI (SCSI Data In)
SCSI Payload (Read Capacity(10) Response Data)
[LUN: 0x0000]
[Command Set:Direct Access Device (0x00) (Using default commandset)]
[SBC Opcode: Read Capacity(10) (0x25)]
[Request in: 27]
[Response in: 29]
LBA: 63 (0 MB)
Block size in bytes: 512
SCSI Response (Read Capacity(10))
[LUN: 0x0000]
[Command Set:Direct Access Device (0x00) (Using default commandset)]
[SBC Opcode: Read Capacity(10) (0x25)]
[Request in: 27]
[Time from request: 0.000252000 seconds]
[Status: Good (0x00)]
Read(10) — Request
Frame 32: 114 bytes on wire (912 bits), 114 bytes captured (912 bits)
Ethernet II, Src: Intel_0f:ea:f5 (00:07:e9:0f:ea:f5), Dst: SuperMic_6c:a9:82 (00:25:90:6c:a9:82)
Internet Protocol Version 4, Src: 192.168.4.13, Dst: 192.168.40.1
Transmission Control Protocol, Src Port: cifs (3020), Dst Port: iscsi-target (3260), Seq: 821, Ack: 693, Len: 48
iSCSI (SCSI Command)
Flags: 0xc1, F, R, Attr: Simple
SCSI CDB Read(10)
[LUN: 0x0000]
[Command Set:Direct Access Device (0x00) (Using default commandset)]
[Response in: 33]
Opcode: Read(10) (0x28)
Flags: 0x00
Logical Block Address (LBA): 64
...0 0000 = Group: 0x00
Transfer Length: 4
Control: 0x00
Read(10) — Response
Frame 33: 214 bytes on wire (1712 bits), 214 bytes captured (1712 bits)
Ethernet II, Src: SuperMic_6c:a9:82 (00:25:90:6c:a9:82), Dst: Intel_0f:ea:f5 (00:07:e9:0f:ea:f5)
Internet Protocol Version 4, Src: 192.168.40.1, Dst: 192.168.4.13
Transmission Control Protocol, Src Port: iscsi-target (3260), Dst Port: cifs (3020), Seq: 693, Ack: 869, Len: 148
iSCSI (SCSI Response)
Flags: 0x80
SCSI: SNS Info
[LUN: 0x0000]
.111 0000 = SNS Error Type: Current Error (0x70)
Valid: 112
0... .... = Filemark: False
.0.. .... = EOM: False
..0. .... = ILI: False
.... 0101 = Sense Key: Illegal Request (0x5)
Sense Info: 0x00000000
Additional Sense Length: 10
Command-Specific Information: 00000000
Additional Sense Code+Qualifier: Logical Block Address Out Of Range (0x2100)
Field Replaceable Unit Code: 0x00
0... .... = SKSV: False
.000 0000 0000 0000 0000 0000 = Sense Key Specific: 0x000000
The iPXE response to the console
Could not open SAN device: Input/output error (http://ipxe.org/1d704039
Could not boot image: Input/output error (http://ipxe.org/1d704039
The message logged on the iSCSI target
Feb 13 09:17:41 san1srvp01 kernel: cmd exceeds last lba 64 (lba 64, sectors 4)
Testing and Troubleshooting
- Identical behavior is observed when attempting to boot a VM using
this iSCSI LUN, ruling out the physical machine and its network card. - The iSCSI device behaves correctly when mounted using the native
Linux initiator, and the device was initially populated over an iSCSI
mount using dd to copy the image file to it. - I created a custom build of iPXE that forces
Read Capacity(16)
andRead(16)
, to no avail. - I have found one documented instance of similar behavior which was
determined to be caused by the operational parameters supplied (or
not) during theLogin
stage, in that case. In response,
I created a custom build with the same parameters and parameter
values as those observed during a working mount using the native
Linux initiator, to no avail. - I have tried moving the fileio backing store image from ZFS to an xfs
volume, to no avail. - I have tried the iPXE sanboot with the block device initialized with
zeros, to no avail. This suggests the problem is not related to
partitions or block device content.
Questions
- Can anyone attest to having a working setup equivalent to this?
- Does anyone know of definite problems with this setup?
- Does anyone know what would cause LIO to behave in this way?
- And the hardest question of all, does anyone know how to fix it?
-TIA
I am the project lead for
Kiwi TCMS
and the current maintainer of pylint-django!
Some of the links contained within this site have my referral id (e.g.,
Amazon),
which provides me with a small commission for each sale. Thank you for your support.
CC-BY-SA &
MIT
2011-2018 ♦ Alexander Todorov
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.
Already on GitHub?
Sign in
to your account
Closed
magnetarnix opened this issue
Aug 5, 2022
· 13 comments
Comments
Hello, when trying to boot some AWS EC2 servers (at least on servers type c6a.large
, c6a.16xlarge
and c6i.32xlarge
) using the latest iPXE AMI I get the error:
iPXE 1.21.1+ (gd3c89) -- Open Source Network Boot Firmware -- https://ipxe.org
Features: DNS HTTP HTTPS iSCSI TFTP SRP AoE ELF MBOOT PXE bzImage Menu PXEXT
Amazon EC2 - iPXE boot via user-data
CPU: AuthenticAMD AMD EPYC 7R13 Processor
net0: 0e:3e:1b:32:0b:a9 using ena-vf on 0000:00:05.0 (Ethernet) [closed]
[Link:up, TX:0 TXE:0 RX:0 RXE:0]
DHCP attempt 1
Could not open net0: Input/output error (https://ipxe.org/1dc94039)
[Same error for 9 attempts]
DHCP failed - rebooting
I recompiled the latest version of master
with debug support to get more info (with make CONFIG=cloud EMBED=config/cloud/aws.ipxe DEBUG=pci:3,ena:3 bin/ipxe.usb
, then uploaded it using ./contrib/cloud/aws-import
) and I get:
iPXE initialising devices...0000:00:00.0 (8086:1237 class 060000) has no driver
0000:00:01.0 (8086:7000 class 060100) has no driver
0000:00:01.3 (8086:7113 class 000000) has no driver
0000:00:03.0 (1d0f:1111 class 030000) has no driver
0000:00:04.0 (1d0f:8061 class 010802) has no driver
0000:00:05.0 (1d0f:ec20) has driver "ena-vf"
0000:00:05.0 has mem febf8000 io 0 irq 0
0000:00:05.0 device not enabled by BIOS! Updating PCI command 0103->0107
0000:00:05.0 latency timer is unreasonably low at 0. Setting to 32.
ENA 0xec278 AQ [bff52400,bff52480) ACQ [bff52380,bff52400)
ENA 0xec278 admin request 0x0:
bff52400 : 00 00 08 01 00 00 00 00-00 00 00 00 00 00 00 00 : ................
bff52410 : 00 01 00 00 00 00 00 00-00 00 00 00 00 00 00 00 : ................
bff52420 : 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 : ................
bff52430 : 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 : ................
ENA 0xec278 admin response 0x0:
bff52380 : 00 00 00 01 00 00 00 00-01 00 00 00 0a 00 00 00 : ................
bff52390 : 86 58 20 14 01 00 00 00-30 00 00 00 40 00 00 00 : .X .....0...@...
bff523a0 : 0e 3e 1b 32 0b a9 00 00-00 24 00 00 00 00 00 00 : .>.2.....$......
bff523b0 : 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 : ................
ENA 0xec278 MAC 0e:3e:1b:32:0b:a9 MTU 9216
0000:00:1f.0 (1d0f:8061 class 010802) has no driver
ok
iPXE 1.21.1+ (gd3c89) -- Open Source Network Boot Firmware -- https://ipxe.org
Features: DNS HTTP HTTPS iSCSI TFTP SRP AoE ELF MBOOT PXE bzImage Menu PXEXT
Amazon EC2 - iPXE boot via user-data
CPU: AuthenticAMD AMD EPYC 7R13 Processor
net0: 0e:3e:1b:32:0b:a9 using ena-vf on 0000:00:05.0 (Ethernet) [closed]
[Link:up, TX:0 TXE:0 RX:0 RXE:0]
DHCP attempt 1
ENA 0xec278 admin request 0x1:
bff52440 : 01 00 03 01 00 02 10 00-00 00 00 00 00 30 f5 bf : .............0..
bff52450 : 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 : ................
bff52460 : 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 : ................
bff52470 : 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 : ................
ENA 0xec278 admin response 0x1:
bff523c0 : 01 00 06 01 00 00 00 00-00 00 00 00 00 00 00 00 : ................
bff523d0 : 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 : ................
bff523e0 : 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 : ................
bff523f0 : 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 : ................
ENA 0xec278 admin response 0x1 status 6:
bff52440 : 01 00 03 01 00 02 10 00-00 00 00 00 00 30 f5 bf : .............0..
bff52450 : 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 : ................
bff52460 : 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 : ................
bff52470 : 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 : ................
bff523c0 : 01 00 06 01 00 00 00 00-00 00 00 00 00 00 00 00 : ................
bff523d0 : 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 : ................
bff523e0 : 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 : ................
bff523f0 : 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 : ................
Could not open net0: Input/output error (https://ipxe.org/1dc94039)
net0: 0e:3e:1b:32:0b:a9 using ena-vf on 0000:00:05.0 (Ethernet) [closed]
[Link:up, TX:0 TXE:0 RX:0 RXE:0]
DHCP attempt 2
[...]
@mcb30 I see that you worked on the ENA driver, maybe you have some ideas of what could be going wrong here?
I am available to do more tests if needed, just let me know what you need.
@magnetarnix Thanks for the bug report. The NIC is being fairly unhelpful: the only information in the response is the error code 6, which is ENA_ADMIN_UNKNOWN_ERROR
according to the Linux kernel driver.
Is this error 100% reproducible on those instance types, or is it intermittent?
If 100% reproducible, you could try tweaking the request parameters constructed by ena_create_cq()
to see if the hardware can be be mollified with some alternative values. It’s unfortunately reasonably common with PCI virtual functions that newer versions of the PF driver will impose additional undocumented constraints on parameter values. Based on experience with e.g. the Intel 40Gig/100Gig family, you might want to try (separately):
- Increase
ENA_TX_COUNT
to a larger power of 2 - Set
req->vector = 1
to request a non-zero MSI-X vector - Set
req->intr = 0x20
to request that interrupts are enabled
Good luck!
Thanks for the quick answer! For those instance types I had this issue 100% of the time, so unless I was very unlucky it seems 100% reproducible.
Following your advice:
- Increasing
ENA_TX_COUNT
to 32: same issue - Increasing
ENA_TX_COUNT
to 64: same issue - In
ena_create_cq()
,req->create_cq.vector = 1;
: same issue - In
ena_create_cq()
,req->create_cq.intr = 0x20;
: same issue - In
ena_create_cq()
, bothreq->create_cq.vector = 1;
andreq->create_cq.intr = 0x20;
: same issue - Increasing
ENA_TX_COUNT
to 32 and bothreq->create_cq.vector = 1;
andreq->create_cq.intr = 0x20;
: same issue
Note than when I say «same issue», I compared the hex dumps to verify that the bytes were indeed modified for each request, but I still got the error message (which stayed exactly identical for each attempt).
Do you have any other idea I could try by chance?
Do you have any other idea I could try by chance?
I have an idea that is cheap to verify: Increase ENA_TX_COUNT
to 128. So eight times the original value. I’m not sure if the original value was 16, but the 32 and 64 made me think so.
Reason for eight times is the MTU of 9216 ( as seen in ENA 0xec278 MAC 0e:3e:1b:32:0b:a9 MTU 9216
) and a more normal MTU of 1500. 9216/1500 is about six.
And while writing down that idea, can another idea: MTU smaller as the maximum of 9216. Do know that I have no idea how to accomplise that. Reason for this message is that I could not resist to ignore the «any other idea I could try».
I have an idea that is cheap to verify: Increase
ENA_TX_COUNT
to 128. So eight times the original value. I’m not sure if the original value was 16, but the 32 and 64 made me think so.
Yes it was set to 16, I tried to increase it to 128 but same issue again! It was a good idea though!
For the MTU it seems like it queries it from the network adapter but doesn’t seem to try to set it at some point, but I might be mistaken.
Observation:
$ grep ^ENA /tmp/text_file_copy_and_paste_debug_attempt_from_issue_openings_message
ENA 0xec278 AQ [bff52400,bff52480) ACQ [bff52380,bff52400)
ENA 0xec278 admin request 0x0:
ENA 0xec278 admin response 0x0:
ENA 0xec278 MAC 0e:3e:1b:32:0b:a9 MTU 9216
ENA 0xec278 admin request 0x1:
ENA 0xec278 admin response 0x1:
ENA 0xec278 admin response 0x1 status 6:
$
Should that be read as admin request 0x1 got two responses?
Note than when I say «same issue», I compared the hex dumps to verify that the bytes were indeed modified for each request, but I still got the error message (which stayed exactly identical for each attempt).
Do you have any other idea I could try by chance?
Thanks for testing. I would probably next add some debug code to the Linux kernel driver to dump out the exact admin queue commands that it submits, and compare those against the (existing) admin queue command dumps from iPXE to see if there are any significant differences.
Should that be read as admin request 0x1 got two responses?
It only got 1 response, but if this response is an error, it prints back the request that triggered the erroneous response and the erroneous response again! So that’s why you see 2 admin response
, and the hex dump of the second one is twice the size, because it contains the request and the response concatenated. See
err: | |
DBGC_HDA ( ena, virt_to_phys ( req ), req, sizeof ( *req ) ); | |
DBGC_HDA ( ena, virt_to_phys ( *rsp ), *rsp, sizeof ( **rsp ) ); |
Thanks for testing. I would probably next add some debug code to the Linux kernel driver to dump out the exact admin queue commands that it submits, and compare those against the (existing) admin queue command dumps from iPXE to see if there are any significant differences.
As I was using iPXE only for a very basic setup, I found out a way to work around the limitation by not using the iPXE image but by having the server boot by default on a Linux distro with a working ENA driver, automatically download the required initrd
/bzImage
files, setup the server bootloader to boot from those downloaded files only for the next boot, and reboot the server. This way each time the server reboots it updates the files and reboots on them, simulating what iPXE does.
This is a «good enough» setup that works for me right now so I’ll stop investigating on my side for now until I have the use for more advanced iPXE features.
Thanks for reporting back.
right now so I’ll stop investigating
Please prevent that this issue keeps dangling open, close it.
I’d like to try to track down or help find a fix for this (I’m sure others will probably come across this issue as time goes by)…
Were there any other potential fixes you’d suggest (I see the possible addition of debug statements to the linux kernel driver mentioned above)?
I still think this is issue should be closed. Either by the original poster or by iPXE community members that have the privilege to close issues.
Other option I see that would help the iPXE project is that this issue gets an assignee.
Goal is getting us beyond «dangling issues are fine».
Regarding
I’d like to try to track down or help find a fix for this. Were there any other potential fixes you’d suggest?
https://ipxe.org/contact#developers
This should be fixed as of the latest series of commits:
stappers@myhost:~/src/mailinglists/ipxe
$ git log --oneline | head -n 5
a8012445 [ena] Increase receive ring size to 128 entries
3b81a4e2 [ena] Provide a host information page
9f81e97a [ena] Specify the unused completion queue MSI-X vector as 0xffffffff
6d2cead4 [ena] Allow for out-of-order completions
856ffe00 [ena] Limit submission queue fill level to completion queue size
stappers@myhost:~/src/mailinglists/ipxe
$ git log 856ffe00...6d2cead4
commit 6d2cead461db7243330f3275ff9ea7ff4607c4f8
Author: Michael Brown <mcb30@ipxe.org>
Date: Fri Aug 26 15:48:52 2022 +0100
[ena] Allow for out-of-order completions
The ENA data path design has separate submission and completion
queues. Submission queues must be refilled in strict order (since
there is only a single linear tail pointer used to communicate the
existence of new entries to the hardware), and completion queue
entries include a request identifier copied verbatim from the
submission queue entry. Once the submission queue doorbell has been
rung, software never again reads from the submission queue entry and
nothing ever needs to write back to the submission queue entry since
completions are reported via the separate completion queue.
This design allows the hardware to complete submission queue entries
out of order, provided that it internally caches at least as many
entries as it leaves gaps.
Record and identify I/O buffers by request identifier (using a
circular ring buffer of unique request identifiers), and remove the
assumption that submission queue entries will be completed in order.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
stappers@myhost:~/src/mailinglists/ipxe
$
That is nice to see. Thanks.
I can confirm that it is working on c6a
/c6i
servers, thanks a lot for the fix it’s much appreciated
РЕДАКТИРОВАТЬ: Существенно обновлено 16 февраля 2019 года, чтобы включить дополнительную информацию по устранению неполадок.
У меня уже много лет используются среды iPXE и iSCSI, но впервые я пытаюсь выполнить загрузку iSCSI, и у iPXE возникают проблемы с обменом данными с целью iSCSI.
Сервер хранения
CentOS Linux release 7.6.1810 (Core)
Linux san1srvp01.********.net 3.10.0-957.5.1.el7.x86_64 #1 SMP Fri Feb 1 14:54:57 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
zfs-0.7.12-1.el7_6.x86_64
Резервный блок
экземпляр
Disk /dev/zpool1/jane: 8422 MB, 8422687232 bytes, 16450561 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0xa9a554b4
Device Boot Start End Blocks Id System
/dev/zpool1/jane1 63 80324 40131 12 Compaq diagnostics
/dev/zpool1/jane2 * 80325 16434494 8177085 7 HPFS/NTFS/exFAT
Резервный fileio
экземпляр
Disk /zpool1/nas1/Media/c0d0.img: 8422 MB, 8422686720 bytes, 16450560 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0xa9a554b4
Device Boot Start End Blocks Id System
/zpool1/nas1/Media/c0d0.img1 63 80324 40131 12 Compaq diagnostics
/zpool1/nas1/Media/c0d0.img2 * 80325 16434494 8177085 7 HPFS/NTFS/exFAT
Упрощенная конфигурация iSCSI Target.Это пример блока с использованием ZFS zvol, и я также пробовал fileio, который ведет себя не иначе.
{
"fabric_modules": [
{
"discovery_enable_auth": true,
"discovery_password": "********************************",
"discovery_userid": "san1srvp01",
"name": "iscsi"
}
],
"storage_objects": [
{
"alua_tpgs": [
{
"alua_access_state": 0,
"alua_access_status": 0,
"alua_access_type": 3,
"alua_support_active_nonoptimized": 1,
"alua_support_active_optimized": 1,
"alua_support_offline": 1,
"alua_support_standby": 1,
"alua_support_transitioning": 1,
"alua_support_unavailable": 1,
"alua_write_metadata": 0,
"implicit_trans_secs": 0,
"name": "default_tg_pt_gp",
"nonop_delay_msecs": 100,
"preferred": 0,
"tg_pt_gp_id": 0,
"trans_delay_msecs": 0
}
],
"attributes": {
"block_size": 512,
"emulate_3pc": 1,
"emulate_caw": 1,
"emulate_dpo": 1,
"emulate_fua_read": 1,
"emulate_fua_write": 1,
"emulate_model_alias": 1,
"emulate_rest_reord": 0,
"emulate_tas": 1,
"emulate_tpu": 0,
"emulate_tpws": 0,
"emulate_ua_intlck_ctrl": 0,
"emulate_write_cache": 0,
"enforce_pr_isids": 1,
"force_pr_aptpl": 0,
"is_nonrot": 1,
"max_unmap_block_desc_count": 1,
"max_unmap_lba_count": 262144,
"max_write_same_len": 65535,
"optimal_sectors": 32768,
"pi_prot_format": 0,
"pi_prot_type": 0,
"queue_depth": 128,
"unmap_granularity": 16,
"unmap_granularity_alignment": 0,
"unmap_zeroes_data": 0
},
"dev": "/dev/zpool1/jane",
"name": "jane",
"plugin": "block",
"readonly": false,
"write_back": false,
"wwn": "8688850f-7200-48a0-ad32-0f4f9397a836"
}
],
"targets": [
{
"fabric": "iscsi",
"tpgs": [
{
"attributes": {
"authentication": 0,
"cache_dynamic_acls": 0,
"default_cmdsn_depth": 64,
"default_erl": 0,
"demo_mode_discovery": 1,
"demo_mode_write_protect": 1,
"fabric_prot_type": 0,
"generate_node_acls": 0,
"login_timeout": 15,
"netif_timeout": 2,
"prod_mode_write_protect": 0,
"t10_pi": 0,
"tpg_enabled_sendtargets": 1
},
"enable": true,
"luns": [
{
"alias": "414d07d6b4",
"alua_tg_pt_gp_name": "default_tg_pt_gp",
"index": 2,
"storage_object": "/backstores/block/jane"
}
],
"node_acls": [
{
"attributes": {
"dataout_timeout": 3,
"dataout_timeout_retries": 5,
"default_erl": 0,
"nopin_response_timeout": 30,
"nopin_timeout": 15,
"random_datain_pdu_offsets": 0,
"random_datain_seq_offsets": 0,
"random_r2t_offsets": 0
},
"chap_mutual_password": "****************",
"chap_mutual_userid": "san1srvp01",
"chap_password": "****************",
"chap_userid": "jane",
"mapped_luns": [
{
"alias": "c8ce872be3",
"index": 2,
"tpg_lun": 2,
"write_protect": false
}
],
"node_wwn": "iqn.1999-10.net.********:jane"
}
],
"parameters": {
"AuthMethod": "CHAP,None",
"DataDigest": "CRC32C,None",
"DataPDUInOrder": "Yes",
"DataSequenceInOrder": "Yes",
"DefaultTime2Retain": "20",
"DefaultTime2Wait": "2",
"ErrorRecoveryLevel": "0",
"FirstBurstLength": "65536",
"HeaderDigest": "CRC32C,None",
"IFMarkInt": "2048~65535",
"IFMarker": "No",
"ImmediateData": "Yes",
"InitialR2T": "Yes",
"MaxBurstLength": "262144",
"MaxConnections": "1",
"MaxOutstandingR2T": "1",
"MaxRecvDataSegmentLength": "8192",
"MaxXmitDataSegmentLength": "262144",
"OFMarkInt": "2048~65535",
"OFMarker": "No",
"TargetAlias": "LIO Target"
},
"portals": [
{
"ip_address": "192.168.40.1",
"iser": false,
"offload": false,
"port": 3260
}
],
"tag": 1
}
],
"wwn": "iqn.1999-10.net.********:san1srvp01"
}
]
}
Сервер PXE / iPXE / TFTP / HTTP
CentOS release 6.10 (Final)
Linux sy1srvp01.********.net 2.6.32-754.10.1.el6.i686 #1 SMP Tue Jan 15 17:33:10 UTC 2019 i686 i686 i386 GNU/Linux
tftp-0.49-8.el6.i686
Реализация iPXE сначала передаст скриптам, совпадающим с именем хоста, uuid или mac именно в таком порядке. Это отдельный сценарий загрузки iPXE для этого Mac mac-0007e90feaf5.ipxe
set username jane
set password ****************
set reverse-username san1srvp01
set reverse-password ****************
set initiator-iqn iqn.1999-10.net.********:jane
sanboot iscsi:192.168.40.1::::iqn.1999-10.net.********:san1srvp01
Инициатор
Compaq ML370 (Generation 0)
BIOS P17 (12/18/2002)
Processor 866/133 Mhz with 256k Cache
RAM 1 GB
Intel Boot Agent GE v1.2.22
Цепная загрузка PXE -> iPXE
PXE 2.1 Build 084 (WfM 2.0), RPL V1.25
PX->EB: PXE! at 9CC2:0070, entry point at 9CC2:0106
UNDI code segment 9CC2:0000, data segment 969B:0000 (602-628kB)
UNDI device is PCI 00:06.0, type DIX+802.3
602kB free base memory after PXE unload
iPXE 1.0.0+ -- Open Source Network Boot Firmware -- http://ipxe.org
Features: DNS HTTP iSCSI TFTP AoE ELF MBOOT PXE bzImage Menu PXEXT
Я использовал трассировку сети для отслеживания iPXE sanboot
Процесс загрузки iSCSI. На высоком уровне это:
- Команда входа в систему (CHAP)
- Тестовый блок готов
- Емкость чтения (10)
- Чтение (10) <- Ошибка
Во-первых, Емкость чтения (10)
возвращает неожиданное значение 63 для LBA вместо 16450560. Затем он пытается выполнить чтение (10) на LBA 64, что предсказуемо завершается неудачей с Адрес логического блока вне диапазона
. Тестирование показывает, что это вызвано конкретным взаимодействием между iPXE и LIO, но точная причина неизвестна.
Емкость чтения (10) — Запрос
Frame 27: 114 bytes on wire (912 bits), 114 bytes captured (912 bits)
Ethernet II, Src: Intel_0f:ea:f5 (00:07:e9:0f:ea:f5), Dst: SuperMic_6c:a9:82 (00:25:90:6c:a9:82)
Internet Protocol Version 4, Src: 192.168.4.13, Dst: 192.168.40.1
Transmission Control Protocol, Src Port: cifs (3020), Dst Port: iscsi-target (3260), Seq: 773, Ack: 637, Len: 48
iSCSI (SCSI Command)
Flags: 0xc1, F, R, Attr: Simple
SCSI CDB Read Capacity(10)
[LUN: 0x0000]
[Command Set:Direct Access Device (0x00) (Using default commandset)]
[Response in: 29]
Opcode: Read Capacity(10) (0x25)
Control: 0x00
Емкость чтения (10) — Ответ
Frame 29: 74 bytes on wire (592 bits), 74 bytes captured (592 bits)
Ethernet II, Src: SuperMic_6c:a9:82 (00:25:90:6c:a9:82), Dst: Intel_0f:ea:f5 (00:07:e9:0f:ea:f5)
Internet Protocol Version 4, Src: 192.168.40.1, Dst: 192.168.4.13
Transmission Control Protocol, Src Port: iscsi-target (3260), Dst Port: cifs (3020), Seq: 685, Ack: 821, Len: 8
[2 Reassembled TCP Segments (56 bytes): #28(48), #29(8)]
iSCSI (SCSI Data In)
SCSI Payload (Read Capacity(10) Response Data)
[LUN: 0x0000]
[Command Set:Direct Access Device (0x00) (Using default commandset)]
[SBC Opcode: Read Capacity(10) (0x25)]
[Request in: 27]
[Response in: 29]
LBA: 63 (0 MB)
Block size in bytes: 512
SCSI Response (Read Capacity(10))
[LUN: 0x0000]
[Command Set:Direct Access Device (0x00) (Using default commandset)]
[SBC Opcode: Read Capacity(10) (0x25)]
[Request in: 27]
[Time from request: 0.000252000 seconds]
[Status: Good (0x00)]
Чтение (10) — Запрос
Frame 32: 114 bytes on wire (912 bits), 114 bytes captured (912 bits)
Ethernet II, Src: Intel_0f:ea:f5 (00:07:e9:0f:ea:f5), Dst: SuperMic_6c:a9:82 (00:25:90:6c:a9:82)
Internet Protocol Version 4, Src: 192.168.4.13, Dst: 192.168.40.1
Transmission Control Protocol, Src Port: cifs (3020), Dst Port: iscsi-target (3260), Seq: 821, Ack: 693, Len: 48
iSCSI (SCSI Command)
Flags: 0xc1, F, R, Attr: Simple
SCSI CDB Read(10)
[LUN: 0x0000]
[Command Set:Direct Access Device (0x00) (Using default commandset)]
[Response in: 33]
Opcode: Read(10) (0x28)
Flags: 0x00
Logical Block Address (LBA): 64
...0 0000 = Group: 0x00
Transfer Length: 4
Control: 0x00
Чтение (10) — Ответ
Frame 33: 214 bytes on wire (1712 bits), 214 bytes captured (1712 bits)
Ethernet II, Src: SuperMic_6c:a9:82 (00:25:90:6c:a9:82), Dst: Intel_0f:ea:f5 (00:07:e9:0f:ea:f5)
Internet Protocol Version 4, Src: 192.168.40.1, Dst: 192.168.4.13
Transmission Control Protocol, Src Port: iscsi-target (3260), Dst Port: cifs (3020), Seq: 693, Ack: 869, Len: 148
iSCSI (SCSI Response)
Flags: 0x80
SCSI: SNS Info
[LUN: 0x0000]
.111 0000 = SNS Error Type: Current Error (0x70)
Valid: 112
0... .... = Filemark: False
.0.. .... = EOM: False
..0. .... = ILI: False
.... 0101 = Sense Key: Illegal Request (0x5)
Sense Info: 0x00000000
Additional Sense Length: 10
Command-Specific Information: 00000000
Additional Sense Code+Qualifier: Logical Block Address Out Of Range (0x2100)
Field Replaceable Unit Code: 0x00
0... .... = SKSV: False
.000 0000 0000 0000 0000 0000 = Sense Key Specific: 0x000000
Ответ iPXE на консоль
Could not open SAN device: Input/output error (http://ipxe.org/1d704039
Could not boot image: Input/output error (http://ipxe.org/1d704039
Сообщение, зарегистрированное на цели iSCSI
Feb 13 09:17:41 san1srvp01 kernel: cmd exceeds last lba 64 (lba 64, sectors 4)
Тестирование и устранение неполадок
- Идентичное поведение наблюдается при попытке загрузить виртуальную машину с использованием
этот iSCSI LUN, исключающий физическую машину и ее сетевую карту. - Устройство iSCSI работает правильно при подключении с использованием собственного
Инициатор Linux, и устройство изначально было заполнено через iSCSI
смонтируйте с помощью dd, чтобы скопировать в него файл образа. - Я создал специальную сборку iPXE, которая принудительно устанавливает
Read Capacity (16)
иRead (16)
, но безрезультатно. - Я нашел один задокументированный случай подобного поведения, которое было
определено, что это вызвано предоставленными эксплуатационными параметрами (или
не) в этом случае на этапеВход в систему
. В ответ,
Я создал кастомную сборку с такими же параметрами и параметрами
значения, наблюдаемые во время рабочего монтирования с использованием собственного
Инициатор Linux, безрезультатно. - Я попытался переместить образ резервного хранилища fileio из ZFS в xfs
том, но безрезультатно. - Я пробовал iPXE sanboot с блочным устройством, инициализированным с помощью
нули, безрезультатно. Это говорит о том, что проблема не связана с
разделы или блокировать содержимое устройства.
Вопросы
- Может ли кто-нибудь подтвердить наличие аналогичной рабочей установки?
- Кто-нибудь знает об определенных проблемах с этой установкой?
- Кто-нибудь знает, что может привести к поведению LIO таким образом?
- И самый сложный вопрос: кто-нибудь знает, как это исправить?
-TIA