Nmi iock error

Environment

• Red Hat Enterprise Linux 6
• Red Hat Enterprise Linux 5

Issue

• The following message is seen on the console

        NMI: IOCK error (debug interrupt?)
        CPU 0 
        Modules linked in: ipt_MASQUERADE iptable_nat ip_nat xt_state ip_conntrack nfnetlink ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables bridge mptctl mptbase bonding be2iscsi ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp bnx2i cnic ipv6 xfrm_nalgo crypto_api uio cxgb3i cxgb3 8021q libiscsi_tcp libiscsi2 scsi_transport_iscsi2 scsi_transport_iscsi dm_round_robin dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec i2c_core dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport joydev sr_mod cdrom hpilo bnx2 serio_raw shpchp pcspkr sg dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod usb_storage qla2xxx scsi_transport_fc ata_piix libata cciss sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
        Pid: 0, comm: swapper Not tainted 2.6.18-194.17.4.el5 #1
        RIP: 0010:[<ffffffff8019d550>]  [<ffffffff8019d550>] acpi_processor_idle_simple+0x14c/0x30e
        RSP: 0018:ffffffff803fbf58  EFLAGS: 00000046
        RAX: 0000000000d4d87e RBX: ffff81061e10a160 RCX: 0000000000000908
        RDX: 0000000000000915 RSI: 0000000000000003 RDI: 0000000000000000
        RBP: 0000000000d4d87e R08: ffffffff803fa000 R09: 0000000000000039
        R10: ffff810001005710 R11: 0000000000000000 R12: 0000000000000000
        R13: ffff81061e10a000 R14: 0000000000000000 R15: 0000000000000000
        FS:  0000000000000000(0000) GS:ffffffff803ca000(0000) knlGS:0000000000000000
        CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
        CR2: 0000000009013954 CR3: 000000060799d000 CR4: 00000000000006e0
        Process swapper (pid: 0, threadinfo ffffffff803fa000, task ffffffff80308b60)
        Stack:  ffff81061e10a000 ffffffff8019d404 0000000000000000 ffffffff8019d404
         0000000000090000 0000000000000000 0000000000000000 ffffffff8004923a
         0000000000200800 ffffffff80405807 0000000000090000 0000000000000000
        Call Trace:
         [<ffffffff8019d404>] acpi_processor_idle_simple+0x0/0x30e
         [<ffffffff8019d404>] acpi_processor_idle_simple+0x0/0x30e
         [<ffffffff8004923a>] cpu_idle+0x95/0xb8
         [<ffffffff80405807>] start_kernel+0x220/0x225
         [<ffffffff8040522f>] _sinittext+0x22f/0x236

        Code: 89 ca ed ed 41 89 c4 41 8a 45 1c 83 e0 30 3c 30 75 15 f0 ff

Resolution

• Use vendor hardware diagnostics software to analyse system health.

• Contact the hardware manufacturer for further assistance.

• Under RHEL6, the kernel.panic_on_io_nmi = 1 sysctl can be set to have the system panic when an I/O NMI is received. Refer to How to set sysctl variables on Red Hat Enterprise Linux? for details.

Root Cause

• Under x86_64, I/O port 0x61 (System Control Port B) holds the reason of an NMI (NonMaskable Interrupt).

• For a detailed description of NMIs, refer e.g. to the Intel® 64 and IA-32 Architectures Software Developer’s Manual Volume 3A: System Programming Guide, Part 1.

• An NMI is a hardware-driven interrupt and it is usually reserved to report serious hardware errors, to a CPU.

File: include/asm-x86_64/nmi.h

    #define get_nmi_reason() inb(0x61)

• In the event of an NMI it is possible to consult the aforementioned port to obtain an indication of what caused the error.

• In the default_do_nmi() function we see why io_check_error() is called and consequently why the NMI: IOCK error (debug interrupt?) message is displayed on the console:

File: arch/x86_64/kernel/traps.c

    asmlinkage __kprobes void default_do_nmi(struct pt_regs *regs)
    {
        unsigned char reason = 0; 
        int cpu; 

        cpu = smp_processor_id();
        ...

        if (!cpu)
            reason = get_nmi_reason();

        ...
        if (reason & 0x40)
            io_check_error(reason, regs);
    }

    static __kprobes void
    io_check_error(unsigned char reason, struct pt_regs * regs)
    {
        printk("NMI: IOCK error (debug interrupt?)n");
        show_registers(regs);
        ...

• As illustrated above, when the 6-th bit (1 << 6 == 0x40) is set, it reports that an I/O Channel Check error (IOCHK) has occurred.
  • Parity and uncorrectable hardware errors are examples of why an IOCHK error could be raised.
  • Most hardware errors should however be reported through the MCE (Machine Check Exception) mechanism. An MCE indicates that the CPU detected an internal machine error or a bus error, or that an external agent detected a bus error. Normally the hardware manufacturer will be able to provide further details.

Diagnostic Steps

• Check dmesg and lspci output to figure out the cause of the NMI