Communications error with the storage controller

HPE MSA 2040 Unresponsive, Dead, or Failed Controller, Controller won’t boot

This weekend I came across a big issue with my HPE MSA 2040 where one of the SAN controllers became unresponsive, and appeared to had failed because it would not boot.

It all started when I decided to clean the MSA SAN. I try to clean the components once or twice a year to remove dust and make sure it’s not getting all jammed up. Sometimes I’ll shut the entire unit down and remove the individual components, other times I’ll remove them while operating. Because of the redundancies and since I have two controllers, I can remove and clean each controller individually at separate times.

Please Note: When dusting equipment with fans, never allow the fans to spin up with compressed air as this can generate current which can damage components. Never allow compressed air flow to spin up fans.

After cleaning out the power supplies, it came time to clean the controllers.

The Problem

As always, I logged in to the SMU to shutdown controller A (storage). I shut it down, the blue LED illuminated it was safe for removal. I then proceeded to remove it, clean it, and re-insert it. The controller came back online, and ownership of the applicable disk groups were successfully moved back. Controller A was now completed successfully. I continued to do the same for controller B: I logged in to shutdown controller B (storage). It shut down just like controller A, the blue LED removable light illuminated. I was able to remove it, clean it, and re-insert it.

However, controller B did not come back online.

After inserting controller B, the status light was flashing (as if it was booting). I waited 20 minutes with no change. The SMU on controller B was responding to HTTPS requests, however you could not log on due to the error “system is initializing”. SSH was functioning and you could log in and issue commands, however any command to get information would return “Please wait while this information is pulled from the MC controller”, and ultimately fail. The SMU on controller A would report a controller fault on controller B, and not provide any other information (including port status on controller B).

I then tried to re-seat the controller with the array still running. Gave it plenty of time with no effect.

I then removed the failed controller, shutdown the unit, powered it back on (only with controller A), and re-inserted Controller B. Again, no effect.

The Fix

At this point I’m thinking the controller may have failed or died during the cleaning process. I was just about to call HPE support for a replacement until I noticed the “Power LED” light inside of the failed controller would flash every 5 seconds while removed.

This made me start to wonder if there was an issue writing the cache to the compact flash card, or if the controller was still running off battery power but had completely frozen.

I tried these 3 things on the failed controller while it was unplugged and removed:

1. I left the controller untouched for 1 hour out of the array (to maybe let it finish whatever it was doing while on battery power)
2. There’s an unlabeled button on the back of the controller. As a last resort (thinking it was a reset button), I pressed and held it for 20 seconds, waited a minute, then briefly pressed it for 1 second while it was out of the unit.
3. I removed the Compact Flash card from the controller for 1 minute, then re-inserted it. In hoping this would fail the cache copy if it was stuck in the process of writing cache to compact flash.

I then re-inserted the controller, and it booted fine! It was not functioning and working (and came up very fast). Looking at the logs, it has no record of what occurred between the first shutdown, and final boot. I hope this post helps someone else with the same issue, it can save you a support ticket, and time with a controller down.

Disclaimer

PLEASE NOTE: I could not find any information on the unlabeled button on the controller, and it’s hard to know exactly what it does. Perform this at your own risk (make sure you have a backup). Since I have 2 controllers, and my MSA 2040 was running fine on Controller A, I felt comfortable doing this, as if this did reset controller B, the configuration would replicate back from controller A. I would not do this in a single controller environment.

Update – 24 Hours later

After I got everything up and running, I checked the logs of the unit and couldn’t find anything on controller B that looked out of ordinary. However, 24 hours later, I logged back in and noticed some new events showed up from the day before (from the day I had the issues):

You’ll notice the event log with severity error:

One thing that’s very odd is that I know for a fact the time is wrong on the error severity log entry, this could be due to the fact we had a daylight savings time change last night at midnight. Either way it appears that it finally did detect that the Storage controller was in an error state and logged it, but it still would have been nice for some more information.

On a final note, the unit has been running perfectly for over 24 hours.

Update – April 2nd 2019

Well, in March a new firmware update was released for the MSA. I went to upgrade and the same issue as above occurred. During the firmware upgrade, at one point of the firmware update process a step had failed and repeated 4 times until successful.

The firmware update log (below was repeated):

During the Storage Controller restarting process, the controller never came back up. I removed the controller 1 hour, re-inserted and the above fix did not work. I then tried it after 2 hours of disconnection.

At this point I contacted HPE, who is sending a replacement controller.

The following day (12 hours of controller removed), I re-inserted it again and it actually booted up, was working with the new firmware, and then did a PFU (Partner Firmware Update) of controller A.

While it is working now, I’m still going to replace the controller as I believe something is not functioning correctly.

18 Responses to “HPE MSA 2040 Unresponsive, Dead, or Failed Controller, Controller won’t boot”

Thank You for the great post it saved my life 🙂
I had exactly the same issue. It has begun with a host port IP setting.
After a few minutes controller B showed up the CopactFlash fault.
First i tryed to reboot the controller and aftert this step instant controller failure.
After it i unplugged the controller ad did the reset via your advice and a miracle is happened 🙂

Glad to hear it helped! Always be careful though, doing this will corrupt or clear the flash and compact flash on the controller… Only do this if the other controller is valid, responding, functioning properly, and has an OK status.

Dear Steve,
We are migrating from Netapp to a HPE MSA 2050. I’ve just started to work with it and I got the error message. I have no experience with the HPE, but it looks fairly straight forward in installing, much better then the Netapp. I will be visiting your blog in the next couple of weeks whilst setting up the HPE. 😉
By the way is there a big difference between the 2040 and 2050?

Have a great day,
André

Glad to hear, the MSA storage arrays are beautiful products!

There is a few differences, the main being that the 205x series are the next generation agyer the 204x series.

Hi Stephen! My name is Alex, i got some problem with my MSA, I hope You can give me good advice. So, i Have two MC on HP MSA, each of MC connected via FibreChannel modules to HP Proliant Servers throught SAN switch. And there are dozen of VMs on servers. I need to replace MC B, without stopping (power off) whole MSA. How do You think, is it possible without data loss?

As long as you configured the unit properly, and have the proper channels of redundancy (which you likely do, if the controller has failed and everything is still working), you just simply follow the standard procedures for swapping the controller.

No data loss should occur if the unit was configured properly, and if you follow standard procedures.

Stephen, thanx for fast answer, yeah, looks like everything configured good, but i have to be sure. Well, i do not really know, witch MC is bad, and a problem i have – a can’t see any mappings via web-interface or CLI, but DG, volumes and other stuff i can see clear. So, HP sent me new MC, i whatnt to make hot replace, but data in MSA very important…

You need to identify which controller is bad first.

Have you logged on to the SMU on both controllers? One should have good information.

No, i cant see any mappings via A and via B MC. And in CLI – too, but “show volume-maps” – still works. And other one symptom – web-interface often unavailable, just disconnecting.

Have you tried to restart one of the management controllers (not the storage controller)?

What did HPe advise?

Are there any amber lights on the back of the storage controllers?

There is no any amber lights. All HP recommendations i made at all (restart of MSA, FW upgrade and downgrade, etc,) – no result. They sent me new MC and wants me to replace it, but i cant stop MSA in september…

Did you restart the Storage Controllers, or Management Controllers?

The storage controllers handle the actual storage, whereas the management controllers provide the CLI and SMU.

Sometimes the management controllers can crash, but everything still works. In this case it would require a restart of only the management controllers.

However, if you’ve restarted the SC’s, the MCs, and the entire SAN with no effect, I’m wondering if there could be an issue with the midplane or something.

When you log on to vSphere, are all paths active? If some paths are offline, you can use this to determine which storage controller may be problematic. In this case, if there are paths offline, I’d replace the storage controller that has the failed paths.

Yes, Stephen, i restarted MCs and SCs, and all the paths online…HPe 2 and 3 level support do not have descision still. As for me – some issue, that common for MCs: Compact Flash card trouble, Firmware trouble…Maybe the fact important that I use virtual volumes.

I have HP MSA 2040 storage which we are going to repurpose. The device is stacked but I am unable to take its serial and it always says COM4/7 doesn’t exists. So, I reset the device and in next attempt I was able to connect through serial port and when i tried to reset it to default settings it started giving me below error.

(none) login: restoredefaults
Password:

2019-09-09 14:59:05: Running MC-only restoredefaults on partner controller.
2019-09-09 14:59:05: Running MC-only restoredefaults on this controller.
2019-09-09 14:59:05: Perform Restore Defaults
/app/default/restoreDefaults.sh: source: line 96: can’t open ‘/mem/relinfo/features.env’

So I dont know its password and i cant reset it as well. Any idea how to fix it?

Can you try reflashing the firmware using the restoredefaults account? I’m not familiar with that account/method, but I’m curious if it’s possible.

Do you have active warranty? I’d check the serial number, and if it has warranty, try submitting a ticket with HPe.

I have HP MSA 2040 storage with two controllers and with D2700 disk enclosure with two controllers.
Now SMU periodically reports warning (The algorithm for best-path routing selected the alternate path to the indicated disk because the I/O error count on the primary path reached its threshold.) and then corresponding disk becomes degraded. I have blinking amber light 2 on controller B of D2700 which indicates SAS port error detected by application client). Now I already have 8 degraded disks of 12 disks on controller storage B of D2700.
How do you think, can I try to restart the storage controller B without any loss of data to fix this issue?

First and foremost, make sure you have an active valid backup of the data.

I’d recommend contacting HPe and submitting a support ticket. It sounds like either a disk, backplane, D2700 controller, or SAS cable might not be functioning.

As for restarting controllers, if the path is failing on that controller, in a dual controller configuration you can simply restart it. However if it’s using that controller, I wouldn’t recommend doing that.

If you’re not going to call HPe support, I might actually recommend restarting the entire storage system properly and see what happens. But make sure you have a backup first as this could make the situation worse.

Источник

Communication error with the storage controller

Yesterday I had an issue with my VMWare farm and my SAN. Im using an HP MSA1000 on a 3 system farm. The farm stopped working altogether and restarting the MSA resolved the issue. In virtual center I brought up the logs and found this entry.

Feb 14 00:15:44 jgssesx01 vobd: Feb 14 00:15:44.125:

Feb 14 00:15:52 jgssesx01 vobd: Feb 14 00:15:52.108:

This repeats tiself about every 15-20 seconds. Is this just an indicator that the storage device is unreachable? I have 2 fiber switch in it for redundancy and find it hard to believe that both switches failed.

Message was edited by: tom howarth to remove the MS Word XML formating

• Mark as New
• Bookmark
• Subscribe
• Mute
• Subscribe to RSS Feed
• Permalink
• Print
• Report Inappropriate Content

Please can you give some infomation as to the version of ESX you are running for your Hosts and the version of vCenter.

Further what firmware is your MSA running, etc

If you found this or any other answer useful please consider the use of the Helpful or correct buttons to award points

Tom Howarth VCP / vExpert

VMware Communities User Moderator

Contributing author on «[VMware vSphere and Virtual Infrastructure Security: Securing ESX and the Virtual Environment|http://www.amazon.co.uk/VMware-VSphere-Virtual-Infrastructure-Security/dp/0137158009/ref=sr_1_1?ie=UTF8&s=books&qid=1256146240&sr=1-1]”.

• Mark as New
• Bookmark
• Subscribe
• Mute
• Subscribe to RSS Feed
• Permalink
• Print
• Report Inappropriate Content

vSphere v4.0.0 build 162856

vCenter v4.0.0 build 162856

I dont have the firmware version for the MSA but the firmware was updated prior to implementation so it shouldn’t be too old if it’s not current.

• Mark as New
• Bookmark
• Subscribe
• Mute
• Subscribe to RSS Feed
• Permalink
• Print
• Report Inappropriate Content

I am moving your post to the dedicated vSphere forums as you will receive a better response

If you found this or any other answer useful please consider the use of the Helpful or correct buttons to award points

Tom Howarth VCP / vExpert

VMware Communities User Moderator

Contributing author on «[VMware vSphere and Virtual Infrastructure Security: Securing ESX and the Virtual Environment|http://www.amazon.co.uk/VMware-VSphere-Virtual-Infrastructure-Security/dp/0137158009/ref=sr_1_1?ie=UTF8&s=books&qid=1256146240&sr=1-1]”.

• Mark as New
• Bookmark
• Subscribe
• Mute
• Subscribe to RSS Feed
• Permalink
• Print
• Report Inappropriate Content

There is a very, very long list of people that have had similar issues with MSA1000/MSA1500 dropping off the planet after extended periods of uptime. This is known to happen on all versions of VMware ESX. I recall having the problem way back on 2.5.0. and also now after I’ve upgraded to 4.0 in late 2009.

It seems to depend on the load generated and is probably related to the number of SCSI reservations, so snapshots, plenty of VM shutdown/startup events or VM deployments would agravate the problem. Some people report as short as a week, while others (including myself) get to run for about 3 months before a crash occurs.

There are workarounds available, such as rebooting the controllers via the serial interface every few days/weeks, but they’re clunky. We’re not that mission critical, so if we remember, we reboot the MSA about every three months, otherwise we just restart it when it crashes.

You can get some excellent details at Eric Miller’s site on http://www.msa1500cs.com on how to work around the problem and address performance and stability in general.

I’ve just noticed that HP finally posted the v7.20 firmware for public download on their web site.

One of the fixes is supposedly:

«Resolved issue of MSA persistent reservation table becoming full when used with various supported levels of VMware.»

Maybe that is (finally) the silver bullet that will resolve the «timebomb» facility the array has. I will be applying it in the next week or two.

Источник

Модераторы: Trinity admin`s, Free-lance moderator`s

FLASH LOADER v12.015 Dec  4 2008 11:32:25

MANAGEMENT CONTROLLER

<Hold down the spacebar for Loader Menu>

Loading controller...

Use new code in primary.
...copy primary...decomp [lzma]...exit loader...


Starting VxWorks ...
initElanRegs Done...
Dumping chipset info...
REVID      0xFFFEF000  0x11
CPUCTL     0xFFFEF002  0x1
HBCTL      0xFFFEF060  0x500
SYSARBCTL  0xFFFEF070  0x2
SYARBMENB  0xFFFEF072  0x8
ARBPRICTL  0xFFFEF074  0x40000F03
PIOFS L    0xFFFEFC20  0x2004
PIOFS H    0xFFFEFC22  0x14
PIO DIR L  0xFFFEFC2A  0x2B9
PIO DIR H  0xFFFEFC2C  0x0
PIO DAT L  0xFFFEFC30  0xEE3F
PIO DAT H  0xFFFEFC32  0xFFFB
Chipset info done.
MC CODE: W441R57 Oct  2 2013 15:31:51



Hold down Spacebar key to enter diags...

Entering POST...
Checking PCI bus integrity...Test passed.
IP  = 010.100.001.201
MAC = 00:c0:ff:10:70:e7
POST Done.
COM 2

-----------------------------------------
Copyright (c) 2006
BUILD TIME: Oct  2 2013  15:31:45
-----------------------------------------
IP Address = 010.100.001.201
IP Mask    = 255.255.255.000
Gateway    = 010.100.001.249
MAC        = 00:c0:ff:10:70:e7
HP StorageWorks MSA2312sa
System Name:
System Location:
Version: W441R57
#

Loading controller...

Use new code in primary.
...copy primary...decomp [lzma]...exit loader...


Starting VxWorks ...
initElanRegs Done...
Dumping chipset info...
REVID      0xFFFEF000  0x11
CPUCTL     0xFFFEF002  0x1
HBCTL      0xFFFEF060  0x500
SYSARBCTL  0xFFFEF070  0x2
SYARBMENB  0xFFFEF072  0x8
ARBPRICTL  0xFFFEF074  0x40000F03
PIOFS L    0xFFFEFC20  0x2004
PIOFS H    0xFFFEFC22  0x14
PIO DIR L  0xFFFEFC2A  0x2B9
PIO DIR H  0xFFFEFC2C  0x0
PIO DAT L  0xFFFEFC30  0xFE3F
PIO DAT H  0xFFFEFC32  0xFFF9
Chipset info done.
MC CODE: W441P28 Oct 19 2010 13:17:12



Hold down Spacebar key to enter diags...

Entering POST...
Checking PCI bus integrity...Test passed.
IP  = 010.000.000.001
MAC = 00:50:13:00:00:aa
POST Done.
COM 2

-----------------------------------------
Copyright (c) 2006
BUILD TIME: Oct 19 2010  13:17:02
-----------------------------------------
IP Address = 010.000.000.001
IP Mask    = 255.255.255.000
Gateway    = 000.000.000.000
MAC        = 00:50:13:00:00:aa
HP StorageWorks MSA2312sa
System Name:
System Location:
Version: W441P28

# show controller
controller-date     controllers
# show controllers
Cannot obtain data lock. Will continue to wait
Cannot obtain data lock. Will continue to wait
Controllers
-----------
Controller ID: A
Serial Number: N/A
Hardware Version:
CPLD Version:
MAC Address: 00:00:00:00:00:00
WWNN: 0000000000000000
IP Address: No_data
IP Subnet Mask: No_data
IP Gateway: No_data
Disks: 0
Vdisks: 0
Cache Memory Size (MB): 0
Host Ports: 0
Disk Channels: 0
Disk Bus Type: Unknown
Status: Running
Failed Over: No
Fail Over Reason: Not applicable

Controller ID: B
Serial Number: N/A
Hardware Version:
CPLD Version:
MAC Address: 00:00:00:00:00:00
WWNN: 0000000000000000
IP Address: No_data
IP Subnet Mask: No_data
IP Gateway: No_data
Disks: 0
Vdisks: 0
Cache Memory Size (MB): 0
Host Ports: 0
Disk Channels: 0
Disk Bus Type: Unknown
Status: Running
Failed Over: No
Fail Over Reason: Not applicable

Error: Communications Error with Storage Controller (-2194) - Could not read SAS info, so skipping enclosure 0.

Error: Communications Error with Storage Controller (-2194) - Could not read SAS info, so skipping enclosure 0.
#
#
# show controllers
Cannot obtain data lock. Will continue to wait
Controllers
-----------
Controller ID: A
Serial Number: N/A
Hardware Version:
CPLD Version:
MAC Address: 00:00:00:00:00:00
WWNN: 0000000000000000
IP Address: No_data
IP Subnet Mask: No_data
IP Gateway: No_data
Disks: 0
Vdisks: 0
Cache Memory Size (MB): 0
Host Ports: 0
Disk Channels: 0
Disk Bus Type: Unknown
Status: Running
Failed Over: No
Fail Over Reason: Not applicable

Controller ID: B
Serial Number: N/A
Hardware Version:
CPLD Version:
MAC Address: 00:00:00:00:00:00
WWNN: 0000000000000000
IP Address: No_data
IP Subnet Mask: No_data
IP Gateway: No_data
Disks: 0
Vdisks: 0
Cache Memory Size (MB): 0
Host Ports: 0
Disk Channels: 0
Disk Bus Type: Unknown
Status: Running
Failed Over: No
Fail Over Reason: Not applicable

Error: Communications Error with Storage Controller (-2194) - Could not read SASPress any key to continue (Q to qu info, so skipping enclosure 0.

Error: Communications Error with Storage Controller (-2194) - Could not read SASPress any key to continue (Q to qu info, so skipping enclosure 0.
#
#
#
#
#
# show system
System Information
------------------
System Name:
System Contact:
System Location:
System Information:
Vendor Name: HP StorageWorks
Product ID: MSA2312sa
Product Brand: MSA Storage
SCSI Vendor ID: HP
Enclosure Count: 0
Health: OK
Supported Locales: English (English), Spanish (español), French (français), German (Deutsch), Italian (italiano), Japanese (日本語), Dutch (Nederlands), Chinese-simplified (简体中文), Chinese-traditional (繁體中文), Korean (한국어)
#

# show debug-log-parameters
Cannot obtain data lock. Will continue to wait
Error: Communications Error with Storage Controller (-2194)
#

In this post we are going to look at basic troubleshooting steps a System Administrator can carry out to identify and potentially resolve a faulty storage controller in a HP MSA 2040.

This post is going to focus more on dual controller setups, with both a Controller A and a Controller B. In my case the faulty controller was brought to my attention by the health ember light on the front of the SAN and using the methods displayed in this article. I was able to gather information and ultimately bring the controller back up.

First step is to log into the SMU (Storage Management Utility) –

Using the SMU you can use the event logs as a first step to identifying the error/problem

In my case I was getting the error below:

Error: Critical Error Fault Type: NMI p1: 0x037454E, p2: 0x0000000 ….. No Cur Thread

Overall there wasn’t a lot of information to go off aside from confirmation that a serious error had occurred on Controller A and as a result Controller B had initiated the failover process.

Controller A was still powered on and ping able, plus I could get to it’s SMU. However, when I tried to log in it would either give an unsuccessful login or state that the controller was ‘initializing’.

Get more information using CLI –

A good way to gather more information is to telnet or ssh into the working controller’s command line interface (CLI). An easy method to do this is to use a piece of software called Putty.

Just enter the controllers IP and press Open

When prompted put in your credentials (the one you would use to log into the SMU)

If a putty connection over your network isn’t working, try the below method to connect via USB.

Connecting to MSA CLI via Serial Connecton using USB to Mini USB –

-Install driver for the USB connection. (http://h20564.www2.hp.com/hpsc/swd/public/detail?swItemId=MTX_8de10954d645450f9e3c0d015d )

-Check COM port on Device Manager from Windows.

-Connect USB to the Mini USB port of the MSA

-Use Putty, or Hyperterminal.

-Hit enter to connect to the MSA.

Useful CLI Commands –

Get information about both controllers

show controllers

Show recent controller events (can copy output into notepad)

show events

In my case the show controllers command displayed a health note advising me to restart the problem controller A.

Health Reason: The controller is not Healthy

Recommendation: – Restart the Storage Controller in this controller module, unless it is performing an operation where it is normal for it to be shut down, such as firmware update.

For all CLI commands, you can refer to HP’s CLI Reference Guide –

Click to access 723979-001.pdf

Re-Seating the Problem Controller –

Ultimately I chose to reseat the problem controller in an attempt to prompt a clean restart. Essentially what this involves is unscrewing the two pins holding the problem controller, followed by pulling it slightly out of the controller slot and putting it back in.

Although doing this caused the controller to show no health issues when re-running the #show controller command on the working Controller (B). I had no connectivity to the problem controller (A).

Progress! At least at this point I didn’t have any errors in my working controller’s (B) SMU and nothing was showing as unhealthy in the CLI.

Restarting the Controller –

All had to do at this point was restart the problem controller (A) via CLI. It came back up healthy with network connectivity.

restart mc a

Additional info concerning CLI restarting: Syntax restart sc|mc a|b|both
Parameters sc|mc
The controller to restart:
– sc: Storage Controller
– mc: Management Controller
a|b|both

I could now connect using putty to Controller A, I could login into it’s SMU and the ember lights had cleared from the physical panel on the SAN. Happy days!

**If a simple reseat, restart, or cold reboot of the problem controller still fails to resolve your issue or you want to try and gather more information as to the preceding events leading up to the error. There is the option to generate detailed logs via CLI.

Get more detailed logs via CLI –

• Enter the command in CLI “#show protocols”
  • If FTP is not Enabled –
  • Enter the command: “#set protocols ftp enabled”
  • Exit the telnet session, putty etc.
  • Once FTP is enabled:
    • Go to Start – Command Prompt – enter “cd ../..” (Go to C:/ Drive or the location where you want to save the logs)
    • Type “#ftp ”
      
    • Then type the user name and password of controller to authenticate.
  • Use the command “#get logs filename.zip”

Once you have your logs, you can use Trace32 to view the logs. (Can use notepad but it’s very difficult to view due to a lack of formatting.

You can download Trace32 here:

Download Microsoft Config Manager tools and choose to only install Common Tools in the wizard.

https://www.microsoft.com/en-us/download/details.aspx?id=9257

Hope this was helpful!

Thanks for reading – feel free to follow and stay updated 🙂 View sysadminguides’s profile on Facebook View GuidesSysadmin’s profile on Twitter View 115372466162675927272’s profile on Google+