Get scsi log sense pages error

I have been evaluating two different HP LTO tape drive models (Ultrium 448 & 960) in addition to two other models from different vendors. Comparing these four drive models, the HP tape drives are the only ones that seem to accumulate a large number of errors in the SCSI error counter pages (hundreds...

Hi Richard,

I have attached a few lines from a log file generated by my test program. The data is from a typical write sequence on a HP Ultrium 960 LTO 3 tape drive. The results are very similar on a HP LTO 2 drive, and in total I have seen these error counts on 10 different HP LTO drives.

My test software was developed for use in environmental tests with LTO drives in hope that I would be able to see when drive performance was beginning to deteriorate. Data transfer rates can reveal a lot of information—especially when a drive is able to operate through an external event with a minimal drop in data throughput. However, error counts are another parameter that can be watched to monitor the drive for failures. With the HP drives I have been using, monitoring error rates has been less useful because they seem to accumulate logged errors in even the most ideal operating environments. (Please do not think that I am using damaged hardware—I have seen this behavior out of the box.) This has led me to wonder if these errors are real, or if HP has implemented these error logs vastly different from other LTO manufacturers. (I do understand that you cannot make a direct comparison of these error counts.)

The upper half of my attached data shows the start of a long write transfer. Notice that errors begin to accumulate almost immediately after data has been written to the tape.

The lower half of my attached data is from the same write transfer—now after transferring more than 24 GB of data. I chose to include this portion of the transfer because it shows the appearance of a number in parameter 1 (write errors corrected with possible delay). As defined in the SCSI specification, this happened after a drop in throughput caused by an environmental shock pulse. This leads me to believe that this particular register (Parameter 1) has been implemented in your current firmware revision.

As for parameters 0 and 4, these numbers continue to grow throughout the entire transfer. The unfortunate thing is that at the time of my testing I did not collect data on the number of data sets written. Therefore, I cannot derive any percentages to compare with the acceptable values that you have given me. Maybe you can make an estimate based on the amount of data transferred?

Again, any information you can provide about these error counters would be much appreciated. Your products work as advertised—it was just alarming to see these large numbers when other drives have few to none. Now that I have this error data, it would be nice to have an explanation to account for these differences. Preferably, I would like to have an explanation more technical than «different vendor, different implementation.»

I’m also open to the idea that maybe HP is more «honest» about the number of times error correction is invoked and it’s really the other drives’ firmware that is reporting low values. Again, I understand it’s all about the implementation. However, if these numbers are something other than error counts, I would suggest that they be corrected, removed, or openly documented.

Let me know if you have any questions about my attached data.

Thanks again,
Brian

Источник

This is from a decade ago, so I thought it time to update the URLs and post it to LJ.

Here is information on how to decode SCSI Sense Data. This revolved around IBM Magstar products since that is where I was first exposed to the guts of SCSI errors.

The AIX Error Report records for TAPE_ERR# (usually 1-6) often include SENSE DATA in the Detail section. A SCSI LOG PAGE 06h can be parsed manually to provide the SENSE KEY, ASC and ASCQ values, as well as the ERROR CODE which will tell us if it is current or past errors being reported. An example Log Page 6 is below:

	0600 0000 0300 0000 FF80 0000 0000 0000 0000 0000 7000 0000 0000 0015 0000 000B 
	0000 0000 001C 7F00 2000 0033 7E58 0000 0000 0000 0000 0000 0000 0000 0000 0000 
	0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 
	0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 
	0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 B041 0000 0000

If you’ll notice, byte 0 is 06. Also note that there are 32 bytes per line, and two hex digits per byte.

Byte 20 represents the SCSI error class. Valid classes are:

    * 70 - Current Error (Direct Access Logical Block NOT From Sense Data).
    * F0 - Current Error (Direct Access Logical Block IS From Sense Data)
    * 71 - Deferred Error (Direct Access Logical Block NOT From Sense Data).
    * 7F - Vendor Spec. Error (Direct Access Logical Block NOT From Sense Data).
    * EE - Encryption Error
    * F1 - Deferred Error (Direct Access Logical Block IS From Sense Data).
    * FF - Vendor Spec. Error (Direct Access Logical Block IS From Sense Data).

In this example, EC (byte 20) is 70, which is valid and means this is a current error.

When the error class is valid, we can get the sense key from byte 22.

In this example, the sense key is 00 (zero) which means «NO ADDITIONAL SENSE». The standard list of sense keys is:

	X0 - No Sense         X6 - Unit Attention    XC - Equal.
	X1 - Recovered Error  X7 - Data Protect      XD - Volume Overflow.
	X2 - Not Ready        X8 - Blank Check       XE - Miscompare.
	X3 - Medium Error     X9 - Vendor Specific   XF - RESERVED.
	X4 - Hardware Error   XA - Copy Aborted
	X5 - Illegal Request  XB - Aborted Command

ASC is at byte 32 (first byte on line 2) and ASCQ is byte 33.

The ASC and ASCQ chart is pretty extensive. Please see the ASC/ASCQ Code Listing from the SCSI Technical Committee for an authoritative reference:

  • http://www.t10.org/lists/asc-num.htm

Note also that sometimes the ASC/ASCQ pair you’re looking up may fall under a different sense key than is expected. The Sense key gives general information, such as «Recovered error», «hardware error», or the like. The ASC/ASCQ pair tells what the actual problem is. This isn’t always 100% helpful, but is close.

Good reference was had from the 3590 Maintenance Information Guide, Msgs section. This gives 90% of what anyone would need to decode SCSI LOG PAGE 06h messages for IBM tape drives. The Jaguar Tape Drives (IBM 3590 & 3592) Information Center is at:

  • http://publib.boulder.ibm.com/infocenter/jagdrive/cust/index.jsp

Included within are how to decode SIM/MIM Records, Log Page 6, and other related information. The 3590 Hardware Reference Guide, Appendix B also shows decent information in regards to non SIM/MIM errors. It makes reference to sense key and ASC/ASCQ bytes. You can acquire PDF copies of tape removable media storage systems’ manuals via the following URLS:

The Magstar Maintenance and Ultrium SCSI Reference books makes reference to «Fault Symptom Codes» which are more definitive; however, due to confidentiality of the 3590 microcode, a complete list of fault symptom codes is not available.

For encryption records, see the Troubleshooting section of the IBM TS3500 Tape Library (IBM 3584) Information Center:

  • http://publib.boulder.ibm.com/infocenter/ts3500tl/v1r0/index.jsp

The above also has general SCSI SENSE KEY/ASC/ASCQ and extended IBM codes under the Reference section.

There are other ways to get this information, but this was easiest for me.

Yours truly,
Josh Davis

Источник

sg_logs: access log pages with SCSI LOG SENSE command

Command to display sg_logs manual in Linux: $ man 8 sg_logs

NAME

sg_logs — access log pages with SCSI LOG SENSE command

SYNOPSIS

sg_logs

[—all] [—brief] [—control=PC] [—help]
[—hex] [—list] [—maxlen=LEN] [—name]
[—page=PG[,SPG]] [—paramp=PP] [—pcb]
[—ppc] [—raw] [—reset] [—select]
[—sp] [—temperature] [—transport]
[—verbose] [—version] DEVICE

sg_logs

[-a] [-A] [-b] [-c=PC] [-h] [-H]
[-l] [-L] [-m=LEN] [-n] [-p=PG[,SPG]]
[-paramp=PP] [-pcb] [-ppc] [-r] [-select]
[-sp] [-t] [-T] [-v] [-V] [-?]
DEVICE

DESCRIPTION

This utility sends a SCSI LOG SENSE command to the DEVICE and then
outputs the response. The LOG SENSE command is used to fetch log pages. Known
log pages are decoded by default. When the —reset and/or
—select option is given then a SCSI LOG SELECT command is issued to
reset parameters (e.g. counters).

In SPC-4 revision 5 a subpage code was introduced to both the LOG SENSE and
LOG SELECT command. At the same time a page code field was introduced to the
to the LOG SELECT command. The log subpage code can range from 0 to 255 (0xff)
inclusive. The subpage code value 255 can be thought of as a wildcard.

This utility supports two command line syntaxes, the preferred one is shown
first in the synopsis and explained in this section. A later section on the
old command line syntax outlines the second group of options.

OPTIONS

Arguments to long options are mandatory for short options as well.

-a, —all
outputs all the log pages supported by the device. This requires a two stage
process: first the «supported log pages» log page is fetched, then for each
entry in the response, the corresponding log page is fetched and displayed.
When used twice (e.g. ‘-aa’) all log pages and subpages are fetched.
-b, —brief
shorten the amount of output for some log pages. For example the Tape
Alert log page only outputs parameters whose flags are set when
—brief is given.
-c, —control=PC
accepts 0, 1, 2 or 3 for the PC argument:

  0 : current threshold values

  1 : current cumulative values

  2 : default threshold values

  3 : default cumulative values

The default value is 1 (i.e. current cumulative values).

-h, —help
print out the usage message then exit.
-H, —hex
The default action is to decode known mode page numbers (and subpage numbers)
into text. When this option is used once, the response is output in
hexadecimal.
-l, —list
lists the names of all logs sense pages supported by this device. This is
done by reading the «supported log pages» log page. When used
twice (e.g. ‘-ll’) lists the names of all logs sense pages and subpages
supported by this device. There is a list of common log page codes below.
-m, —maxlen=LEN
sets the «allocation length» field in the LOG SENSE cdb. The is the maximum
length in bytes that the response will be. Without this option (or LEN
equal to 0) this utility first fetches the 4 byte response then does a second
access with the length indicated in the first (4 byte) response. Negative
values and 1 for LEN are not accepted. LEN cannot exceed
65535 (0xffff). Responses can be quite large (e.g. the background scan
results log page) and this option can be used to limit the amount of
information returned.
-n, —name
decode some log pages into ‘name=value’ entries, one per line. The name
contains no space and may be abbreviated and the value is decimal unless
prefixed by ‘0x’. Nesting is indicated by leading spaces. This form
is meant to be relatively easy to parse.
-O, —old
switch to older style options.
-p, —page=PG[,SPG]
log page code to access. PG is expected to be a decimal number between
0 and 63 inclusive. A hexadecimal number can be specified by a leading «0x»
or a trailing «h». Common log page codes are listed below. Optionally
SPG, a subpage code, can be given. SPG is expected to be a
decimal number between 0 and 255 inclusive.
-P, —paramp=PP
PP is the parameter pointer value to place in a field of that name in
the LOG SENSE cdb. A decimal number in the range 0 to 65535 (0xffff) is
expected. When a value greater than 0 is given the —ppc option
should be selected. The default value is 0.
-q, —pcb
show Parameter Control Byte settings (only relevant when log parameters
being output in ASCII).
-Q, —ppc
sets the Parameter Pointer Control (PPC) bit in the LOG SENSE cdb. Default
is 0 (i.e. cleared).
-r, —raw
output the response in binary to stdout. Error messages and warnings are
output to stderr.
-R, —reset
use SCSI LOG SELECT command (PCR bit set) to reset the all log pages (or
the given page). [SPC-4 (rev 6) doesn’t say that a given log (sub)page can
be reset yet.] Exactly what is reset depends on the accompanying SP
bit (i.e. —sp option which defaults to 0) and the
PC («page control») value (which defaults to 1). Supplying this option
implies the —select option as well. This option seems to clear error
counter log pages but leaves pages like self-test results, start-stop cycle
counter and temperature log pages unaffected. This option may be required to
clear log pages if a counter reaches its maximum value since the log page in
which the counter is found will remain «stuck» until something is done.
-S, —select
use a LOG SELECT command. The default action (i.e. when neither this option
nor —reset is given) is to do a LOG SENSE command. When this option
is given, the SP bit (i.e. —sp option which defaults to 0), the
PC («page control») value (which defaults to 1) and the PCR bit (i.e.
—reset option which defaults to 0) are placed in the LOG SELECT
cdb. At some stage the log page and subpage options may also be active [but
SPC-4 (rev 6) doesn’t say that].
-s, —sp
sets the Saving Parameters (SP) bit. Default is 0 (i.e. cleared). When set
this instructs the device to store the current log page parameters (as
indicated by the DS and TSD parameter codes) in some non-volatile location.
Hence the log parameters will be preserved across power cycles. This option
is typically not needed, especially if the GLTSD flag is clear in the
control mode page as this instructs the device to periodically save all
saveable log parameters to non-volatile locations.
-t, —temperature
outputs the temperature. First looks in the temperature log page and if
that is not available tries the Informational Exceptions log page which
may also have the current temperature (especially on older disks).
-T, —transport
outputs the transport (‘Protocol specific port’) log page. Equivalent
to setting ‘—page=18h’.
-v, —verbose
increase level of verbosity.
-V, —version
print out version string then exit.

NOTES

This utility will usually do a double fetch of log pages with the SCSI LOG
SENSE command. The first fetch requests a 4 byte response (i.e. place 4 in
the «allocation length» field in the cdb). From that response it can
calculate the actual length of the response which is what it asks for
on the second fetch. This is typical practice in SCSI and guaranteed to
work in the standards. However some older devices don’t comply. For
those devices using the —maxlen=LEN option will do a single fetch.
A value of 252 should be a safe starting point.

Various log pages hold information error rates, device temperature,
start stop cycles since device produced and the results of the last
20 self tests. Self tests can be initiated by the sg_senddiag(8) utility.
The smartmontools package provides much of the information found with
sg_logs in a form suitable for monitoring the health of SCSI disks and
tape drives.

Here is a list of log pages that are decoded by this utility. [The code
values can be given to ‘—page=’ as is, with a trailing «h» instead of
the leading «0x», or as their decimal equivalents.]:

0x0 Supported log pages

0x0,0xff Supported log pages and subpages

0x1 Buffer overrun/underrun

0x2 Write error counter

0x3 Read error counter

0x4 Read reverse error counter

0x5 Verify error counter

0x6 Non-medium error

0x7 Last n error events

0x8 Format status (sbc-2)

0xb Last n deferred errors or asynchronous events

0xc Logical block provisioning (sbc-3) or Sequential access device (ssc-2)

0xd Temperature

0xe Start-stop cycle counter

0x10 Self-test results

0x11 Solid state media

0x15 Background scan results (sbc-3)

0x16 ATA pass-through results (sat-3)

0x17 Non-volatile cache (sbc-3)

0x18 Protocol specific port (SAS transport)

0x19 General statistics and performance

0x1a Power condition transitions

0x2f Informational exceptions

0x37 Seagate cache (vendor, disk)

0x3e Seagate factory (vendor, disk)

In the 2.4 series of Linux kernels the DEVICE must be a SCSI
generic (sg) device. In the 2.6 series block devices (e.g. SCSI disks
and DVD drives) can also be specified. For example «sg_logs -a /dev/sda»
will work in the 2.6 series kernels.

EXIT STATUS

The exit status of sg_logs is 0 when it is successful. Otherwise see
the sg3_utils(8) man page.

OLDER COMMAND LINE OPTIONS

The options in this section were the only ones available prior to sg3_utils
version 1.23 . In sg3_utils version 1.23 and later these older options can
be selected by either setting the SG3_UTILS_OLD_OPTS environment variable
or using ‘—old’ (or ‘-O) as the first option.

Options with arguments or with two or more letters can have an extra ‘-‘
prepended. For example: both ‘-pcb’ and ‘—pcb’ are acceptable.

-a
outputs all the log pages supported by the device.
Equivalent to —all in the main description.
-A
outputs all the log pages and subpages supported by the device.
Equivalent to ‘—all —all’ in the main description.
-c=PC
Equivalent to —control=PC in the main description.
-h
suppresses decoding of known log sense pages and prints out the
response in hex instead.
-H
same action as ‘-h’ in this section and equivalent to —hex in
the main description.
-l
lists the names of all logs sense pages supported by this device.
Equivalent to —list in the main description.
-L
lists the names of all logs sense pages and subpages supported by this
device. Equivalent to ‘—list —list’ in the main description.
-m=LEN
request only LEN bytes of response data. Default is 0 which is
interpreted as all that is available. LEN is decimal unless it has
a leading ‘0x’ or trailing ‘h’. Equivalent to —maxlen=LEN in
the main description.
-n
Equivalent to —name in the main description.
-N
switch to the newer style options.
-p=PG[,SPG]
PG is the log page code to access. Should be a hexadecimal number
between 0 and 3f inclusive. If given SPG is the log subpage code.
SPG should be a hexadecimal number between 0 and ff inclusive. The
subpage code of ‘ff’ can be thought of as a wildcard.
-paramp=PP
PP is the parameter pointer value (in hex) to place in command.
Should be a number between 0 and ffff inclusive.
-pcb
show Parameter Control Byte settings (only relevant when log parameters
being output in ASCII).
-ppc
sets the Parameter Pointer Control (PPC) bit. Default is 0 (i.e. cleared).
-r
use SCSI LOG SELECT command (PCR bit set) to reset the all log pages (or
the given page). Equivalent to —reset in the main description.
-select
use a LOG SELECT command. Equivalent to —select in the main
description.
-sp
sets the Saving Parameters (SP) bit. Default is 0 (i.e. cleared).
Equivalent to —sp in the main description.
-t
outputs the temperature. Equivalent to —temperature in the main
description.
-T
outputs the transport (‘Protocol specific port’) log page. Equivalent
to —transport in the main description.
-v
increase level of verbosity.
-V
print out version string then exit.
-?
output usage message then exit.

AUTHOR

Written by Douglas Gilbert

REPORTING BUGS

Report bugs to <dgilbert at interlog dot com>.

COPYRIGHT

Copyright © 2002-2013 Douglas Gilbert

This software is distributed under the GPL version 2. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Источник

  • #61

I have now put in a new motherboard in the chassis and started a scrub on the full pool of 24 6TB disks configured as 12 mirrors.

  • #62

I still get errors when running of off the new motherboard.

The thing is, that now, the errors still occur on the same twos disks (da0 & da6). Looking at my notes, I can see, that the errors used to occur on random disks.

I’m now thinking, the problem could be with some slots on the backplane(s).

I will now move da0 & da6 to new slots and scrub again.

Code:

...
...
(da6:mps0:0:17:0): READ(10). CDB: 28 00 a4 40 25 68 00 00 b0 00 length 90112 SMID 1017 terminated ioc 804b scsi 0 state 0 xfer 0
(da6:mps0:0:17:0): READ(10). CDB: 28 00 a4 40 25 68 00 00 b0 00
(da6:mps0:0:17:0): CAM status: CCB request completed with an error
(da6:mps0:0:17:0): Retrying command
(da6:mps0:0:17:0): READ(10). CDB: 28 00 a4 40 24 a8 00 00 c0 00
(da6:mps0:0:17:0): CAM status: SCSI Status Error
(da6:mps0:0:17:0): SCSI status: Check Condition
(da6:mps0:0:17:0): SCSI sense: MEDIUM ERROR asc:11,0 (Unrecovered read error)
(da6:mps0:0:17:0): Info: 0xa44024a8
(da6:mps0:0:17:0): Error 5, Unretryable error
...
...
(da0:mps0:0:9:0): READ(10). CDB: 28 00 ac 39 98 e8 00 00 d8 00 length 110592 SMID 851 terminated ioc 804b scsi 0 state 0 xfer 0
(da0:mps0:0:9:0): READ(10). CDB: 28 00 a8 6a 7e 88 00 00 08 00 length 4096 SMID 204 terminated ioc 804b scsi 0 state 0 xfer 0(da0:mps0:0:9:0): READ(10). CDB: 28 00 ac 39 98 e8 00 00 d8 00
(da0:mps0:0:9:0): CAM status: CCB request completed with an error
(da0:mps0:0:9:0): READ(10). CDB: 28 00 a8 6c 5e f0 00 00 08 00 length 4096 SMID 678 terminated ioc 804b scsi 0 state 0 xfer 0(da0:
mps0:0:9:0): Retrying command
(da0:mps0:0:9:0): READ(10). CDB: 28 00 a8 6a 7e 88 00 00 08 00
(da0:mps0:0:9:0): CAM status: CCB request completed with an error
(da0:mps0:0:9:0): Retrying command
(da0:mps0:0:9:0): READ(10). CDB: 28 00 a8 6c 5e f0 00 00 08 00
(da0:mps0:0:9:0): CAM status: CCB request completed with an error
(da0:mps0:0:9:0): Retrying command
(da0:mps0:0:9:0): READ(10). CDB: 28 00 ac 39 98 10 00 00 d8 00
(da0:mps0:0:9:0): CAM status: SCSI Status Error
(da0:mps0:0:9:0): SCSI status: Check Condition
(da0:mps0:0:9:0): SCSI sense: MEDIUM ERROR asc:11,0 (Unrecovered read error)
(da0:mps0:0:9:0): Info: 0xac399810
(da0:mps0:0:9:0): Error 5, Unretryable error
...
...

  • #63

The errors occur at «all» slots in the backplanes. There seems to be no bad slots.

I have reseated all power connections and moved the HBA (ServeRAID) to a different slot.

Now, I get a LOT of errors on alot of disk right when I begin my scrub. But now, they are a «type» I have not seen before (Information unit iuCRC error detected):

Code:

...
...
Jul 27 15:27:41 ultraman	 (da4:mps0:0:15:0): READ(10). CDB: 28 00 42 eb 33 a0 00 01 00 00 length 131072 SMID 125 terminated ioc 804b scsi 0 state 0 xfer 0
Jul 27 15:27:41 ultraman (da4:mps0:0:15:0): READ(10). CDB: 28 00 42 eb 33 a0 00 01 00 00 
Jul 27 15:27:41 ultraman (da4:mps0:0:15:0): CAM status: CCB request completed with an error
Jul 27 15:27:41 ultraman (da4:mps0:0:15:0): Retrying command
Jul 27 15:27:41 ultraman (da4:mps0:0:15:0): READ(10). CDB: 28 00 42 eb 34 a0 00 00 60 00 
Jul 27 15:27:41 ultraman (da4:mps0:0:15:0): CAM status: SCSI Status Error
Jul 27 15:27:41 ultraman (da4:mps0:0:15:0): SCSI status: Check Condition
Jul 27 15:27:41 ultraman (da4:mps0:0:15:0): SCSI sense: ABORTED COMMAND asc:47,3 (Information unit iuCRC error detected)
Jul 27 15:27:41 ultraman (da4:mps0:0:15:0): Retrying command (per sense data)
...
...

  • #64

Now, I get a LOT of errors on alot of disk right when I begin my scrub. But now, they are a «type» I have not seen before (Information unit iuCRC error detected).

I have reseated the disk cables on the HBA. The iuCRC error errors appear to have gone.

  • #65

I still get SCSI errors after having reseated all power cables (I could find).

Two of my disks are way more likely to have errors happen on them (As I have noticed before). Surely, that can not be coincidence? I will remove the disks (currently) called da0 & da6, and replace them with other disks.

Trying to quantify it, I use grep to get all lines relating to a SCSI error from dmesg, and see how many times each drive has been mentioned. As shown below, da0 & da6 have way more mentions than other disks. I’m guessing, each error results in 9 mentions. That puts da0 at 270/9 = 30 errors and da6 at 342/9 = 38 errors.

Code:

dmesg | grep -E "^(da[0-9]+" -o | sort | uniq -c

 270 (da0
   9 (da11
   9 (da12
   9 (da13
  18 (da17
   9 (da18
   9 (da20
  18 (da21
   9 (da23
 342 (da6

In other news: I have ordered a replacement power distribution unit for the machine.

  • #66

I can only speak for myself, but i think i understand what fixed the issue for me.
Before I went on vacation I replaced the SAS->SATA cable hoping I fixed the issue. With me not home for 2 weeks and a server without errors I thought that was the problem.
But when i arrived home within days the errors started coming back….

But after a few more weeks with during daytime (trying) to check the console every hour to see when the error arrived I started to see a pattern.
When I am working behind my PC I changed the fanspeed of the server to medium instead of high (the server is like 3 feet next to me).
So almost everytime I was working for a longer period of time behind my pc I recieved an SCSi error.

Right now i keep the the Fanspeed at high and i am allready error free for ~3+ weeks.

I can’t speak for your situation.
But in my case I have the feeling it was an heat issue..

  • #67

Right now i keep the the Fanspeed at high and i am allready error free for ~3+ weeks.

Have you logged the temperature of your disks? Running «smartctl -a /dev/daX» should tell you.

If you did, maybe you could tell what impact your fans have on drive temperature.

  • #68

Could also be SAS controller heat.

@Sir SSV recently posted how to update the firmware on an sm backplane. Iirc

  • #69

Have you logged the temperature of your disks? Running «smartctl -a /dev/daX» should tell you.

If you did, maybe you could tell what impact your fans have on drive temperature.

No I don’t have the temperature logged.
But I think it is the SAS controller (HBA, onboard on my X11SSL-cf mobo) which was to hot.

Even now it feels hotter compared to my chipset heatsinks (the heatsinks are the same) but HBA’s are somehow known to run quite hot…
I dont have a Heatgun to see how hot the heatsink actually is…

I can try and run a command on it to see how high the temperature is (if someone knows the right command)
:)

*EDIT*
I know for sure that the drives are running under 40 degrees C.
I have a mailnotification set up when they run hotter than that.

Last edited: Aug 6, 2017

  • #70

Last time, i replaced two disks, which had a lot more errors than others. I replaced them with different disks. Now the two new disks are the ones with the most errors. That’s really odd (I think). Earlier, I have moved disks around, and the problems do not appear to be tied to specific slots in the backplane.

The two new disks are da0 & da4. This the the number of mentions of disk errors pr. disk from dmesg:

Code:

dmesg | grep -E "^(da[0-9]+" -o | sort | uniq -c

157 (da0
   9 (da11
   9 (da12
   9 (da13
  13 (da18
   6 (da20
  67 (da4

I am replacing da0 with a new disk. The new one is WD Gold. Old was WD Red.

Will the new disk inherit the trait of having much more SCSI errors than other disks? I’ll report back.

  • #71

I have now installed a brand new PDU (Power Distribution Unit) in the machine. I have started a scrub, and I am now waiting for the result with bated breath.

  • #73

Still getting errors. 14% into the scrub :mad:. I need to look in to the firmware on the backplanes.

From /var/log/messages:

Code:

...
...
Aug 21 12:38:18 ultraman	 (da3:mps0:0:14:0): READ(10). CDB: 28 00 78 39 af a8 00 00 a8 00 length 86016 SMID 711 terminated ioc 804b scsi 0 state 0 xfer 0
Aug 21 12:38:18 ultraman (da3:mps0:0:14:0): READ(10). CDB: 28 00 78 39 af a8 00 00 a8 00
Aug 21 12:38:18 ultraman (da3:mps0:0:14:0): CAM status: CCB request completed with an error
Aug 21 12:38:18 ultraman (da3:mps0:0:14:0): Retrying command
Aug 21 12:38:18 ultraman (da3:mps0:0:14:0): READ(10). CDB: 28 00 78 39 af 00 00 00 a8 00
Aug 21 12:38:18 ultraman (da3:mps0:0:14:0): CAM status: SCSI Status Error
Aug 21 12:38:18 ultraman (da3:mps0:0:14:0): SCSI status: Check Condition
Aug 21 12:38:18 ultraman (da3:mps0:0:14:0): SCSI sense: MEDIUM ERROR asc:11,0 (Unrecovered read error)
Aug 21 12:38:18 ultraman (da3:mps0:0:14:0): Info: 0x7839af00
Aug 21 12:38:18 ultraman (da3:mps0:0:14:0): Error 5, Unretryable error
...
...
Aug 21 12:54:43 ultraman	 (da3:mps0:0:14:0): READ(10). CDB: 28 00 7c 21 08 f8 00 00 b8 00 length 94208 SMID 111 terminated ioc 804b scsi 0 state 0 xfer 0
Aug 21 12:54:43 ultraman (da3:mps0:0:14:0): READ(10). CDB: 28 00 7c 21 08 f8 00 00 b8 00
Aug 21 12:54:43 ultraman (da3:mps0:0:14:0): CAM status: CCB request completed with an error
Aug 21 12:54:43 ultraman (da3:mps0:0:14:0): Retrying command
Aug 21 12:54:43 ultraman (da3:mps0:0:14:0): READ(10). CDB: 28 00 7c 21 08 40 00 00 b8 00
Aug 21 12:54:43 ultraman (da3:mps0:0:14:0): CAM status: SCSI Status Error
Aug 21 12:54:43 ultraman (da3:mps0:0:14:0): SCSI status: Check Condition
Aug 21 12:54:43 ultraman (da3:mps0:0:14:0): SCSI sense: MEDIUM ERROR asc:11,0 (Unrecovered read error)
Aug 21 12:54:43 ultraman (da3:mps0:0:14:0): Info: 0x7c210840
Aug 21 12:54:43 ultraman (da3:mps0:0:14:0): Error 5, Unretryable error
...
...

Pool status:

Code:

  pool: ultraman
state: ONLINE
  scan: scrub in progress since Mon Aug 21 10:50:43 2017
		5.85T scanned out of 39.2T at 809M/s, 11h59m to go
		176K repaired, 14.94% done
config:

	NAME											STATE	 READ WRITE CKSUM
	ultraman										ONLINE	   0	 0	 0
	  mirror-0									  ONLINE	   0	 0	 0
		gptid/2e00ac23-183d-11e7-ae9d-0025901ef244  ONLINE	   0	 0	 0
		gptid/6e71919e-1618-11e7-a3b7-0025901ef244  ONLINE	   0	 0	 0
	  mirror-1									  ONLINE	   0	 0	 0
		gptid/6f22c98c-1618-11e7-a3b7-0025901ef244  ONLINE	   0	 0	 0
		gptid/6fe54bfe-1618-11e7-a3b7-0025901ef244  ONLINE	   0	 0	 0
	  mirror-2									  ONLINE	   0	 0	 0
		gptid/70bfd5c6-1618-11e7-a3b7-0025901ef244  ONLINE	   0	 0	 0
		gptid/424dc93e-7923-11e7-b91b-0cc47a5312f0  ONLINE	   0	 0	 0  (repairing)
	  mirror-3									  ONLINE	   0	 0	 0
		gptid/7ad0f185-1619-11e7-a3b7-0025901ef244  ONLINE	   0	 0	 0
		gptid/9e899578-183c-11e7-ae9d-0025901ef244  ONLINE	   0	 0	 0
	  mirror-4									  ONLINE	   0	 0	 0
		gptid/427c2189-184c-11e7-ae9d-0025901ef244  ONLINE	   0	 0	 0
		gptid/4342a98c-184c-11e7-ae9d-0025901ef244  ONLINE	   0	 0	 0
	  mirror-5									  ONLINE	   0	 0	 0
		gptid/a1ab9a69-184c-11e7-ae9d-0025901ef244  ONLINE	   0	 0	 0
		gptid/a2851364-184c-11e7-ae9d-0025901ef244  ONLINE	   0	 0	 0
	  mirror-6									  ONLINE	   0	 0	 0
		gptid/0dcbcccd-184d-11e7-ae9d-0025901ef244  ONLINE	   0	 0	 0
		gptid/0e9ed582-184d-11e7-ae9d-0025901ef244  ONLINE	   0	 0	 0
	  mirror-7									  ONLINE	   0	 0	 0
		gptid/2b56cf1c-184d-11e7-ae9d-0025901ef244  ONLINE	   0	 0	 0
		gptid/2c2a1c62-184d-11e7-ae9d-0025901ef244  ONLINE	   0	 0	 0
	  mirror-8									  ONLINE	   0	 0	 0
		gptid/69718320-184d-11e7-ae9d-0025901ef244  ONLINE	   0	 0	 0
		gptid/6a4e6afa-184d-11e7-ae9d-0025901ef244  ONLINE	   0	 0	 0
	  mirror-9									  ONLINE	   0	 0	 0
		gptid/8fccb2c6-184d-11e7-ae9d-0025901ef244  ONLINE	   0	 0	 0
		gptid/90b48d70-184d-11e7-ae9d-0025901ef244  ONLINE	   0	 0	 0
	  mirror-10									 ONLINE	   0	 0	 0
		gptid/aeda4f88-184d-11e7-ae9d-0025901ef244  ONLINE	   0	 0	 0
		gptid/afb5bfc3-184d-11e7-ae9d-0025901ef244  ONLINE	   0	 0	 0
	  mirror-11									 ONLINE	   0	 0	 0
		gptid/bca6e4f0-1b71-11e7-ae9d-0025901ef244  ONLINE	   0	 0	 0
		gptid/d06f5f86-7b79-11e7-b91b-0cc47a5312f0  ONLINE	   0	 0	 0

errors: No known data errors

SMART data from da3 (The disk with errors):

Code:

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME		  FLAG	 VALUE WORST THRESH TYPE	  UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate	 0x002f   200   200   051	Pre-fail  Always	   -	   0
  3 Spin_Up_Time			0x0027   205   204   021	Pre-fail  Always	   -	   8708
  4 Start_Stop_Count		0x0032   100   100   000	Old_age   Always	   -	   15
  5 Reallocated_Sector_Ct   0x0033   200   200   140	Pre-fail  Always	   -	   0
  7 Seek_Error_Rate		 0x002e   100   253   000	Old_age   Always	   -	   0
  9 Power_On_Hours		  0x0032   099   099   000	Old_age   Always	   -	   975
 10 Spin_Retry_Count		0x0032   100   253   000	Old_age   Always	   -	   0
 11 Calibration_Retry_Count 0x0032   100   253   000	Old_age   Always	   -	   0
 12 Power_Cycle_Count	   0x0032   100   100   000	Old_age   Always	   -	   15
192 Power-Off_Retract_Count 0x0032   200   200   000	Old_age   Always	   -	   13
193 Load_Cycle_Count		0x0032   200   200   000	Old_age   Always	   -	   22
194 Temperature_Celsius	 0x0022   110   107   000	Old_age   Always	   -	   42
196 Reallocated_Event_Count 0x0032   200   200   000	Old_age   Always	   -	   0
197 Current_Pending_Sector  0x0032   200   200   000	Old_age   Always	   -	   0
198 Offline_Uncorrectable   0x0030   100   253   000	Old_age   Offline	  -	   0
199 UDMA_CRC_Error_Count	0x0032   200   200   000	Old_age   Always	   -	   0
200 Multi_Zone_Error_Rate   0x0008   200   200   000	Old_age   Offline	  -	   0

  • #74

I finally took a look at the firmware on my two backplanes (SAS2-826EL1 & SAS2-846EL1).

Supermicro provide tools to flash the firmware here:
ftp://ftp.supermicro.com/utility/ExpanderXtools_Lite/

Windows and Linux versions are available. I prefer Linux, so I download the 64 bit Linux version:
SMC ExpanderXtoolsLite v1.5_Linux64.zip

The only file I need from the zip file is the binary xflash.

I boot my server from a Debian 8 install USB stick. I enter rescue mode and enter a shell.

I copy the xflash binary to different USB stick. I mount it using the Debian shell.

Let’s find the SAS addresses of my backplanes:
xflash -i get avail

********************************************************************************
Xflash

LSI SAS Expander Flash Utility

Version: 7.0.0.0

Copyright (c) 2010 LSI Corporation. All rights reserved.
********************************************************************************

Initializing Interface.
Expander: SAS2x28

1) SAS2x28 (50030480:0157F6FF) (0.0.0.0)
2) SAS2x36 (50030480:0120CDFF) (0.0.0.0)

Let’s see what firmware is on the backplanes:

xflash -i 500304800120CDFF get ver

********************************************************************************
Xflash

LSI SAS Expander Flash Utility

Version: 7.0.0.0

Copyright (c) 2010 LSI Corporation. All rights reserved.
********************************************************************************

Initializing Interface.
Expander: SAS2x36

Firmware Region Version: 55.07.23.00

Some time spent Googling, seems to indicate that the latest version of the firmware is called 55.14.18.0. Apparently, the FW is not freely available from Supermicro (?!). The way to get it, seems to be to download it from some unknown FTP site! Well, I’m desperate, so I do it even though I do not like it.
ftp://211.214.88.34/bios/BP_FW/FW_55.14.18.0.zip

In the zip file, the firmware is in the file sas2xfw_55.14.18.0.fw.

Let’s flash the firmware to the backplane(s). The SAS address must be entered with out the colon in the middle, hence:

xflash -i 500304800120CDFF down fw sas2xfw_55.14.18.0.fw 0 -y

Looks good (Screendump). It worked!

To make sure, I reboot the machine back in to Linux, and check firmware on both backplanes with the command xflash -i <SAS_ADDRESS> get ver (This returns the active firmware as fas as I understand). The firmware on both backplanes is reported as 55.14.18.0.

I boot up FreeNAS again and start to scrub the pool.

Within a few minutes, I get a SCSI error!!

Code:

(da3:mps0:0:14:0): READ(10). CDB: 28 00 4b e9 b8 30 00 01 00 00 length 131072 SMID 493 terminated ioc 804b scsi 0 state 0 xfer 0
(da3:mps0:0:14:0): READ(10). CDB: 28 00 4b e9 b8 30 00 01 00 00
(da3:mps0:0:14:0): CAM status: CCB request completed with an error
(da3:mps0:0:14:0): Retrying command
(da3:mps0:0:14:0): READ(10). CDB: 28 00 4b e9 b7 50 00 00 e0 00
(da3:mps0:0:14:0): CAM status: SCSI Status Error
(da3:mps0:0:14:0): SCSI status: Check Condition
(da3:mps0:0:14:0): SCSI sense: MEDIUM ERROR asc:11,0 (Unrecovered read error)
(da3:mps0:0:14:0): Info: 0x4be9b750
(da3:mps0:0:14:0): Error 5, Unretryable error

So it is still not fixed! I have now installed:

  • Two new power supplies.
  • Two new SAS cables between HBA and backplanes.
  • A new Power Distribution Unit
  • New firmware on both backplanes
  • New motherboard

I’m now considering trashing the whole machine. I have no more things left to try.

Ericloewe


  • #75

This is beyond weird. Is the behavior the same on a different OS?

At this point, the only thing left are the drives, but everyone and their mother uses WD Reds. Plenty of 6TB models in the mix, too.

Now that I think about it, could there be something intermittently shorting out something on the backplane? What’s the output of smartctl -x /dev/daX, for a representative drive or two?

  • #76

Let’s see the number of «SCSI Status Error»s in dmesg after 3 scrubs during the weekend:

Code:

[root@ultraman] ~# dmesg | grep -i scsi status error | sort | uniq -c

   1 (da10:mps0:0:22:0): CAM status: SCSI Status Error
   1 (da11:mps0:0:23:0): CAM status: SCSI Status Error
   1 (da12:mps0:0:24:0): CAM status: SCSI Status Error
   1 (da16:mps0:0:28:0): CAM status: SCSI Status Error
   1 (da17:mps0:0:29:0): CAM status: SCSI Status Error
   1 (da19:mps0:0:31:0): CAM status: SCSI Status Error
   1 (da20:mps0:0:32:0): CAM status: SCSI Status Error
   1 (da22:mps0:0:34:0): CAM status: SCSI Status Error
   9 (da3:mps0:0:14:0): CAM status: SCSI Status Error
   1 (da7:mps0:0:19:0): CAM status: SCSI Status Error

Most errors occur on /dev/da3. This was also the case before updating the firmware on the backplanes.

In fact, the disks reported are mostly the same as before upgrading the backplane firmware, as seen here:

Code:

[root@ultraman] ~# dmesg | grep -i scsi status error | sort | uniq -c

   1 (da10:mps0:0:22:0): CAM status: SCSI Status Error
   1 (da11:mps0:0:23:0): CAM status: SCSI Status Error
   1 (da12:mps0:0:24:0): CAM status: SCSI Status Error
   1 (da16:mps0:0:28:0): CAM status: SCSI Status Error
   1 (da17:mps0:0:29:0): CAM status: SCSI Status Error
   1 (da19:mps0:0:31:0): CAM status: SCSI Status Error
   1 (da20:mps0:0:32:0): CAM status: SCSI Status Error
  19 (da3:mps0:0:14:0): CAM status: SCSI Status Error
   1 (da7:mps0:0:19:0): CAM status: SCSI Status Error
   1 (da9:mps0:0:21:0): CAM status: SCSI Status Error

Here is an example of an error on /dev/da3:

Code:

(da3:mps0:0:14:0): READ(16). CDB: 88 00 00 00 00 01 7b 45 17 b0 00 00 00 c8 00 00 length 102400 SMID 853 terminated ioc 804b scsi 0 state 0 xfer 0
(da3:mps0:0:14:0): READ(16). CDB: 88 00 00 00 00 01 7b 45 17 b0 00 00 00 c8 00 00
(da3:mps0:0:14:0): CAM status: CCB request completed with an error
(da3:mps0:0:14:0): Retrying command
(da3:mps0:0:14:0): READ(16). CDB: 88 00 00 00 00 01 7b 45 16 e8 00 00 00 c8 00 00
(da3:mps0:0:14:0): CAM status: SCSI Status Error
(da3:mps0:0:14:0): SCSI status: Check Condition
(da3:mps0:0:14:0): SCSI sense: MEDIUM ERROR asc:11,0 (Unrecovered read error)
(da3:mps0:0:14:0): Info: 0x17b4516e8
(da3:mps0:0:14:0): Error 5, Unretryable error

As suggested, Let’s see what smartctl has to say about /dev/da3 (The drive with the most errors logged).

What’s the output of smartctl -x /dev/daX, for a representative drive or two?

Code:

[root@ultraman] ~# smartctl -x /dev/da3

smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:	 Western Digital Red
Device Model:	 WDC WD60EFRX-68L0BN1
Serial Number:	WD-WX11DC61YDPV
LU WWN Device Id: 5 0014ee 263cb5ce4
Firmware Version: 82.00A82
User Capacity:	6,001,175,126,016 bytes [6.00 TB]
Sector Sizes:	 512 bytes logical, 4096 bytes physical
Rotation Rate:	5700 rpm
Device is:		In smartctl database [for details use: -P show]
ATA Version is:   ACS-2, ACS-3 T13/2161-D revision 3b
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:	Mon Aug 28 09:32:18 2017 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is:   Unavailable
APM feature is:   Unavailable
Rd look-ahead is: Enabled
Write cache is:   Enabled
ATA Security is:  Disabled, NOT FROZEN [SEC1]
Wt Cache Reorder: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00)	Offline data collection activity
					was never started.
					Auto Offline Data Collection: Disabled.
Self-test execution status:	  (   0)	The previous self-test routine completed
					without error or no self-test has ever
					been run.
Total time to complete Offline
data collection:		 ( 2384) seconds.
Offline data collection
capabilities:			 (0x7b) SMART execute Offline immediate.
					Auto Offline data collection on/off support.
					Suspend Offline collection upon new
					command.
					Offline surface scan supported.
					Self-test supported.
					Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:			(0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:		(0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine
recommended polling time:	 (   2) minutes.
Extended self-test routine
recommended polling time:	 ( 678) minutes.
Conveyance self-test routine
recommended polling time:	 (   5) minutes.
SCT capabilities:			(0x303d)	SCT Status supported.
					SCT Error Recovery Control supported.
					SCT Feature Control supported.
					SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME		  FLAGS	VALUE WORST THRESH FAIL RAW_VALUE
  1 Raw_Read_Error_Rate	 POSR-K   200   200   051	-	0
  3 Spin_Up_Time			POS--K   204   204   021	-	8775
  4 Start_Stop_Count		-O--CK   100   100   000	-	19
  5 Reallocated_Sector_Ct   PO--CK   200   200   140	-	0
  7 Seek_Error_Rate		 -OSR-K   100   253   000	-	0
  9 Power_On_Hours		  -O--CK   099   099   000	-	1138
10 Spin_Retry_Count		-O--CK   100   253   000	-	0
11 Calibration_Retry_Count -O--CK   100   253   000	-	0
12 Power_Cycle_Count	   -O--CK   100   100   000	-	19
192 Power-Off_Retract_Count -O--CK   200   200   000	-	17
193 Load_Cycle_Count		-O--CK   200   200   000	-	27
194 Temperature_Celsius	 -O---K   114   107   000	-	38
196 Reallocated_Event_Count -O--CK   200   200   000	-	0
197 Current_Pending_Sector  -O--CK   200   200   000	-	0
198 Offline_Uncorrectable   ----CK   100   253   000	-	0
199 UDMA_CRC_Error_Count	-O--CK   200   200   000	-	0
200 Multi_Zone_Error_Rate   ---R--   200   200   000	-	0
							||||||_ K auto-keep
							|||||__ C event count
							||||___ R error rate
							|||____ S speed/performance
							||_____ O updated online
							|______ P prefailure warning

General Purpose Log Directory Version 1
SMART		   Log Directory Version 1 [multi-sector log support]
Address	Access  R/W   Size  Description
0x00	   GPL,SL  R/O	  1  Log Directory
0x01		   SL  R/O	  1  Summary SMART error log
0x02		   SL  R/O	  5  Comprehensive SMART error log
0x03	   GPL	 R/O	  6  Ext. Comprehensive SMART error log
0x04	   GPL,SL  R/O	  8  Device Statistics log
0x06		   SL  R/O	  1  SMART self-test log
0x07	   GPL	 R/O	  1  Extended self-test log
0x09		   SL  R/W	  1  Selective self-test log
0x0c	   GPL	 R/O   2048  Pending Defects log
0x10	   GPL	 R/O	  1  SATA NCQ Queued Error log
0x11	   GPL	 R/O	  1  SATA Phy Event Counters log
0x21	   GPL	 R/O	  1  Write stream error log
0x22	   GPL	 R/O	  1  Read stream error log
0x30	   GPL,SL  R/O	  9  IDENTIFY DEVICE data log
0x80-0x9f  GPL,SL  R/W	 16  Host vendor specific log
0xa0-0xa7  GPL,SL  VS	  16  Device vendor specific log
0xa8-0xb6  GPL,SL  VS	   1  Device vendor specific log
0xb7	   GPL,SL  VS	  54  Device vendor specific log
0xbd	   GPL,SL  VS	   1  Device vendor specific log
0xc0	   GPL,SL  VS	   1  Device vendor specific log
0xc1	   GPL	 VS	  93  Device vendor specific log
0xe0	   GPL,SL  R/W	  1  SCT Command/Status
0xe1	   GPL,SL  R/W	  1  SCT Data Transfer

SMART Extended Comprehensive Error Log Version: 1 (6 sectors)
Device Error Count: 76 (device log contains only the most recent 24 errors)
	CR	 = Command Register
	FEATR  = Features Register
	COUNT  = Count (was: Sector Count) Register
	LBA_48 = Upper bytes of LBA High/Mid/Low Registers ]  ATA-8
	LH	 = LBA High (was: Cylinder High) Register	]   LBA
	LM	 = LBA Mid (was: Cylinder Low) Register	  ] Register
	LL	 = LBA Low (was: Sector Number) Register	 ]
	DV	 = Device (was: Device/Head) Register
	DC	 = Device Control Register
	ER	 = Error register
	ST	 = Status register
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 76 [3] occurred at disk power-on lifetime: 1135 hours (47 days + 7 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  40 -- 51 00 00 00 01 5c 2b 8f 00 40 00  Error: UNC at LBA = 0x15c2b8f00 = 5841325824

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  60 00 b8 00 08 00 01 5c 2b 8f b8 40 00  2d+14:27:14.165  READ FPDMA QUEUED
  60 00 b8 00 00 00 01 5c 2b 8f 00 40 00  2d+14:27:14.165  READ FPDMA QUEUED
  60 00 c8 00 08 00 01 5c 2b 8e 38 40 00  2d+14:27:14.153  READ FPDMA QUEUED
  60 00 c8 00 00 00 01 5c 2b 8d 70 40 00  2d+14:27:14.153  READ FPDMA QUEUED
  60 00 c8 00 08 00 01 5c 2b 8c a8 40 00  2d+14:27:14.142  READ FPDMA QUEUED

Error 75 [2] occurred at disk power-on lifetime: 1132 hours (47 days + 4 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  40 -- 51 00 00 00 01 7b 45 16 e8 40 00  Error: UNC at LBA = 0x17b4516e8 = 6363092712

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  60 00 c8 00 08 00 01 7b 45 17 b0 40 00  2d+11:38:04.384  READ FPDMA QUEUED
  60 00 c8 00 00 00 01 7b 45 16 e8 40 00  2d+11:38:04.384  READ FPDMA QUEUED
  60 00 c0 00 00 00 01 7b 45 16 28 40 00  2d+11:38:04.383  READ FPDMA QUEUED
  60 00 c8 00 08 00 01 7b 45 15 60 40 00  2d+11:38:04.372  READ FPDMA QUEUED
  60 00 c8 00 00 00 01 7b 45 14 98 40 00  2d+11:38:04.372  READ FPDMA QUEUED

Error 74 [1] occurred at disk power-on lifetime: 1130 hours (47 days + 2 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  40 -- 51 00 00 00 01 34 2f d9 b8 40 00  Error: UNC at LBA = 0x1342fd9b8 = 5170518456

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  60 00 e0 00 08 00 01 34 2f da 98 40 00  2d+09:36:26.840  READ FPDMA QUEUED
  60 00 e0 00 00 00 01 34 2f d9 b8 40 00  2d+09:36:26.840  READ FPDMA QUEUED
  60 00 e0 00 08 00 01 34 2f d8 d8 40 00  2d+09:36:26.829  READ FPDMA QUEUED
  60 00 d8 00 00 00 01 34 2f d8 00 40 00  2d+09:36:26.828  READ FPDMA QUEUED
  60 00 e0 00 08 00 01 34 2f d7 20 40 00  2d+09:36:26.817  READ FPDMA QUEUED

Error 73 [0] occurred at disk power-on lifetime: 1127 hours (46 days + 23 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  40 -- 51 00 00 00 00 d8 12 4c 50 40 00  Error: UNC at LBA = 0xd8124c50 = 3625077840

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  60 01 00 00 08 00 00 d8 12 4d 50 40 00  2d+06:27:33.777  READ FPDMA QUEUED
  60 01 00 00 00 00 00 d8 12 4c 50 40 00  2d+06:27:33.777  READ FPDMA QUEUED
  60 00 d8 00 00 00 00 d8 12 4b 78 40 00  2d+06:27:33.775  READ FPDMA QUEUED
  60 01 00 00 08 00 00 d8 12 4a 78 40 00  2d+06:27:33.764  READ FPDMA QUEUED
  60 01 00 00 00 00 00 d8 12 48 40 40 00  2d+06:27:33.764  READ FPDMA QUEUED

Error 72 [23] occurred at disk power-on lifetime: 1106 hours (46 days + 2 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  40 -- 51 00 00 00 01 15 c4 27 30 40 00  Error: UNC at LBA = 0x115c42730 = 4660143920

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  60 00 c8 00 08 00 01 15 c4 27 f8 40 00  1d+09:50:45.002  READ FPDMA QUEUED
  60 00 c8 00 00 00 01 15 c4 27 30 40 00  1d+09:50:45.002  READ FPDMA QUEUED
  60 00 b8 00 08 00 01 15 c4 26 78 40 00  1d+09:50:44.990  READ FPDMA QUEUED
  60 00 c0 00 00 00 01 15 c4 25 b8 40 00  1d+09:50:44.990  READ FPDMA QUEUED
  60 00 d0 00 00 00 01 15 c4 24 e8 40 00  1d+09:50:44.989  READ FPDMA QUEUED

Error 71 [22] occurred at disk power-on lifetime: 1099 hours (45 days + 19 hours)
  When the command that caused the error occurred, the device was doing SMART Offline or Self-test.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  40 -- 51 00 00 00 00 78 39 af 00 40 00  Error: UNC at LBA = 0x7839af00 = 2017046272

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  60 00 a8 00 08 00 00 78 39 af a8 40 00  1d+03:04:06.588  READ FPDMA QUEUED
  60 00 a8 00 00 00 00 78 39 af 00 40 00  1d+03:04:06.588  READ FPDMA QUEUED
  60 00 a0 00 08 00 00 78 39 ae 60 40 00  1d+03:04:06.577  READ FPDMA QUEUED
  60 00 a8 00 00 00 00 78 39 ad b8 40 00  1d+03:04:06.577  READ FPDMA QUEUED
  60 00 a8 00 08 00 00 78 39 ad 10 40 00  1d+03:04:06.535  READ FPDMA QUEUED

Error 70 [21] occurred at disk power-on lifetime: 1080 hours (45 days + 0 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  40 -- 51 00 00 00 00 28 27 35 28 40 00  Error: UNC at LBA = 0x28273528 = 673658152

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  60 00 d0 00 08 00 00 28 27 36 00 40 00	 07:48:14.170  READ FPDMA QUEUED
  60 00 d8 00 00 00 00 28 27 35 28 40 00	 07:48:14.170  READ FPDMA QUEUED
  60 00 d8 00 00 00 00 28 27 34 50 40 00	 07:48:14.169  READ FPDMA QUEUED
  60 00 d0 00 08 00 00 28 27 33 80 40 00	 07:48:14.158  READ FPDMA QUEUED
  60 00 d8 00 00 00 00 28 27 32 a8 40 00	 07:48:14.158  READ FPDMA QUEUED

Error 69 [20] occurred at disk power-on lifetime: 1076 hours (44 days + 20 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  40 -- 51 00 00 00 00 a8 2b b0 38 40 00  Error: UNC at LBA = 0xa82bb038 = 2821435448

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  60 00 b8 00 08 00 00 a8 2b b0 e8 40 00	 03:23:48.845  READ FPDMA QUEUED
  60 00 b0 00 00 00 00 a8 2b b0 38 40 00	 03:23:48.845  READ FPDMA QUEUED
  60 00 98 00 08 00 00 a8 2b af a0 40 00	 03:23:48.834  READ FPDMA QUEUED
  60 00 c0 00 00 00 00 a8 2b ae e0 40 00	 03:23:48.833  READ FPDMA QUEUED
  60 00 c0 00 00 00 00 a8 2b ae 20 40 00	 03:23:48.832  READ FPDMA QUEUED

SMART Extended Self-test Log Version: 1 (1 sectors)
Num  Test_Description	Status				  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline	Completed without error	   00%	  1102		 -
# 2  Extended offline	Completed without error	   00%	   941		 -
# 3  Extended offline	Completed without error	   00%	   762		 -
# 4  Extended offline	Completed without error	   00%		20		 -

SMART Selective self-test log data structure revision number 1
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
	1		0		0  Not_testing
	2		0		0  Not_testing
	3		0		0  Not_testing
	4		0		0  Not_testing
	5		0		0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

SCT Status Version:				  3
SCT Version (vendor specific):	   258 (0x0102)
SCT Support Level:				   1
Device State:						Active (0)
Current Temperature:					38 Celsius
Power Cycle Min/Max Temperature:	 38/43 Celsius
Lifetime	Min/Max Temperature:	 25/45 Celsius
Under/Over Temperature Limit Count:   0/0
Vendor specific:
01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

SCT Temperature History Version:	 2
Temperature Sampling Period:		 1 minute
Temperature Logging Interval:		1 minute
Min/Max recommended Temperature:	  0/60 Celsius
Min/Max Temperature Limit:		   -41/85 Celsius
Temperature History Size (Index):	478 (406)

Index	Estimated Time   Temperature Celsius
407	2017-08-28 01:35	38  *******************
...	..( 48 skipped).	..  *******************
456	2017-08-28 02:24	38  *******************
457	2017-08-28 02:25	41  **********************
...	..(305 skipped).	..  **********************
285	2017-08-28 07:31	41  **********************
286	2017-08-28 07:32	40  *********************
...	..( 44 skipped).	..  *********************
331	2017-08-28 08:17	40  *********************
332	2017-08-28 08:18	39  ********************
...	..( 26 skipped).	..  ********************
359	2017-08-28 08:45	39  ********************
360	2017-08-28 08:46	38  *******************
...	..( 45 skipped).	..  *******************
406	2017-08-28 09:32	38  *******************

SCT Error Recovery Control:
		   Read:	 70 (7.0 seconds)
		  Write:	 70 (7.0 seconds)

Device Statistics (GP Log 0x04)
Page  Offset Size		Value Flags Description
0x01  =====  =			   =  ===  == General Statistics (rev 2) ==
0x01  0x008  4			  19  ---  Lifetime Power-On Resets
0x01  0x010  4			1138  ---  Power-on Hours
0x01  0x018  6	 26384963078  ---  Logical Sectors Written
0x01  0x020  6	   134865765  ---  Number of Write Commands
0x01  0x028  6	182479148242  ---  Logical Sectors Read
0x01  0x030  6	   861594856  ---  Number of Read Commands
0x03  =====  =			   =  ===  == Rotating Media Statistics (rev 1) ==
0x03  0x008  4			1136  ---  Spindle Motor Power-on Hours
0x03  0x010  4			 875  ---  Head Flying Hours
0x03  0x018  4			  45  ---  Head Load Events
0x03  0x020  4			   0  ---  Number of Reallocated Logical Sectors
0x03  0x028  4			   0  ---  Read Recovery Attempts
0x03  0x030  4			   0  ---  Number of Mechanical Start Failures
0x04  =====  =			   =  ===  == General Errors Statistics (rev 1) ==
0x04  0x008  4			  76  ---  Number of Reported Uncorrectable Errors
0x04  0x010  4			   0  ---  Resets Between Cmd Acceptance and Completion
0x05  =====  =			   =  ===  == Temperature Statistics (rev 1) ==
0x05  0x008  1			  38  ---  Current Temperature
0x05  0x010  1			  40  ---  Average Short Term Temperature
0x05  0x018  1			  39  ---  Average Long Term Temperature
0x05  0x020  1			  45  ---  Highest Temperature
0x05  0x028  1			  32  ---  Lowest Temperature
0x05  0x030  1			  43  ---  Highest Average Short Term Temperature
0x05  0x038  1			  34  ---  Lowest Average Short Term Temperature
0x05  0x040  1			  39  ---  Highest Average Long Term Temperature
0x05  0x048  1			  38  ---  Lowest Average Long Term Temperature
0x05  0x050  4			   0  ---  Time in Over-Temperature
0x05  0x058  1			  60  ---  Specified Maximum Operating Temperature
0x05  0x060  4			   0  ---  Time in Under-Temperature
0x05  0x068  1			   0  ---  Specified Minimum Operating Temperature
0x06  =====  =			   =  ===  == Transport Statistics (rev 1) ==
0x06  0x008  4			  28  ---  Number of Hardware Resets
0x06  0x010  4			   0  ---  Number of ASR Events
0x06  0x018  4			   0  ---  Number of Interface CRC Errors
								|||_ C monitored condition met
								||__ D supports DSN
								|___ N normalized value

SATA Phy Event Counters (GP Log 0x11)
ID	  Size	 Value  Description
0x0001  2			0  Command failed due to ICRC error
0x0002  2			0  R_ERR response for data FIS
0x0003  2			0  R_ERR response for device-to-host data FIS
0x0004  2			0  R_ERR response for host-to-device data FIS
0x0005  2			0  R_ERR response for non-data FIS
0x0006  2			0  R_ERR response for device-to-host non-data FIS
0x0007  2			0  R_ERR response for host-to-device non-data FIS
0x0008  2			0  Device-to-host non-data FIS retries
0x0009  2			0  Transition from drive PhyRdy to drive PhyNRdy
0x000a  2			1  Device-to-host register FISes sent due to a COMRESET
0x000b  2			0  CRC errors within host-to-device FIS
0x000d  2			0  Non-CRC errors within host-to-device FIS
0x000f  2			0  R_ERR response for host-to-device data FIS, CRC
0x0012  2			0  R_ERR response for host-to-device non-data FIS, CRC
0x8000  4	   238278  Vendor specific

Last edited: Aug 28, 2017

  • #77

Considered trying to update the firmware on the HDs?

  • #78

Considered trying to update the firmware on the HDs?

The firmware on /dev/da3 is 82.00A82 according to SMART. Is there a newer firmware? All I could find was WD Red SMART load/Unload utility 3.5″ Drive, but the release date is 12/20/2013 (My drive is probably newer), and the page does not seem to mention the firmware version for comparison?

Update:

Looking around for firmware updates, I found this post which may be relevant to me. What do you (eager readers) think?

Code:

My problem is that the drives are not behaving the same (APM wise) with the
same WDidle3 settings and it can cause sync problems when different bios
drives are sitting on the same SAS lane. Therefore, I had to separate the
drives to different SAS ports and it seems to working stable now (FreeBSD).

Last edited: Aug 28, 2017

  • #80

Perhaps you should tabulate which drives in which bays have which firmware?

Источник

Понравилась статья? Поделить с друзьями:
  • Get s m a r t command error reading s m a r t перевод
  • Get proxy error
  • Get play link error
  • Get pdo error message
  • Get last error code