Marco Peereboom wrote:
quoted text
>> I created it with bioctl, but my version is from a September 1 snapshot  
>> so it is before your fix.
> 
> There is a good chance that the hotspare does not work prior to that
> fix.  I'd say this explains what you see.
> 
>> The server ran flawlessly for 2 years now, and I'll bet it's been a year  
>> since I've even slid it forward enough in the rack to get the cover off.  
>>  Do cables go bad with use?
> 
> No but they might have been dislodged when you moved the server.
> 
>> I may try to replace the safte0 enclosure this weekend then.
> 
> It could be that it is simply hung (GEM chips tend to do that).  Power
> cycle the whole thing to see what happens.  If the GEM is hung there is
> a chance that the SCSI bus is hung too.  This can easily lead to the
> issues you have been seeing (I have seen this in the past on other
> enclosures with a GEM chip).
> 

I power cycled the server after my users went home, checked the cables, 
and after about a hour's worth of hair-pulling nvram/disk configuration 
mismatches I finally got the system back up with sd0 in degraded mode 
and the other two optimal.

Brought the system up to the most recent snapshot and did a bioctl -H 
1:0 sd0 and the rebuild kicked off immediately.

jross@nirvana:/home/jross $ sudo bioctl -v -i ami0
Volume  Status               Size Device
  ami0 0 Degraded      72999763968 sd0     RAID1
       0 Failed                  0 0:9.0   safte0 <>
                                                  'unknown serial'
       1 Online        73403465728 0:10.0  safte0 <HITACHI 
HUS103073FL3800 SA1B>
                                                  'V3W09L5A0050B499004B'
  ami0 1 Online        72999763968 sd1     RAID1
       0 Online        73403465728 0:11.0  safte0 <HITACHI 
HUS103073FL3800 SA1B>
                                                  'V3W06MNA0050B4AD01D3'
       1 Online        73403465728 0:12.0  safte0 <HITACHI 
HUS103073FL3800 SA1B>
                                                  'V3W0A6VA0050B4A80C0C'
  ami0 2 Online        72999763968 sd2     RAID1
       0 Online        73403465728 1:4.0   safte1 <HITACHI 
HUS103073FL3800 SA1B>
                                                  'V3VZV2JA0050B4AX04C2'
       1 Online        73403465728 1:1.0   safte1 <HITACHI 
HUS103073FL3800 SA1B>
                                                  'V3W0726A0050B49W01CB'
  ami0 3 Unused        73407899648 0:13.0  safte0 <HITACHI 
HUS103073FL3800 SA1B>
                                                  'V3W093EA0050B44V0578'
  ami0 4 Unused        73407899648 1:0.0   safte1 <HITACHI 
HUS103073FL3800 SA1B>
                                                  'V3W07PSA0050B4710207'

jross@nirvana:/home/jross $ sudo bioctl -H 1:0 sd0
jross@nirvana:/home/jross $ sudo bioctl -v -i ami0
Volume  Status               Size Device
  ami0 0 Rebuild       72999763968 sd0     RAID1 0% done
       0 Rebuild       73403465728 1:0.0   safte1 <HITACHI 
HUS103073FL3800 SA1B>
                                                  'V3W07PSA0050B4710207'
       1 Online        73403465728 0:10.0  safte0 <HITACHI 
HUS103073FL3800 SA1B>
                                                  'V3W09L5A0050B499004B'
  ami0 1 Online        72999763968 sd1     RAID1
       0 Online        73403465728 0:11.0  safte0 <HITACHI 
HUS103073FL3800 SA1B>
                                                  'V3W06MNA0050B4AD01D3'
       1 Online        73403465728 0:12.0  safte0 <HITACHI 
HUS103073FL3800 SA1B>
                                                  'V3W0A6VA0050B4A80C0C'
  ami0 2 Online        72999763968 sd2     RAID1
       0 Online        73403465728 1:4.0   safte1 <HITACHI 
HUS103073FL3800 SA1B>
                                                  'V3VZV2JA0050B4AX04C2'
       1 Online        73403465728 1:1.0   safte1 <HITACHI 
HUS103073FL3800 SA1B>
                                                  'V3W0726A0050B49W01CB'
  ami0 3 Unused        73407899648 0:13.0  safte0 <HITACHI 
HUS103073FL3800 SA1B>
                                                  'V3W093EA0050B44V0578'
  ami0 4 Unused        73407899648 1:0.0   safte1 <HITACHI 
HUS103073FL3800 SA1B>
                                                  'V3W07PSA0050B4710207'

So it looks like 0:9 is still a bad disk--I'll try to check that out 
tomorrow in the new server.

The cables were all tight connections but I wonder about the quality. 
They are both loose bundled round cables with 5 connectors but I'm only 
using one for the safte.  They were what I  had at the time, but I'd 
sure take an suggestions for a good vendor and to get some really good 
cables.

Thanks again,

Jeff