I certainly don't know if this is a freeswitch specific issue, but we have several systems in the field running freeswitch on top of CentOS 5.5 and others on Ubuntu 10.04. Some of the systems, after a weeks of performance, go into a read only state on the hard drive, at which point, a restart is the only fix. After the reboot, the drive seems OK - I've run smartctl to verify that all seems good. Sometimes, in readonly mode, it even gives an I/O error when trying to execute the /sbin/shutdown and we have to manually reboot. At this point, I am trying to figure out what may be the root cause. Has anyone else experienced something this? Any suggestions? Thanks, Troy
Sounds like hardware level, not FS-specific. Every time I've had this it's been a bad drive or bad motherboard/raid controller. Try checking the drive for bad blocks. Since it's multiple servers perhaps it could also be a driver bug. Regards, -Steve On 15 October 2010 19:53, Troy Anderson <freeswitch@tlainvestments.com> wrote:
I certainly don't know if this is a freeswitch specific issue, but we have several systems in the field running freeswitch on top of CentOS 5.5 and others on Ubuntu 10.04. Some of the systems, after a weeks of performance, go into a read only state on the hard drive, at which point, a restart is the only fix. After the reboot, the drive seems OK - I've run smartctl to verify that all seems good. Sometimes, in readonly mode, it even gives an I/O error when trying to execute the /sbin/shutdown and we have to manually reboot.
At this point, I am trying to figure out what may be the root cause. Has anyone else experienced something this? Any suggestions?
Thanks, Troy
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Troy Anderson wrote:
At this point, I am trying to figure out what may be the root cause. Has anyone else experienced something this? Any suggestions? Your best bet on this is to look at the log file (/var/log/messages and among others in the same sub-dir) and see if you can identify what triggered the HD into a RO mode.
----- don't and stop are the ONLY two 4-letter words considered offensive to men, but not when used together. -- View this message in context: http://freeswitch-users.2379917.n2.nabble.com/Read-only-filesystem-tp5640425... Sent from the freeswitch-users mailing list archive at Nabble.com.
Many times there isn't anything suspect in any logs, but recently on one system, I saw the following (repeated many times over time): Oct 13 19:40:13 localhost kernel: ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen Oct 13 19:40:13 localhost kernel: ata2.00: cmd e5/00:00:00:00:00/00:00:00:00:00/00 tag 0 Oct 13 19:40:13 localhost kernel: res 40/00:01:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout) Oct 13 19:40:13 localhost kernel: ata2.00: status: { DRDY } Oct 13 19:40:17 localhost kernel: ata2: soft resetting link Oct 13 19:40:17 localhost kernel: ata2.00: configured for UDMA/33 Oct 13 19:40:17 localhost kernel: sd 1:0:0:0: timing out command, waited 7s Oct 13 19:40:17 localhost kernel: ata2: EH complete Oct 13 19:40:17 localhost kernel: SCSI device sda: 234441648 512-byte hdwr sectors (120034 MB) Oct 13 19:40:17 localhost kernel: sda: Write Protect is off Oct 13 19:40:18 localhost kernel: SCSI device sda: drive cache: write back Can anyone help me with pointers on how to determine what they mean? On Oct 15, 2010, at 4:34 PM, mazilo wrote:
Troy Anderson wrote:
At this point, I am trying to figure out what may be the root cause. Has anyone else experienced something this? Any suggestions? Your best bet on this is to look at the log file (/var/log/messages and among others in the same sub-dir) and see if you can identify what triggered the HD into a RO mode.
----- don't and stop are the ONLY two 4-letter words considered offensive to men, but not when used together. -- View this message in context: http://freeswitch-users.2379917.n2.nabble.com/Read-only-filesystem-tp5640425... Sent from the freeswitch-users mailing list archive at Nabble.com.
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
See below. Steve on iPhone On 16 Oct 2010, at 06:59, Troy Anderson <freeswitch@tlainvestments.com> wrote:
Many times there isn't anything suspect in any logs, but recently on one system, I saw the following (repeated many times over time):
Oct 13 19:40:13 localhost kernel: ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen Oct 13 19:40:13 localhost kernel: ata2.00: cmd e5/00:00:00:00:00/00:00:00:00:00/00 tag 0 Oct 13 19:40:13 localhost kernel: res 40/00:01:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
The driver asked the drive to do something and got no response from the drive. The numbers tell you what, but is only really useful if you're a developer of the driver.
Oct 13 19:40:13 localhost kernel: ata2.00: status: { DRDY } Oct 13 19:40:17 localhost kernel: ata2: soft resetting link
Chipset is disconnecting and reconnecting the drive, since that can sometimes fix a problem by resetting the drive.
Oct 13 19:40:17 localhost kernel: ata2.00: configured for UDMA/33
Fallen speed back to UDMA 33 (slowest UDMA), since some problems are caused by the drive running faster than the drive/cable can cope with.
Oct 13 19:40:17 localhost kernel: sd 1:0:0:0: timing out command, waited 7s
Something timed out - not sure if this is the reset or the failed action that triggered the reset.
Oct 13 19:40:17 localhost kernel: ata2: EH complete Oct 13 19:40:17 localhost kernel: SCSI device sda: 234441648 512-byte hdwr sectors (120034 MB) Oct 13 19:40:17 localhost kernel: sda: Write Protect is off Oct 13 19:40:18 localhost kernel: SCSI device sda: drive cache: write back
Drive is back.
Can anyone help me with pointers on how to determine what they mean?
On Oct 15, 2010, at 4:34 PM, mazilo wrote:
Troy Anderson wrote:
At this point, I am trying to figure out what may be the root cause. Has anyone else experienced something this? Any suggestions? Your best bet on this is to look at the log file (/var/log/messages and among others in the same sub-dir) and see if you can identify what triggered the HD into a RO mode.
----- don't and stop are the ONLY two 4-letter words considered offensive to men, but not when used together. -- View this message in context: http://freeswitch-users.2379917.n2.nabble.com/Read-only-filesystem-tp5640425... Sent from the freeswitch-users mailing list archive at Nabble.com.
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Sounds like a firmware issue with either the Drive Controller or the hard drive itself. On Sat, Oct 16, 2010 at 7:37 AM, Steven Ayre <steveayre@gmail.com> wrote:
See below.
Steve on iPhone
On 16 Oct 2010, at 06:59, Troy Anderson <freeswitch@tlainvestments.com> wrote:
Many times there isn't anything suspect in any logs, but recently on one system, I saw the following (repeated many times over time):
Oct 13 19:40:13 localhost kernel: ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen Oct 13 19:40:13 localhost kernel: ata2.00: cmd e5/00:00:00:00:00/00:00:00:00:00/00 tag 0 Oct 13 19:40:13 localhost kernel: res 40/00:01:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
The driver asked the drive to do something and got no response from the drive. The numbers tell you what, but is only really useful if you're a developer of the driver.
Oct 13 19:40:13 localhost kernel: ata2.00: status: { DRDY } Oct 13 19:40:17 localhost kernel: ata2: soft resetting link
Chipset is disconnecting and reconnecting the drive, since that can sometimes fix a problem by resetting the drive.
Oct 13 19:40:17 localhost kernel: ata2.00: configured for UDMA/33
Fallen speed back to UDMA 33 (slowest UDMA), since some problems are caused by the drive running faster than the drive/cable can cope with.
Oct 13 19:40:17 localhost kernel: sd 1:0:0:0: timing out command, waited 7s
Something timed out - not sure if this is the reset or the failed action that triggered the reset.
Oct 13 19:40:17 localhost kernel: ata2: EH complete Oct 13 19:40:17 localhost kernel: SCSI device sda: 234441648 512-byte hdwr sectors (120034 MB) Oct 13 19:40:17 localhost kernel: sda: Write Protect is off Oct 13 19:40:18 localhost kernel: SCSI device sda: drive cache: write back
Drive is back.
Can anyone help me with pointers on how to determine what they mean?
On Oct 15, 2010, at 4:34 PM, mazilo wrote:
Troy Anderson wrote:
At this point, I am trying to figure out what may be the root cause. Has anyone else experienced something this? Any suggestions? Your best bet on this is to look at the log file (/var/log/messages and among others in the same sub-dir) and see if you can identify what triggered the HD into a RO mode.
----- don't and stop are the ONLY two 4-letter words considered offensive to men, but not when used together. -- View this message in context: http://freeswitch-users.2379917.n2.nabble.com/Read-only-filesystem-tp5640425... Sent from the freeswitch-users mailing list archive at Nabble.com.
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Jeffrey Leung suggested:
Sounds like a firmware issue with either the Drive Controller or the hard drive itself.
This can also happen with consumer-grade drives when they detect and remap a bad block (or chunk of bad blocks). This is especially problematic with RAID. The drive is busy fixing the problem, and it appears to the driver that the drive is no longer responding. If this happens frequently, it might be an indication that the drive is failing. The drive could also be overheating. mose
Have you ruled out hard drive failure using smartctl? Do a long selftest and review the results. Also, swap out your SATA cables or at least ensure they are fully seated. Finally, check to see if there are known issues with the SATA interface you are using (is it onboard, add-in card, raid card?) and the linux kernel you have installed. On Sat, Oct 16, 2010 at 12:59 AM, Troy Anderson < freeswitch@tlainvestments.com> wrote:
Many times there isn't anything suspect in any logs, but recently on one system, I saw the following (repeated many times over time):
Oct 13 19:40:13 localhost kernel: ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen Oct 13 19:40:13 localhost kernel: ata2.00: cmd e5/00:00:00:00:00/00:00:00:00:00/00 tag 0 Oct 13 19:40:13 localhost kernel: res 40/00:01:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout) Oct 13 19:40:13 localhost kernel: ata2.00: status: { DRDY } Oct 13 19:40:17 localhost kernel: ata2: soft resetting link Oct 13 19:40:17 localhost kernel: ata2.00: configured for UDMA/33 Oct 13 19:40:17 localhost kernel: sd 1:0:0:0: timing out command, waited 7s Oct 13 19:40:17 localhost kernel: ata2: EH complete Oct 13 19:40:17 localhost kernel: SCSI device sda: 234441648 512-byte hdwr sectors (120034 MB) Oct 13 19:40:17 localhost kernel: sda: Write Protect is off Oct 13 19:40:18 localhost kernel: SCSI device sda: drive cache: write back
Can anyone help me with pointers on how to determine what they mean?
On Oct 15, 2010, at 4:34 PM, mazilo wrote:
Troy Anderson wrote:
At this point, I am trying to figure out what may be the root cause.
Has
anyone else experienced something this? Any suggestions? Your best bet on this is to look at the log file (/var/log/messages and among others in the same sub-dir) and see if you can identify what triggered the HD into a RO mode.
----- don't and stop are the ONLY two 4-letter words considered offensive to men, but not when used together. -- View this message in context: http://freeswitch-users.2379917.n2.nabble.com/Read-only-filesystem-tp5640425... Sent from the freeswitch-users mailing list archive at Nabble.com.
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- -Rupa
Rupa Schomaker wrote:
Also, swap out your SATA cables or at least ensure they are fully seated.
That's a good suggestion. I had purchased some cheap SATA cables. They were fine at the beginning and over the next year or two they started failing. The drive would just drop out of a RAID array without warning, because it appeared to fail. Testing the drive didn't show any errors. Reseating the cables didn't help. Replacing the cable was the next thing to try, and it fixed the problem. I didn't track down why the cables failed, because it wasn't worth the time. The connector might have oxidized or something. mose
It could also be bad soldering on either ends of the connectors, or lead-free solder growing 'legs' and causing a short somewhere. I wished they stopped marketing RoHS products... There's a reason why to this date, no sane avionics technician will ever use RoHS solder on an airplane. On Sat, Oct 16, 2010 at 10:01 AM, Russell Mosemann <Russell.Mosemann@cune.org> wrote:
Rupa Schomaker wrote:
Also, swap out your SATA cables or at least ensure they are fully seated.
That's a good suggestion. I had purchased some cheap SATA cables. They were fine at the beginning and over the next year or two they started failing. The drive would just drop out of a RAID array without warning, because it appeared to fail. Testing the drive didn't show any errors. Reseating the cables didn't help. Replacing the cable was the next thing to try, and it fixed the problem. I didn't track down why the cables failed, because it wasn't worth the time. The connector might have oxidized or something.
mose
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
participants (6)
-
Jeffrey Leung
-
mazilo
-
Rupa Schomaker
-
Russell Mosemann
-
Steven Ayre
-
Troy Anderson