Server: log of incidents

2016 Aug 25 Hardware upgrade 5mn Built-in thernet card seemed flaky, replaced by a PCIe one
2015 Dec 13 Hardware failure 10mn System disk not responding. Was a bad contact in the cable
2015 Nov 19 Power down 6h Electricity was down a short time, but the server network failed on unattended restart due to some DHCP issue with the ADSL box.
2014 Jul 16 Phone line failure 10h Wall phone socket died...
2014 Mar 11 Hardware upgrade 3h CPU, RAM, Disk massive upgrade
2013 Aug 12 OS upgrade 10mn To Debian v7.0 Wheezy
2012 Jan 26 ADSL Upgrade 1h From the Freebox v5 to the Freebox v6
2011 May 24 ADSL Down 17h Phone line was down 2011-05-24 13h30 to 2011-05-25 06h00
2011 Fev 19 Kernel upgrade 2h As part of upgrade to Debian 6.0 "squeeze"
2010 Jun 06 Kernel upgrade 5mn
2009 Dec 29 ADSL Down 13h ISP(Free) major routing problems, 21h50 to 10h50
2009 Oct 03 Kernel upgrade 5mn
2009 Aug 17 ADSL Down 3h ISP(Free) server problems, 18h to 21h
2009 Feb 15 OS Upgrade 1mn Upgraded to the new Debian 5.0, "Lenny".
2008 Dec 06 Hardware upgrade 30mn Memory bars came back from warranty, installed back. Upgraded to 4GB RAM.
2008 Oct 01 Hardware failure 8 hours A Memory bar failed (OCZ) and froze the server. removed at 8h00 on Oct 02.
2008 Sep 28 Hardware upgrade 2h Upgraded to an Intel Core 2 duo E8400 (3G), 2G ram, Gigabyte EG43m-SH2 motherboard, AHCI activated.
2008 Sep 09 Hardware failure 3 days Power supply failed as I was away from home. It was my mistake: I mistakenly left a mediocre power supply brand (Silverstone) in the server, thinking I had upgraded it with the fabulous Seasonic ones, like all my other machines ;-( Down from 2008-09-08.10:23 to 2008-09-12.13:45
2008 Sep 04 ADSL Down 2h23 My France Telecom phone line went down, cutting my ADSL access... From 08h51 to 11h14
2008 Aug 27 ADSL Down 22h My France Telecom phone line went down, cutting my ADSL access... From 2008-08-27.15:11 to 2008-08-28.12:53
2008 Jul 12 Hardware upgrade 20h Upgraded to 4x 640G SATAII Samsung disks (2 platters each to reduce vibrations), and had to change the Motherboard to an Asrock 939N68PV-GLAN to be able to use them. Had to use a development debian kernel (2.6.25) to handle the chipset, though, as the standard 2.6.18 saw only the first disk.
2008 May 18 Hardware upgrade 6h Upgraded to an AMD64 3500 CPU, 2G Ram, gigabit ethernet, with tthe samw 4x 400G Hitacgi IDE disks. Upgrade went bad, took me time to detect the cause (a bad IDE disk cable). Then had some isssues on how to configure the gigabit network card.
2007 Nov 01 Kernel upgrade 2h Upgraded to debian Etch with standard kernel. Had to battle as my motherboard was too old to support grub, reverted to lilo.
2007 Oct 09 Hardware failure 10mn Spare network card failed (D-Link DFE-530TX). replaced with another spare.
2007 Oct 05 Modem upgrade 10mn Migration from Freebox v3 to freebox v5 HD.
2007 Sep 13 ADSL down 10mn No more ADSL. Had to hard reboot the modem. Outage from 09h08 to 09h18
2007 Sep 12 ADSL down 20mn No more ADSL. Had to hard reboot the modem. Outage from 09h18 to 09h40
2007 Jul 09 ADSL down 2h No more ADSL. Had to hard reboot the modem. Outage from 05h48 to 08h00
2007 Apr 23 ADSL down 36h Hardware problem at my Free DSLAM, ADSL down from 2007-04-23 15h45 to 2007-04-25 09h00
2007 Apr 03 UPS died 2h30 Power outage due to the ... UPS that died (after 5 1/2 years of use). Down from 17h00 to 19h40
2007 Jan 08 Hardware crash 6h30 Machine crashed at 02h30. Restarted at 09h00. Unknown cause (bad RAM? bad block in swap?)
2006 Dec 31 DSN server problems3h dnsmasq was down this morning. I set up a cron job to restart it everyday to prevent this to happen again
2006 Dec 17 Hardware upgrade 15mn Power supply (LC Power with 14cm fan) was noisy. Changed to a Seasonic S12 500W
2006 Dec 11 DSN server problems1h The dnsmasq I was using as LAN DNS server misbehaved suddenly. Turned out the Debian stable (sarge) version, 2.22 has some bugs, especially with Windows clients, had to switch to 2.35
2006 Dec 08 Power failure 3h Due to big storms, lots of electric failures, total downtime: 3 hours.
2006 Dec 05 ADSL modem reboot5mn ADSL was super slow, and tracroute showed that it was close to the modem. Rebooted the freebox and everything was back to normal
2005 Jan 18 ADSL disconnection 15mn paquets werent routed. a freebox reboot fixed it (or it was repaired at the same time)
2005 Jan 06 ADSL disconnection 36h ADSL was cut Friday 14:00 and came back in a crippled way (slower) Saturday 7th, 20:00. Full speed was resumed on Tue 10th morning. I now use Free's "freebox" V4 model
2005 Nov 29 ADSL disconnection 10 days On Wed Nov 29th, no more ADSL! It was a strange problem, that puzzled the ISP. Fortunately (but perhaps the bug was due to some preparation of this move), On Dec 6th I was notified by Free that I was being "Degrouped", and miracle! on Fri Dec 9th I was back with adsl access with a better connection (no more France Telecom messing with my packets), at 4000/1000 bps in dl/ul with Free. My IP address changed also in the process.
2005 Sep 07 Kernel upgrade 2mn to 2.4.31
2005 Aug 29 Debian upgrade 1h To the new "stable" debian, aka "sarge" instead of "woody"
2005 Mar 22 Hardware Maintainance 10mn CPU was a bit hot (48C), I removed the old pad and put thermal compound to gain nearly 10C
2005 Mar 17 Hardware + kernel upgrade 2h Upgraded to an used Motherboard (ASUS TUSL2-C) + CPU (celeron 1.3Ghz), still the same 512M ram (the max of what is supported by the MB). Kernel upgraded to 2.4.29.
2004 Nov 21 Kernel upgrade 2mn to 2.4.28
2004 Oct 19 Disk upgrade 1/2h added one Hitachi 160g IDE (gained by reorganizing backups).
2004 Oct 17 Disk upgrade 4h upgrade: 4 Seagate 160g -> 3 Hitachi 400g IDE. I recommend using the Hard Disk Upgrade Mini How-To.
2004 Oct 16 Kernel upgrade 2mn To 2.4.27. Uses now ext3 journal on disks.
2004 Sep 15 Disk failure 3h A disk failed (non-recoverable bad block) on a 160G seagate 7002.7, freezing the machine. Disk (hdb, /temp3) replaced by a 160G hitachi. Down 6h45 -> 9h45. Actually I could recover the disk thanks to smartmontools, a mandatory read!
2004 Jul 14 Crash 1h Secondary IDE controler was in error, why? (I guess it was the bad blocks in an inode table on a disk, see above) Down 6h30->7h30.
2004 May 01 Kernel upgrade 1h to install lm-sensors-2.8.6 & i2c-2.8.4 for temperature monitoring
2004 Apr 25 Kernel upgrade 1h to 2.4.26, (from 2.2.20) but still in debian stable (to try to get rid of hardware crahses) - it worked! The previous "Hardware Crash" were thus software crashes due to the 2.2 kernel on large partitions.
2004 Apr 13 Hardware Crash 1h Kernel panic.
2004 Apr 10 Hardware Crash 30mn Kernel panic.
2004 Mar 28 Hardware Crash 1h Unexplained linux kernel crash (18h->19h)... either the motherboard is growing old, or the overclock is troublesome with 4 disks, or the power supply is flaky. I supressed the overclock, we'll see.
2004 Mar 26 Hardware Crash 1h Disk#4 failed. Some rest brought it back up again
2004 Mar 15 Hardware Crash 4h Unexplained linux kernel crash :-(
2004 Mar 09 ADSL upgrade 3mn Free upgraded my line to 1024/128
2004 Feb 25 Crash 20mn Experimenting with samba to read windows disks failed
2004 Feb 23 Disk upgrade 3h upgrade: 2 -> 4 Seagate barracuda 7200.7 160g, and new case: antec sonata + antec power supply
2003 Sep 14 Reboot 15mn Malfunctioning printer made the server freeze.
2003 Jun 21 Disk upgrade 4h 2 x Seagate barracuda IV 80g -> 2 x Seagate barracuda 7200.7 160g
2003 May 15 ADSL down 9h France Telecom messed with their DSLAM: down from 03h36 to 14h47
2003 Apr 19 ISP Switch 1s I now use for my ADSL access, with a fixed IP.
2003 Mar 27 Hardware problem 9h The power supply died in the night (01h30). Replaced (by a full-scale model, not the puny barebones model used before). Back up at 10h30.
2003 Mar 6 Move 2h I moved home in a new house, but downtime was minimal.
2002 Jul 24 ADSL problem4h Regional ADSL down from 10h to 12h.
2002 Jul 21 Reboot 2mn Failed attempt at hacking USB on my 2.2 for a device supported in 2.4 froze the kernel.
2002 Feb 24 DNS problem10h Wanadoo DNS servers down (again) from 4h00 to 16h00
2002 Feb 22 TC setup1mn Setting up IP traffic control via TBF to avoid choking the ADSL modem with outbound traffic. Used info from Linux Advanced Routing & Traffic Control HOWTO. Reboot just to be sure to have a clean IP stack after a lot of tweaking. For those interested, the magic command for sustaining full simultaneous upload and download on netissimo 1 is:
tc qdisc add dev ppp0 root tbf rate 15kbps latency 30ms burst 1504
See also:
2002 Feb 09 Disk firmware upgrade25mn Reboot (in DOS) to fix the noise that plagued my Seagate Barracuda ATA IV disks with the smscan utility. From 16h16 to 16h41.
2002 Feb 08 ADSL problem1h15 Regional ATM backbone down, from 12h45 to 15h03
2001 Dec 16 Kernel upgrade2h Hardware maintainance: Harddisks put into a grounded custom-built aluminium rack with ventilation tubes, rolled into automobile phonic foam, with a fan at the end: no more vibrations (not fixed to case), 5 degrees colder, and absolutely no noise (case was transmitting vibrations before). Flex factor case was ripped of the (now useless) floppy drive and the disk racks. Downtime: 13h30 to 15h30.
Added an InfraRed receiver (ASUS) + kernel support (hence recompilation)
2001 Dec 04 Kernel upgrade2mn Kernel upgrade (add scsi subsystem for tests of usb disk)
2001 Dec 02 Kernel upgrade2mn Kernel upgrade to 2.2.20
2001 Nov 30 Disk upgrade 5mn IBM 60g -> 2 x Seagate barracuda IV 80g
2001 Nov 29 ADSL problem4h30 Wanadoo disconnection, from Nov 28 20:35 to Nov 29 01:01
2001 Nov 3 Kernel upgrade 2mn recompilation to add QoS modules for traffic control (23h50)
2001 Sep 12 ADSL problem4h Modem was disconnected and could not reconnect to ISP (Wanadoo), although line sync was Ok. A manual reboot of the modem fixed it. Script checking connection enhanced to force reboot in this case.
2001 Sep 1 UPS install1h Installation of a MGE UPS ellipse (Onduleur). Downtime: 10h25 to 11h25.
2001 Aug 22 Kernel upgrade 2mn reconfiguration of the kernel to get sound (08h30)
2001 Aug 21 DNS problem4h IP changed, but a bug in my dynamic DNS updater to failed to update the DNS of from 05h00 to 09h00
2001 Aug 10 Memory upgrade 32mn 256 M => 512 M (in 2 dimms)
2001 Aug 01 Reboot 17mn unrelated electrical work in the house (new fridge :-)
2001 Jul 30 Web Server starts - as
2001 Jul 26 Disk upgrade 21h IBM 40g -> IBM 60g (60GXP)
2001 May 31 Server online - on ADSL, only client at first, no inbound connections allowed. Server is a Celeron 800 (overclocked by 12%), with an ATX Flex form factor motherboard (ASUS CUSI-FX), and a small Flex case. OS is a stable (2.2 potato) Debian Linux system.
Times are French times (GMT + 1)