SysAdmin Day Contest Winner: Saving the Day
Reason #7959 to thank you sysadmin: Finding and implementing solutions at 1 am on New Year’s Day. Bejoy Mathews tells the tale:
SysAdmin to the Rescue
I’ll tell you a story which saved one of my old organizations a great deal of data loss. This was on a New Year’s Eve. I handled support for PeopleSoft on RedHat Linux—OS/Web/App. Though DB was on Oracle, I supported the OS part. The main database administrator did not backup for a week, and the data was crucial for the whole organization’s daily basic requirements—especially HR. The storage device cracked one of the hard disk drives some time earlier, and since that was not monitored properly we did not put the proper HDD in in time. One more HDD of the three went down and the database crashed. This was a huge question mark for the team of sysadmins and database admins. Many did not turn up due to fear of losing their jobs. The storage company told us the only way to correct this was to format the whole data on the storage. We all were really worried—we had already crossed the Dec 31st 11:59pm line, and it was January 1st, sitting in the Data Centre at 1-2 am in the morning. Even though the storage company advised us to format, we thought of using one of their own web-based utilities to see if we could recover. That worked really well—we were able to do something that even the storage owners couldn’t guide us to. I did not have to format it. And once the storage was back on line, I was able to retrieve data through Dec 30th at least, with only one day of lost data. That brought smiles to everyone and brought many out of their hiding places. Sysadmin skilled experience was a real help, when all seemed lost! Servers/Apps/DBs all had their respective halos on. It was truly a Happy New Year for me at least!
Check back later this week for the last of our sysadmin stories.
Related posts:
SysAdmin Day Approacheth: Are you Ready?
The Big Red Button: The Dream of a SysAdmin
System Administrator Appreciation Day: Thank You, Thank You, Thank You
1 Comment
Live Every Day like it's SysAdmin Day | Puppet Labs
[...] SysAdmin Day Contest Winner: Saving the Day Tweet [...]