server outage, 7/8/21

This is the general discussion area. Everyone is welcome, but you must register to post.

Moderator: RLG MGMT Team

Post Reply
Hammer
Posts: 5217
Joined: 11 May 2005, 14:50

server outage, 7/8/21

Post by Hammer »

Most of you may not have noticed but I know Softball/Staark and Gator did (while getting Softball prepped to join us in DCS!).

The datacenter where the servers are hosted had a power outage yesterday starting at about 8PM Pacific. It was quite an outage in a business area, and the generators did not kick in like they should have (personally I think when adding a 3rd generator recently they messed things up and did not properly test).

I was the first one in my company to know about it thanks to Softball!

We started working immediately to assess the situation and plan the bring-up once power came back online. Power came back online about 10PM and myself and another starting working remotely to evaluate the damage while a 3rd waiting with an angry mob outside the data center door to get in (they did not let anyone in as all cabinet doors were open due to the chilled water plant being offline and too warm to have closed cabinets). We did have everything up successfully and running mostly by 1AM with some issue systems and switches taking the rest of the night to repair/bring up. Needless to say I worked all night with no sleep until it was time to prep for the Naval Retirement Ceremony I was invited to today.
Helmut
Hammer
Posts: 5217
Joined: 11 May 2005, 14:50

Re: server outage, 7/8/21

Post by Hammer »

The DCS server and teamspeak servers are back up, of course.
Helmut
User avatar
Grifter
Posts: 2516
Joined: 30 Jun 2002, 07:02

Re: server outage, 7/8/21

Post by Grifter »

That…..sucks. I’m so sorry that you were up all night working to repair the damage, and the night before the ceremony too. Man, you can’t catch a break lately. Thanks for getting the servers back up, although I can’t fly here on OGT beach.
Image
Hammer
Posts: 5217
Joined: 11 May 2005, 14:50

Re: server outage, 7/8/21

Post by Hammer »

Yah - kinda sucked and I am still a bit tired today. But I have been up for several days at a time with a few hour long catnaps (think military, simulated combat ops), so not too big a deal. :)
Helmut
Post Reply