A night to remember...

Sigh, please excuse if this blog is a little on the short and tired sounding side. We've been up for a while now as you may have noticed upgrading the db hardware among other things. As of this writing EVE is up again and so far so good. I´ll explain in more detail tomorrow but just thought I´d post a few lines to share some details on what happened.

What we were trying to accomplish was replacing a set of scsi diskboxes with a fiber storage server on the db. The migration went according to plan at first, somewhat late but considering all the steps involved the original estimate was way to optimistic. When restarting the db shortly after four o´clock performance was frankly abyssmal. After a lengthy session of debugging we decided to close down the servers and rearrange the setup.

To make a long story short the root of the problems was three things. The switch in front of the db was wrongly configured causing degraded network performance. The number of drives in the fiber box handling db io were fever than on the scsi box. We doubled the number of drives. And then we replaced drivers on the fiber box with another version boosting read performance fivefold. Now time will tell how this works but so far so good. I will blog some more tomorrow.