More hardware, more fixes and some freeze.

"Every day you may make progress. Every step may be fruitful. Yet there will stretch out before you an ever-lengthening, ever-ascending, ever-improving path. You know you will never get to the end of the journey. But this, so far from discouraging, only adds to the joy and glory of the climb."

Sir Winston Churchill said some interesting things and I believe this one describes our current situation to a certain extent. Today is our freeze day for Kali here at CCP, yesterday we deployed about 15 hotfixes to Tranquility and we have more hardware coming.

But we're nowhere near done yet and we're sure that the hurdles we face will be no less of a challenge than the ones already passed - or the ones we're trying to pass right now.

But, more hardware?

Yeah, as we mentioned before, we have boatloads of hardware coming, everything from the extreme RAMSAN to new network equipment and internet connections all affect the performance on Tranquility.

Not only that, we're always ordering something new and we don't always mention everything - but it's days like today which are fun to mention, because we have some big batches of hardware coming from IBM over the next months, resulting in more than 30 servers being added to Tranquility (that's 60 more nodes).

But you're fixing too, right?

Yeah, we continue to fix and deploy hotfixes to Tranquility. Often, these are fixes which you shouldn't really notice, where we're gaining small performance here or addressing a certain issue there, but we also deploy tools. Yesterday was one of those days, where we wished we had deployed the tools before the weekend because we would have gotten some good data from the weekends node crashes.

We do have a rule though to never deploy on Fridays, we don't take the risk anymore of ruining weekends, because there is always a risk involved with deploying fixes. Some problems simply don't show themselves before Tranquility and the most extreme case being that Tranquility simply doesn't start again after a fix.

Those bastards, they killed Tranquility

On the other hand, we also have fixes which we can't deploy to Tranquility except as part of major relases. These usually rely on some very fundamental changes in the whole EVE infrastructure. Kali is a good example of that, where Redundancy mentioned one of the low-level client improvements in one of his blogs.

Ohnoes, not more bugs!

However, this is a double-edged sword. Kali by it's simple nature has a lot of new and changed code in it which can make things go wrong. This is one of the reasons we're adding a big chunk of hardware now, to increase the headroom available till we are able to address the problems which only show themselves on Tranquility.

But to address the problems that emerge outside Tranquility, Kali will be entering public testing next week on Singularity and will stay there until we feel comfortable about Kali being ready for Tranquility. We'd appreciate any help you can offer in flushing out the issues on Singularity the next weeks.

To sum it up, in Samuel L. Jackson's immortal words:

"We're on the m***********"