Crisis update from EVE CSS (Updated)

Hello everyone.

It's been a trying day for just about anyone involved with EVE, be it the players, support staff or the devs. Here's a very brief synopsis of what has been going on behind the scenes today. Omitted are screams of despair and curses that would turn your hair white.

Just before today's downtime, we had confirmed that there was a problem with the server, resulting in stuck problems, lag and other symptoms. Our GM tools also lagged heavily, but we had hopes that the downtime maintenance would resolve this problem. This was not the case.

After downtime, we lost all connection to our petition answering tools and as reports of an ever deepening crisis situation began to flood in, we could only monitor it and work with the devs to try and isolate the problem. As of now, work is still ongoing and we do not have any confirmed diagnosis or ETA on a fix. We do have access to the petitions interface and will be putting all our effort into clearing the queue. Please note that there will be delays in answering your ingame petitions and emails, but as always, we will answer everyone as soon as we possibly can.

We sincerely apologize to everyone for the inconvenience and assure you that we have all hands on deck, working to fix the problem. Thank you for your patience and understanding.

Update: We eventually traced the fault to a hardware malfunction, which was causing a number of our systems to start failover procdures, removing nodes from the cluster. We should be out of harms way but continue to monitor the situation.