No Downtime on Wednesday, December 4th

Visit this thread to discuss the No Downtime experiment.

Please visit this thead to provide feedback on the No Downtime experiment. (Thread locked until 11:00 UTC on Wednesday, December 4th).

Every day I think about downtime and wait for it to pass.

Downtime today is not like downtime was a few years ago when it was usually up to an hour, enough time for a long lunch, including dessert. These days Tranquility's auto-reboot on weekends takes approximately 4 minutes and 20-40 seconds, just enough for a quick cup of tea. But it is still an inconvenience since you must time your activities to be at a safe place at 11:00 UTC. Downtime waits for no one.

Therefore, for the first time in the more than six thousand days of EVE Online, there will be no downtime on Wednesday, 4 December! We will begin the first 48 hour run ever of EVE at 11 o'clock on Tuesday 3 December and end it on Thursday, 5 December.

This will be the first small step to evaluate what needs to be done for us to improve the EVE Online experience for players in different time zones around the world. Even a short downtime has an impact on your experience and to set the game up for yet another thriving decade then we must address that.

First, let’s take a step back and look at the reasons why we have downtime in the first place:

• We have excessive memory consumption and lack of clean-up in certain areas, and we don't necessarily refresh cache since the daily reboot will take care of it all.
• We still have daily database jobs that run during downtime.
• There are certain things that must be done regularly, and it is most convenient to do them during startup when there are no players online.

This is our dirty laundry. We have documented all the things we know that can go wrong. Then there are the things we don't know about and testing is the only way to find out…

We know that some things will be less than ideal, e.g., asteroid belts will probably be barren in the last 24 hours. Other things may not get updated (e.g. average market prices) or may break, even badly, and we are fully prepared to have to emergency reboot Tranquility in the evening on the 4th. Time will tell.

So why take this chance, why not try and move downtime? Wouldn't some other time be better? Well, not really. The only real solution is no downtime. Everything else is just trying to pick the least-worst solution.

EVE players are all over the world, and roughly grouped into three areas by time zones:

• Europe, including Western Russia, and Africa
  • Separated by the Atlantic from America
• America
  • Separated by the Pacific from Asia
• Asia, including Eastern Russia, and Oceania
  • Separated by the east part of the Mediterranean / west part of the Indian Ocean from Europe (yeah, I know this is a stretch, I'm picking oceans here to try and fit a pattern)

"Pacific Downtime"

The primary play sessions are visible in the online graphs:

The first instinct would be to say that downtime should be at 7 o'clock when the online population is the lowest. This is the "Pacific Downtime"; after the American play session and before the Asian play session. I'll come back to why it isn't the right choice.

"Atlantic Downtime"

The population is, however, not the same as an activity. Using CPU usage on Location Nodes (the nodes in the Tranquility computing cluster that host the solar systems) as a proxy for activity then a different picture emerges:

There are noticeable dips and peaks here, and downtime is not where you think it is.

If you look closely at the peaks, then you will notice a small dip at each one of them. That's downtime at 11 o'clock. There is a decline in activity between 10:40 and 11:20 UTC each day, which fits nicely with Central European lunch. EVE is real and requires food.

The lowest dips, however, are at 23 o'clock (around 23:30 UTC actually). This is the "Atlantic Downtime"; after the European play session and before the American play session.

But downtime at 23 o'clock in between the primary play sessions of the two largest player timezone groups may be a terrible idea even if this is most often the overall low point of activity. For example, wars at a time suitable for both player groups would happen at that time.

There is more than can be read from these graphs such as that CPU usage overnight and at 7 o'clock, during the lowest population, is quite high, so activity per player is high during the American play session. Americans also play longer into the night than, say, Europeans. Shortening the American play session at the tail end with the "Pacific Downtime" is, therefore, not the right choice.

"Mediterranean Downtime"

What about the "Mediterranean Downtime"? It is probably the worst spot of the three possibilities with a high online population and activity.

So all choices are bad, but it must be acknowledged that Asia currently gets the short end of the stick.

We could nudge downtime a bit earlier into the morning but realistically no earlier than approximately 9:30 UTC, and perhaps we will. Rest assured, we are always looking at our metrics and wondering if downtime can be placed better.

But honestly, then there is no ambition in moving downtime around. As I said at the beginning, the only real solution is no downtime.

No downtime on Wednesday, 4 December, will be a singular experience! That being said, then I don't want players to log in to try and break any online records. I really want as close to a typical Wednesday as possible.

We want to gauge the impact to inform our priorities. What should we fix to try and get to a place where we can normally run Tranquility for, say, 48 hours at a time and perhaps have four downtimes each week. And then continue to increase the run-time and reduce the downtimes by focusing on the pain points as they surface.

See you in space,

CCP Explorer