Observing the "Burn Jita" Player Event

Things have been afoot in the Jita solar system recently...

WHAT IS JITA?

For quite some time, the Jita solar system has been the most populated solar system in EVE Online on a sustained basis, having organically risen as the main player trade hub in game due to its location and the basic human need to gather together and trade stuff.

It's had a varied reputation as [wretched hive of scum, villainy and scammers], [representative product of EVE's emergent sandbox game design], [indefatigable cog of economic prosperity], and [rallying point for impassioned protestors].

And, starting last Friday, it was on fire.

Actually, Jita itself wasn't on fire, but thousands of players who normally find it a "safe" haven of trade and economic gameplay found their ships on fire - everything from a lonely hauler on up to the simply massive, highly-armored freighters moving thousands of USD equivalents of ships and goods.

As you can see from the below "kill" heatmap of the EVE universe (where each dot is one of the 5,431 solar systems in EVE's known universe), it was a pretty big deal.

Click image to enlarge

And it all happened because a few players wanted to make it happen and then, after convincing thousands of others to join them from around the world, they made it happen.

"BURN JITA"

For several months now, a coalition of thousands of players have been planning the "Burn Jita" event, aimed at disrupting the biggest trade hub in game in order to wreak some terrific, universe-wide changes by shaking the very pillars of EVE's economy at its metaphorical heart. Surely there was some sort of larger, meta-economic objective that likely would cause them great profit. This planning happened both in game and outside of game, via voice over IP chat, secret forums, Twitter and more. A sci-fi plot of the new information age if there ever was one.

Using game mechanics and some very impressive logistics, the coalition, lead by a group known as Goonswarm, amassed a tremendous amount of manpower and ships in a concentrated effort that would probably make many real life military strategists jaw-drop. Bravado-saturated propaganda was also a pretty big part of their plot.

Fast forward to Friday:  Player fleets comprised of high-damage-dealing ships began to destroy high-value industrial ships during a sustained campaign with single, coordinated volleys, sacrificing themselves in waves because the in-game "police" warps in almost immediately with harsh, reprimanding justice.

Amidst this orchestrated chaos, opportunistic salvagers profited from the wrecks left behind and EVE's deep, talented player community began blogging about, filming, screenshotting, livestreaming and celebrating/decrying the event. Some opposing fleets rose up to fight the invading hordes as well - to mixed success.

All of it occured in the same framework of unrestricted player movement and limitless player choice in a single, shared game universe. As our senior producer put it "f*cking brilliant". As a participating player put it: "There’s just something special about building 15,000 spaceships and loading their guns with 1 round of ammo to shoot. And doing it right in front of the police."

As developers we watched in awe at another amazing thing our players brought to the universe we created. Yet there was a lot more going on behind the scenes on our end besides popcorn and a comfy chair...

THE VERY TECHNICAL SIDE

Since the "Burn Jita" event was announced well in advance and CCP wants to support player-driven events (as long they are within the rules of the game), we reinforced the Jita solar system on our beefiest hardware, further reinforced all neighbouring systems and set out to monitor the event and provide the best experience we could to willing (and unwilling) participants.  Then things started to unfold a day earlier than announced on Friday morning. We gathered data and fine-tuned the systems and as CCP Veritas put it: "It's okay, didn't want that Friday night anyway."

As we talk about the players' experience, I want to give you a bit of an insight into two projects in EVE in relation to the "Burn Jita" event: A different perspective on Time Dilation and a brief introduction to EVE Metrics.

We introduced Time Dilation to be able to handle large-scale fleet fights in a more graceful manner.  Basically it slows down overall "game time" to make sure each player action is properly handled and executed in order by the servers - keeping it fair and orderly. CCP Veritas of the EVE Dev group called Team Gridlock was the driving force behind Time Dilation (TiDi) and has written a number of devblogs on TiDi:

• Introducing Time Dilation (TiDi)
• Time Dilation Video Demo
• Time Dilation – How's That Going?

Time Dilation was born from the need to better support large-scale fleet fights in 0.0, the lawless area of space where player alliances fully rule and frequently battle in numbers that dwarf entire server populations in other online games. While we have had Time Dilation kick in in low-sec (slightly less lawless area of space) then this is the first time that Time Dilation kicks in for any sustained periods in high-sec (where the police show up) and it's been very beneficial for us to observe TiDi in this environment - a digital gold mine of sorts for our dev team.

The server CPU profile and general usage profile of Jita has always been very different from the rest of the universe. There are 4 large groups of server-load profiles:

1. 0.0 fights - where players typically bring big numbers to the dynamically shifting battlefields of lawless, alliance controlled space
2. mission hubs - where lots of players are running missions for popular NPC agents for fun and profit
3. market hubs - where the financiers, traders and manufacturers do a lot of collective business (Jita, Amarr, Rens and more...)
4. general solar systems - just your "average" solar system

We configured Time Dilation to deal with 0.0 fights although those same parameters mostly work for trade hub server loads like Jita - but could be improved.  As a specific example: TiDi has to kick in very aggressively and very quickly during large fleet jump-ins, where a fleet simultaneously transfers between two solar systems.  This "use case" doesn't typically apply to Jita. Instead we have to deal with Crime Watch monitoring (i.e., who is "allowed" to kill whom, which also applies to low-sec but not 0.0) and CONCORD response (i.e., the police, which doesn't apply to low-sec or 0.0) for short-burst smaller engagements (but potentially multiple such activities). Different code and different usage profile.

So we reconfigured TiDi specifically for Jita, only on the node hosting Jita.  CCP Veritas provides the details:

Here's how we handled the server load of a few hundred murderous savages valued customers making ships explode in the most populated system in EVE Online.

Having a lot of people in Jita is nothing new.  Being EVE's main market hub these days, many folks come to sell their produced goods or fit out their spaceships for adventures elsewhere.  Sometimes, the strain from all that commerce has been enough to overload the server it runs on, and so we've made a single exception in our "no population caps" rule so that things can remain nice and orderly.  This is generally considered an okay thing, since preventing someone from jumping in to make some deals isn't nearly as bad as preventing people from jumping into a huge fleet fight.  I'm just giving some historical perspective, of course, because ever since the Inventory Setification optimization that accidently halved Jita's normal load profile, we haven't really been at the cap very much.

In any case, going into this event we had two dials to adjust - the application of Time Dilation and where the population cap is set.  Let the graphs begin!

Click image to enlarge

The event kicked off in the very early morning on Friday, 27 April.  We weren't entirely sure when exactly it was going to happen, so we had Jita configured just like any other normal day, with a 2,300 player cap and generic TiDi parameters.  I probably would have lowered the population cap once things got down into the 20% range, but I was getting my ANSI Standard amount of sleep(), so that didn't happen.

However, if you showed me this graph without the title and said it was from a fleet fight out in 0.0, I would believe you.  The only thing that looks out of place is that the load was 60% before the fighting opened.  TiDi is set up to keep the fleet fight load profile in the 80-100% CPU utilization range, which as you can see here it did quite well.  From what I hear, the early hours of the event were heavy with counter-play, so I suppose it makes sense that the performance looks like a fleet fight.

Fast forward a few hours 'til after downtime (and coffee), and we're back up with a much more aggressive population cap - 1,850:

Click image to enlarge

We wanted to keep things fairly low at this point as we had found a few bugs in how Crime Watch worked under Time Dilation.  It's not something it's asked to do very often so these issues hadn't come up since TiDi's introduction months ago.  However, even with such a low cap (1,850 is low these days, how far we've come...), TiDi is kicking in.  Unfortunately, it's not keeping load in the nice 80-100% band like I like to see, it's holding around 60-90%, with just a couple spikes here and there keeping it down.

Later that afternoon, CCP Masterplan hot-patched Jita with a fix for the nastier issue we had found with Crime Watch and I got to work tweaking TiDi and upping the population cap.  Essentially, we're looking at a load profile with high ambient load and small spikes.  The trick was to find the sweet spot that would keep the ambient load at around 80% CPU and have TiDi be less sensitive to spikes as they were almost guaranteed to be small.  Normally we have to be much more aggressive when a spike starts because it could be a huge jump-in or a really effective bomber run, but in this particular use case we're limited to ganks or smartbombings involving tens of ships, not thousands.

Basically speaking, I made TiDi's tendency to slow down time more relaxed and less frequent, and made its mechanism to speed back up slightly more aggressive.  Here's the results from Saturday with a 2,200 player cap:

Click image to enlarge

Sweet spot achieved!

So how was the experience in Jita?  Lazarus Telraven, one of Goonswarm's main Fleet Commanders, says "This weekend has been a milestone for EVE, the first time anyone has declared war on a solar system. The game worked wonderfully. Among the Jita Burners complaints are few and praise is plentiful, but we all agree that without TiDi none of this would have been possible. In fact, after asking all of our allies what they thought I only received one complaint!  The only real problems that I was asked to mention was blackscreening on undock and not being able to make a war target only overview. When I asked our fleets to offer comments on the performance the response was overwhelmingly positive. GSF member Bagdon said that "opinion on performance for ccp: fucking brilliant. all brackets on, all effects on, I might even start playing more after this!"  The number of people in Jita spiked between 1500-2300 pilots the entire weekend with fighting, fleet warping, fleets undocking, and just loads of everything in the system but it all worked beautifully as TiDi was up and down in prime times. With everything being said we can't wait for the rest of the changes coming with Inferno."

The comments on the client performance ("fucking brilliant. all brackets on, all effects on") were heart-warming to us since we have recently put a lot of effort into client performance, per CCP Veritas' devblog Your Client, Made Gooder, starting with the release of EVE Online: Crucible 1.2 in February.  In part, that effort focuses on increasing the client performance but we are also spending good time on identifying and removing frame rate spikes and drops; making the client run more smooth, i.e., be more responsive and more consistent.

THE METRICS

For those that are interested in some numbers here are some statistics about the weekend in Jita gathered using a new system, called EVE Metrics, that is currently in development.  CCP Atlas provides the details:

First off, the top 20 weapons/ammunition used over these three days:

Next, let's compare weapon module activations this past weekend with the previous weekend:

Click image to enlarge

Click image to enlarge

Click image to enlarge

Click image to enlarge

Note that these graphs do not show how often these modules fired but rather how often people clicked the button to start them cycling.  The total damage inflicted over these three days amounted to 45,117,952 hit points delivered in 249,021 distinct hits.

Here are the damage and hits timelines this past weekend:

Click image to enlarge

Click image to enlarge

Note that in all the graphs above we're looking at one hour moving averages.

Finally, here we see the number of market orders placed in Jita per day since the beginning of March until this past weekend:

Click image to enlarge

The red dotted line signifies what we would have expected to see during a normal weekend.

So while many ships were destroyed this weekend, it sure feels good to be the hosts of such massive amounts of spaceship carnage. We'll always be prepping for the next big event our players throw at us.

Make your stories happen!

• CCP Explorer  //  Software Director  //  Follow on: Twitter / Google+

(Devblog co-authored by CCP Atlas, CCP Manifest and CCP Veritas)

New to EVE? Start your 14-day free trial today.
Returning pilot? Visit Account Management for the latest offers and promotions.