The EVE Client - A Love Story | EVE Online

The EVE Client - A Love Story

2008-12-11 - 发布者 CCP Atlas

The EVE Client - A Love Story

Greetings fellow New Eden denizens. I wanted to write a few words to recap some of the issues we have been having with client performance as of late and explain our ongoing strategy for addressing these problems.

The short version

(without technobabble)

With Quantum Rise, we kind of messed up the performance of the EVE client. It's sad, but it's true...

With Quantum Rise 1.0.1 and 1.0.2 we made it better again. With Quantum Rise 1.0.3, scheduled for this week, we are making it awesome.

The long version

(with technobabble)

Here is the actual client love story you were promised in the title. It follows the Hero's Journey pretty well with our protagonist, Eva Litla, first receiving a great injustice at the hands of the antagonist (we will call him Mr. Bug). In the second act, Eva Litla starts realizing her potential and in the climatic end scene she comes back to slay Mr. Bug. Anyways, I don't want to spoil the ending, but it's pretty cool.

Table of Contents

Act 1 - Quantum Rise 1.0.0

Over the past year or so we have been aggressively improving our server performance on multiple fronts. We have added great new hardware, we have migrated to 64-bits, we have introduced StacklessIO and made a great number of performance tweaks on the server and database, large and small.

The general feeling is that the server cluster has never been as healthy as it is right now and that feeling is backed up by data: Jita lag is a thing of the past, mission hubs are generally relaxed and couple of weeks ago we saw a 1,000 person fleet fight take place on Tranquility, which is something we have dreamt of for a long time. In that fight there was considerable lag, but the performance of the node degraded gracefully and we had consistent quality of service. Big pat on the back for us then.

Not so fast. After the release of the Quantum Rise expansion on 11 November (which by the way included several fantastic server and database optimizations) the client seemed to have been left out in the cold.

We received numerous complaints from our players that client performance had generally become worse than before. The problem was that we did not manage to reproduce the same results in-house and it had completely slipped though our net before release. There seemed to be few bits and bobs left over, but it's always the same when you try a bit of do-it-yourself, isn't it?

We finally tracked down some of the problems a few days after the Quantum Rise release. A number of the newer under-the-hood features in the Trinity graphics engine were causing issues under high-stress situations and our resource handling had become worse than before. This was something that was especially apparent on single-core machines and when running multiple clients.

In hindsight, we should have caught this, but we didn't. Needless to say the week after Quantum Rise wasn't exactly our proudest moment. Although we had improved server and database performance in Quantum Rise, due to the client issues the perception was that overall performance had degraded. And perception is reality, doubly so in virtual worlds.

Act 2 - Quantum Rise 1.0.1

Okay, now for the good news. Once we had an internally reproduced the performance issues we immediately put some very smart people into action working on these problems.

It turned out we had extensive issues with memory and state leaks but the largest problem stemmed from the use of a specific resource lock which went out-of-process. This did not show up on multi-core tests and really just manifested itself under certain conditions such as when running two clients on a single-core machine or when the computer was under extreme load.

So, on Saturday 22 November, 11 days after we deployed the Quantum Rise expansion we shipped Quantum Rise 1.0.1, which was a client-only patch which addressed the critical performance issues from Quantum Rise 1.0.0.

We are very happy with that patch and especially that we managed to get it out the door before that weekend's pew pew (1337 speak for shooting each other with big guns). It took an enormous effort on the part of the development teams involved and some pretty long hours, but luckily we had plenty of candy so it wasn't that bad.

Act 3 - Quantum Rise 1.0.2

We put a lot of work into the Quantum Rise 1.0.1 patch, with people pulling in evenings, nights and weekends to fix things. Our teams generated a whole lot of performance and usability fixes that needed more testing than we could manage for the Saturday patch. It was decided not to jeopardize that delivery, but rather put out another full server and client patch a few days later with all those non-critical fixes.

Quantum Rise 1.0.2 was released on 27 November and contained a large number of smaller changes to address lingering issues, a lot of them having to do with usability. With the release of Quantum Rise 1.0.2 we believe that we completely made up for the client-side sins of the initial Quantum Rise release.

Act 4 - Quantum Rise 1.0.3

Despite the performance being back up to pre-Quantum Rise levels (right?), we were not quite done yet.

This week we will release Quantum Rise 1.0.3, which is jam packed with performance improvements. We have spent a lot of time profiling the client and identifying bottlenecks. Most of the client performance enhancements we are making in Quantum Rise 1.0.3 are to address older issues and should make the client much snappier than you have seen it before.

We are primarily looking at client lockups which occur here and there when there is a lot going on at the same time. Software locking up while it is thinking about things is really something out of 1997 and we will be working hard on continuing to address this in future patches as well. This is quite specific and not something people generally encounter during their normal play time. However, the great thing is that by looking at these specific scenarios we are really getting a glimpse into the overall performance of the client. We really have to stress the client to its breaking point and beyond to pick up things that, when added up, can lead to an overall degraded performance.

A number of fleet fights have been conducted on our testing servers where our players basically showed us how they play the game and allow us to gather information on specific usage profiles. We used this information to construct new ways to benchmark the client under the unique stresses of fleet fights. That is the work that has lead us to Quantum Rise 1.0.3. Despite the fact that fleet fights and high-stress engagement scenarios form the basis of our performance attack plan, we do expect to see some major client performance gains under a lot of different scenarios.

I cannot stress enough how important these fleet tests and player reports are. It has been amazing to see our player base jump into the fray to help us make the client better. It's truly a testament to how strong the EVE Community is.

Anyways, back to the story. In particular, we are attacking client performance on these fronts:

  • Overall graphics performance bottlenecks.
  • Premium-specific graphics performance bottlenecks.
  • UI performance bottlenecks.
  • Identifying and implementing user-settings to optimize performance.

Overall graphics performance bottlenecks

When VTuning the client we noticed that under certain conditions the code was spending an abnormal amount of time in RtlReAllocateHeap (see Kernel32.HeapReAlloc, which is forwarded to NTDLL.RtlReAllocateHeap). This did not happen immediately, but rather after a while under specific load, and only on Windows XP. On Windows Vista we got completely different behavior with consistent memory allocation every time.

What this means in the real world is that after a certain amount of time loading resources, the memory started to become fragmented, which lead to degraded performance when copying resources in memory. We do a lot of copying resources in memory, from putting a new ship on the screen to loading up a menu, so after a while that process became slower and slower. This resulted in across-the-board performance drops after playing for a while with a specific playing style. This did not affect all players on Windows XP and not all hardware seemed to be affected equally which is why it was pretty hard to find. Also, this did not happen on Windows Vista.

After some investigation we found that the regular process heap on Windows XP was not optimal for our resource loading needs. We needed to replace it with a specialized Low Fragmentation Heap. This change made the time spent in loading resources much more consistent. Also, the resource object heap is now separate from the other process heaps which allows the memory manager to better manage the blocks for its needs.

The performance on Windows Vista in this regard is still considerably better than on Windows XP but the performance on Windows XP should now be better and much more uniform than before.

In unearthing this our teams also discovered several other things that could be tweaked and resource and memory leaks that we could plug, resulting in a much better handling of resources.

Premium-specific graphics performance bottlenecks

When the Premium edition of EVE was introduced in the Trinity expansion (not to be confused with the Trinity graphics engine mentioned above (don't ask...)) the logic for loading up turrets was different between Classic and Premium. In the Classic edition turrets are always loaded up lazily. However, the belief was (and was backed up by metrics) that the performance of the Premium edition was so awesome that it wasn't necessary to have this lazy turret loading in place. Rather the turret models and locators were loaded up and initialized as soon as the parent model (ship) came into the scene and there was no way for the player to turn them off.

We have now found that our original assumptions and performance tests were faulty and under real-life situations (i.e., space game) the cost of real-time loading is too high. The framerate suffers from too many turrets and loading up a heavy scene with a lot of turrets (e.g., warping into a fleet fight) is much more expensive.

So, we are replicating the lazy turret loading from Classic into Premium. This means that turrets will now only be shown for ships that satisfy one of the two following conditions:

  • You are doing 'look at' on the ship.
  • The ship is shooting its turrets and Turret Effects are enabled (default).

What this means is that ships will not show their turret models until the point when they are fired. If you disable Turret Effects through the settings menu (the escape menu) then you will generally only see turrets on your own ship.

If you really want to see the turrets of a ship you just right-click and pick 'look at'. The turrets will be loaded until the ship leaves the scene.

UI performance bottlenecks

We are doing some fantastic (and well overdue) performance tweaks to the UI. For the Quantum Rise 1.0.3 patch the most noticable part is the way containers (such as windows and buttons) are rendered. Fewer draw calls are being made and performance is very noticeably increased when you have a lot of windows open, especially on lower-end computers or computers under heavy load.

Drawing and loading brackets has been greatly optimized. Loading of brackets can be up to an order of magnitude faster than it was previously. This is especially relevant when you are loading up a great number of brackets.

We are continuing this work throughout the rest of the year and will probably put in more fixes for UI performance after the New Year or in the next expansion.

New user-settings to optimize performance

From the fleet testing, we have been doing with our wonderful players on the test servers, several bottlenecks have been identified that can't really be fixed as such, but are things that cause issues in high-load situations. Keep in mind that if you are a high-sec carebear (like me) these new options will probably make no difference to your client performance. However, if you are pew-pewing with a thousand people you will want to keep reading.

We have added the following options to the settings menu:

  • Disable drone models.
    • This will cause your client not to render any drone models, improving performance on Classic and Premium significantly if you have a scene with many drones.
  • Disable Explosion Effects.
    • Explosion Effects can drain your system and cause stuttering if there are a lot of them on the screen at the same time.
  • Disable Tactical Messages.
    • You can now shut off messages about warp scrambling, self destruct, etc. This does not make a big impact on performance but every little bit helps.
  • Disable camera shake.
    • We put this in because of numerous requests but it has not been shown that this has an impact on performance.

In addition we now have an easy way to quickly disable all brackets in the same manner as 'show all brackets' worked before.

If you disable all of these options then EVE won't look as nice but it might make the difference between life and death in fleet fights.

We have put a lot of effort into Quantum Rise 1.0.3 in a short amount of time. Given some of the low-level changes we are making there is the possibility of some issues cropping up, but we felt that people would much rather have this amazing performance patch now and give us a bit of leeway to fix issues rather than waiting until after the New Year. Please continue working with us after this patch is released to get continued performance and usability benefits into the client.

Pretty graphs

A large portion of the Quantum Rise 1.0.3 development push revolved around getting better benchmarks and improving our metrics. To that end we have been measuring client performance in a wider range of circumstances than before. This has lead to the identification of a number of problems. Once the problems are identified, the fixes are more often than not relatively straightforward for the extremely smart people that work in that area of the code.

Here you can see some of the graphs that were generated to find ugly spots in the performance and to gauge it against the solution we came up with in each instance.


Figure 1: Extreme example of launching drones comparing Empyrean Age vs. Quantum Rise 1.0.3 and showing the predictable results of disabling models.


Figure 2: Same example as in Figure 1 when completely zoomed out when models are LOD-ed out. We no longer get a performance drop with increased items if they are not being rendered.


Figure 3: Before the client froze for 60 seconds on our test machine when toggling on all brackets for 2000 ships. Now that time is down to 12 seconds.


Figure 4: Because we are no longer applying turrets until just-in-time we see a big change in the loading time and frame rate when warping to turret ships.


Figure 5: The same example as in Figure 4 when zoomed out. There is only a minor client hiccup when compared to Emperyan Age.


Figure 6: Here we see the effects of the resource load optimizations and Low Fragmentation Heap on the Classic client.


Figure 7: The Low Fragmentation Heap has a better effect on the Premium client, but resource loading was already much better than in Classic.


Figure 8: Smartbombing a great number of NPCs at the same time shows us how expensive the explosion effect is. Now it can be turned off.

It should be noted that all of these examples here are very extreme. They do not directly represent the playing experience. At best they show us an indication as to the problem areas we have been looking at. The before and after graphs will probably not correspond directly to anything you'll experience.

That being said, the changes are awesome :-)

Act 5 - Quantum Rise 1.0.4

(This part was added later for completeness.)

There were a few isolated problems in Quantum Rise 1.0.3. Some users experienced memory leaks and there was a crash issue in the resource cache.

We were very well prepared this time. We started working on fixing things as soon as we received the first report of problems in 1.0.3 and one week later we released the 1.0.4 patch.

This patch centered on a few isolated things:

  • The resource cache was not initialized properly so it always used 256 MB of memory, which looked like a leak.
  • There was a memory leak in alt-tabbing out of a fullscreen game that stemmed from an improper cleanup of the graphics device.
  • An old memory leak in the market's price history graph was found and fixed.
  • We added a memory usage graph to the in-game monitor debug tool (control-shift-alt-m). Now there is no place for those memory leaks to hide.
  • We fixed a crash issue that occurred sometimes when loading resources.

The Quantum Rise 1.0.4 patch is a direct continuation of the 1.0.3 release and completes this round of performance fixes for the client. We hope you like it :-)

Interlude - Fleet fight survival guide

With the changes that we are making in Quantum Rise 1.0.3 thousand person fleet fights should be much more enjoyable (at least to the winning side, there's not much we can do for the other poor guys giggle... ehem, sorry).

Here is a list of things you should do in order to make sure that your framerate is sufficient to at least see when you are podded. It is important for you to configure the client before you enter the fight because some of these settings will only take effect when a new scene is loaded and if you have some of the heavier options enabled your client might not recover from the warp-in before you find yourself back at HQ with Aura's friendly voice telling you about the latest advances in cloning technology.

If you are happy with your frame rate and loading times with any of these items toggled on then by all means continue to run your client that way. However, if the tradeoff is between performance and prettiness I'm sure you will choose performance. The list below is ordered by priority with the most important bits on top.

  • Only show the exact brackets you are interested in (at the very least disable drone and wreck brackets).
  • Disable drone models.
  • Disable Turret Effects.
  • Keep your camera zoomed out all way when warping in.
  • Disable Explosion Effects.
  • Disable Effects.
  • Disable audio (this does not include EVE Voice, there should not be a performance impact there).
  • Disable Damage Messages and Tactical Messages.
  • Disable shadows and HDR.
  • Consider using Windows Vista (people might hate it but it has considerably better native heap management).

You will note that this list doesn't contain an item to 'run in classic mode'. Pre-Quantum Rise 1.0.3 that would have been one of the things to do but the performance is very much comparable now and if you have modern hardware that can handle the Premium client well then normally there should really be no reason to switch to Classic for fleet fights. You might actually get better performance with the Premium client.

Sample chapter from the sequel

We will continue to work on client performance after Quantum Rise 1.0.3. We already have some great performance improvements in our pipeline that we simply cannot get into this patch because of time constraints. But worry ye not, there will be client patches after this one.

Some of the things we are looking at include:

Lazy Model Loading:
We intend to eliminate completely the client stutter that you get when warping into heavy scenes. That familiar feeling you get (much mitigated in Quantum Rise 1.0.3) when your client stops responding just before you see your little blinky pod thing will be gone. This will be an enormously important change for everybody, from the fleet fighters to the mission runners because every time you warp into a scene with multiple objects you will see a difference.

More resource performance improvements:
There is plenty of work to be done in resource loading. We will continue to improve the process and streamline common operations.

Simulation performance improvements:
We have found some performance bottlenecks in the physics simulation engine that runs the world update loop. These are mostly relevant to the client that runs its own simulation loop and will have a performance benefit in scenes with a lot of objects in close proximity of each other.

More UI performance fixes:
A lot of work is continuing to be done on the UI framework to improve performance of windows and brackets. We will see considerable improvements to UI performance in the near future.

Epilogue

This has been an awesome time for us here with putting in great fixes and working closely with the community to identify problem areas with client performance. We hope that we will continue to deploy performance patches when they are needed. The teams here have done an amazing job in getting these fixes out quickly and the wonderful EVE Online community has been extremely patient and helpful during this period.

In conclusion, without wanting to sound immodest I must say that with Quantum Rise 1.0.3 the EVE Online client will be by far the best performing piece of software ever constructed in the entire history of mankind :-)

The End.

Jon Bjarnason
Technical Director
EVE Online, CCP Games

必读好贴