Tyrannis Performance Improvements

Introduction

Part of our job is to seek ways to improve performance, and as there's always room for improvement; this is a continuous process.

During the Tyrannis production cycle, we organized a few mass fleet fights at regular intervals on the test server (Singularity) in order to investigate and scrutinize any performance issues by actively monitoring these massive engagements. Nearing the launch of Tyrannis, it was clear that client performance in fleet fights had gotten pretty bad. We had reports of very high client memory usage as well as some probable performance bottlenecks.

Resource Loading

During the Singularity mass test on May 15, 2010, we noticed that our resource caching system was at times reporting negative memory usage. When this happened, resources were never released and the client would continue to accumulate them until it ran out of memory.

The negative resource usage was quite puzzling and was not experienced by everyone in the fleet fight. When we closely examined the resource manager we found a couple of issues.

It turned out that there was an issue with the way the resource manager assigned a size to textures that had not been fully loaded. This caused the resource manager to hold on to too much memory; this was fixed shortly after it was discovered. We also found and fixed a memory leak in the module responsible for reading resources. At that time, during internal testing, we started noticing that some objects from our physics simulation weren't being cleaned up as well as they should have been. We fixed that, as well.

As we went into the mass test on Singularity on May 20, 2010, we knew we had made some progress but were still not sure whether the weird negative memory issue would again rear its ugly head. We saw very early on in the test that the changes to the resource manager had not done the trick.

When the client had just started up, the resource usage was correctly reported. However, something was causing it to turn negative as time went on. After the mass test, we noticed that warping to a small fleet of ships with a fresh client could reproduce the issue. We also noticed that one of the ships was a Tengu, a Tech 3 ship. By stalking the poor pilot for most of the night we were fairly sure that the Tech 3 ship was somehow causing this problem, and at midnight we finally had solid reproduction steps.

It turned out that the size variable of the geometry was uninitialized. This meant that we would get unpredictable values, even negative ones, when generating a mesh, such as the Tech 3 ships. This was fixed and another mass testing was scheduled on Singularity that same day, May 21. The negative resource problem was fixed!

Before fix

After fix

With the resource usage fixed we saw that we still had our work cut out for us as there were objects still being left in the memory, even after warping away and docking.

Dead Balls and Destiny

Memory leaks in EVE are always difficult to debug. Because of the way our graphics engine and physics simulation interact, we actually rely on circular references. Also, by looking at specific objects you are invariably increasing the references to them, making them hang around longer.

A primer on Destiny:

  Destiny: Our physics simulation system.

  Balls: An atomic in our physics solution.

Behind the scenes we have these "Destiny balls" (yeah, I know) which we use to determine among other things the position and velocity of everything in space. We attach our graphics to the Destiny balls and rely on them being removed to clean up any related graphics assets. An example would be when ships explode or warp away.

During the fleet tests, we noticed that not all Destiny balls were being removed as expected and there were a lot of them left in memory, even though the ships they represented were gone. These extra Destiny balls exist only in the client and are commonly referred to as dead balls. Having these objects that we attach our graphics to hang around and never get cleaned up can obviously have a noticeable memory impact.

When you warp into a gate camp, aggressive ships are loaded before others to better help you realize what is going on. There was a bug in this part of the loading code that could result in multiple instances of graphics being added to the scene. When these ships were removed, only one of the instances was removed and the others were left behind. This meant that the extra graphics were referencing Destiny balls which were supposed to be gone, making them become dead balls. The error was not easy to spot, but thankfully easy to fix.

We have also changed the way Destiny removes Destiny balls so now there are no references back to the ball from the graphics when the ball is removed. The first mass test on these changes was on May 22 and the results were as we had hoped for, as no dead ball was discovered after a 200 ship fleet fight.

These changes were included in Tyrannis 1.0.2

Overview changes

During these Singularity tests, the top issues reported by players were that the overview was updating too frequently and felt too "jerky," and that the fleet window was causing client lag. Upon investigation, we found out that fleet changes were causing a lot more UI updates than were necessary. Every time a player joined or left your fleet, your flat fleet member list and the overview were reloaded, and, in addition, all the brackets were refreshed as well as all the overview entries.

This was only discovered a few days before Tyrannis was released, so we did not want to change too much at the time. What we did change was to stop reloading the overview when a player joined/left your fleet, and instead just added or removed the player from the overview (based on the overview settings you had for fleet members). This change introduced a bug where the overview stopped updating correctly (oops), but that was fixed very shortly after the release. In the case of the fleet member list, we found a way to reload it 6-8 times faster than before and chose to leave it at that since it was a rather risk-free change that dramatically improved the fleet window's performance.

After Tyrannis was released, we kept working on these issues. In Tyrannis 1.0.2, we deployed changes that do more targeted updates of overview entries and brackets when a player joins/leaves your fleet, rather than updating them all. The fleet window performance was further improved by adding/removing the player from the member list rather than reloading the whole list.

The Star Map

When opening up the star map the graphics were loaded and generated and kept around for easy access later. This resulted in a bit of extra memory usage, but allowed the map to be opened up quite fast.

It turned out that by throwing away the graphics when closing the map and regenerating them when opening it, we could release 80 Mb of memory. The change we made means that the map takes a bit longer to start up, but the benefit is that you are not using additional memory for the map when it is closed.

Renovating old systems

We are always trying to improve upon and refactor old and slow code. We have started using a new rendering scheme, but as we haven't been able to renovate everything in EVE, some of the things in EVE are still using our original scheme.

The dust fields, the particles that pass your ship when you are flying at sub warp speeds, were one of those things. The old implementation was heavily CPU bound and did not scale well. By using our newer particle system we decrease the time updating and rendering particle systems by a third than we did before.

The old inefficient particle system was also used for the old suns and we moved the suns into the new rendering scheme for performance reasons. We still do not consider them renovated; they will get their chance to shine in the future. 

Although these changes aren't huge, every millisecond saved helps with your FPS, and thus the smoothness of your experience in EVE.

There's more to come!

There has been constant performance improvement work going on since Tyrannis was released and more changes are on their way. We're replacing our old math library with a new version that's many times faster and old systems are still being renovated.

All the changes mentioned in this blog are client-side only, and will not solve any server-client lag that might exist. However, that should make playing a more stable, faster and more enjoyable experience.

We would like thank everyone that participated in the mass tests on Singularity and encourage everyone to participate in future tests. We couldn't have done it without the Singularity community.

• CCP Blaze