Reflection on Revelations

Happy new year from CCP QA.

Revelations is by far the biggest expansion CCP has released since the dawn of EVE and getting it out the door was a major undertaking to say the least. I want to use this opportunity to reflect back on the release, how QA at CCP has been changing lately and what our improvement plans are for this year.

The QA tale for the Revelations patch was somewhat different from the release of previous patches. Volunteer bug hunters helped out big time as soon as the new features started appearing on Singularity. When the crunch really hit QA, the last 5 weeks or so before release, folks from other departments swung into QA action and helped out. We outsourced all platform tests, localization tests, usability tests and even some functional tests to 3rd party QA companies.

This gave the core QA team at CCP Iceland room for improvement. We had more time to analyze hard to reproduce cases, write-up detailed test scripts, organize better test events on Singularity, analyze logs better and so forth.

All critical and major bugs we knew about were fixed before the patch, but defect tracking still has a number of minor and usability defects to be fixed. Development for the next phase of Kali is in its infancy and we’re putting most of our development effort into fixing the minor bugs. Also, there are more performance improvements underway so we can handle 50 thousand concurrent users in the near future.

Performance issues

There is a QA issue we will address this year. We lack the ability to perform good performance and load analysis before we deploy to Tranquility. Many performance issues are first discovered when the code reaches Tranquility, and we franticly have to hotfix the server and this impacts the players which is not good.

When Revelations was first deployed, the server was consuming a lot of memory and we had a series of irregular CPU spikes we had not seen before. Proxies were maxing out and crashing, database load spikes were occurring, and jump queues kicked in unnecessarily. Then we had to reset the cluster-load balancer which caused unbalanced load for a couple of weeks causing more problems. The petition queue rose to a new high of 10,000 petitions.

We’re making progress with measuring performance before the code is on Tranquility, but the reality is such that many performance bugs do not surface until we have thousands of people online.

To put things in perspective, Singularity, our test bed, is such a small cluster (8 servers compared to 84 servers on Tranquility) and we get at most 500-1000 users on Singularity on a good testing day.

Improving QA

Last year, QA went through some major restructuring; we got more organized, utilized 3rd party testing companies for grunt work, and put more focus on empowering volunteers willing to help out with testing on Singularity.

We split the QA team into 3 focus groups we call “cells” – test management-, deployment- and monitoring-cell. Each cell has a specific focus providing better service and more professionalism and growth opportunities for the QA staff.

• The test management cell organizes bigger testing tasks on Singularity with the involvement of other testers, programmers and volunteers. They review and write test plans and make sure they are executed in an organized manner.

• The deployment cell prepares hotfixes and patches for Tranquility and Serenity (the Chinese cluster). They keep test servers up to date, build scripts and so forth.

• The monitoring cell works closely with the operations folks; they monitor server logs, forums, analyze performance issues and work closely with the game masters on current issues.

Having employees run the same manual sanity tests (checklist) for every patch makes the people involved overlook UI and other bugs when running the same tests again and again. About 60% of the fixes we verify can be verified by bug hunters and outsourced to 3rd party companies, giving QA room to focus on the bigger and more complex tasks.

This year we are making automated load test tools to simulate high load on certain parts of the system and performing benchmark tests. We’re hiring professionals in this area to help us out and look forward to the results.

Volunteer Bug Hunters

QA is building an even more powerful network of bug hunter volunteers and we are paying better attention to the team than we had been doing for the last couple of years. We have given the bug hunters a greater role and we are recruiting more bug hunters. We have now 18 strong bug hunters all over the world, our goal is to double the size of the team and increase the scope of what they do.

The volunteer teams are a huge benefit to EVE. Volunteer QA adds dynamics to the QA process that we could never achieve internally or with outsourcing. Bug hunters are experienced players interested in QA and love the game. They offer a fresh perspective we in QA hunger for. They point out possible design flaws, help to reproduce bugs and verify bug fixes. They analyze bug reports from players, help organize complex tests and perform organized function tests (we call sanity tests).

Bug Hunters Wanted

We are looking for more bug hunters to join us. If you are an experienced EVE player, organized, have a good eye for bugs and possible improvements then you may be a good candidate for the bug hunter team.

What bug hunters get out of this is basically

• Access to development with the ability to influence the game.
• A free account on Tranquility.
• EVE store credit for good work.

If you are interested in becoming a bug hunter, please submit your application to ISD here

Have fun. Enjoy EVE.