More news from the testing pits

Boy Howdy!

Well, I‘ve got a lot to cover, so I‘ll just jump right in.

Bugs site and other improvements (zomg, moar!!!)

We‘ve already pushed the feedback system for the bugs site live and we‘re very pleased with the initial results, but we‘re not stopping there. My plan is to constantly add to and improve on what we already have. I‘m quite pleased with the effect that the feedback system has had so first we will be focusing on improving that and increasing the amount of information viewable on the bug reports they‘ve submitted. Here are some ideas we currently have on the drawing board for improvements to bug reporting and player feedback from the test servers:

Planned:
• "My Bug Reports”:
We will give you the ability to view the bug reports you’ve already submitted, see when they were submitted, and if they have been acted upon yet. These are your bug reports and I know that, to many of you, it’s important what happens to them and if the issue is fixed yet or not. We want to make more information about what happens to your individual bug reports available as well as eventually provide information showing you exactly which patch the bug you reported was, or will be, fixed in.

• Overview thread:
A thread where I will post, and update various bits of statistical info relating to bugs, bug reports, defects, progress in testing, etc. I.E. number of defects closed, number of bug reports submitted, how complete the testing on feature X is, etc. This will be a part of the point below.

• Revamping informational posts:
There are plenty of informational posts in the Game Development forums, a few of which are out of date, and many which I feel could use more information - this will be addressed. We will eventually have a small series of on-going informational threads that contain all relevant information as well as links to helpful posts, dev blogs, statistics, and other neat stuff.

Under Discussion:
• Upgrades to logserver:
Integrate more logging and information gathering, such as automating portions of the bug report submission process. This should make BR submission easier. more efficient and should remove much of the guess work, while increasing the information obtained from log submissions.

• Testing roundtables:
Players and testers get together once every few weeks to discuss bugs, known issues and workarounds, concerns, test server related stuff etc. This would take place on IRC, or in-game on Sisi and be open to anyone from the EVE community.

I would also like to point everyone to this thread where we have asked for your direct input and suggestions for how we can continue to improve the bug reporting system and the bugs site in general. Please drop by and give us your feedback.

Umm.. what about Revelations II stuffz?!

Yes, yes, I was getting to that…

A lot of work on the part of QA (and everyone else too!) went into RevelationsII. In total we spent over 11,000 man-hours on testing, not including the efforts of the Bug Hunter volunteer team. To put that into perspective the “average” amount of hours one person will work in one year is 2000 hours. We’ve also increased the size of the testing team, again. We now have 5 times more dedicated testers than we did for Revelations I, and we are still looking at hiring more!

For Revelations II we processed about 4700 bug reports, and have closed around 2000 defects. Keep in mind that the bulk of those defects are caught and resolved before they ever get to the live server. In addition to this we performed 4 separate extensive regression tests on the patch candidate before approving it for release to TQ. We have also stepped up the amount of our data-gathering so we can better see how each new patch is affecting CPU usage, memory usage, server load, server stability, and other important trends. Finally, we have stepped up the filtering of bug reports, on average we are now able to process anywhere from 50 to 300 bug reports per day.

What all of this means for you is we have more people, more resources, and better processes to ensure higher game quality than ever have before and we are aiming even higher. I am sure that, over time, you will find that constant improvement is a big point with me. One should always strive to outperform the job they did the last time around.

Improving our abilities to fight the evil lag-monster.

As part of our ongoing efforts to improve stability, fight lag and other load issues we are acquiring and writing new tools, to replace and suppliment what we are using currently. These tools will eventually become an entire testing suite of simulation and benchmarking tools that will allow us to properly simulate load on a server in various ways, as well as collect statistical data that we can use to see how the performance of EVE is being effected by various situations.

One of these tools will be is an application capable of logging in with a series of special clients, say 300-500 at one time, to simulate a variety of actions. This can range from a simulated fleet battle where you have 2 sides firing on various targets, or to hundreds of people doing more "day to day" activities in a system, ie. Jita.

Another part of the suite will also include various benchmarking tools and clients so we can gather relevant data from the live server for analysis.

This suite will allow us to pull both server-side as well as client-side statistics so that we can more quickly and effectively see which changes will have the best results on server and client performance and allow us to perform better benchmarking / data gathering on the entire server-client package. Overall this should allow us to push out more effective changes on a much shorter time table; the end result being the EVE community will see better improvements faster than we have been able to do in the past.

Lingorm, our newest addition to the CCP QA staff, has been hired specifically to handle and manage the team that is working on this suite, improvements to existing tools (such as the bugs site, and others), and to produce more helpful tools that will help us make EVE better.