Random thoughts from QA...
Who am I? My name is Stuart 'Mephysto' Webb, and I'm one of the Testers in the QA Department. I used to be a bughunter from May 2004, and I joined CCP's QA team last August. I thought I'd share with you a few facts and figures about the QA dept and the work we do on Eve...
The normal day here is 8hrs like most working people, although you can generally find at least 2 members of QA in the office at any time from 8am to 8pm, and sometimes even later, as well as a fair number other devs. During an average day, our tasks will vary from reviewing newly created defects from the bughunter team, checking out the latest features and fixes submitted by the programmers and content developers, our own on-running tests of the client/servers, and most importantly our hunting down and podding of you, our valued customers on TQ :D. Yes, we do play eve, and many of us are pvp'ers.
Some figures for you (as promised). Our defect tracking system currently has listed 10745 defects, of which only 827 remain unfixed. Or to put it another way, we have fixed 92% of the errors that have been found in Eve. It should be noted that a large proportion of these bugs never reach TQ, as they are identified and fixed on internal test servers way before a test patch is made public.
So, I expect your next question would be something like, "why are so few issues listed on the known issues list?" Ok, ok, so I know its more likely to be "OMG why havent you fixed them!" but I'm not answering that (yet). As for the known issues list, I plan to improve and update it as soon as I have some "free" time.
Most of the defects identified are either problems in test builds which you have not nor will you see, problems with the server code which have no effect on gameplay but cause issues for us, or exploits which we arent going to tell you about (hey, we have to keep our pvp chars funded someh... er... I didnt say that...).
There are some players saying bugreports are a black hole, with no response, and seemingly nothing happening to the reports once they are made. Some have even specualted that we have an automated system for deleting them. Theres only so many times you can read things like (this example is an actual received bugreport):
Title: Drones window
Description: There is no drone window!
Reproduction Steps: It is not present
... before you want to shoot people, or at least not have to read another report like that. The bughunters do a fantastic job for QA by screening the bugreport lists, filtering out the rubbish from the good reports, and then adding those good reports into our main system for the appropriate devs to see.
So, why havent we fixed them? Well, this comes down to priority. A harsh fact of life is that a bug on TQ, while a problem for players, may be considered unimportant internally when compared to, say, a huge memory leak on the cluster. It may suprise you to know that Tranquility still has a memory leak from the Bloodlines deployment, and the senior programmers have been searching for the cause since they became aware of it. As such, if there's another problem that requires the same programmer to fix it, it's likely to be deemed less of a priority. This doesnt mean it wont get fixed, just that it wont be fixed yet.
Oh yes, I almost forgot. Patch testing. Before each patch is released, QA do a "sanity" check on the code (normally 3, actually) to ensure that everything is working as it should. A single sanity check will take the entire QA dept (8 people, with more being recruited) a minimum of 6 hours to complete. A check consists of going through every aspect of the client and checking it is working as intended.
Despite this testing, some bugs do slip past us. Each time one does, we expand our checklists. Why do bugs slip by? Primarily because the 8 of us in QA cannot hope to test every last way of doing things that is used by 117,000+ players, and because some bugs do not appear until you have a couple of thousand people online. Also, we cannot replicate the network setup on TQ without duplicating the hardware and code. An example of this is the issue whereby some people were unable to warp to mission bookmarks. This came about because the mission's destination was on a different server node than the node the agent was located on, and the information on the bookmark wasnt being transfered correctly. Due to the smaller test servers, this exact situation was not specifically tested. It will be next time though.
Anyway, I need to go back to making my client crash (depressingly easy at present), and since talking to the programmers and asking nicely hasnt got it fixed yet, I think I'll confiscate their coffee.
Soon to be lynched, Meph.
