Current Issues & The Inbox Monster

As you may have noticed, we've started putting more effort into Known Issues. I like it, because we try to be open about EVE and this is a step forward. I can also see that people feel that after they saw this, there are more bugs in the Dragon patch. Well, that's not entirely true, this is the first time we've made a concerted effort to list that many of the known issues. Our internal system shows considerably less issues in Dragon that say Bloodlines, which is interesting compared to how comprehensive the rewrites were.

We don't need no stinkin' queues!

However, I'd like to address some specific concerns. First, there are these queues which there are two types of. The log-on and the jump. The way the log-on queue is disabled is that it's limit is set very high, allowing thousands to log on every minute. It only kicks in at startup, preventing the crash of the cluster and if the system you are trying to log into isn't loaded yet.

You may notice the "system not loaded" mostly in character selection, but in reality it was always there. Before this mechanism came in, you sat in a black screen, and waited a long time for the system to load till you finally got in. This resulted in many clients timing out while waiting or even worst, get stuck. Now, you simply aren't allowed to wait in that black state, shouldn't time out and shouldn't get stuck. In short, it's designed to prevent bad things from happening and so far it has.

The jump queue on the other hand is far more tricky. We believed it was effectively "disabled" by setting it to kick in at 95% or more CPU usage. This is a very bad node state and means that you are more than likely to get stuck if you get into a system like that. The first problem was that the system discarded the setting and used it's default, 85% CPU, which is quite common. In cases of up to 8 constellations being on the same node, that would effectively close down about 80 solar systems or so.

The second parameter used is more than 700 people in a single solar system. We haven't hit that yet and that usually only happens in solar systems which are already on a dedicated node, so it only affects itself.

The reason this was set in was to prevent solar systems from overloading and getting players stuck. This was especially dangerous with the large solar systems, because when these nodes with the high load solar system died, they would start a chain reaction that would eventually take the whole cluster down.

But, this is not without its drawbacks. Even though we're seeing systems like Jita only running at 55%-75% CPU - even on Sundays - after the Dragon patch, we still had abnormal behavior which brought spiked CPU usage up and thus caused jump queueing.

You see, after Dragon, the codebase is much more stable, using far less CPU and we have a more clear picture of what is causing a lot of load spikes. Call it a good signal-to-noise ratio. It probably won't surprise you, but Bookmarks is pretty high on the list, so high in fact that we will be putting in even more drastic limitations to bookmarks. Until we of course implement the replacement for the gameplay which bookmarks introduced. What's in the coming patch is simply to address the load associated with copying large amounts of them. More on that in another blog which is on the way.

We're still tuning the queues to do what they are supposed to do and to prevent them from being a nuisance to everyone that wants to fly around space (or worse, getting you shot while waiting or can't move in a system with no pilots in it). We have a patch in Testing on Singularity that addresses some of the Known Issues that have fixes we could safely test and deploy, while others will have to wait for a better time. If all goes well, we will see it on Tranquility tomorrow.

Other issues and them forums

Going over the forums really creates a big list of things you want to answer, but being a dev and answering the same question 27 times, while the some in the thread saying "zOMG DEV NOT REPLY YOU DON'T CARE!!!" doesn't really provide an encouraging environment. I laughed when someone demanded I answer immediately at 5 in the morning. We do sleep occasionally so I'm sorry, it won't happen again :)

You see, forums suck at delivering information to the whole community, simply because not everybody reads them. We read them, occasionally reply but we can't (and won't) reply to every thread there. If I spend an hour each day on the forums, that 7 hours a week I'm not working on making EVE better. Actually, I put in that many hours anyways that I do read forums about 1 hour a day :P

Anyways, the dev blogs have shown far better results in spreading the word and opinions of developers and thats the main reason we try to emphasize more on dev blogs.

Take the EIB scam. We answered these very same questions the last time the [Insert any big scam here] was in the spotlight. We do not interfere, unless there was a violation of the EULA or Terms of Service. I mean, people transfered this guy their ISK. He then doctored screenshots and whatnot to prove his legitimacy. Then shows screenshots of how big the scam was. Hmm.

We're not going to replace trust, nor are we going to fix situations where people send their money to some guy in hopes of getting it all back. Then again, we'd like to support IPOs and other such models, but that's down the road.

Cloaking. What an annoying bug Dragon brought about. The Dragon issue raised the question whether it's to overpowered. Or with the first of the fixes if it's too underpowered. Or overpowered. Or ... you get the drift. We managed to get this fix in tomorrow's patch because it was low risk. Be patient, there is more to fix and more patches to come and we'll address cloaking properly.

Also regarding the misunderstandings on the forums, we have 117 people on staff, misunderstandings happen. In an effort to maintain our openness with the community that many of you have said is one of our big strengths compared to other companies in the MMOG industry, we run the risk of miscommunication when we allow more than one person to post on the forums. We accept the risk, I hope you do too.

Log-on and log-offs tactics. Tricky thing too. In a world where the most important thing for many are their reputation, others don't really care and act like cowards. That's right. That's my not-so-humble opinion. You need to resort to log-on traps to kill your enemy, you get no respect. But then again, that's your choice.

We've changed these mechanics quite often over the last 3 years, the main issue being to balance between connection loss and other legitimate situations to prevent the exploitation of the loopholes. Changing this mechanic affects everyone, and that's the problem. However, we are indeed looking into it. Good suggestions welcome.

Blobby Starbase Ping Pong Wars. No, we don't like them either. We are working on solutions, most likely a part of Kali Two, where we work mainly on warfare and player infrastructure, the basis for large scale warfare. We're also doing a lot with combat in general and you will see a lot of that already in Kali One. Things like changes to ECM, relative adjustments to hitpoints vs. class and a general hitpoint boost, some likely changes to range and so on.

When is Kali One coming, you ask? I'd like to see it in the end of September, but Dragon still needs maintenance on TQ, so we can't assign as many resources over to the final push of Kali One as we wanted. Yes, we do have more people than me working on this game. ;)

Talking about working on the game. Some of you believe we're spending all our time on China. That's certainly not true. It was partly true, as I explained some time ago but as you can see, the Dragon code build, which all this work was used on, is performing considerably better.

The Dragon deployment also signifies the point where we're releasing patches according to our original plans, where China is on a delayed release schedule, our efforts now focused on fixing Dragon on Tranquility and wrapping up Kali, while our partners in China start translations of the newest Dragon code base. In other words, almost all development benefits both worlds and China enabled us to hire even more people to work on EVE.

Recent downtimes and patch deployment. Well, before patch deployment, the old Tranquility code base was really starting to show its age. Bugs would crop up, which started filling logs, which then eventually started filling the hardware, which then crashed the whole thing. We also saw a lot of load related things that took down Tranquility and we have had some networking problems to top it all off. Unfortunate timing, Murphy sticks his hand in again. Dragon resolved a lot of the load issues, but also got blamed for a lot it wasn't responsible for.

The late patch deployment was all caused by one bug. One single, solitary bug. The cluster just would not start because of it. The funny thing was that the fix we deployed to get Tranquility running prevented all other servers from starting up. This is especially ironic since the Dragon patch worked on all the other clusters including Serenity, just not TQ.

When it rains, it pours I guess. We should be through the roughest part now, though last night's issue was due sizeone of our database drives running out of memory. That was pure management bork, it should have had enough room for an incident like that. So to fix it in more ways than one, we bought another RamSan. Had to sell my car again though :(

Oh, and this just in. The incident last night required some more work during yesterday's downtime, which required us to fail-over our database cluster, which caused a lot of cache in our game servers to invalidate, which caused the startup problems today. Damn it. Then our network providers went haywire and the London hosting center was disconnected. Someone give me something to bash Murphy with!

Anyways. We’re also hiring a lot of more GMs to handle the petition flow. The response time is far from being satisfactory and we know it. So check that out if you are interested.

There are a lot of more, but this is getting really long. I'll close this off with ... The Inbox Monster

Some guy once said, "show me what you read and I'll tell you who you are". Or something. I think that if I show you some of what happens in my inbox, it might tell you a little bit of who we are. These are things I found in the oldest mails of the Game Design folder. Here goes. It's DD/MM/YYYY.

15.4.2004 - Kieron says Microcontrollers gone haywire. NPC supply for them were incorrectly edited. Fixed.

15.4.2004 - t20 thinks insurance shouldn't cover wars. Nice.

16.4.2004 - LeKjart sees even more insane types on the market that go for good profits. Fixed. Wonder if he made a profit himself ...

16.4.2004 - TomB says it's way too easy to chain farm an NPC spawn since he was with a group of players doing it himself last night. Clever. Never forgave him for not bringing me along. We randomized the spawn areas and changed the logic a bit.

18.4.2004 - TomB doesn't like all them sentries in 0.0-0.3. I think he got griefed by some trying to gank some noobs. Wouldn't surprise me judging by the fact that he managed to gank the dev that sat on the opposite side of the table at launch. Bastard.

19.4.2004 - LeKjart on insurance again. Perhaps it shouldn't cover CONCORD offenses? The griefer in Solaris says the death penalty was high enough and that most people CONCORDOKENED got shot because of bugs or mistakes. (HARR FOR US PIWATS!)

19.4.2004 - The highway situation. PapaSmurf says everybody was flying through Yulai now and we had to do something. We decentralized the highways. Yulai died.

19.4.2004 - Zrakor wants to create Spy drones which you attach to ships so you can trace them. We find too many drawback. Crap, nice feature to hunt down people.

19.4.2004 - RealX had heard the tutorial was supposed to be revamped and wanted to help. He said it was so much crap it couldn't be revamped and needed a totally new one. He was right.

19.4.2004 - Fuhry completes build 1421 of Castor. Rejoice. Retort. Counter.

20.4.2004 - Zrakor doesn't like the exploitability of the static deployment of missions. More dynamic jump count implemented, more variety introduced. Yay, I guess, cause it sent a lot of people to lower security. >)

20.4.2004 - GMs say that CONCORD and Sentries aren't responding fast enough. Can and criminal flagging is first born to allow players to do something about it (and ended with kill rights extension), but the intermediate route is TomB equips CONCORD with WMDs.

20.4.2004 - Zrakor proposes subtle changes to the faction standing table since none of the pirates were really enemies and such. So we made them enemies.

21.4.2004 - Loot tables break and t20 gets lots of weird stuff, with premonitions of doom for the EVE economy if it wasn't fixed since we can't have too many Plasma physics skills out there. Turns out it was one NPC that was borked, in a mission t20 was doing. t20's pirate reputation was scarred for life.

22.4.2004 - TomB wants racial ship brackets. Nothing happened, but comes in when we do the overview much later on.

22.4.2004 - TomB suggest we give all NPCs a secondary attack, like a short range and long range attack. Ends up being far more complicated, with support ships, e-war and such stuff. Comes in over a long period of time.

22.4.2004 - Hellmar asks if Scrapmetal Processing is seeded with NPCs. It isn't and I call him a carebear.

23.4.2004 - TomB doesn't like people in the NPC corporations and wants them all thrown out. Now, everybody ends up in the newbie corporations.

23.4.2004 - PapaSmurf analyzes TQ and finds out the client and the server were getting hammered with all the standing lookups. Needs a new design, talks about some programmer looking formulas like O( A * C1 * C2) and O( N^3 ). I can only think about beer at Sirkus later on.

26.4.2004 - t20 sends yet another post about research slots, from hell. Says of the 15456 slots available in the game, only 1233 are available. Points out that Curse Alliance are somehow missing the point since 800 of the 1233 are in Curse. I spotted a financial opportunity, but settled for the thought of beer next Friday. We end up with some slot and pricing system, eventually ending in the next-gen system available today.

26.4.2004 - PapaSmurf points out that he'd like crews on ships, so we can do "PaperClippy the Crewmember". We hate the Paperclip in a certain help feature. But crews was something we always wanted to do, so tabbed for a later look.

27.4.2004 - Hellmar is worried that you don't get a notification if you are cargo scanned by someone. He doesn't like all these Kestrels targetting him in his Mammoth. Internally dismissed as carebear whine, we quietly moved our Kestrels to another system.

28.4.2004 - t20 suggests a gang and fleet proposal with gang abilities. We think it's a really good proposal, at least one of the best so far. It evolves and a derivate of that has started coming into EVE today. Talk about a slow birth.

28.4.2004 - TomB wants to show damage done, seperated into damage type, with nice colors and stuff. It was one of those nice 'to haves' that's still lingering around on a todo list.

29.4.2004 - Caiman finishes the first "dungeon" in EVE and writes a ToDo. It's all text commands and it takes weeks to create one small one. We decide to hire in people to do a real tool and create a lot of them. They end up as Complexes, mostly in Deadspace.

30.4.2004 - LeKjart has changes to the physics lined up. I read it and see some really complex math stuff (he has a doctorate in chaos math). We survived.

2.5.2004 - I send out a huge word document. Nobody bothers to read it. Gets split into pieces with a mental note to not write so much. Ends with addition of NPC Officers and Commanders, Officer and Faction loot, Implant sets, Industrial Operations, Faction Ships and such. Didn't get the NPCs smacktalking in tho. Escalating paths first mentioned. That one's coming!

3.5.2004 - We decide to change missiles, since everybody is now equipping Kestrel with Cruise Missiles. Damnit.

3.5.2004 - LeKjart introduces new market, market in space, new restrictions and new skills. There was much rejoicing!

6.5.2004 - Dancer of EVE-I fame requests a cool feature, a news-wire type of service, spooling out in-game events such as "5 X corp ships killed in system X" which can be used for news items, kind of like an RSS feed. Really cool. Said we'd do it one day. It might actually get in.

9.5.2004 - I mail about more rare ships like the Magnates and Imperials. It's put on the todo list. Hmmm....

10.5.2004 - TomB suggests we do better incorporating gangs in criminal flagging, that wars spill over and supportive actions also included. Lots of big badaboom after that.

12.5.2004 - GMs request the Message of The Day (MOTD) in the client. They got their wish. Now we just had to find someone to put updated information in there.

17.5.2004 - Lickspittle points out the exploitability of mines. They are removed and will be introduced as Starbase structures laying mine-fields. Uhm ... right.

That's enough for this round folks. I have lots more where that came from. Of course, please note that these have been considerably simplified since the discussion usually involved an e-mail war of ~20 emails and some spicely adjusted for humor.