Carbon and the Core Technology Group
Over the last few days you will have seen the blogs coming out from the boffins in our engineering teams about Mass Testing, Long Lag and Thin Clients. In some of these blogs you will have seen reference to ‘Core' teams and so I wanted to take this chance to introduce the Core Technology Group (CTG), let you know what it is we do and where we sit within the CCP development structure.
At the end of June, our CEO Hilmar Veigar Petursson referred to CCP's Core Technology Platform at his opening keynote of China GDC. For some time now, we have been internally branding the framework we use to build all of our games as "Carbon."
Giving the framework a name helps us a lot with communication internally and of course to you, the wonderful players of EVE Online. There is also the old Icelandic saying of "if you know its name, you can kill it", which basically talks about the power of knowing the names of things and how that gives you the ability to control said phenomena. As complex and multifaceted as our Carbon framework is then we certainly benefit from all the help we can get to wield it. And the custodians of Carbon are the Core Technology Group.
This video shows some of the Carbon technology in action as was presented by two CTG members at GDC this year.
So, why have Carbon and what is it really?
As CCP and EVE continue to grow, it makes sense for us to consider our existing technologies and re-use these if appropriate across our projects as part of the Carbon framework. This becomes more appealing if those technologies are proven (or, as we call it, ‘battletested') in the crucible that is a game being used by a passionate, resourceful and sometimes devious player base. Furthermore, as our new projects continue to mature, we can take the technologies developed by them and apply them to EVE. This means that the EVE development teams get some great new technologies which it hasn't had to spend any developer time creating. Having the CTG take the strain making this happen ensures the EVE developers can concentrate on developing EVE and make best use of this common technology.
It also makes sense for a central group (the CTG) to build brand new Carbon technologies which we know can be shared. This group can then deploy and support the technology across CCP to whoever needs or wants to use it.
So what is the Carbon technology framework?
Well, it is mostly things which operate at a low level in our games. The idea is that the framework will consist of all of the key technology pieces for an MMO that our game teams can pick up and use. The Trinity2 graphics engine was the first piece of Carbon technology (even if we didn't know it as Carbon back then) introduced back in late 2007 by the newly formed Core group. Since then, various parts of our technology have been ‘Carbonised' such as the graphics refresh that happened as part of Apocrypha. You can refer back to these Dev Blogs for details of things which have been made part of Carbon in the past.
Of course, just creating and deploying new or existing technology, even if it is ‘battletested', is only part of the story. The CTG also spends a significant portion of its time re-working, re-writing and re-engineering parts of our codebase in a continuous effort to keep it up to date as technology advances. This also allows us to prepare the codebase to accept new technologies into the Carbon framework, allowing us to keep pushing the boundaries of what EVE Online can be.
Who is in the Core Technology Group?
Well, the CTG started, as mentioned earlier, with a few graphics programmers working under the direction of our Chief Technology Officer to produce the next generation of the EVE graphics engine. From that small beginning, we have been hiring new people into CCP in order to fill a number of teams. The CTG is a separate group within CCP, it is not part of EVE. We do not count as part of the EVE headcount and we have a separate budget and hiring plan. However, in reality we work very closely with EVE although Core does not work on game features. We provide the reprocessed minerals that the game teams then use to, in the case of EVE, build your serious internet spaceships.
We currently have 5 teams within the CTG, although we do use carefully chosen 3rd parties when needed. Futuremark is a good example where we are working with them on a new part of the graphics rendering engine. It must be noted that whilst the CTG teams all work on Carbon, some parts of Carbon are not maintained by the CTG. These are usually specialist areas which can be managed by the team which has that expertise. Animation is a prime example. The animation team is based in Atlanta and the work they are doing will be part of Carbon.
The 5 Core Technology Group teams consist of the following...
2 x Core Graphics Teams (12 people):
These teams are those working in the bowels of Trinity2, making it perform better, provide sexier visuals and use more up-to-date technology. The Carbon Character Technology video from earlier in this blog demonstrates a small portion of their work-in-progress. As you can see, this team has been working on the graphics technology for avatars and the environments they will inhabit. For EVE that's Incarna. These guys also develop and maintain our in-house tools system, ‘Jessica', which is used by pretty much every developer in CCP and contains the Trinity2 rendering engine. As requests for improvements, new functionality or bug fixes come in from the EVE developers, the Core Graphics guys get on the case and deliver. Jessica is also used by the EVE video team in making all of our trailers, allowing them serious time-saving shortcuts in staging assets for dramatic narrative effect.
Core EVE Graphics Team (3 people):
This team consists of three Core Graphics developers who are 100% assigned to EVE, working as part of a nine person EVE development scrum. Due to the demands of working in a cross-project, multinational company, parts of the CTG must occasionally shift its focus onto a particular project on a temporary basis. When this concentration is not on EVE (currently it is), the Core EVE Graphics team provides the link and continuity of Core involvement in the graphics side of EVE. By staying 100% laser focused, they make sure EVE continues to get the Core graphics programmer support it needs. You may have seen the recent Dev Blog by CCP Blaze about Tyrannis Performance Improvements who is on this team. These guys are instrumental in working closely with the EVE Art team to bring you new ships, planets and other graphical wonders.
Core Infrastructure Team (4 people):
This team is really at the heart of everything we are able to put out to our customers at CCP. These guys have created a common set of Productivity software, tools and technology which allows us to build more efficiently EVE and its patches from the source code, significantly reducing the time it takes to make EVE. This team has also built the repair tool which CCP Mandrake blogged about recently. In addition, this team has been working for a long time on delivering a way for our developers to be able to stress test EVE when they are developing new features or investigating and fixing bugs. We are now rolling this out and you can read more in the Dev Blog from CCP Atropos about the Thin Clients. I believe this is a significant step forward in how we are able to build and maintain EVE, allowing us to deliver a virtual world that we are much more confident is able to operate as we intend when we have record numbers of players hammering our servers and software.
Core Cluster Team (7 people)
What is one of the cornerstones of EVE? Lots of people interacting in a single universe without sharding. Obviously, EVE has scaled to dizzying heights over the last 7+ years and we now have more players than ever immersed in New Eden. However, we know that there are problems and we know we have to be constantly fighting to help the EVE cluster scale well beyond where it is now. This important task rests on many people all across CCP, from operations and virtual world staff to the EVE software developers to the EVE game designers and beyond. A key element in the fight to improve how we scale and reduce ‘Lag' in the game is the work of the Core Cluster team. This team has the high level goal of developing our cluster technology to allow EVE to scale well into the future, even as we put more demands on it with more pilots inhabiting a more dynamic and developed EVE universe. This is no overnight fix and there have been some great discussions (as well as some painful ones) on EVE-O and other forums. Many of these threads have a good grasp, if not of the specifics, then of the general idea that we are trying to solve some of the hardest problems in a number of very complex disciplines. CCP Warlock works as the Distributed Systems Architect for CCP and the Core Cluster team and recently released an excellent blog on some of the challenges we are facing.
As recent blogs have started to describe, we are doing things right now to attack lag. We have people looking at specific bugs and issues which have been plaguing fleet fights. We are well aware that there are lag issues around large fleet fights (and some not so large ones) which are reducing your enjoyment of this part of EVE. We know because you have been telling us about your experiences regarding things like module lag, jump in lag and blackscreens to name just a few. We also know because we, as players ourselves (long time lurker, first time poster checking in after 5 years playing) experience these issues. I will leave the specifics of our progress against these problems to the actual developers working on "lag" who can get low down and dirty with some real techy blogs.
Genuine progress on these issues is being made thanks to the tools being produced by Core Infrastructure that are being used by the Core Cluster and EVE development teams. The nature of the lag problem means that it takes time to diagnose and address the problems, but we have invested significantly in getting the right tools and people to help us identify and improve the situation as fast as possible. Once fixes have been deployed to TQ and we have real evidence that they are making a positive difference to your playing experience, you will see more Dev Blogs detailing the processes and fixes.
So in terms of teams and what we work on, that is the Core Technology Group as it stands right now. You can find some more information here. We will continue to grow proportionately to CCP's development teams and the growing scope of the Carbon framework. We are actively recruiting into the CTG and if you are interested, you can apply at the usual place, the CCP jobs page.
Now that you hopefully have a bit more clarity on how CCPs Core teams are structured, it is important that you, the EVE players, know that the focus of the Core Technology Group is on supporting EVE and the experience of playing it. Through our Carbon strategy we make sure that all future product development at CCP feeds back into Eve, as they all integrate back to Carbon. We also make sure that CCP has a robust battletested core framework to win our war against the impossible.
