apocrypharrrrrdware!
reported by CCP Mindstar | 2009.02.16 10:54:45 | Comments

With all of these fabulous new blogs about the fancy new things coming to EVE in the rapidly approaching Apocrypha expansion, I thought I would take some time to give you all an update from down here at the Tranquility command console of Virtual World Operations.

As EVE moves towards its 7th year of operation, we are continuing to improve and expand the hardware of the servers running the game, ensuring that everyone gets the most lag free experience possible.   The preparations we are making for Apocrypha are no different - we have been working quietly behind the scenes with quite a large number of significant improvements that allow us to keep growing our user count.  Of course, with the retail launch of EVE just around the corner, we are making sure that when thousands more players come into EVE, everyone still gets the best performance ever!

A Brand new RamSan!

First and foremost - we are making preparations to ensure that our database server will be able to keep up with this ever increasing number of users.   As the true heart of Tranquility, any lag felt at the database level reflects across the entire universe, and it is highly important that we keep our database completely lag free.

For the hardware geeks out there, the WOW Gold Billig database runs on a Microsoft SQL Server 2005 Active/Standby cluster.  It has two identical database servers, with the database stored over two RamSan 400s and a DS4800 fiber channel array.

Our database servers themselves are more than capable of handling the levels of load we generate at present, with CPU usage quite acceptable, however we have noticed that our fiber channel disk array is starting to get pushed to a bit more load than we would like. This is not causing any visible issues just yet, but if left alone it will start to slow us down a bit - it's time to look to the future.

So we bought a 2TB RamSan 500 :-)

The new RamSan has been online for around a week now, running a few minor database tables while we ensure that everything is humming along smoothly (which it is) and making friends with our two existing RamSans.  On Wednesday (18th Feb), we will be moving ALL data currently on our fiber channel drives over to this new RamSan, meaning the entire EVE database will be run from solid-state storage!  Awesome!

That's nice, but I don't know what that technobabble was all about, what does it mean for the average player?

Having much faster storage means that we will see a number of improvements in the future:

Shorter daily downtimes
Shorter expansion deployments
Maintaining virtually nonexistent database lag
And why do we need a whole 6 hours downtime to plug this thing in?

It actually doesn't take very long at all to plug the new RamSan in - that was done offline a week or so back.  The reason we need to take Tranquility and associated websites down is that we have to take the EVE database offline to move its files around.

Moving the data files from our fiber channel disks to the RamSan is a fairly simple "copy->paste" operation, however due to the sheer amount of data being moved (1.1 TB), it will take us around 4 - 5 hours just to copy!  Add in an hour or so of buffer and testing, and you have a 6 hour downtime.

Yummy, I like new stuff - so whatever happened to those super IBM blade thingamajigs you were all excited about last year?

It has been a pretty busy few months for us in Virtual World Operations, to say the least.  There was a dev blog last year outlining some new super-powered blades that we had ordered.  Well, to say the least, we have not stopped in our war on lag!  The 6 x 3.3ghz Intel Wolfdale powered blades mentioned in that blog arrived soon after, performing very nicely indeed (a fact the regular 1000+ players in Jita would probably attest to).  They performed so well, in fact, that we ordered a few more as a bit of a Christmas present...

Just prior to Christmas, we took delivery of two brand new bladecentres packed full of 28 of these 3.3Ghz, 16GB RAM monster blades!  I and CCP Claw were on site in London to welcome them.  Over one hardcore weekend of Windows installer madness we put these 28 new servers straight into service, retiring 56 of our old AMD blades in the process.

In total, we now have 34 of these ultra powerful blades in the cluster - 26 running as SOL blades, and 8 running as Proxy blades.  That gives us a total of 104 out of 208 nodes running on brand new hardware!  That's right, half of Tranquility is running on shiny new top of the line hardware as of Christmas ;)

But it doesn't stop there...

We have all sorts of things on the horizon that focus on our continued efforts attacking lag, latency and server side performance from all angles.

Our network engineers are busily finalizing the plans on some Cisco 7606-S routers which will be placed as the primary routing points within our game server network, and they are also revising our Internet peering strategy to expose Tranquility to our customers in the least number of hops around the globe.  Lower ping times mean faster pew pew!

We have still more juicy hardware about to be delivered to us!  This time, we are going to be among the first in the world to try out Intel's new "Nehalem" based Xeon processors.  We have a test blade en-route to Tranquility and will be benchmarking that against the performance gains we have had from our Wolfdales.

CCP Claw and I are itching to get back out to London for some more hardcore Windows installer action :-)

All the best, and Fly Safe...