EverQuest 2 Game Update Blog

Rothgaar tagged along with Dracos Argent last night during our raid to view the lag that was plaguing many of Butcherblock raid and heroic instances.

While we  were fighting Waansu he was able to diagnose the lag after seeing a 6 second lag spike in during one of our pulls. After doing some research he found that instances from Butcherblock were being loaded on to servers already heavily loaded due to an issue with the load balancing server.

Rothgaar explained the following during our raid.

Rothgar Shouts, “It appears that all of butcherblocks instanced content is being routed to an overloaded dynamic cluster, but all of the overland zones seem to be on the standard cluster.”

“I have looked at many butcherblock instances and they all are running on the same overloaded cluster while the normal cluster has available processing power. So its defiantly an issue with the world manager balancing zones in the wrong place.”

Later he posted on the official forums the following:

The fix is complete and I’m seeing instances for Lucan and Butcherblock being properly load-balanced now.  Any existing instances will need to shut down and be restarted for them to get moved to less-busy servers.  New instances will be starting up just fine and should enjoy a somewhat lag-free environment.

It generally takes an instance about 20 minutes to shut down after everyone leaves so if your zone is currently unplayable and don’t mind a 20 minute break, you should be able to resume shortly.

We’re very sorry for the poor performance this undoubtedly caused over the weekend for BB and LDL players.  I’ll see if I can put Brenlo on the spot for some bonus XP.

Unfortunately, he followed his post in the forums with this:

Sorry to be the bearer of bad news, but this issue was something that started on Friday.  We’re aware that there are still lag issues and I by no means meant to imply that this last fix “fixed lag”.  There was a definite problem with load balancing that was making the problem much worse and that has been fixed.  You’ll probably still see lag issues that were there prior to Friday and we are working on tracking those down.

http://forums.station.sony.com/eq2/posts/list.m?start=75&topic_id=476210

I would like to thank Rothgaar for his time to come to our raid to view the lag issue  and be able to put the fix in so we can raid again!

I hope a fix to the remaining lag issues get resolved quickly as possible, such as the long zone times, the periodic broker lag and worst of all, the lag while fighting contested mobs in overland zones.

This is a PM from Rothgaar to a Dracos Argent member with some insight of the server system architecture:

We actually do have tools in place to monitor server health. It’s a pretty amazing tool and would be fun to show it to you and what all it can do. We also have people monitoring server status 24 hours a day so you’d think they would be made aware of this type of issue. Unfortunately with so many games to monitor and so many different types of health conditions to watch for, we have to create specific alerts to notify the Operations department when something is outside of ideal working parameters. We have many alerts created for things but in this particular instance, there was no alert that would have caught the problem. The first thing that came to mind when I saw the issue was “why don’t we alert on number of processes per machine?” So today you can bet I’ll be working with Operations to get that alert created so this won’t happen in the future. I’m sure there are other situations out there that we could be watching for as well, but its difficult to imagine some of them until they happen. This particular problem was a new one that had never occurred before. Ideally the quad-core servers should be running around 4 processes per machine so they each have their own dedicated CPU. But in this case some machines were running as many as 16 processes. It was pretty insane and no wonder why those processes were starved for CPU cycles.