New Reticles
Just a minor update to bring you this week, new reticles!
From left to right: Rocket Launcher, Grenade Launcher, Chaingun, Laser Rifle.
Hopefully these reticles should work better over a variety of terrain colours, whilst still being fairly close to the original set which most people are used to.
New US Servers
After some weeks of suboptimal gameplay on our current US game servers located in Kansas City, we've decided to get a replacement server in a better location.
You should now see [US] Justice servers in the list of available games, these are located in Chicago, Illinois. We're hoping that this server should offer better pings, fewer connectivity issues and no freezes during the middle of games. Whilst this server is 500 miles further east than Kansas City, the connectivity is likely to be better for the majority of users, including west coast players.
We've reduced the number of servers running from Hotswap, leaving a couple of servers running incase there are any issues with our new provider. Please give feedback on the forums, as we're looking to eventually retire the old servers.
Staff Updates
Thought it'd be a good time to list some recent staff changes.
Vincismurf, Coding, Art, Everything; First, please welcome Vincismurf to the team. At first, I wasn't sure what he did, so I asked BugsPray, and the answer was 'everything'. So, after some digging, I found out that Vincismurf basically does do everything. He's got a decade's worth of Torque (and its various iterations) experience, an artist, and has a really cool accent.
Anti-Kryst, Community Test Lead; A long time and popular Legions player, Kryst, will be helping to organize and promote discussion from the private testers, as well as being a vital communication link between Devs and the private testers. He'll be taking over this role from Darklord who returns as a regular private tester.
Daphinicus, Developer Test Lead, Writer, Level Design; Please welcome, Daphinicus, to the Legions: Overdrive development team! You've probably seen this well-spoken nerdbomber in the forums, and we're pleased to have him on the Legions: Overdrive team. He is a professional writer and will be helping develop Legions lore, level design/development, and will be working directly with Kryst to collaborate ideas/feedback between private testers/developers.
Please take the time to welcome our newest staff members!
The Magnificent Day of SeymourGore's Birth
In case you missed the forum thread, recently our most prized and intensely attractive Legions: Overdrive developer, SeymourGore, celebrated his birthday on January 8! Please take the time to wish our beloved SeymourGore a belated happy birthday (and cheese, give him your cheese, preferably marble).
Gorge Launches, Core Returns
We've been teasing you with screenshots of Gorge for awhile, and we're excited to announce that Gorge and Core are ready to be released for the Live servers!
Gorge introduces some new elements to Legions: Overdrive gameplay with team specific Jump Pads being one of the more exciting additions.
Also, releasing alongside Gorge is the old fan favorite Core, you'll notice some revisions have been done to Core to help with its flow and improve its playability.
Keep an eye on the servers as these maps have been added to the current rotation when there are higher player counts.
When will these maps be available? Soon.
And soon, as in right now! Hohohohohoho!
One other change is the splash damage calculation. In one of the previous silent releases, we had changed the way splash damage is calculated. We've tweaked the splash damage to be closer to the way it was before, but with the new equation.
Fixing Performance Issues: An Example
By Mabel
1. Introduction
A warning: This is going to be a technical post, so if you're not interested in programming, you can probably skip this one. A few of you are aspiring programmers, and this is for you guys. I'm going to walk through the process of finding and fixing a particularly irritating performance issue.
Legions is made up of about 800k lines of C++ code, 20k lines of TorqueScript (a slow and simple C-like scripting language), and a few scattered lines of Assembly. The game is controlled mostly by TorqueScript and leaves the heavy lifting to C++.
2. Identifying the Problem
In order to fix something, we need to be able to identify a reproduction case (or repro case for short). If we don't know when a problem happens, good luck finding out how to fix it.
Now that we know what we need to do, let's chase the bug. While playing a pickup, I noticed more of a certain kind of stutter than usual. This was an instant red flag: Anomalies usually happen for a reason, so what was different this match? One of the captains had decided to use the (particularly ridiculous) strategy of having three or four Sentinels on defense. What's so special about the Sentinel?
Cluster grenades of course! To try to fix the problem, I SSHed into the server and executed the following in the game's interactive interpreter:
A minute later (now that everybody has respawned and nobody has clusters grenades, just the normal grenade launcher), the server mysteriously started behaving better. Brilliant! Now we know exactly what to chase.
3. Fixing the Problem
The Torque engine has an instrumented profiler. Unfortunately, it's difficult to use and poorly instrumented (so it doesn't give us nearly enough information). Luckily for us, the fact that the game simulation is almost entirely single-threaded means we can use unsophisticated (and free!) single-threaded profilers like Very Sleepy. If we needed a multithreaded profiler I'd have to use something like VTune, but we don't.
The next step is to identify what part of the code is causing the game to lag. Since the problem could be either client-side or server-side (we don't know yet!), we'll start up a listen server so we can get both at once.
To make sure the problem actually happens, let's amplify the cluster a little bit...
and then see what happens...
Let's profile the problem. My game actually stalled for 10+ seconds while processing this enormous lag monster. That's a good thing, though. Since the profiler will tell me how much time is being taken in all the code, it's a bit fiddly to try and isolate one problem unless you can make it a significant amount of time. Now that we've captured the CPU activity during that lag, let's see what's taking so...
Oh. My. God.
That's a lot of time. The game is taking ages casting rays. But why? Taking a look at the top of the "Called From" list, we see that 80% of the calls are coming from inside ccalcExplosionCoverage. Functions prefixed with a "c" like that are generated in a macro that helps create console functions, so let's check the script to see how this is called.
Oh, how lovely, this is an easy one. Let's break this down: We start a container search for objects in range, and we filter by type. One problem -- calcExplosionCoverage is an expensive function call and it's not necessary to do it all the time. This function does a bit of testing to see if the object is occluded or not. This is a necessary step, but only when you're actually going to apply damage. So let's try and eliminate as many cases as possible before we do the coverage check.
We need to do damage in two cases. The first is when we're shooting either a ShapeBase object (Flags and Players) or an armed cluster grenade. Good -- these are checks that can easily be done. Let's rearrange that a bit.
Okay! We just took an ubercluster down from six to three seconds by simply rearranging the code so it executes less often! Let's take a look at the profiler again.
There's a lot of information here, and most of it is painful. There are a few very telling points, though: First, you can see CodeBlock::exec is taking up almost 70% of the CPU time during this lag. Omigawd. Given the sheer volume of the query, using script at all has become the main bottleneck on performance. We can see all sorts of standard library functions rocking the top of this chart sucking down huge amounts of CPU time. A quick inspection of these functions reveals that they are all string operations. The scripting language actually stores all values as strings, then bounces them to and from CPU native values using atof, sprintf, strtol and a bunch of others I'm sure I've missed.
Wait a minute....
Fo reelz. Looks like to speed this up, we're going to need to bring this into C++. Let's do that now.
Much cleaner! I found it interesting that the C++ function is much easier to read than the TorqueScript function (even with all that ugly code to convert to and from strings!). It's also a whopping 6x faster. The ubercluster now takes less than a second to resolve and takes the same time to resolve a 480-explosion cluster as it took to resolve a 16-explosion cluster. Nice. Let's take another look at the profiler.
Alright! Switching over to native code has cut down our execution time immensely and evened out the profile a lot. We still have one particularly pointless and nasty function: the quicksort. We can see it's doing lots of vector length operations, which is another red flag, since calculating vector length is an expensive operation (it needs a square root). Fortunately for us, it's rarely necessary. Vector length can be thought of as the Pythagorean Theorem generalized to n-dimensions (n in this case is 3). So x^2 + y^2 + z^2 = magnitude^2. While we can obviously use the magnitude (and incur the wrath of the sqrt) we can (slightly less obviously) use the squared magnitude for fast comparisons. So we avoid the entire nasty function.
This is one of the easiest, most common optimizations you should never forget to perform, and the fact that this code originally didn't is rather mind boggling. Fortunately, we have an even better one.
Don't do it. We don't need to sort the list at all. None of the code assumes the list is sorted, so we're just transforming data for no good reason. Let's take care of that now.
We introduce a new parameter which will sort the search results. None of the other code that calls this functions assumes we iterate through them in any particular order, but we add this in case they need to. We also perform the lenSquared optimization to speed up the sort. No reason to leave a minefield for other people. And let's check the profile one last time...
Close enough for jazz.
Game Modes
As we work to improve Legions: Overdrive, we wanted to get a feel for what the Community wanted to see as available gametypes. CTF is definitely the staple and most popular gametype, but surely there's room for more.
So, hop on to the forums and visit this thread to let your voice be heard. Want a dedicated deathmatch server to release your stresses? Want to relive your favorite 'Mapubare' moment playing Rabbit? Think there's a place for Arena-style gameplay in Legions: Overdrive? Let us know!
Donations
There have been many concerns brought up about how Legions: Overdrive development will maintain itself, especially since Legions: Overdrive won't be sold for profit. And this is a valid concern, as love alone only runs so far.
There has been interest from community members regarding a donation system for Legions: Overdrive to help maintain server costs, and we're ready to have a system in place to accept donations. All donations from the community will go directly to maintaining the servers and the operating costs associated with them.
More details on how to donate, along with what your donations fund, can be found on our Donate page.
So, as long as there's interest (and donations) from the Legions: Overdrive community, Legions: Overdrive will carry on.
New IRC Channel
We've recently decided to launch a new IRC channel on Quakenet - say hello to #legions.info.
With new players joining Legions, and lots of old players returning, sometimes it's been a bit busy with our main IRC channel. All too often it goes off-topic, but as we all know what happens in an IRC channel rarely matches the channel name! So to help people who have questions about Legions, need a bit of help to get going with the game, we recommend that they visit #legions.info.
It's a strictly Legions only channel - no off-topic talk is allowed. Expect this channel to be strictly moderated, if you aren't helpful to other users or talk about things which don't have anything to Legions - we'll ask you to move it to #legions.
We still expect #legions to be the more popular channel, and continue to be our main community hub. However we hope that our new channel will have a friendly atmosphere which allows users who aren't too familiar with IRC to talk about the thing they joined the channel for - Legions!
All the relevant links and channel rules can be found on our new IRC page.
Gorge Screenshots
Ahoy hoy,
Hope everyone is enjoying their Holidays, as a present from the Legions: Overdrive development team to you, please take the time and enjoy these screenshots of Gorge! Gorge is nearing completion and we're hoping to have the map available to the public shortly.
The Netfix and its Impact on Legions
1. Introduction
So there has been a lot of talk about the netfix and a lot of (uninformed) talk about what it means. I'm going to try to clear up what exactly it is and what it actually means for the game. Due to the nature of the topic this will get a little technical. I'll try my best to explain everything clearly, but be aware there's only so much I can simplify.
2. Background
To understand what the netfix is one must first understand a bit about how games are simulated. There are lots of ways a game can run, but we'll assume we're going to talk about a game like Legions (which is the simplest and most secure model for a multiplayer game). The game world you move around is a collection of objects that are updated in discrete calculations called "ticks". This means that there are a finite (and consistent!) amount of positions a person is in each second, this is important and we'll see why later on. The tick rate of Legions was 1 tick every 32 milliseconds, or 31.25 ticks per second.
The game uses what's called a client/server architecture. This means that there is one server that has the authoritative simulation that determines what happens in the game. So each tick the game will do a variety of functions.
A client will collect its input and then send it to the server. A server will gather all the input packets it has received, react to this input by spawning projectiles or applying forces to their player or camera, check for collisions with objects, correct positions based on said collisions, and move all objects according to the forces acting on them at the time.
3. The Problem
3.1. Projectile Movement Delay
So as it was Legions had a few nasty little bugs with the way it handled updating objects. When a new object was created it would not be updated immediately. Instead it would be added to a list of objects to update, and would be updated the next time the server tried to update its objects. This would create a lag between the time a projectile was fired and the time a projectile would start to move. This essentially added 32ms to everybody's ping.
3.2. Packet Queuing
Legions also had a nasty bug in the code that would send packets over the network. The game would send packets in an inconsistent fashion, instead of sending a steady stream of packets it would (not really) randomly queue a packet to be sent on the next tick. This meant that packets would be sent at an inconsistent rate (1, 0, 2, 1, 1, 0, 2, 0, 2, etc) which could be devastating, especially with packet loss.
3.3. The Tick Rate Itself
Finally, the tick rate was pretty slow given how fast the game is. Players can move fast enough that their collision boxes aren't contiguous volumes, this means that you can miss a player by shooting in the space between where they would have been had the simulation been more detailed. More on this one later.
4. The Solution
4.1. Process Objects Immediately
The first fix is a pretty obvious one. Instead of adding objects to a list that will be executed the next tick, we can process them at the end of the current tick. This is the fix that contributes the most to having to aim behind you to rocket jump.
4.2. No Packet Queuing
This one is an easy fix as well! The netcode for legions was designed for the days of 56k, which have (thankfully) long since passed. We have much higher throughput now, and we can afford to throw way more information over the pipes and still come within a reasonable amount of traffic that can be handled by a lowly DSL line. We simply bypass the queuing logic and send packets without worrying about not sending too often.
4.3. Speed Up The Tick Rate
I decreased the tick interval from 32ms to 15ms. This increases the number of ticks per second to 66.66 (repeating).
5. The Effects
5.1. Projectiles Feel Good
Projectiles actually feel (and look!) like they come out of your gun. They behave more "as you expect" and go more "where you want them to". Midairs are way, way easier, chaingun doesn't look like it's coming from way behind you.
5.2. The Game Is More Consistent
Both your client and the server would suffer from the queuing issue, and at different times. Shots happen when you expect them to more often. The human mind is pretty bad at picking out exactly when something's not immediate, so this one is really hard to pin down. This certainly makes the server simulations behave better and helps with "non-regs" when trying to body-block somebody or dodge a body-blocker.
5.3. Everything
Oh this one is really my favorite, I don't even know where to start. There are lots of interesting properties that this one has, lots of them having to do with interaction with netcode.
Speeding up the tick rate actually makes the game feel significantly different because of the relatively unsophisticated and shoddy physics code. The game feels significantly less "floaty" and more "immediate" because it has more accurate representations of acceleration (from your jets and from gravity). The way that the math works out your jets will push you farther faster and gravity will pull you down faster. This feels significantly more "real".
If you think back to the way that ticks happen you can see all sorts of nasty little edge cases that crop up. Because of the (piss poor) way that collision is handled in the engine there isn't a unified step where you can test projectiles against moving objects. Since objects moved in 32ms increments, you could quite easily miss a player if they had too high a transverse velocity by shooting the space between their calculated positions. Giving the simulation a finer granularity helps to hide these issues without overhauling huge amounts of the game's collision code.
This one requires a bit more information about how players are displayed on your screen. Your game tries to provide to you the illusion of smooth movement of other players by displaying players as moving between the last two positions you receive from the server. You can see that this means that all player representations on your screen are delayed by at least 32 ms. By reducing the tick time to 15 we halved that mandatory delay time and made players' representations more accurate and more resilient to network issues like dropped packets.
This also has a funny side effect of making your listed ping drop by 17ms.
6. So What Does This Mean?
In short, the game's netcode is now far more reliable, and the game behaves much more like it should. I'd estimate that your ping has been reduced by ~30ms on movements, and ~60ms on shots, and non-registers should happen much, much less often.
You're welcome!
7. Other Reading
For more information on the basics of game networking check out What every programmer needs to know about game networking.
For more information on why changing the timestep alters the physics so radically check out Euler's Method. As far as I know Legions uses Euler's method exclusively.