Diablo 3 and Path of Exile Networking Breakdown

Blizzard is visiting DigiPen on Thursday (9/27/2018) and to market myself to them I’m going to do a small scale network analysis on Diablo 3 (my favorite of their games). In order to do so, I’ll be comparing everything I find in Diablo 3 to what I can find in Path of Exile, Diablo 3’s main competitor. Let’s get to it.

Latency

First thing to get out of the way: both Diablo and PoE do their networking over TCP. This incurs an amount of latency due to Nagle’s algorithm and naive packet loss handling. Like WoW and other games that use TCP for their networking both Diablo 3 and PoE set the PUSH flag to avoid Nagle’s for packets being sent by the server, but packets generated by the client aren’t setting the PUSH flag and therefore are using Nagle’s. Setting the PUSH flag is a registry thing in windows so doing so is outside the realm of a normal user. Using Nagle’s client side algorithm is going to cause some latency, but not anymore than a game on UDP that buffers gameplay inputs. In general, ARGs like Diablo and PoE’s player interaction isn’t frequent so the latency doesn’t have to be incredibly low. The only examples I can think of from Diablo are buffs, debuffs, and crowd control. These games can get away with a bit more latency than high-paced shooters like Overwatch, CoD, and Fortnite. The aforementioned games run server updates at 60Hz to enforce fairness in the competitive environment whereas in Diablo and PoE the majority of the game is PvE so they can get away with lower refresh rates.

So for these two games:

Game Client/Server Send Rate while idle Client/Server Send Rate just walking Client/Server Send Rate full gameplay Movement Response Time Ability Response Time Game Reported Ping ICMP RTT
Diablo 3 5/5 Hz 15/15 Hz 30/40 Hz 81 ms 65 ms 95 ms 38 ms
Path of Exile 20/33 Hz 30/35 Hz 32/40 Hz 136 ms 111 ms 40 ms 38 ms

Latency tests done by recording gameplay and keyboard footage at 120 FPS and then counting the frames it takes for the character to begin their animation. Not the most scientific, but it’s something.

The actual game session servers blocks ICMP so I pinged the closest ip according to tracert (24.105.30.129) for the listed RTT.

PoE’s latency might be a little skewed due to the game being poorly optimized and running slowly on my computer, but I don’t feel bad because with my GTX 1070 I’m still an above average consumer.

PoE’s packets Up/Down have high amounts of variance during gameplay. The client bounces between 26 Hz and 36 Hz frequently and the server bounces between 35 Hz and 50 Hz.

Bandwidth

Game Up Down
Diablo 3 1533 bits/s 38k bits/s
Path of Exile 748 bits/s 32k bits/s

Both games use very little bandwidth. That’s pretty impressive for the amount of enemies that are on the screen at once.

Network Topology

* This section contains a lot of conjecture. *

Diablo sets a great example.

The first connection is to a Battle.net login server. You log in over 443 which makes me think its an HTTPS REST call which is a great way to do authentication (according to Facebook and Google). You then get sent to the regional server to download your character info and connected to various other servers to manage global chat, Battle.net friends lists, etc. When you start a session you’re given a session server to play on which allows Blizzard to enforce DRM and loot validity. I remember the Borderlands games had big issues with players tampering with loot because that game was peer-to-peer so moving to client-server in order to maintain fairness (especially since Diablo shipped with an auctionhouse) is a good choice. Blizzard also hosts their own servers since they’ve been doing this before “The Cloud” was a big thing. Simple, scalable, everything you’d want from a well designed network topology.

This is where I become disappointed. Path of Exile is really bad at this.

The login process is normal. From what I can tell it’s custom socket stuff, but it could just be REST on a different port. You’re regional server manages only the character lists. When you start playing you are then moved to an in-game location specific server. Yes, in an ARPG where 95% of the game is either solo or a party of up to 8, the network topology of PoE is more similar to that of a full-on MMO. Sll the chat and friends lists features also get funneled through the location specific server. The town areas are the only areas where the player can see and interact with random players also in that town. The big problem is even if I’m playing by myself every time I move to a new area I have to do a host migration. This makes me believe that the original plan for PoE was more MMO-like with players encountering each other in the world more often, but something changed or there was a technological limitation so they switched gears.

These instance servers are hosted by a cloud hosting company called INAP. Apparently Hi-Rez also uses INAP so it doesn’t seem that bad; they have well placed servers across NA, EU, Oceania, but they seem to lack servers in South America and Africa which is a bummer. PoE’s network topology made me think I was looking at a different game and that saddened me. I hope they eventually move to a better model, but that would definitely cost a lot of money.

Conclusion

Comparatively, the games preform very similarly from both a latency and bandwidth perspective, but PoE’s network topology doesn’t make much sense for the game they shipped so I prefer Diablo'3 networking model overall. Kind of glad I reached that conclusion too; it would have been awkward to apply to Blizzard with a blog post containing a dis-favorable comparison to one of their largest competitors in the field.

Both game’s could still improve. Their state replication layers seem to work fine for the kind of games they are, but if they start to dabble with more time sensitive gameplay mechanics or some form of serious PvP combat they should consider moving off of TCP and increasing server update rates. Also, PoE should get its networking topology together, but it never will because it’s already shipped so I’m just going to have to live with this knowledge for the rest of my life.