As part of our infrastructure migration project, I had planned to use OVH’s IP Load Balancing to balance traffic between datacenters.
Things didn’t go as planned
The initial idea was simple enough: spread requests across 2 web front-end servers located in 2 different datacenters (Strasbourg and Roubaix).
Why 2 different datacenters? Once you start thinking “redundant infrastructure”, you get paranoid about everything that could take you offline, and having a redundant setup spread over 2 different geographic locations greatly reduces the risk of an outage.
So I signed up for the multi-datacenter compatible IP LB option, but after declaring my two backends (the web front ends) so the IP LB could spread the load, I quickly noticed that only the Roubaix server was getting any requests. Why? Before answering that, let’s start with a rage tweet about an infrastructure OVH had signed off on.
L'option "compatible multidatacentre" pour l'ip load balancé ne permet pas de faire du load balancing entre datacenter. gros WTF. @ovh_fr
— Tristan Bessoussa (@sf_tristanb) August 18, 2014Translation: “The ‘multi-datacenter compatible’ option for load-balanced IPs doesn’t let you load balance between datacenters. Big WTF.”
Anycast IP, the culprit
Anycast is an addressing and routing technique that sends data to the “nearest” or “most efficient” server, depending on the routing policy.
So with anycast, you can give your Strasbourg server a weight of 1 and your Roubaix server a weight of 8 (so that Roubaix gets 8 times more requests than Strasbourg) all you like: Strasbourg will be completely ignored. Given how close Roubaix and Strasbourg are, I’ll let you imagine my reaction.
A blessing in disguise
As it turns out, this whole anycast IP thing is actually a very good idea for companies offering services worldwide. Here is an infrastructure idea that would work with multi-datacenter IP load balancing:
In the end, I moved the 2 offending servers back to Roubaix and everything fell into place. The lower latency inside a single datacenter, over a vRack network, will even let me set up a Galera or Percona cluster, but that’s another story.
Conclusion
IP Load Balancing lets you load balance within different datacenters, not between datacenters.
Edit: Octave Klaba, OVH’s CEO, responded to this post: apparently you can load balance between datacenters if you ask support:
@sf_tristanb tu as 2 options avec IP LB "Anycast": master/slave ou master/master. Par défaut, c'est master/slave. L'autre on le fait au mano
— Octave Klaba / Oles (@olesovhcom) August 27, 2014Translation: “You have 2 options with IP LB ‘Anycast’: master/slave or master/master. Master/slave is the default. The other one we set up by hand.”