iRacing Explains The Daytona 24 Server Issues

In honor of the real world Rolex 24 race event, yesterday iRacing.com organized it’s annual virtual 24 Hours of Daytona. Many iRacing members had prepared themselves to compete in the popular team-based virtual 24-hour endurance race, taking part in a Corvette C7 DP, Audi R8 GT3 LMS, Mercedes AMG GT3, or BMW Z4 GT3.

Sadly enough, some of the race servers suffered some disconnection issues, booting active drivers out of the live race. Taking into account the amount of simultaneous connections in the various races and the fact that the servers farms have to manage international connections for hours on end, it does not seem surprising that a technical issue is a possibility.

However, many iRacing members seemed to have different thoughts regarding the matter and flooded the media channels such as Facebook and Twitter with complaints, and accusations. Some out of disappointment, some out of frustration, and some because they thought it was the perfect time to start a sh*tstorm.

Facebook Server Issues thread

The iRacing staff posted an official reply, explaining what went wrong throughout the weekend. Sadly enough this generated an even bigger negative response.

From our point of view, technical issues are a reality of every online and offline adventure. Whether it is your own hardware failing, or having internet connection issues, experiencing software bugs, PC’s locking up, you name it. We all have been there. Why would it be so strange to see a big international multi-user event suffer the same faith?

 

 

Yesterday we experienced major disconnection issues throughout the iRacing 24 Hours of Daytona. This is hugely frustrating to us, especially for it to happen during a team event like a 24-hour race that requires extensive practice and preparation from our members. Since the event, our network team has been reviewing what went wrong throughout the weekend.

 

This morning, our VP of Technology and overall networking guru Chris Page offered the following statement:

 

“From an iRacing perspective, when we see these issues, the general observation is a drop in network traffic to the servers, and sometimes an accompanying short disruption in our non-race traffic between data centers. So it isn’t a case that our resources are overwhelmed by a DDoS attack or just high usage.

 

Because it isn’t impacting only race traffic, we know that it is network related in general. It isn’t the sim client or race server code. Some members have observed this themselves. One member reported issues with Teamspeak at the same time there were connection issues with iRacing. When one resolved, so did the other.

 

We are very sensitive to carrier issues. We’ve had issues like this in the past, where frequent connection issues were resolved when our ISP dropped a peering partner. I don’t think that there is a network application more sensitive to network issues. With video, it is a one way delivery of data that is easily buffered up. With voice, a loss of connection for a few seconds might go unnoticed if no one is talking at that moment, or a voice might get choppy for a number of seconds, but then all is good again. These events aren’t uncommon, and are common enough that we hardly register or retain their occurrence. It is a meme how laggy things can get in first person shooters, or online RPGs, but even they are forgiving as generally the character stays in position. Racing is different from every other application out there. I could go for a walk in a mall, stop abruptly and close my eyes for 5 seconds without giving it a second thought. The same idea is terrifying to think about doing while driving in a group of cars on the highway.

 

I can’t speak for network carriers, and I don’t know how much it impacts their bottom line, or how high of a priority it is if they have a port that is intermittently failing. They know most applications can deal with it.

 

Well, it impacts us, and we care a whole lot. If you want to help us find the problem, in the other thread I’ve suggested that you can run a tracer command before an event, and after issues, should you have them. If there are changes, it might indicate that one of the changes is related to the issues. It isn’t a smoking gun, but it is something that we can look more closely at for trends.

It isn’t necessarily the change of route itself that is the cause of the problem, but it might be that the initial route was oversubscribed and is dropping traffic. The route changing is an indicator that there might have been problems on the route. It doesn’t mean that there were problems. Similarly, just because a route doesn’t change, it doesn’t mean that there were no problems. It probably means the issues resolved, or are ongoing.

 

Meanwhile we’ll continue to investigate and better all things under our control. This is so frustrating for us all.

 

Edit: I remember interviewing a network engineer about his projects. He shared with me that their projects are front loaded. With the start of a new year, the have access to budgets to implement their upgrades and expansions. As a result, there tends to be more network changes during the beginning of the year compared to the end.”

 

Official Webpage – iRacing.com