BigFoot Networks Killer NIC: Killer Marketing or Killer Product?
by Gary Key on October 31, 2006 2:00 AM EST- Posted in
- Networking
Testing Methodology
As we alluded to on the front page, designing a set of testing methodologies that would be consistent and repeatable for the Killer NIC is nearly impossible. It still is to be quite honest. However, we feel like the test criteria we devised is fair for both BigFoot Networks and our readers. There is a phrase that we had to abide by during testing and it makes sense when reviewing our results: Sometimes close enough is good enough.
How do you test a card when the two most critical variables in the online gaming experience, our network connection and the game's server performance, are totally out of our control? Besides these two variables it is impossible to guarantee the movements, actions, experience level, character class, and quantity of players on the servers we connected to would be the same or at the very least close enough for each of our tests. Our testing methodology is by no means perfect we found it to be good enough so this is what we decided to do.
We set up two identical systems for testing as our main focus is to determine if the $279.99 Killer NIC provides a better overall online gaming experience in a wide variety of games than the onboard and essentially free NIC on the Asus P5N32-SLI Premium. Each system had the exact same software image with the only differences being the driver load for the Killer NIC and the nForce 590SLI. Each system was connected to a D-Link Gamer Lounge router with all GameFuel options disabled. We have both Cable and DSL service but decided to strictly use the DSL connection to further reduce the variables. Our Cable connection speed varies greatly depending upon the time of day so this option was not viable. We have enough variables to contend with and adding yet another one was not in our best interest.
We utilized FRAPS to capture our frame rates in each game and the ping time measurements came from the games internal ping rate value where applicable. The ping rates were captured by a third party every sixty seconds and then averaged over the length of game play. We played each game for an average of fifteen minutes online for a total of ten separate sessions at different periods during the day or night. We typically logged on to servers that were fully populated when possible or were close to the maximum amount of players for the selected map.
We utilized each test system at the same time, each player played as the same character class when possible, and connected to the same game server simultaneously. Each player stayed together, followed the same path, and performed the same actions as much as possible during testing. If one player was killed during action then we started the process over until we reached our fifteen minute mark except in Counter Strike: Source and Quake 4 where the time limits ranged anywhere from seven to twenty minutes. Our World of WarCraft play time was 30 minutes in each section due to our play locations and character travel requirements.
Our testing routine was fairly methodical in nature for the FPS and MMO games. We will not mention names but one particular tester took a significant amount of time to learn how to stay out of harms way in most games. It was so bad at one point that we thought about calling Dick Cheney to give him a lesson on how to use a shotgun. Overall, we spent about 300 hours online to bring you around a 100 hours worth of results.
In our RTS testing we had to change our testing methods slightly. Each player played the same map and as the same class each time in a single versus single player game. We scripted out a series of events for each player to follow and probably had a 98% success rate in following the script. We also tried random two on two player matches on the same map and server to see if there would be any noticeable differences in our results. There were not any real differences so we will present our one on one player results. Our outside test party hosted the game utilizing the same DSL service and had a similar system setup on their end. Ping rates could not be accurately measured in our RTS games so they will not be presented, although they were typically very low at game startup due to our test routine.
We also will present results with our D-Link Gamer Lounge DGL-4100 router with GameFuel technology implemented. This router basically promises to accomplish several of the Killer NIC features on the router. This includes providing a platform that reduces latency and boosts network efficiency and performance while intelligently managing and automatically prioritizing network traffic. Sounds familiar right? We are including results from our D-Link DGE-550T PCI Gigabit network adapter to show the results from using a standard PCI NIC in your system with and without our D-Link DGL-4100 optimized.
Since we did not have four identical systems our test plan for the Gaming Router and PCI NIC consisted of powering down our test systems, inserting our D-Link PCI NIC card into each system, removing the Killer NIC and disabling the nForce 590SLI NIC in the BIOS, swapping out the hard drives for a D-Link NIC image, rebooting, and then setting up the D-Link router for optimized operations with our games. We then tested each machine in the same manner listed earlier. The difference being that we ran our tests with GameFuel on and then ran the tests again with GameFuel off. This meant there was usually a ninety minute difference in test results between our Killer NIC / nForce 590SLI combination and our D-Link Router / D-Link NIC results.
A lot can happen in ninety minutes when playing online but fortunately our variables were not off the map after each individual test session. The greatest variability we noticed in the results actually came from the time differences between running our D-Link NIC with GameFuel enabled and disabled. Please be aware of this when reviewing the numbers. However, considering our results are averaged from ten different sessions the overall variability is fairly minimal between test sessions. We set the Killer NIC to Game mode for all of our gaming benchmarks.
There were so many different combinations that we could have tried but in the end we decided to utilize this combination of components to present our results. We could have utilized a standalone PCI-E NIC, used a different video card or processor, and tested strictly on a LAN to remove most variables. We actually did all of that and the results were different but the percentage differences in scores between each solution were always about the same except when using a single processor in World of WarCraft. We will explain that issue shortly. It is time to take a look at the benchmarks and see how well this Killer NIC performs in a high-end gaming system.
As we alluded to on the front page, designing a set of testing methodologies that would be consistent and repeatable for the Killer NIC is nearly impossible. It still is to be quite honest. However, we feel like the test criteria we devised is fair for both BigFoot Networks and our readers. There is a phrase that we had to abide by during testing and it makes sense when reviewing our results: Sometimes close enough is good enough.
How do you test a card when the two most critical variables in the online gaming experience, our network connection and the game's server performance, are totally out of our control? Besides these two variables it is impossible to guarantee the movements, actions, experience level, character class, and quantity of players on the servers we connected to would be the same or at the very least close enough for each of our tests. Our testing methodology is by no means perfect we found it to be good enough so this is what we decided to do.
We set up two identical systems for testing as our main focus is to determine if the $279.99 Killer NIC provides a better overall online gaming experience in a wide variety of games than the onboard and essentially free NIC on the Asus P5N32-SLI Premium. Each system had the exact same software image with the only differences being the driver load for the Killer NIC and the nForce 590SLI. Each system was connected to a D-Link Gamer Lounge router with all GameFuel options disabled. We have both Cable and DSL service but decided to strictly use the DSL connection to further reduce the variables. Our Cable connection speed varies greatly depending upon the time of day so this option was not viable. We have enough variables to contend with and adding yet another one was not in our best interest.
We utilized FRAPS to capture our frame rates in each game and the ping time measurements came from the games internal ping rate value where applicable. The ping rates were captured by a third party every sixty seconds and then averaged over the length of game play. We played each game for an average of fifteen minutes online for a total of ten separate sessions at different periods during the day or night. We typically logged on to servers that were fully populated when possible or were close to the maximum amount of players for the selected map.
We utilized each test system at the same time, each player played as the same character class when possible, and connected to the same game server simultaneously. Each player stayed together, followed the same path, and performed the same actions as much as possible during testing. If one player was killed during action then we started the process over until we reached our fifteen minute mark except in Counter Strike: Source and Quake 4 where the time limits ranged anywhere from seven to twenty minutes. Our World of WarCraft play time was 30 minutes in each section due to our play locations and character travel requirements.
Our testing routine was fairly methodical in nature for the FPS and MMO games. We will not mention names but one particular tester took a significant amount of time to learn how to stay out of harms way in most games. It was so bad at one point that we thought about calling Dick Cheney to give him a lesson on how to use a shotgun. Overall, we spent about 300 hours online to bring you around a 100 hours worth of results.
In our RTS testing we had to change our testing methods slightly. Each player played the same map and as the same class each time in a single versus single player game. We scripted out a series of events for each player to follow and probably had a 98% success rate in following the script. We also tried random two on two player matches on the same map and server to see if there would be any noticeable differences in our results. There were not any real differences so we will present our one on one player results. Our outside test party hosted the game utilizing the same DSL service and had a similar system setup on their end. Ping rates could not be accurately measured in our RTS games so they will not be presented, although they were typically very low at game startup due to our test routine.
We also will present results with our D-Link Gamer Lounge DGL-4100 router with GameFuel technology implemented. This router basically promises to accomplish several of the Killer NIC features on the router. This includes providing a platform that reduces latency and boosts network efficiency and performance while intelligently managing and automatically prioritizing network traffic. Sounds familiar right? We are including results from our D-Link DGE-550T PCI Gigabit network adapter to show the results from using a standard PCI NIC in your system with and without our D-Link DGL-4100 optimized.
Since we did not have four identical systems our test plan for the Gaming Router and PCI NIC consisted of powering down our test systems, inserting our D-Link PCI NIC card into each system, removing the Killer NIC and disabling the nForce 590SLI NIC in the BIOS, swapping out the hard drives for a D-Link NIC image, rebooting, and then setting up the D-Link router for optimized operations with our games. We then tested each machine in the same manner listed earlier. The difference being that we ran our tests with GameFuel on and then ran the tests again with GameFuel off. This meant there was usually a ninety minute difference in test results between our Killer NIC / nForce 590SLI combination and our D-Link Router / D-Link NIC results.
A lot can happen in ninety minutes when playing online but fortunately our variables were not off the map after each individual test session. The greatest variability we noticed in the results actually came from the time differences between running our D-Link NIC with GameFuel enabled and disabled. Please be aware of this when reviewing the numbers. However, considering our results are averaged from ten different sessions the overall variability is fairly minimal between test sessions. We set the Killer NIC to Game mode for all of our gaming benchmarks.
There were so many different combinations that we could have tried but in the end we decided to utilize this combination of components to present our results. We could have utilized a standalone PCI-E NIC, used a different video card or processor, and tested strictly on a LAN to remove most variables. We actually did all of that and the results were different but the percentage differences in scores between each solution were always about the same except when using a single processor in World of WarCraft. We will explain that issue shortly. It is time to take a look at the benchmarks and see how well this Killer NIC performs in a high-end gaming system.
Facebook
Twitter
RSS
87 Comments
View All Comments
LoneWolf15 - Tuesday, October 31, 2006 - link
Almost forgot my one other point: As others have said, Vista's TCP stack is completely different. If the Killer NIC is designed largely around the way previous versions of Windows work, even with Vista drivers it could turn out to be a high-priced piece of hardware that functions no better than a regular PCI nic.TonyB - Tuesday, October 31, 2006 - link
Instead of paying $300 to improve your World of Warcraft PIng Times you can simply do a little research. find out which WoW server is located in your geographic zone. If you live in California, look for a West coast server, if you're in New York, look for a East coast server , if you're in Chicago look for a Central server. Pay the $25 character transfer fee and move your account to the new server, viola!! decreased ping times.ps: this only works if you aren't playing in a server thats in your geographic zone already.
VooDooAddict - Tuesday, October 31, 2006 - link
Because to you... Free = $25 transfer fee and no longer playing with the same group of friends?? Not an option for most people.I aplaud BigFoot for targeting this need with thier product. Riding the coat tails of something like WoW is a tried and true buisness practice. However, it's a failed execution and from the looks of it due mainly to the price. Everyone else agree that if priced for $49 or less they could start enjoying part of that big pile of money called WoW?
$49 for a card that could help keep things running smoothly in large raids with teamspeak running full tilt? They could drop the giant K and the embedded linux to help reduce costs as 95% of the WoW target wouldn't care about those features.
Spacecomber - Tuesday, October 31, 2006 - link
I'll try not to repeat what has already been said about this NIC and Anandtech's review of it; I did have a few random thoughts after skimming through the article.I felt like I had to give BigFoot credit for apprarently seeking out this review from Anandtech. Although they had a lot to gain from a good review, because of the extent of Anandtech's readership and reputation, they must have known that their card would have to deliver on what was advertised in order to get any sort of positive grade (and in the end it did fail to deliver on all but a insignificant fraction of what it promised).
On the other hand, this gets us to the meat of the article. Besides the issue of price, the main problem with this card is that it just isn't ready for prime-time, yet. This means that anyone who does spend $280 on this card is essentially signing up as a beta tester for a product this is still in development. Although gamers probably are familiar with this role (being the first to own some new hardware or new game means being the first to encounter unresolved and frustrating bugs), it still seems a bit perverse to have to pay so much money for this dubious honor.
Given that this seems to be current state of affairs with "cutting edge" games and hardware, I couldn't help imaginging how it would make more sense to put the (presumably) free products into the hands of users who know that they are expected to beta test and work with tech/support staff to make a potentially good product better, rather than the apparent current practice of putting new products into the hands of people who are only required to evangelize for the company.
I guess somewhere along the way the marketing people won out, and comapnies now find their money better spent on marketing rather than product development. Personally, I can't believe that viral marketing is going to be more helpful at making this product a success than would money spent on further testing and development.
Essentially, this is what BigFoot got by their providing a card to Anandtech for testing (constructive feedback and a willing partner to test out potential fixes - in the form of new drivers). But, I suppose BigFoot has bills to pay, now, before they can even worry about getting this card fully up to specification, and this means that someone has to pay.
WileCoyote - Tuesday, October 31, 2006 - link
Review was too wordy! Just large paragraphs of text without any type of organization. Dullest Anandtech article ever.goinginstyle - Wednesday, November 1, 2006 - link
Dullest response ever and it did not take a PhD to write my response. The article flowed from an introduction, with understandable technology descriptions that were not in other reviews, to results with a subjective/objective based ending. It had a couple of wordy sections but it was a lot better than most of the one line sentence descriptions you see in articles today. So please go buy a card, test it, and then provide us your perfect review. If you are unwilling to do this then at least tell us how it should be organized so anthropologists from around the world can understand it. I am sure Eric R. Wolf is turning over in his grave today because an anthropologist did not like the organization of an article.
WileCoyote - Thursday, November 2, 2006 - link
Chill out! Don't get so stressed out over 3 sentences and someone else's opinion. Big deal, I think it's a poorly written article when compared to previous ones at Anandtech. And I'm not questioning the author's education or net worth. I have left dozens of positive comments for previous Anandtech articles/authors so I'm not trolling either.goinginstyle - Thursday, November 2, 2006 - link
The fact is you have not offered an opinion on how to improve the article. You bash it but have not replied as to how you would have written it. If you think it is poorly written and have no issue critizing the author then at least leave a few comments on how you would have changed it. He at least asked for your opinion and others on how to improve the article. If you cannot respond in a professional manner to that request then you are trolling. Tell us how you would have organized the article? Better yet, test the card, and post your own review. You could simply take the time and email Gary with your revisions. He has already edited the article and the sections I thought were a little wordy are gone. Of course I emailed him and relayed my thoughts about the subject instead of dropping a one line dump in the forums.
WileCoyote - Friday, November 3, 2006 - link
Based on your replies I have learned the following:1. If you think an article is poorly written, buy the item and review it yourself.
2. If you do not write in a "professional manner" do not write anything at all. Ha!
3. If you say an article is "too wordy" people will not understand that the solution is to write less words.
4. Talking with people here is a big waste of time. They think they are always right and smarter than you.
The article was boring to me and I dropped a note to say that. I'm sorry for any hurt feelings as a result. If you know me better than I do and want to argue that the article was not boring to me, then feel free. Everything else you have said has nothing to do with what I originally said. I don't know, maybe you had a bad day at work and need to take it out here? Or maybe you want to aim at the easiest target in a thread... say, mine? Go find something challenging and worth spending time on. Unless this is the only way you feel smart? By the way, this post was too wordy. Boring even...yawn.
goinginstyle - Wednesday, November 29, 2006 - link
You must work in a really small cube. Yawn......