BigFoot Networks Killer NIC: Killer Marketing or Killer Product?
by Gary Key on October 31, 2006 2:00 AM EST- Posted in
- Networking
Testing Methodology
As we alluded to on the front page, designing a set of testing methodologies that would be consistent and repeatable for the Killer NIC is nearly impossible. It still is to be quite honest. However, we feel like the test criteria we devised is fair for both BigFoot Networks and our readers. There is a phrase that we had to abide by during testing and it makes sense when reviewing our results: Sometimes close enough is good enough.
How do you test a card when the two most critical variables in the online gaming experience, our network connection and the game's server performance, are totally out of our control? Besides these two variables it is impossible to guarantee the movements, actions, experience level, character class, and quantity of players on the servers we connected to would be the same or at the very least close enough for each of our tests. Our testing methodology is by no means perfect we found it to be good enough so this is what we decided to do.
We set up two identical systems for testing as our main focus is to determine if the $279.99 Killer NIC provides a better overall online gaming experience in a wide variety of games than the onboard and essentially free NIC on the Asus P5N32-SLI Premium. Each system had the exact same software image with the only differences being the driver load for the Killer NIC and the nForce 590SLI. Each system was connected to a D-Link Gamer Lounge router with all GameFuel options disabled. We have both Cable and DSL service but decided to strictly use the DSL connection to further reduce the variables. Our Cable connection speed varies greatly depending upon the time of day so this option was not viable. We have enough variables to contend with and adding yet another one was not in our best interest.
We utilized FRAPS to capture our frame rates in each game and the ping time measurements came from the games internal ping rate value where applicable. The ping rates were captured by a third party every sixty seconds and then averaged over the length of game play. We played each game for an average of fifteen minutes online for a total of ten separate sessions at different periods during the day or night. We typically logged on to servers that were fully populated when possible or were close to the maximum amount of players for the selected map.
We utilized each test system at the same time, each player played as the same character class when possible, and connected to the same game server simultaneously. Each player stayed together, followed the same path, and performed the same actions as much as possible during testing. If one player was killed during action then we started the process over until we reached our fifteen minute mark except in Counter Strike: Source and Quake 4 where the time limits ranged anywhere from seven to twenty minutes. Our World of WarCraft play time was 30 minutes in each section due to our play locations and character travel requirements.
Our testing routine was fairly methodical in nature for the FPS and MMO games. We will not mention names but one particular tester took a significant amount of time to learn how to stay out of harms way in most games. It was so bad at one point that we thought about calling Dick Cheney to give him a lesson on how to use a shotgun. Overall, we spent about 300 hours online to bring you around a 100 hours worth of results.
In our RTS testing we had to change our testing methods slightly. Each player played the same map and as the same class each time in a single versus single player game. We scripted out a series of events for each player to follow and probably had a 98% success rate in following the script. We also tried random two on two player matches on the same map and server to see if there would be any noticeable differences in our results. There were not any real differences so we will present our one on one player results. Our outside test party hosted the game utilizing the same DSL service and had a similar system setup on their end. Ping rates could not be accurately measured in our RTS games so they will not be presented, although they were typically very low at game startup due to our test routine.
We also will present results with our D-Link Gamer Lounge DGL-4100 router with GameFuel technology implemented. This router basically promises to accomplish several of the Killer NIC features on the router. This includes providing a platform that reduces latency and boosts network efficiency and performance while intelligently managing and automatically prioritizing network traffic. Sounds familiar right? We are including results from our D-Link DGE-550T PCI Gigabit network adapter to show the results from using a standard PCI NIC in your system with and without our D-Link DGL-4100 optimized.
Since we did not have four identical systems our test plan for the Gaming Router and PCI NIC consisted of powering down our test systems, inserting our D-Link PCI NIC card into each system, removing the Killer NIC and disabling the nForce 590SLI NIC in the BIOS, swapping out the hard drives for a D-Link NIC image, rebooting, and then setting up the D-Link router for optimized operations with our games. We then tested each machine in the same manner listed earlier. The difference being that we ran our tests with GameFuel on and then ran the tests again with GameFuel off. This meant there was usually a ninety minute difference in test results between our Killer NIC / nForce 590SLI combination and our D-Link Router / D-Link NIC results.
A lot can happen in ninety minutes when playing online but fortunately our variables were not off the map after each individual test session. The greatest variability we noticed in the results actually came from the time differences between running our D-Link NIC with GameFuel enabled and disabled. Please be aware of this when reviewing the numbers. However, considering our results are averaged from ten different sessions the overall variability is fairly minimal between test sessions. We set the Killer NIC to Game mode for all of our gaming benchmarks.
There were so many different combinations that we could have tried but in the end we decided to utilize this combination of components to present our results. We could have utilized a standalone PCI-E NIC, used a different video card or processor, and tested strictly on a LAN to remove most variables. We actually did all of that and the results were different but the percentage differences in scores between each solution were always about the same except when using a single processor in World of WarCraft. We will explain that issue shortly. It is time to take a look at the benchmarks and see how well this Killer NIC performs in a high-end gaming system.
As we alluded to on the front page, designing a set of testing methodologies that would be consistent and repeatable for the Killer NIC is nearly impossible. It still is to be quite honest. However, we feel like the test criteria we devised is fair for both BigFoot Networks and our readers. There is a phrase that we had to abide by during testing and it makes sense when reviewing our results: Sometimes close enough is good enough.
How do you test a card when the two most critical variables in the online gaming experience, our network connection and the game's server performance, are totally out of our control? Besides these two variables it is impossible to guarantee the movements, actions, experience level, character class, and quantity of players on the servers we connected to would be the same or at the very least close enough for each of our tests. Our testing methodology is by no means perfect we found it to be good enough so this is what we decided to do.
We set up two identical systems for testing as our main focus is to determine if the $279.99 Killer NIC provides a better overall online gaming experience in a wide variety of games than the onboard and essentially free NIC on the Asus P5N32-SLI Premium. Each system had the exact same software image with the only differences being the driver load for the Killer NIC and the nForce 590SLI. Each system was connected to a D-Link Gamer Lounge router with all GameFuel options disabled. We have both Cable and DSL service but decided to strictly use the DSL connection to further reduce the variables. Our Cable connection speed varies greatly depending upon the time of day so this option was not viable. We have enough variables to contend with and adding yet another one was not in our best interest.
We utilized FRAPS to capture our frame rates in each game and the ping time measurements came from the games internal ping rate value where applicable. The ping rates were captured by a third party every sixty seconds and then averaged over the length of game play. We played each game for an average of fifteen minutes online for a total of ten separate sessions at different periods during the day or night. We typically logged on to servers that were fully populated when possible or were close to the maximum amount of players for the selected map.
We utilized each test system at the same time, each player played as the same character class when possible, and connected to the same game server simultaneously. Each player stayed together, followed the same path, and performed the same actions as much as possible during testing. If one player was killed during action then we started the process over until we reached our fifteen minute mark except in Counter Strike: Source and Quake 4 where the time limits ranged anywhere from seven to twenty minutes. Our World of WarCraft play time was 30 minutes in each section due to our play locations and character travel requirements.
Our testing routine was fairly methodical in nature for the FPS and MMO games. We will not mention names but one particular tester took a significant amount of time to learn how to stay out of harms way in most games. It was so bad at one point that we thought about calling Dick Cheney to give him a lesson on how to use a shotgun. Overall, we spent about 300 hours online to bring you around a 100 hours worth of results.
In our RTS testing we had to change our testing methods slightly. Each player played the same map and as the same class each time in a single versus single player game. We scripted out a series of events for each player to follow and probably had a 98% success rate in following the script. We also tried random two on two player matches on the same map and server to see if there would be any noticeable differences in our results. There were not any real differences so we will present our one on one player results. Our outside test party hosted the game utilizing the same DSL service and had a similar system setup on their end. Ping rates could not be accurately measured in our RTS games so they will not be presented, although they were typically very low at game startup due to our test routine.
We also will present results with our D-Link Gamer Lounge DGL-4100 router with GameFuel technology implemented. This router basically promises to accomplish several of the Killer NIC features on the router. This includes providing a platform that reduces latency and boosts network efficiency and performance while intelligently managing and automatically prioritizing network traffic. Sounds familiar right? We are including results from our D-Link DGE-550T PCI Gigabit network adapter to show the results from using a standard PCI NIC in your system with and without our D-Link DGL-4100 optimized.
Since we did not have four identical systems our test plan for the Gaming Router and PCI NIC consisted of powering down our test systems, inserting our D-Link PCI NIC card into each system, removing the Killer NIC and disabling the nForce 590SLI NIC in the BIOS, swapping out the hard drives for a D-Link NIC image, rebooting, and then setting up the D-Link router for optimized operations with our games. We then tested each machine in the same manner listed earlier. The difference being that we ran our tests with GameFuel on and then ran the tests again with GameFuel off. This meant there was usually a ninety minute difference in test results between our Killer NIC / nForce 590SLI combination and our D-Link Router / D-Link NIC results.
A lot can happen in ninety minutes when playing online but fortunately our variables were not off the map after each individual test session. The greatest variability we noticed in the results actually came from the time differences between running our D-Link NIC with GameFuel enabled and disabled. Please be aware of this when reviewing the numbers. However, considering our results are averaged from ten different sessions the overall variability is fairly minimal between test sessions. We set the Killer NIC to Game mode for all of our gaming benchmarks.
There were so many different combinations that we could have tried but in the end we decided to utilize this combination of components to present our results. We could have utilized a standalone PCI-E NIC, used a different video card or processor, and tested strictly on a LAN to remove most variables. We actually did all of that and the results were different but the percentage differences in scores between each solution were always about the same except when using a single processor in World of WarCraft. We will explain that issue shortly. It is time to take a look at the benchmarks and see how well this Killer NIC performs in a high-end gaming system.
Facebook
Twitter
RSS
87 Comments
View All Comments
rqle - Tuesday, October 31, 2006 - link
I never like to bash a company product because always believed there a niche market, but at its very best, this product doesnt seem to justified the $300 price cost. I do believe many would buy it at a lower price.My current broadband is 3.0mbps/512kb, i can pay $5 more a month more for Verizon Fiop 15mbps/2mbp, i would rather go that route for my improve network + other capibilities. As for the side processing function, a cheap $200 (Athlon64 3200+ computer system) can do a whole lot more. Am not saying it a bad product, it just price way to high for me.
cornfedone - Tuesday, October 31, 2006 - link
SOS, DD.Just as we've seen with Asian mobos in the last few years, this NIC card is an over-hyped, under-performing POS. Just as mobos from Asus, DFI, Sapphire, Abit, and more have suffered from vcore, BIOS, memory, PCI slot and many other issues, the BigJoke NIC card is more defective goods with zero customer support. I'm sure everyone looks forward to a wiped hard drive image... due to a poorly designed NIC card.
Sooner or later consumers are gonna wise up and stop buying these defective products. Until they vote with their wallet instead of their penis, unscrupulous companies will continue to ship half baked POS products and fail to provide proper customer support. If consumers stop buying these defective goods, then the company will either correct their problems or go tits up.
autoboy - Tuesday, October 31, 2006 - link
I didn't buy a NIC to make up for my small penis, I bought a Porsche. Chicks don't notice my NIC.The K does look pretty cool. Maybe it will fit on my Porsche.
Frumious1 - Tuesday, October 31, 2006 - link
I dare say few if any people have purchased AGEIA or Killer NIC cards. As for your whining about ASUS, DFI, etc. I guess you're one of the people running a budget $50 mobo that can't understand what it's like to really push a system? Or are you the other extreme: you overclocked by 50% or more and are pissed that the system wasn't fully stable?slashbinslashbash - Tuesday, October 31, 2006 - link
You talk a lot about how it's virtually impossible to test something whose entire performance is based almost entirely on an Internet connection which is inherently variable. That's why you need some more control over the variables in your test. Namely, a LAN.For example... have 4 computers, each with exactly the same configs except for the network cards. One of them has the KillerNIC, the rest have different NICs. They are all running Unreal Tournament. You also have one computer set up as the server, on the same Gigabit Ethernet switch as the 4 "player" machines. The level is a small plain square room with no doors, trenches, or any other features. Each of the "players" is running the same script where they have unlimited ammo and a machine gun, running circles around and around, shooting constantly from the minute they respawn. Let the scripts run for 100 hours and capture the framerate and pings on each machine.
So you have 4 computer players running around in circles in a small square room, shooting each other for 100 hours. Yes, there will still be randomness, but over the course of 100 hours it should cancel out, and this test should be replicable. Any real differences between the NICs would come out over time. Run it again to make sure.
Or maybe that's not the best way of doing it. I don't even play UT2003 so I don't know what's really possible and what's not, but I've heard of people doing scripts and stuff. Maybe there are better ways of doing it, but you can eliminate the variable of the Internet connection by limiting your testing to a LAN.
Gary Key - Wednesday, November 1, 2006 - link
We tested over a LAN, the results were not that different, in fact the NVIDIA NIC and Intel PRO/1000 PT cards had better throughput and latencies the majority of the time. We did not show these results as the card is marketed to improve your Online Gaming experience. If the card had been marketed as a must have product to improve your gaming capability on a LAN then it would have been reviewed as such.
When we tested on the LAN the steps you outlined were basically followed from a script viewpoint in order to ensure the variables were kept to a minimum. We did not provide these results, maybe we should have in hindsight. Our final opinion of the card would not have changed.
Thanks for the comments. :)
Frumious1 - Tuesday, October 31, 2006 - link
The product is targeted at gamers. Look at the marketing material. now, while a LAN party goer might get some advantage out of it, there are FAR more people playing games from home using broadband connections. If this only improves performance in a LAN environment (clearly NOT what is being advertised), then it's already a failure. I like what Gary did here: look at real world testing and let us know how it turned out. Who gives a rip about controlled environments and theoretical performance increases if the reality is that the product basically doesn't help much? What's really funny is that they even show a ping "advantage" in FEAR of maximum 0.40ms and average 0.13ms. WTF!? Like anyone can notice a .13ms improvement in ping times! The frame rate improvements might be good (if they were available in many games)... still not $270+ good, though.Bladen - Tuesday, October 31, 2006 - link
Or as you touched on, do the test for a long time, or many many repeats, and let the averages soeak for themselves.shoRunner - Tuesday, October 31, 2006 - link
You pay almost $300 for the ultimate NIC card, and its PCI so it can't even get anywhere near gigbit throughput. AND the CPU utilization isn't even better than an onboard solution. PLEASE. If anyone is truely thinking about buying this, send me a PM I've got some beautiful ocean front property in Montanta to sell you for pennies on the dollar.mlau - Tuesday, October 31, 2006 - link
You underestimate this card greatly. This is the ultimate network card for linux:it could theoretically offload almost all of linux' network stack (including linux'
advanced filtering/routing capabilites and protocols). It's a firewall-router on a
card. IMHO the card is targeted for the wrong crowd (although I understand it somewhat,
since gamers are usually stupid enough to buy 2 video cards and other completely
unnecessary stuff [ageia comes to mind])