BigFoot Networks Killer NIC: Killer Marketing or Killer Product?
by Gary Key on October 31, 2006 2:00 AM EST- Posted in
- Networking
Testing Methodology
As we alluded to on the front page, designing a set of testing methodologies that would be consistent and repeatable for the Killer NIC is nearly impossible. It still is to be quite honest. However, we feel like the test criteria we devised is fair for both BigFoot Networks and our readers. There is a phrase that we had to abide by during testing and it makes sense when reviewing our results: Sometimes close enough is good enough.
How do you test a card when the two most critical variables in the online gaming experience, our network connection and the game's server performance, are totally out of our control? Besides these two variables it is impossible to guarantee the movements, actions, experience level, character class, and quantity of players on the servers we connected to would be the same or at the very least close enough for each of our tests. Our testing methodology is by no means perfect we found it to be good enough so this is what we decided to do.
We set up two identical systems for testing as our main focus is to determine if the $279.99 Killer NIC provides a better overall online gaming experience in a wide variety of games than the onboard and essentially free NIC on the Asus P5N32-SLI Premium. Each system had the exact same software image with the only differences being the driver load for the Killer NIC and the nForce 590SLI. Each system was connected to a D-Link Gamer Lounge router with all GameFuel options disabled. We have both Cable and DSL service but decided to strictly use the DSL connection to further reduce the variables. Our Cable connection speed varies greatly depending upon the time of day so this option was not viable. We have enough variables to contend with and adding yet another one was not in our best interest.
We utilized FRAPS to capture our frame rates in each game and the ping time measurements came from the games internal ping rate value where applicable. The ping rates were captured by a third party every sixty seconds and then averaged over the length of game play. We played each game for an average of fifteen minutes online for a total of ten separate sessions at different periods during the day or night. We typically logged on to servers that were fully populated when possible or were close to the maximum amount of players for the selected map.
We utilized each test system at the same time, each player played as the same character class when possible, and connected to the same game server simultaneously. Each player stayed together, followed the same path, and performed the same actions as much as possible during testing. If one player was killed during action then we started the process over until we reached our fifteen minute mark except in Counter Strike: Source and Quake 4 where the time limits ranged anywhere from seven to twenty minutes. Our World of WarCraft play time was 30 minutes in each section due to our play locations and character travel requirements.
Our testing routine was fairly methodical in nature for the FPS and MMO games. We will not mention names but one particular tester took a significant amount of time to learn how to stay out of harms way in most games. It was so bad at one point that we thought about calling Dick Cheney to give him a lesson on how to use a shotgun. Overall, we spent about 300 hours online to bring you around a 100 hours worth of results.
In our RTS testing we had to change our testing methods slightly. Each player played the same map and as the same class each time in a single versus single player game. We scripted out a series of events for each player to follow and probably had a 98% success rate in following the script. We also tried random two on two player matches on the same map and server to see if there would be any noticeable differences in our results. There were not any real differences so we will present our one on one player results. Our outside test party hosted the game utilizing the same DSL service and had a similar system setup on their end. Ping rates could not be accurately measured in our RTS games so they will not be presented, although they were typically very low at game startup due to our test routine.
We also will present results with our D-Link Gamer Lounge DGL-4100 router with GameFuel technology implemented. This router basically promises to accomplish several of the Killer NIC features on the router. This includes providing a platform that reduces latency and boosts network efficiency and performance while intelligently managing and automatically prioritizing network traffic. Sounds familiar right? We are including results from our D-Link DGE-550T PCI Gigabit network adapter to show the results from using a standard PCI NIC in your system with and without our D-Link DGL-4100 optimized.
Since we did not have four identical systems our test plan for the Gaming Router and PCI NIC consisted of powering down our test systems, inserting our D-Link PCI NIC card into each system, removing the Killer NIC and disabling the nForce 590SLI NIC in the BIOS, swapping out the hard drives for a D-Link NIC image, rebooting, and then setting up the D-Link router for optimized operations with our games. We then tested each machine in the same manner listed earlier. The difference being that we ran our tests with GameFuel on and then ran the tests again with GameFuel off. This meant there was usually a ninety minute difference in test results between our Killer NIC / nForce 590SLI combination and our D-Link Router / D-Link NIC results.
A lot can happen in ninety minutes when playing online but fortunately our variables were not off the map after each individual test session. The greatest variability we noticed in the results actually came from the time differences between running our D-Link NIC with GameFuel enabled and disabled. Please be aware of this when reviewing the numbers. However, considering our results are averaged from ten different sessions the overall variability is fairly minimal between test sessions. We set the Killer NIC to Game mode for all of our gaming benchmarks.
There were so many different combinations that we could have tried but in the end we decided to utilize this combination of components to present our results. We could have utilized a standalone PCI-E NIC, used a different video card or processor, and tested strictly on a LAN to remove most variables. We actually did all of that and the results were different but the percentage differences in scores between each solution were always about the same except when using a single processor in World of WarCraft. We will explain that issue shortly. It is time to take a look at the benchmarks and see how well this Killer NIC performs in a high-end gaming system.
As we alluded to on the front page, designing a set of testing methodologies that would be consistent and repeatable for the Killer NIC is nearly impossible. It still is to be quite honest. However, we feel like the test criteria we devised is fair for both BigFoot Networks and our readers. There is a phrase that we had to abide by during testing and it makes sense when reviewing our results: Sometimes close enough is good enough.
How do you test a card when the two most critical variables in the online gaming experience, our network connection and the game's server performance, are totally out of our control? Besides these two variables it is impossible to guarantee the movements, actions, experience level, character class, and quantity of players on the servers we connected to would be the same or at the very least close enough for each of our tests. Our testing methodology is by no means perfect we found it to be good enough so this is what we decided to do.
We set up two identical systems for testing as our main focus is to determine if the $279.99 Killer NIC provides a better overall online gaming experience in a wide variety of games than the onboard and essentially free NIC on the Asus P5N32-SLI Premium. Each system had the exact same software image with the only differences being the driver load for the Killer NIC and the nForce 590SLI. Each system was connected to a D-Link Gamer Lounge router with all GameFuel options disabled. We have both Cable and DSL service but decided to strictly use the DSL connection to further reduce the variables. Our Cable connection speed varies greatly depending upon the time of day so this option was not viable. We have enough variables to contend with and adding yet another one was not in our best interest.
We utilized FRAPS to capture our frame rates in each game and the ping time measurements came from the games internal ping rate value where applicable. The ping rates were captured by a third party every sixty seconds and then averaged over the length of game play. We played each game for an average of fifteen minutes online for a total of ten separate sessions at different periods during the day or night. We typically logged on to servers that were fully populated when possible or were close to the maximum amount of players for the selected map.
We utilized each test system at the same time, each player played as the same character class when possible, and connected to the same game server simultaneously. Each player stayed together, followed the same path, and performed the same actions as much as possible during testing. If one player was killed during action then we started the process over until we reached our fifteen minute mark except in Counter Strike: Source and Quake 4 where the time limits ranged anywhere from seven to twenty minutes. Our World of WarCraft play time was 30 minutes in each section due to our play locations and character travel requirements.
Our testing routine was fairly methodical in nature for the FPS and MMO games. We will not mention names but one particular tester took a significant amount of time to learn how to stay out of harms way in most games. It was so bad at one point that we thought about calling Dick Cheney to give him a lesson on how to use a shotgun. Overall, we spent about 300 hours online to bring you around a 100 hours worth of results.
In our RTS testing we had to change our testing methods slightly. Each player played the same map and as the same class each time in a single versus single player game. We scripted out a series of events for each player to follow and probably had a 98% success rate in following the script. We also tried random two on two player matches on the same map and server to see if there would be any noticeable differences in our results. There were not any real differences so we will present our one on one player results. Our outside test party hosted the game utilizing the same DSL service and had a similar system setup on their end. Ping rates could not be accurately measured in our RTS games so they will not be presented, although they were typically very low at game startup due to our test routine.
We also will present results with our D-Link Gamer Lounge DGL-4100 router with GameFuel technology implemented. This router basically promises to accomplish several of the Killer NIC features on the router. This includes providing a platform that reduces latency and boosts network efficiency and performance while intelligently managing and automatically prioritizing network traffic. Sounds familiar right? We are including results from our D-Link DGE-550T PCI Gigabit network adapter to show the results from using a standard PCI NIC in your system with and without our D-Link DGL-4100 optimized.
Since we did not have four identical systems our test plan for the Gaming Router and PCI NIC consisted of powering down our test systems, inserting our D-Link PCI NIC card into each system, removing the Killer NIC and disabling the nForce 590SLI NIC in the BIOS, swapping out the hard drives for a D-Link NIC image, rebooting, and then setting up the D-Link router for optimized operations with our games. We then tested each machine in the same manner listed earlier. The difference being that we ran our tests with GameFuel on and then ran the tests again with GameFuel off. This meant there was usually a ninety minute difference in test results between our Killer NIC / nForce 590SLI combination and our D-Link Router / D-Link NIC results.
A lot can happen in ninety minutes when playing online but fortunately our variables were not off the map after each individual test session. The greatest variability we noticed in the results actually came from the time differences between running our D-Link NIC with GameFuel enabled and disabled. Please be aware of this when reviewing the numbers. However, considering our results are averaged from ten different sessions the overall variability is fairly minimal between test sessions. We set the Killer NIC to Game mode for all of our gaming benchmarks.
There were so many different combinations that we could have tried but in the end we decided to utilize this combination of components to present our results. We could have utilized a standalone PCI-E NIC, used a different video card or processor, and tested strictly on a LAN to remove most variables. We actually did all of that and the results were different but the percentage differences in scores between each solution were always about the same except when using a single processor in World of WarCraft. We will explain that issue shortly. It is time to take a look at the benchmarks and see how well this Killer NIC performs in a high-end gaming system.
Facebook
Twitter
RSS
87 Comments
View All Comments
WileCoyote - Friday, November 3, 2006 - link
I said it's too wordy. That means take out words. Are you people for real?Frumious1 - Tuesday, October 31, 2006 - link
That post was too short. Three sentences with no real support to back it up. Lamest comment ever.Look, sorry you have poor reading skillz, but a lot of us are quite educated and like to know the WHY and HOW besides just the WHAT. My first thought was "how in the hell is this product supposed to do ANYTHING for frame rates!?" Gary answered that by looking into the details more than superficially. Oh, sure, it doesn't really deliver -- a 5% increase on a moderately high-end system is pretty silly for a $270 product -- but that it can affect frame rates at all is surprising to me. I now have hope for Windows Vista, at least on the networking stack side of things.
What's really odd is that the review is pretty clearly advising people to NOT go out and buy this "killer" product, because it just isn't that great. If I saw a review on AnandTech that bashed a product without any backing material, I'd feel I was reading something at HardOCP. Well, okay, they back up their complaints sometimes, but their testing methodology is worse than suspect. Remember the Core 2 "launch" reviews where they used a midrange GPU config to conclude that it did nothing for gaming performance?
Anyway, there seem to be quite a few new names on here for this article. Wonder how many are employed by BigFoot? It's like there are a bunch of people bashing Gary for even reviewing the product at all, another group that's bashing him for not liking the card, a third group bashing him for not being able to do a 1 page writeup, and then a few people that think: "nice review; didn't surprise me much with the results, but at least we can now know that Killer can help in certain situations, even if it's overpriced as hell."
I'm sure Gary was tested a lot more than is shown in this article. I've conversed with him in the past, and I'd wager he was pulling his hair out over this article. How many times did you write it Gary? Ten? More? I think you ought to post here and set the record straight, because my bet is that in terms of improving gaming performance the Killer NIC is as good as a NIC is going to get... which means the other NICs from Intel, 3COM, etc. are basically all as good as onboard solutions if that. There's really not much you can do to truly improve performance of a NIC for online gaming when you're looking at maybe 5ms worst case being delay added by the hardware and OS.
As I said above in an earlier comment, it would be nice to have more developers like Jon Carmac around, because he apparently knows how to already perform well without lots of extra hardware. Can't say his games are the greatest, but the Doom 3 engine and networking aspects (client/server architecture) are clearly ahead of most other FPS solutions.
WileCoyote - Wednesday, November 1, 2006 - link
I was trying to keep my comment short, simple, and to the point. I don't need to write a page defending my opinion on something as trivial as a product review. The numbers speak for themselves in this review. And it wasn't so much the quantity but rather the quality - there is little to no structure in the review. That's fact, not opinion. There is a better way to write a review - I've read hundreds of them on Anandtech.I guess my Bachelor's Degree in Anthropology from a well-respected U.S. university doesn't count for much and neither does the successful business I created/own which pays a six-figure salary.
Gary Key - Wednesday, November 1, 2006 - link
Apparently my MBA and the fact I am already retired (at 44 and doing this because I really enjoy it) does not count for very much either. LOL.....
Seriously, I am always looking to improve. How would you change the structure of the article?
Aikouka - Tuesday, October 31, 2006 - link
Being that I run an FTP server that tends to see a decent amount of internal traffic, it actually sounds like a TNIC (or simply making a dedicated server) could actually be beneficial to me. Although, I sure have no desire to pay $300 for that card when I could easily spend the money on a second video card for SLi purposes or such. Also, I know switching from single to dual-core really helped to off-set the issue of FTP uploading on the local intranet. It really won't matter as when I build a new PC next month, I'm simple setting my old PC as a dedicated server to offload those annoying tasks.Also, Mlau, there are some games where "real estate" matters. World of Warcraft is a great example of this and I'm glad that I play the game in 1650x1080, because in certain situations, there's so much junk on your screen (I may call it junk but it actually helps :P), that you need all the extra room you can get. You may be happy in 1024x768, but that gives you no right to vehemently demean people for wanting to play in higher resolutions, which doing so also provides a better quality picture without wasting resources on Anti-Aliasing. Almost all the time, enjoying a game the way the developers designed/envisioned it can be an enriching experience for the gamer.
Gary, your comment about WoW being limited to 64 FPS... I think you may've left Vertical Sync on :D. I can easily get 90FPS on my dated Athlon 64 X2 4400+ with a GeForce 6800GT OC playing in 1650x1080 with max graphics settings. Albeit, I don't get a constant 90FPS, but it can be easily attainable in non-expansive places. So yeah, if your refresh rate on your LCD/CRT was set to 60Hz, you probably would see your game hover around 60-64FPS or somwhere between that and 30FPS. I know I turned VSync off on mine as I couldn't constantly achieve 60FPS, so the game lowered my FPS to about 30 with VSync turned on. Simply turning it off raised me to an easy 45 minimum with no tearing evident. I know that in the Hillsbrad/Alterac area, I would probably get around 45-60 depending on how far into the distance I could see.
Gary Key - Wednesday, November 1, 2006 - link
http://forums.worldofwarcraft.com/thread.html;jses...">Blizzard ResponseHi,
We tried your suggestions during testing and nothing helped. We used both LCD and CRT monitors with vsync off. This was with several different video cards and Core 2 Duo/AM2 X2 processors. The frame rates were always capped to 64 until we switched to a single core processor on either system. We contacted Blizzard directly and they confirmed the dual core bug. The link above has their response on line item 6. Are you using FRAPS to capture the frame rates? If so which version please?
Thanks!!!
goinginstyle - Thursday, November 2, 2006 - link
I downloaded FRAPS 2.81 and sure enough my 4800+ X2 is capped at 64FPS.otherwise - Tuesday, October 31, 2006 - link
You can get an 10/100 ethernet card with a TOE dirt cheap for much less then $300 if you really want one. With most people who actuially need a TOE also demanding 10/100/1000 support, there is a glut of 10/100 TOE NICs.dijuremo - Tuesday, October 31, 2006 - link
You will probably get more out of $300 if you get a hardware raid controller for your system (Areca comes to mind) which will not only provide a speed upin storage, but also redundancy for your system. I know it does nothing for your network performance, but is money better spent which is my point.I considered getting a Killer NIC for my new system, but did not do it because of price, no PCI-Express support (Mobo only has 2 PCI slots used for sound card and HDTV tuner and has 4 PCI Express slots, one used currently for Nvidia 7950GX2), plus performance gains were not that good (I read another review of this card about a month ago elsewhere).
As for the person saying sli/xfire is useless, you are totally wrong. At 1920x1080 (using the LVM-37w3 LCD monitor - 1080p native) you need sli or xfire to have decent speeds to play games with AA and AF. If you don't play games or play at 1280x1024 or less it does not matter, but above that you really need sli or xfire.
I also agree the article was a bit too long and actually more effective that a Lunesta overdose. Not sure if the writer is trying to avoid what just happened to HEXUS where they reviewed an Alienware PC and got e-mail back from the company saying they would not get any more harware because of the bad review...
heated snail - Tuesday, October 31, 2006 - link
I don't mean to be a jerk, and I appreciate any sincere and fact-finding test/review article. However:I'm amazed, was this really a review of a basic hardware item? Because instead it reads like a mini-novel about all the difficulty the testers/reviewers had in doing their job. Is it too much to ask for a more verbally efficient writing style? About two paragraphs briefly acknowledging that this product has been hyped in the media, and acknowledging that testing was a little more difficult than usual... then get straight to the tests on page two and conclusions on page three. I can't believe how long it took to read through this whole thing with its very repetitive descriptions and self-references.