City of Villains (Beta) Tests
Update: We would like to clarify that the beta version of City of Villains was stated as non-optimized in the release notes. The test server is open to any CoV subscriber and it does support PhysX hardware, but it is very possible for performance improvements to be made to the implimentation before their production release of the Issue 7 patch. In fact, we have been told by AGEIA that performance on the final code will absolutely be better than on the test server. We look forward to testing and reporting on PhysX support under production CoV code/servers as soon as possible.As with any MMOG, City of Villains has been difficult to test. The fact that PhysX hardware is only supported in a very specific area of gameplay makes it even more complicated. Luckily, we found a way around these issues. As City of Villains makes use of instanced areas for missions (multiple parties entering a single zone will each play the game in their own copy of that area), we were able to eliminate any outside influence from other players and go on a mission solo. Also key in our ability to test the Mayhem Missions (the part of City of Villains in which the PhysX hardware is used) is the fact that it is currently on the test server.
Normally, before a gamer gets a Mayhem Mission, he or she will have to do a bunch of other missions, level up a bunch of times, and unlock a contact called a "broker." The broker gives you access to "newspaper missions." After doing 3 of the newspaper missions, a broker will offer you a special job: a heist. In the future, and on the test server, this is where mayhem missions will be available. Each mayhem mission, once begun, has a 15 minute time limit. This limit can be extended by destroying things in the city, but there is no way to get enough time to actually benchmark multiple configurations of hardware. Doing 3 other newspaper missions and getting another mayhem mission isn't an option because this takes a huge amount of time (the missions can be different every time as well).
Fortunately, NCSoft offers the ability to copy your character to the test server at any time, multiple times. Our solution was to do enough missions to get to a point where we could be offered a mayhem mission (even though the missions are currently only on the test server). Then we copy our character over to the test server a bunch of times. Now we are able to accept the same mission any time we want.
This is great for testing a feature like PhysX, but doing a full test of the game is still hard. Working with a team fighting a huge number of enemies is a key element in the game. It's something that just can't be reliably repeated barring the assembly of an AnandTech guild built solely for choreographing, reenacting, and benchmarking scenes hundreds of times. For now, this is all we need. We included a screenshot of the game options with and without a PhysX card installed:
We created a couple videos to show the difference between hardware accelerated and software physics. Here are the numbers that resulted from our tests.
What we see clearly shows that the PhysX card is putting quite a strain on performance. Yes we get more mail or packing peanuts spewing forth from our rampant destruction. Yes, we do think it looks great with more stuff going on. But we are really not happy with the performance we are seeing here. It appears that our GPU is not overly loaded at these resolutions, as the difference between 800x600 and 1600x1200 gives very little in the way of performance gain without PhysX hardware installed. This implies that the bottleneck in performance is in the rest of the system. This is confirmed by the fact that performance drops considerably when running the test on a much slower CPU.
Whether on a low resolution with a fast CPU or a high resolution with a slow CPU, the PhysX hardware gives us low average framerates, very low minimum instantaneous framerates, and adds a bit of stutter to the movement of the game. Unlike the Ghost Recon test we showed last week, playing the game with the PhysX card feels much less satisfying. Multiple frames seem to take a long time to render (rather than just one or two) giving movement a choppy feel to it.
That being said, it is very important that we point out the fact that we are benchmarking code running on a test server. NCSoft makes it clear that code running on this server is not optimized and players may see degraded performance. We hope very sincerely that the performance issues will be resolved; right now it just isn't worth it.
So why include these numbers at all if it's on a test server? Well, the fact that they back up the results we saw in GRAW last week do lend some credibility the tests. What we seem to be seeing is that games which use the PhysX processor to tack on special effects physics take a performance hit when those effects are triggered. Of course, that only holds if the driver didn't fix the performance issues we saw in Ghost Recon.
Facebook
Twitter
RSS
67 Comments
View All Comments
apesoccer - Wednesday, May 17, 2006 - link
That's a good question as well...especially for those of us using other additional pci cards...mbhame - Wednesday, May 17, 2006 - link
You guys are giving Ageia WAY too much slack. :(Call a spade a spade and save face.
apesoccer - Wednesday, May 17, 2006 - link
There's no use in throwing in the towel before we get in the ring...mbhame - Wednesday, May 17, 2006 - link
Throwing in the towel...? How do you infer that from what I said?I said "Call a spade a spade". Whether Anandtech.com chooses an easy-out path of "Currently the PPU sucks..." (not in so many words) or not, there is tremendous grace extended to Ageia around here, and frankly, it stinks.
Obviously there is a fine line between journalism with respect (which 99% of other websites are ignorant of) and brown-nosing, or needing to get a pair. All I'm saying is it's not very clear where this site's stand is amongst these possibilities.
Trisped - Wednesday, May 17, 2006 - link
I think everyone is being cautiously optimistic that the tech will improve. I wasn't on the 3d accelerator screen when that first happened, but from what I hear those cards were expensive and actually were worse then not having them. But now they are required for every game and windows vista.We want to wait to see if they can work out the bugs, give us better comparisons, and to compare it to the GPU only systems that are suppose to be coming. Once we have all the facts we can pass a final verdict, until then everything is guess work.
apesoccer - Wednesday, May 17, 2006 - link
There's alot of grace given to it everywhere...I have yet to see an article bash them. There has been a lot of interest in this product, and frankly, the general concensis is that we want to see it succeed. That aside, i don't think they can make a precise statement saying...This product is going to suck balls...or this is going to be the next Sliced Bread...My problem with it, is the lack of depth to the findings (and your statement "Call a spade a spade"...), I wish they had tried more kinds of CPU's with different kinds of GPU's, at several resolutions at both the same settings hardware/software and different ones. Without those tests, you can't really say you've tested the product.
Basically...because they haven't done enough work with it yet [imo](due to time restraints or whatever...), we can't make any real statements about this product. Other then, at the one hardware setting they ran it at, compared to the different software setting ( >< ), the software setting scored better in fps. Which tells us what? The ppu uses overhead cpu cycles when doing at least 3x the amount of work the cpu would be doing at the lower sofware settings. So lets see some different settings (and some of the hardware/software running at the same), so we can get a better idea of the big picture.
mbhame - Wednesday, May 17, 2006 - link
I don't agree with your assessment on the general consensus. My circles vehemently want it to fail as it's an additional cost to our PCs, an additional heat source, an additional power requirement... and for what?I think you're kidding yourself if you think some other CPU:GPU combination would yield appreciably-different results.
DerekWilson - Wednesday, May 17, 2006 - link
we're working very hard to find ways to more extensively test the hardware. you've hit the nail on the head with why people haven't been tearing this part up. we are certainly suspicious of its capabilities at this point, but we just don't have the facts to draw a hard line on its real value (exept in the context of "right now").mbhame - Wednesday, May 17, 2006 - link
Well then make stronger statements in the present-tense. Just because someone makes a product designed to do "X", it doesn't mean that they'll succeed in doing so. You guys come across as if it's a given the PPU *will be* a success and in doing so generate a level of expectation of success. As it stands now this is a total flop - treat it as such. Then IF and when they DO make it worthwhile for some appreciable reason then we can marvel at their about-face collectively.It's not cynicism, it's reality.
AnnonymousCoward - Friday, May 19, 2006 - link
Why do you need stronger criticism? You've been able to determine what's going on, and that's because the performance charts speak for themselves. I'd rather read what's currently on the conclusion page, instead of the obvious "This product's performance sucks with current games."