City of Villains (Beta) Tests
Update: We would like to clarify that the beta version of City of Villains was stated as non-optimized in the release notes. The test server is open to any CoV subscriber and it does support PhysX hardware, but it is very possible for performance improvements to be made to the implimentation before their production release of the Issue 7 patch. In fact, we have been told by AGEIA that performance on the final code will absolutely be better than on the test server. We look forward to testing and reporting on PhysX support under production CoV code/servers as soon as possible.As with any MMOG, City of Villains has been difficult to test. The fact that PhysX hardware is only supported in a very specific area of gameplay makes it even more complicated. Luckily, we found a way around these issues. As City of Villains makes use of instanced areas for missions (multiple parties entering a single zone will each play the game in their own copy of that area), we were able to eliminate any outside influence from other players and go on a mission solo. Also key in our ability to test the Mayhem Missions (the part of City of Villains in which the PhysX hardware is used) is the fact that it is currently on the test server.
Normally, before a gamer gets a Mayhem Mission, he or she will have to do a bunch of other missions, level up a bunch of times, and unlock a contact called a "broker." The broker gives you access to "newspaper missions." After doing 3 of the newspaper missions, a broker will offer you a special job: a heist. In the future, and on the test server, this is where mayhem missions will be available. Each mayhem mission, once begun, has a 15 minute time limit. This limit can be extended by destroying things in the city, but there is no way to get enough time to actually benchmark multiple configurations of hardware. Doing 3 other newspaper missions and getting another mayhem mission isn't an option because this takes a huge amount of time (the missions can be different every time as well).
Fortunately, NCSoft offers the ability to copy your character to the test server at any time, multiple times. Our solution was to do enough missions to get to a point where we could be offered a mayhem mission (even though the missions are currently only on the test server). Then we copy our character over to the test server a bunch of times. Now we are able to accept the same mission any time we want.
This is great for testing a feature like PhysX, but doing a full test of the game is still hard. Working with a team fighting a huge number of enemies is a key element in the game. It's something that just can't be reliably repeated barring the assembly of an AnandTech guild built solely for choreographing, reenacting, and benchmarking scenes hundreds of times. For now, this is all we need. We included a screenshot of the game options with and without a PhysX card installed:
We created a couple videos to show the difference between hardware accelerated and software physics. Here are the numbers that resulted from our tests.
What we see clearly shows that the PhysX card is putting quite a strain on performance. Yes we get more mail or packing peanuts spewing forth from our rampant destruction. Yes, we do think it looks great with more stuff going on. But we are really not happy with the performance we are seeing here. It appears that our GPU is not overly loaded at these resolutions, as the difference between 800x600 and 1600x1200 gives very little in the way of performance gain without PhysX hardware installed. This implies that the bottleneck in performance is in the rest of the system. This is confirmed by the fact that performance drops considerably when running the test on a much slower CPU.
Whether on a low resolution with a fast CPU or a high resolution with a slow CPU, the PhysX hardware gives us low average framerates, very low minimum instantaneous framerates, and adds a bit of stutter to the movement of the game. Unlike the Ghost Recon test we showed last week, playing the game with the PhysX card feels much less satisfying. Multiple frames seem to take a long time to render (rather than just one or two) giving movement a choppy feel to it.
That being said, it is very important that we point out the fact that we are benchmarking code running on a test server. NCSoft makes it clear that code running on this server is not optimized and players may see degraded performance. We hope very sincerely that the performance issues will be resolved; right now it just isn't worth it.
So why include these numbers at all if it's on a test server? Well, the fact that they back up the results we saw in GRAW last week do lend some credibility the tests. What we seem to be seeing is that games which use the PhysX processor to tack on special effects physics take a performance hit when those effects are triggered. Of course, that only holds if the driver didn't fix the performance issues we saw in Ghost Recon.
Facebook
Twitter
RSS
67 Comments
View All Comments
Tephlon - Wednesday, May 17, 2006 - link
yeah, no. I know that havoc is doing genaric physics. And light poles DO normally bend without the card. Cars do shake and explode. Cans can be kicked. All that stuff is normally threre.I'm just saying the card seems to accentuate all of it. Not just more particles, but better explosions. Better ragdoll. Pots break a bit different, etc.
It was definately there before, but I think it all looks better with the physx. My roommate said he noticed the difference as well. I let him borrow it for a while while I was at work.
Again, I know I have no proof, atleast not to show you atm... but to me it all seems better than before.
If I get a chance I'll fraps a run through a level once with and once without, and throw the links up here. I personally have seen several sites' comparison vids, but I don't feel they show everything very well.
Again, I'd heard it only adds particles to explosions, like you did, but I swear I can see the difference with everything.
Anyone ever heard Ageia say EXACTLY what difference there is for GRAW with their card?
DerekWilson - Wednesday, May 17, 2006 - link
Perception of experiences can greatly be affected by expectations. None of us are ever able to be 100% objective in all cases.That being said, in your first post you mention not "feeling" the performance impact. If you'll take a look at our first article on PhysX (and the comments) you will notice that I reported the same thing. There aren't really huge slow downs in GRAW, but only one or two frames that suffer. If the errant frame(s) took a quarter of a second to render, we would definitely notice it. But while an AVERAGE of 15 frames per second can look choppy having a frame or two take 0.066 seconds to render is not going to significantly impact the experience.
Minimum framerates are important in analysing performance, but they are much more difficult to properly understand than averages. We want to see high minimum framerates becuase we see that as meaning less slow-down or stutter. But generally (in gpu limited situations) minimum framerates aren't outliers to the data set -- they mark a low point where framerate dips down for a good handful of frames. In the case of GRAW with PhysX, the minimum is really non contiguous with the performance of the rest of the frames.
CoV is another story. The framerate drops several times and we see stuttering. It's definitely something easily "felt" during gameplay. But CoV Issue 7 is still beta, so we might see some performance imporvements when the code goes live.
Tephlon - Wednesday, May 17, 2006 - link
Derek, I totally agree. I wasn't arguing about anything technicial the article, or the relativity minimum fps has on the 'feel' or 'playablilty'. It just doesn't seem like most readers (here and elsewhere) understand it. I also won't hide the fact that I DO WANT this tech to succeed, partly because I heard them speak at quakecon and I liked what I heard/saw, and partly because I've dropped $300 in good faith that my early adoption will help the cause and push us forward in the area of true physics in gaming. And even though my perception is undoubtably affected because of my expectations, its not entirely misled either. Even with my bias I can be realistic and objective. If this thing did nothing for visuals/gameplay and killed my experiance with crappy performance, I'd of course have a different opinion on the matter.I was simply saying that readers seem to lose sight of the big picture. Yeah, its in the rough stages. Yeah, it only works with a few games. I'm not here to pitch slogans and rants to make you buy it, I just wanted people to understand that device 'as it is now' isn't without its charm. It seems the only defense that's brought in for the card is that the future could be bright. It DOES have some value now, if your objective about it and not out to flame it immediately. I like what it does for my game, even if its not revolutionary. I just hope that there are enough people objective enough to give this company/card a chance to get off the ground. I DO think its better for industry if the idea of a seperate physics card can get off the ground.
I dunno, maybe I see too much of 3DFX in them, and it gets me nostalgic.
Again, Derek, I wasn't knocking on the report at all, and I hope it wasn't taken that way. I think it said just what it was supposed to or even could say. I was more trying to get the readers a balenced look at the card on the use side, since straight numbers send people into frenzies.
Did all that get to what I was trying to convey? I dunno, I confuse myself sometimes. I wasn't meant to be an author of ANYTHING. In any case, good luck to you.
Good luck to all.
DerekWilson - Wednesday, May 17, 2006 - link
lol, I certainly didn't take it as a nagative commentary on anything I said. I was trying to say that I appreciate what you were saying. :-)At a basic level, I very much agree with your perspective. The situation does resemble the 3dfx era with 3d graphics. Hardware physics is a good idea, and it would be cool if it ends up working out.
But is this part the right part to get behind to push the industry in that direction?
AnandTech's first and foremost responsibility is to the consumer, not the industry. If the AGEIA PhysX card is really capable of adding significant value to games, then its success is beneficial to the consumer. But if the AGEIA PhysX card falls short, we don't want to see anyone jump on a bandwagon that is headed over a cliff.
AGEIA has the engine and developer support to have a good chance at success. If we can verify their capabilities, then we can have confidence in recommending purchasing the PhysX card to people who want to push the agenda of physics hardware. There is a large group of people out there who feel the same way you do about hardware and will buy parts in order to benefit a company or industry segment. If you've got the ability and inclination, that's cool.
Honesly, most people that go out and spend $300 on a card right now will need to find value in something beyond what has been added in GRAW, CoV, and the near term games. If we downplayed the impact of the added effects in GRAW and CoV, its because the added effects are no where near worth $300 they cost. It is certainly a valid perspective to look towards the future. You have the ability to enjoy the current benefits of the hardware, and you'll already have the part when future games that make more compelling use of the technology come out.
We just want to make sure that there is a future with PhysX before start jumping up and down screaming its praises.
So ... I'm not trying to say that anything is wrong with what you are saying :-)
I'm just saying that AnandTech has a heavy responsibility to its readers to be more cautious when approaching new markets like this. Even if we would like to see it work out.
Tephlon - Thursday, May 18, 2006 - link
true. I do get your point.And again, you're right. With a more balenced perspective on the matter, I sure can't see you suggesting a 300 dollar peice of hardware on a hunch either. I do respect how your articles are based on whats best for the little guy. I think I'd honestly have to say, if you were to suggest this product now AS IF it was as good as sliced bread... I would be unhappy with my purchase based on your excitment for it.
teheh. Yeah, you made the right call with your article.
Touche', Derek. TOUCHE'
thehe. I guess not everyone can gamble the $300, and thats understandable. :-(
Like I said... here's hopin'. :-D
RogueSpear - Wednesday, May 17, 2006 - link
I'm not an expert on CPUs, but all of this has me wondering - isn't physics type code the kind of mathematical code that MMX and/or SSE and their follow-ons were supposed to accelerate? I'm sure physics was never mentioned way back then, but I do remember things like encryption/decryption and media encoding/decoding as being targets for those technologies. Are game developers currently taking advantage of those technologies? I know that to a certain point there is parity between AMD and Intel CPUs as far as compatibility with those instruction sets.apesoccer - Wednesday, May 17, 2006 - link
Seems like this was a pretty limited review...Were you guys working with a time table? Like 4hrs to use this card or something?I think i would have tried more then just single core cpu's...since we're heading mostly towards multicore cpus. I also would have run tests at the same lvl (if possible; it feels like we're intentionally being kept in the dark here) to compare software and hardware with the same number of effects, at different levels and at resolutions...At low res, you're maxing the cpu out right? Well, then if the ppu uses 15% of the cpu but outputs 30% more effects, you're being limited by the cpu even more...You should see greater returns the higher the resolution you go...Since you're maxing your gpu's out more (rather then the cpus) the higher the res. All of this is moot if the overhead cpu usage by the ppu can be run on a second cpu core...since that's where the industry is headed anyway. And making software/hardware runs on a dual core should give us a better idea of whether or not this card is worth it.
peternelson - Wednesday, May 17, 2006 - link
To the people who say it's a decelerator. It is a little slower but it is NOT doing the same amount of work. The visual feast is better in the hardware accelerated game than without the card. But we need a way to quantify that extra as just "fps" ignores it.Second, Anandtech PLEASE get yourselves a PCI bus analyser, it need not be expensive. I want to know the % utilisation on the PCI bus. At 32 bit 33MHz it is potential max 133 MByte/sec.
How much of that is being used to talk to and from the PHYSX card, and is it a bottleneck that would be solved by moving to PCI Express? Also in your demo setups, considering what peripherals you are using, are you hogging some of the PCI bandwidth for (say) a PCI based soundcard etc which would be unfair on the physx card.
ALSO one of the main purposes of THIS review I would say is to COMPARE the ASUS card with the BFG card. You don't seem to do that. So assuming I want a physx card, I still don't know which of the two to buy. Please compare/contrast Asus vs BFG.
DerekWilson - Wednesday, May 17, 2006 - link
honestly, the asus and bfg cards perform identically, pull about the same ammount of power and produce similar levels of noise.If you are trying to decide, buy the cheaper one. There aren't enough differences to make one better than the other (unless blue leds behind fans really does it for you).
We didn't do a more direct comparison because we have an engineering sample ASUS part, while our BFG is full retail. We generally don't like to make direct comparisons with preproduction hardware in anything other than stock performance. Heat, noise, power, pcb layout, and custom drivers can all change dramatically before a part hits retail.
We will look into the pci bus utilization.
peternelson - Wednesday, May 17, 2006 - link
Thanks, so, I will have to choose on features like the nice triangle box on the BFG ;-)
In gaming on older machines where both the sound and network and possibly other things are all on the same PCI bus, then either the physx or the other stuff could suffer from bus contention.
I hope you can either ask or do some analysing to watch the amount of traffic there is.