Gaming in VR at 60 fps (Patreon)

If you're here, reading this, it's likely that you already know how grandiose it is to play your favorite games in VR. Just to be there, inside the virtual world. To be able to crane your neck and peek behind a corner. Friend or foe? See and admire details that would never even register with your brain if you were playing on a flat screen.

And yet, if you're here reading this, it's also pretty likely that your favorite game is not in VR (at least officially), because publishers are so scared to invest in the sector, the poor things. I pity them. No, seriously, I do. I mean, it must be hard sitting on billions of dollars and giving up the opportunity of a lifetime, to be part of a revolution which is much greater than when movies transitioned from black and white to color, possibly deeper and more meaningful than the invention of movies themselves... just because you're so fixated on the bottom line that thinking of investing like 0.1% of your revenue only to build a market, without an immediate profit guarantee, has you running for the hills while mumbling "a fad... it's just a passing fad..."

But as you know, the number one item on my not-so-hidden agenda is "demolish any rationalization that can be used to resist the inevitable".

We already went a long way together with the rationalization "Aww, but I cannot possibly redesign my whole game to implement motion controls, they're so hard! Everything needs to be made grabbable and physics-responsive, all my costly animations for the thousands of actions that the main character can use to interact with the world will have to be redone or thrown away, don't even get me started on the nightmare which is to redesign the inventory, EVERYTHING in the game will need to be tested again..."

Ah, no. We proved that playing immersed, with exactly the same control scheme and interaction model as were built for the original 2D version, is perfectly feasible and greatly enjoyable.

What is the next wall of resistance that we need to break down?

Performance, or "you can select any frame rate for our game, as long as it's 60 fps"

When Oculus and Valve started the current iteration of VR, they boldly posited that every VR game should run at 90 fps. Well, that's kind of reasonable, methinks. Not so much harder than running at 60 fps as every other game does (erm, PC games at least: console versions still have a tendency to drag their feet at 30 fps).

But then, when the memo gets to the engineering team of a studio, their reply is that 90 fps is 50% more performance, or 33% shorter frame time. It takes months of work to shave down 5-10% of execution time from a well-optimized engine; 33% is a very big ask, they say. Also, since the time quantum in a Windows system is exactly 1 ms (Windows is not a true realtime operating system), and there are only 11 ms in a frame at 90 fps, there's really not much room left for maneuvering.

That's before you tell them that all of that will have to happen at the highest possible resolution that the engine supports, because everything in VR is life-sized and each pixel gets magnified and put under close scrutiny.

And before you tell them that actually there will be two full frames to render (one  for each eye) in those meager 11 milliseconds of time. And no, don't even think of deep-pipelining the GPU, because latency is BAD!

As those of you who have been following me for some time know already well, I began to tackle this hairy issue since my GTA V mod by using Alternate Eye Rendering, which put simply means having the game draw only one eye in each frame time (so ideally in those 11 ms allowed by the headset refresh period). The other eye is reprojected, which means that the previous image for it gets rotated around the user's head in such a way as to simulate the relative orientation that the world would have had for the current frame when seen from the "stale" eye, if that eye view had in fact been drawn by the game engine.

That takes care of smoothing things out when you turn your physical head while in VR, and with a few extra engineering tricks (what you might have seen called "camera rotation compensation" or "yaw folding") you can also get a smooth rotation for the game camera with the gamepad right stick or mouse, as long as you're in first person.

But everything else is problematic. Strafing (whether it's your character that's doing it or other players or NPCs in the game), orbiting the camera around your avatar when playing in third person, moving your physical head from side to side, in general any kind of relative lateral motion between your eyes and the virtual objects you're looking at, all of that will trigger a psychovisual illusion caused by the rapid flashing of the low-persistency headset display, where it looks like the borders of objects get duplicated or triplicated. I simulated the effect, which happens in the viewer's brain so can't actually be captured on video, in my post from Oct 31: https://www.patreon.com/posts/74012973

This "ghosting" effect becomes worse as the game frame rate gets lower, because on average your vision will be hit by more flashes of the same repeated image.

If you manage to squeeze 90 fps from your game (drawing alternate eye views each time) the ghosting is barely noticeable, and you can get used to it and learn to "unsee" it in just a few minutes.

If the game runs at 70-80 fps, the ghosting is there but only annoying when NPCs move from side to side very close to you.

However (and that's the case for quite a few people), if your PC is not top-shelf and you can only get the game to run at the canonical 60 fps or even less, the ghosting starts to become an issue.

In fact, one question I get asked very frequently is: how come some native VR games render at around 60 fps and look very fluid with ASW for Oculus or motion smoothing for SteamVR, while your mods ghost like crazy when I hit 60 fps?

The difference, as Brian from Flat2VR never tires of explaining (thanks my friend <3) is that when native VR games, which are typically built using lower-complexity assets and dialing down quite a lot on the engine settings, say they're rendering at 60 fps, they mean 60 fps per eye, or 120 fps total. When a modded AAA game is rendering at 60 fps, it means 60 fps total, or 30 fps per eye. At 30 fps per eye, every existing reconstruction system (ASW, motion smoothing, etc.) breaks down completely and the game becomes unplayable.

Well, every existing system... until today.

AER v2

I can finally announce that I have my prototype working, and I'm already beginning to give it out in beta testing. Depending on how many problems are found in the beta, I will publish the first release next week or the one after that.

It's been a monumental endeavor (I lost track of the sleepless nights, it's all become a blur), and it still needs lots of TLC. In particular, two crucial points still need to be addressed. As I mentioned in one of my previous posts, the tech is conceived to be fully autonomous and adaptive. In its final version, it will be able to automatically adapt to the characteristics of the game, the system you're running it on, and possibly even to the complexity of the scene that you're playing at a given moment, and work out on its own the best configuration to apply dynamically to maintain the illusion of full frame rate within the current constraints.

I'm just one guy though, and implementing and testing all of that will take several more weeks or months, while instead I'm itching to go back to modding games for us all to play (yes, I'd also like to find some time to play my own mods every now and then!).

So I will be taking an incremental approach, releasing or re-releasing one title each time with the new tech customized for it, and gradually adding in the new features as they become available.

The first title, which already WorksForMe(TM), is Final Fantasy VII Remake Intergrade.  With the upcoming re-release for this beautiful game, under the "Advanced" section of the "R.E.A.L. VR" tab of the mod overlay, you will have access to two new rendering modes. They are simply called "1/2 rate" and "1/3 rate". Apart from that, you will of course still be able to fall back on the good old Alternate Eye Rendering mode (which will now be called Legacy AER) and on Mono. Mono's mostly to keep Brandon's happy []-D

Except for possible coding errors on my part, the two legacy modes should behave absolutely identically to how they always did. It adds a small maintenance cost to me of course, keeping the old behaviors in place, but I don't like pulling out options and choices from under my users' feet, even though we're in 2022 and it seems that everybody operating in software does that systematically and shamelessly.

But let's talk about the new things!

The "1/2 rate" and "1/3 rate" modes will try to lock the game frame rate to the respective frequency, so for instance if you're using a Quest 2 set to refresh at 90 Hz (the recommended rate for my mods), which means 180 fps when running at full rate and drawing both eyes each frame, the "1/2 rate" will cap the game at 90 fps or 180/2, while "1/3 rate" will cap the game at 60 fps or 180/3. To make another example, if you have a Valve Index set to 120 Hz, the caps will be at 240/2=120 fps and 240/3=80 fps respectively (always remember that the starting rate is double the frequency of the headset, because the headset needs both eye views refreshed on each tick).

It is important to avoid stuttering or missed frames that no other external limiters are competing to also tell the game how fast it should render, so as always with my mods you should keep V-Sync off and not use any third-party limiters like RivaTuner. The internal fps limiter of the game is fine as long as it's set to a frequency above or equal the one you're targeting, so for FF7R it's all good because its frame limiter defaults to 120 Hz.

(Side note: a few people asked how to remove this limitation for FF7R. Don't do that.  The game can't render faster than that anyway, especially at VR resolutions, and I observed that if you remove the 120 fps cap something in the driver or the game itself panics and hard-limits the game to 60 fps synced to the monitor, which is way worse than where you started from).

The "1/2" and "1/3" fractions are also important to keep in mind because they represent how many of the frames sent to the headset are actually rendered by the game, as opposed to those which are being synthesized (interpolated) by my new implementation. So, with 1/3 rate, each eye will get one frame rendered by the actual game engine camera, then two frames interpolated by my tech, and then again one frame rendered and so on.

So, which one should you choose? "1/2 rate", normally meaning 90 fps, or "1/3 rate" for 60 fps? The answer is yours to give, in the sense that it depends on the power of your system and on what balance you prefer to strike in the triangle clarity/fluidity/latency.

If you need your game to be super-responsive, and your system does not have lots of GPU power under the hood, then choose 1/2 rate and lower the resolution, or the game graphics options, until the mod reports a stable 90 fps. If you prefer nicer eye candy and don't mind a few more milliseconds of latency, choose "1/3 rate" and you will have a lot more headroom to move your graphics knobs. And of course, if you are the lucky owner of an RTX 4090, choose "1/2 rate" with all the bells and whistles and enjoy your perfectly smooth and insanely detailed VR :-D

The new paradigm ("give your GPU some breathing room") and what to expect

When the tech is functioning properly, basically every frame rate (90 fps with "1/2 rate", 60 fps with "1/3 rate", and further upcoming modes that will have even stronger interpolation capabilities) will look and feel as you were playing a native VR game, i.e., there will be no ghosting and no juddery motion whatsoever, because the headset will always be fed a steady 180 fps stream of images. That magic will last as long as you're hitting the stipulated target, and that can only occur reliably if you have some GPU headroom left.

In other words: if you ask too much from your system, and your GPU usage hits, say, 97% or higher, your poor graphics card will no longer be capable of completing all its required tasks on time. Visible effects will be: motion in the virtual world beginning to appear juddery or jittery; the runtime mirror window, or even the headset itself, starting to lose frames; tracking problems; frame drops in the capture if you're recording or streaming, and so on and so forth.

If that only happens for a second or two, no problem, as you might've just hit a rough spot in the game: let it pass and keep on truckin'. But if that occurs too often or even constantly, it indicates that you need to choose a more conservative setting on the interpolation scale. For now as I mentioned the mod will expose two, but in my secret lab under the [REDACTED] of the [REDACTED], I had acceptable results going as low as "1/6 rate". That means the unheard-of possibility of gaming in VR while pumping out as little as 30 fps total!

For the moment though you'll able to access a minimum of 60 fps, so if you want perfectly smooth interpolation and your system is unable to keep up a steady 60 fps you'll need to lower the resolution, close other apps, reduce your graphics settings and so on (*cough* upgrade *cough*) until you have a steady fps meter.

Incidentally: some aspect of my framework are long overdue for a restyling. One of those is the framerate counter, which is usable but nothing to write home about. One of these days (or nights) I will rework it into a visualization that can tell you more about what is causing you to miss or drop frames.

And to answer an important question that quite a few of you posted in the comments and personal messages during these weeks: the new AER v2 tech will NOT leverage DLSS3, for two reasons of importance and one incidental. The incidental one being, DLSS3 was not publicly available when I started working on this, a problem which could have been avoided just by waiting a bit.

The two important reasons are: a. I didn't want to restrict the benefits of the revision to only users with the super-new, super-expensive last-gen cards, and b. I wanted the two technologies to be usable together, so when new games start to come out that can barely hit 60 fps with DLSS3 switched on (it's going to happen, as sure as the sun rises, and sooner than you think) my tech can still be added to save them ;-)

Bonus

As if this post wasn't long enough yet. Told ya you would soon regret the days when I was being silently productive ;-)

The new releases, starting from the FF7R reissue with AER v2, will finally sport the magical slider you always dreamed of and never dared to ask (just kidding, many of you asked!): I worked hard on it these last few days, despite the brain fog, and it finally came together just this morning. All the reprojection/filtering/interpolation passes will have built-in automatic sharpening, which will be fully controllable by the user.

Again, to clarify. This is not just post-processing sharpening, which kinda works but can never quite restore the detail that was lost during previous operations. On the contrary, all texture manipulations will be done in such a way as to preserve all the detail in the original rendered images, and if requested with a sort of "overdrive" that allows for a sharper image than provided by the game itself.

Happy VR'ing!

(I apologize for all the unanswered comments and personal messages these days, I have been feeling really unwell. I seem to be getting better though, so I hope to be able to catch up soon!)