Xbox Series X DirectML: A Next-Generation Game Changer
Microsoft’s DirectML has the potential to become a game-changer for the next-generation Xbox console. It should allow developers to create next-generation games that run at 4K resolution but also run at a higher frame rate, offering ray tracing at the same time.
Xbox Series X supports Machine Learning for games with DirectML, a component of DirectX. DirectML leverages unprecedented hardware performance in a console. Benefiting from over 24 TFLOPS of 16-bit float performance, and over 97 TOPS (trillion operations per second) of 4-bit integer performance on Xbox Series X. Machine Learning can improve a wide range of areas, such as making NPCs much smarter, providing vastly more lifelike animation, and greatly improving visual quality. Today we will focus on improving visuals.
AI Upscale of Low-Resolution Textures In Real-time
First let’s look at the work of Playfab, a company that Microsoft acquired back in early 2018. They have been working on making tools for the back-end of games supported in the cloud. Playfab is using the power of Microsoft’s Azure servers & AI to upscale low-resolution textures in real-time. Playfab’s GM, James Gwertzman talked with VentureBeat on some of the things he and his team are working on.
“One of the studios inside Microsoft has been experimenting with using ML models for asset generation. It’s working scarily well. To the point where we’re looking at shipping really low-res textures and having ML models uprez the textures in real-time. You can’t tell the difference between the hand-authored high-res texture and the machine-scaled-up low-res texture, to the point that you may as well ship the low-res texture and let the machine do it”
Xbox Series X Machine Learning – SDR to HDR Conversion
Back in March Eurogamer’s Digital Foundry was presented with the Xbox Series X using machine learning to convert SDR to HDR in a few games including Halo 5 in real-time.
“We got to see the Xbox One X enhanced version of Halo 5 operating with a very convincing HDR implementation, even though 343 Industries never shipped the game with HDR support. Microsoft ATG principal software engineer Claude Marais showed us how a machine learning algorithm using Gears 5’s state-of-the-art HDR implementation is able to infer a full HDR implementation from SDR content on any back-compat title.”
Machine Learning Upscaling
Machine learning upscaling could potentially be the most impactful of the features as it improves both performance and visuals. We already have a great implementation of this tech on PC. It’s called DLSS 2.0.
“Deep Learning Super Sampling (DLSS) uses artificial intelligence and machine learning to produce an image that looks like a higher-resolution image, without the rendering overhead. AI algorithm learns from tens of thousands of rendered sequences of images that were created using a supercomputer. That trains the algorithm to be able to produce similarly beautiful images, but without requiring the graphics card to work as hard to do it.”
This means that developers will have an option to render a game at 1080p or 1440p. Then they can upsample it to 4K with little loss to image quality. Rendering a game in 1080p will also allow devs to utilize spare GPU resources for better graphics. Resources such as graphical effects, Ray Tracing, and/or better frame rate. How developers use it is up to them, and I’m sure it will depend on the type of game they make. DLSS on PC is usually used in games that use Ray tracing, allowing games to run at a high frame rate and with high image quality- upscaled 1440p or 4K.
According to developers, there may be even more uses for Machine Learning in gaming. Developers are still figuring out how to properly use DirectML, so I’m excited to see what the future will bring. Improved AI and physics would be a huge win for me.
SOURCE: EuroGamer, Digital Trends, VentureBeat
Awesome
I’m excited for game AI to be running on NNs.
Much better opponents to play against, ones your can’t tell aren’t human opponents.
That’s going to be the big change, as games are getting pretty stale.
Like, same game better graphics. That’s all we’ve had for 15 years
Based on my own experience with DLSS 2 as well as using an ai upscaling system on my DVDs of Deep Space 9 I’m actually kind excited about this.
I was playing control on PC and even with a 2080ti ftw3 ultra I couldn’t get above 30fps (with all ray tracing on) unless I rendered at 1080p. With DLSS 1 it looked ok on my 4k OLED but I could see too many issues and figured I’d wait for the next round of hardware to play the game.
Then they added DLSS 2 to it and suddenly that 1080p rendering resolution basically looked identical to a native 4k image on a 65 inch display. I couldn’t believe it.
So this kind of tech along with other features of dx12u and the hardware itself I’m actually pretty excited because it’s going to let this console punch well above it’s weight so to speak. Plus Microsoft doesn’t need supercomputers to do the learning they have azure which is the world’s 2nd largest cloud compute platform and more powerful than 1000 super computers due to its virtual nature.
Going to be exciting, especially as devs find all the tricks on how to use this stuff best combined with the usual tricks they always find in consoles to keep games looking better and better. I mean compare a year 2 PS4 or Xbox one game to a modern one. You would never have imagined the difference in quality
This ‘new’ tech, MS ML, is really best served for backwards compatibility games to ensure older games look even better on newer consoles, provided the AI has been trained and some kind of patch is introduced. Otherwise anyone that wants better looking games will be using the 8k lossless scaling of Unreal 5 and that includes their lighting tech, which is really a partial RT, instead of full on performance heavy brute force RayTracing.
Well really as it says in the article its good for both. Instead of shipping 4k assets they could ship low resolution ones and save tons of storage space and bandwidth then use the DML to bring them back to original quality…
So a good example is you could have a multi platform game that’s say 50 gigs on ps5 (or pc) but only 30 on Xbox because of this technology.
Plus devs will find fun ways to use it outside its intended use..
As for backwards compatible stuff… It won’t help nearly as much because you can’t add detail that was never there in the first place. It’s taking the original high resolution asset and then allowing them to drastically lower the resolution which removes most detail. Then DML brings all that detail back. It’s not adding new detail.
They have other tools to enhance older games but the thing this would be used for in those is making them run at higher resolution and still look clean and also add HDR to something that never had it.
It will also benefit pc gamers in similar.
While it’s great to include a cross platform initiative, I’m not sure it will bring much to the table in its current form and the way its presented. A lot of MS PC/console users like to complain about the ‘lack of visual fidelity’ using dynamic resolution, checkerboard and DLSS techniques.
Dynamic Res is a way to maximize performance while maintaining high resolution textures when CPU/GPU bottlenecks are hit,
DLSS is a way to use AI to upscale from very low quality textures to better looking textures and hopefully allow the CPU/GPU to maintain reasonable performance while providing better quality lighting. Also assumes the textures have been AI trained first to get the better image quality.
Checkerboard is a way to use reasonable quality HD textures to produce ~80-85% of a 4k texture but do so with a small 15-20% performance demand over handling a HD texture vs a 400% demand for a 4k one, thereby allowing for more performance to be available for other things,like lighting, better physics, better AI NPC’s, etc.
MS ML is DLSS generic and lossy. Nvidia is not required.
Why not just go checkerboard and/or use the upcoming Unreal 5 engine with 8k textures and lossless scaling (dynamic res might be part of this)? Unreal 5 seems to have the scaling already there with an 8k texture. Real thing scaled to your system without losing any image fidelity. The ML/DLSS upscaling requires the AI is trained to look at garbage textures and matched to their 4k version so that it can scale them up. It only works if the AI is already trained with that texture. It isn’t magic.
So you will pay the licensing costs / usage fees for everyone using UE5 then, since DirectML is free of usage and not bound to any explicit engine, what will make it exclusive to that engine?!
What a ridiculous comment.
How much do you think the licensing fees are? LOL
Devs pay nothing until they make more than 1 million $. After that it’s 5%. Distribute digitally to cut out disk production and you’re making more money as a dev than someone not using the UE5 tech/textures and physical disks. And you are right…YOUR comment is ridiculous.