Archives For virtual reality

Memory Management in UE4

Michaël Dubé —  October 22, 2015 — 8 Comments

Hey there!

Well, it’s been a while since I’ve written something for the blog; we’ve been busy at Frima building a cool update to our Oculus Connect 2 demo, which Vincent and fellow programmer Marc-André attended. We now have facial animations and kick-ass voice overs into the game, which bring it to a whole new level. There’s also an amazing article published on Polygon about FATED, so things are getting exciting! Click here to read the article.

Since we are only two programmers on the project, and since we are needed for pretty much every technical aspect of the game, sometimes stuff falls into the cracks. That’s pretty much what happened with memory management for FATED, and since we were both new to Unreal 4, we made some rather critical mistakes on top of not spending a minute on investigating what was loaded, and when.

When the rush to get our OC2 demo out of the door was over, it was taking a whole 75 seconds (!) to load our demo scene. That’s pretty insane for any kind of game. It was time to dig a bit deeper into what was actually taking so long to load, and maybe add a bit of asynchronous loading in there. So while this post may be somewhat of a given for Unreal veterans, I think it could be helpful for some developers with a bit less exposition to the tech.

 

WHAT’S UNDER THE HOOD?

First things first: identify the culprits. Unreal is pretty dope when it gets to commands you can input to investigate every aspect of the game, and memory statistics are no exception. One of the most useful commands out there is “memreport –full”. The “full” parameter will dump more information that could prove useful but is not required per se. Running this command will create a .memreport file in your “Saved\Profiling\MemReports” project folder. It’s a custom file extension, but this is actually a text file representing a snapshot of your game’s current memory state.

You’ll probably want to look first at the object list, which is the complete list of UObject you have loaded and information on the amount of memory space they take up. You will have a line looking like this for every object type:

AnimSequence          36        16628K           16634K           7578K             7578K

Texture2D                  150      212K               219K               260996K         96215K

AnimSequence (or Texture2D) is the name of the class of the object, while the number next to it (36 or 150) is the number of instances of that class. What you want to look into after that are mostly the first and last of the four other numbers you see up there. The first is the actual size of the objects while the last one is the size of the assets that these objects directly reference exclusively. For example, the Texture2D objects do not take much memory as objects holding the textures metadata (212K) as opposed to the actual texture data (pixels) it references exclusively (meaning no other objects references to that data), which amount to 96215K.

The two lines above were extracted from a memreport for this scene:

MemoryTestsMapThat’s right, there is absolutely nothing in there except the floor, a player start, and the skybox. So where the heck are those 36 animations and 150 textures coming from?

 

FINDING ASSET REFERENCES

Now that I knew there was something clearly wrong there, I had to find out which assets were loaded. Sure, there are 36 animation sequences in memory, but which ones and why? Again, UE4 can show that to you quite easily.

The memreport command is in fact a list of other commands executed one after the other to form a “complete” report of your game memory. One of the commands it runs is “obj list”. This is what outputs the list of objects and the number of instances, but this command can also use some parameters, one of which is the class of the objects.

 

obj list Class=AnimSequence

 

This will output all of the animation sequences in memory at that moment, in order of what uses the most memory (the first number). Like this:

 

AnimSequence /Game/Character/Horse/Animation/AS_HorseCart_idle.AS_HorseCart_idle      

491K       491K       210K       210K

 

Now, if you’re like me, you might want to make this look a bit clearer, and fortunately it’s quite easy to write your own Exec function that can output exactly what you want. It can, for instance, sort by exclusive resource size instead of object size. Here is a sample of code to get you started in that direction:

for (TObjectIterator<UAnimSequence> Itr; Itr; ++Itr)

{

            FString animName = Itr->GetName();

            int64 exclusiveResSize = Itr->GetResourceSize(EResourceSizeMode::Exclusive);

}

 

Reference viewerThe Visual Reference Viewer

Now that I could pinpoint exactly which objects were loaded, I was able to investigate with more ease what was referencing them in that seemingly empty scene. There are two ways to view object references in Unreal. First, there is the visual reference viewer (that you can see in the picture above) in the editor; this will show all potential references, not necessarily what is referencing the actual loaded asset in memory right now. Of course, there is also an easy way to figure out the current shortest reference with yet another console command.

 

obj refs name=AS_HorseCart_idle

 

This will output a chain of references on what is loaded in memory (it can take some time), and following this chain will usually lead to the culprit. In our case, we were doing one mistake that was responsible for a lot of those loose asset references: direct asset references in some of our native class constructors. Like this one for example:

 

static ConstructorHelpers::FObjectFinder<UMaterial>

TextMaterial(TEXT(“Material’/Game/UI/Materials/UITextMat.UITextMat'”));

 

In the above example, this is a direct reference to a material: this material will always be present in memory. This is quite logical when you think about it, since the constructor is not only called when an instance of said class is created, but also at startup as it is run to set default properties when the static version of the class is created. So avoid those if it’s not something you need at all times in memory! Instead, opt for a UPROPERTY that can be assigned in the blueprint version of that class, even if it’s always the same asset. At least that way if you don’t have an instance of that object loaded in your scene, it won’t be in memory.

 

ASYNCHRONOUS LOADING USING THE STREAMABLE MANAGER

The wrongly referenced assets were not the only reason why it was taking so long to load our demo, so there was still more work to do. We had some remaining temporary assets, some pretty huge textures (I’ll talk about that in more detail below) but, more importantly, we still had a lot to load and we were trying to load everything at once. Let’s see how to asynchronously load assets using what Unreal calls the Streamable Manager.

If you have an actor that references multiple Animation blueprints, for example, they will all be loaded when you instantiate it, even if you only use one at a time.

 

TSubclassOf<UAnimInstance> mAnimBPClass01;

TSubclassOf<UAnimInstance> mAnimBPClass02;

TSubclassOf<UAnimInstance> mAnimBPClass03;

 

The above lines should become this:

TAssetSubclassOf<UAnimInstance> mAnimBPClass01;

TAssetSubclassOf <UAnimInstance> mAnimBPClass02;

TAssetSubclassOf <UAnimInstance> mAnimBPClass03;

After this modification, the animation blueprint that mAnimBPClass01 references will only be loaded when you specifically ask for it to be loaded. Doing so is quite simple: you need to use the FStreamableManager object. Just make sure to declare it somewhere that will always be in memory, an object that won’t ever be deleted (like the GameInstance of your game, for example). In my case, I dedicated a special “manager” object to it, which is created at game start and never deleted. It handles everything that is dynamically loaded in our game.

 

UPROPERTY()

FStreamableManager mStreamableManager;

 

There is more than one way to load an asset asynchronously, but here is one example: mArrayOfAssetToLoad being a TArray of FStringAssetReference.

 

mStreamableManager.RequestAsyncLoad(mArrayOfAssetToLoad,

FStreamableDelegate::CreateUObject(this, &UStreamingManager::LoadAssetDone));

 

FStringAssetReference are the string representation of the full path of the asset in Unreal’s internal file system. Using a TAssetSubclassOf<> pointer, you can get it by calling ToStringReference() on it.

Furthermore, if you are using Wwise for your audio management, Audio Banks can become quite huge and long to load. Fortunately, a wise (get it?) bank subdivision and using LoadAsync() on the UAkAudioBank instead of Load() will fix that for you. Be sure to uncheck AutoLoad on the banks in the editor before you do! Also, for some reason the LoadAsync() call is not exposed in Blueprint, so you need to do that in native code or expose it yourself.

 

LEVEL COMPOSITION

Assets loading asynchronously is one thing, but you probably also want to split your levels into chunks to load separately. Unreal allows that using “Level Composition”. Level loading used to be done on the main game thread of Unreal, but it is now possible to split the loading on different threads.

 

In the DefaultEngine.ini file of your project, add this:

[Core.System]

AsyncLoadingThreadEnabled=True

 

We are still on Unreal 4.8.2, but from what I found on the subject, this may already be by default in 4.9. Anyways, this should help make the asynchronous loading of levels smoother. If you are enabling that feature, however, you need to be careful about what you do in your class constructor, as some operations are not thread-safe, and using them could result in a lock or even crash the game.

Image3

Level streaming can be done “automatically” using volumes or a simple distance factor, but we decided to do it manually in our case. In the picture above, unchecking Streaming Distance will allow every sub-level linked to that layer to be loaded manually. This can be done in blueprint using Load Stream Level or in C++ using UGameplayStatics::LoadStreamLevel().

Image4

 

Texture Mip Bias

In pretty much every game, textures are what end up using a lot of your space. In pretty much every game, there is also a bunch of oversized textures that are far from optimized pixel density-wise. Fortunately, instead of reimporting every such texture, UE4 offers a really simple way to trim the fat without needing to reimport everything: LOD Bias.

Image6

Image5

You can see that the difference in resource size is considerable when we drop 1 mip, especially when the texture is in 4096X4096! Of course, we couldn’t do that for everything, but there was a lot of stuff that was in 4096 that really did not need to be.

Of course there is a lot more to memory optimization and management, but this is pretty much what I have done to get our demo from taking 75+ seconds to load to somewhere around 10 seconds. Unreal is a great tool, and I keep learning and getting better at it. I hope this will help some of you out there in creating your own awesome content. If you have any questions or comments, I’ll be happy to answer them! In the meantime, I’ll go back to working on FATED. Stay tune for more info!

 

Mick / Lead Programmer / @Mickd777
www.fatedgame.com

In the wake of this week’s Oculus Story Studio announcement of the animation project Henry, we wanted to share our own favorite VR short film, Butts. It’s more of a must Watch than a must play really. It’s not the only VR animation short out there but its execution is flawless and it’s sure to put a smile on your face.

We love how the creators describe it on their website: “Butts is an animated VR short about love, trust, and learning what it means to be truly free.” It’s an adaptation of an animated short that you can also watch on their website. It’s interesting to compare the VR experience to the original short and see how virtual reality affects the storytelling.

What do you think? Have you tried other VR animated films?

http://www.buttsvr.com/

Hey again,

The excitement for Virtual Reality continues! We saw some amazing stuff coming from the RIVER start-ups (powered by the Rottenberg Ventures), special mention to FOVE and their awesome VR eye tracking system! They just launched a Kickstarter campaign, back it up!

day21

The Project Morpheus demo with the guns was a big star here. It was really fun just to watch the great reactions from the players. We got to try a Sixense demo, also with guns… it was just great! Having controllers definitely adds to the VR experience.

day22

A few more people played our FATED demo. Even though it’s still pretty early in Alpha, it’s still making quite an impression. Quote of the day: “Best Demo I’ve played during the event and it was in a f**king Hotel Room!”

day23

We finished the day with a great talk from our friends at Epic. It was good to hear that their Showdown demo worked without any modification @120 Hertz on the Project Morpheus. This is really high end stuff and it was running on a very high end PC rig. The PS4 is a powerful machine and mustn’t be underestimated.

But as all good things must come to an end, we need to get back to work now.

See you later folks!

Vincent Martel / Producer / @Vincent_Martel

http://store.steampowered.com/app/302710/

This game has been out for a while but they recently added VR support. The experience is amazing and refreshing as it is not placing the player in first person but rather in a top-down view. It works pretty well. I’ll leave you with the impressions of GameMuscleVideos.

There is so much more to VR than first-person perspective.

http://stv.re/chicken-walk/

The team got to try this silly demo this week. Taking the role of a chicken just living the chicken life sure is interesting. What struck us though was the clever use of what we call the “Virtual Nose”. By creating the impression that the player has a nose (or a beak in this case), you reduce the effects related to simulation sickness. As we discussed in one of our first entry, simulation sickness is something we need to eliminate as much as possible.

Ars technica explains that: “Offering a fixed object that doesn’t shift as you move around a virtual world has been shown to help anchor many VR users, reducing the apparent difference between visual and sensorimotor stimuli that can lead to simulation sickness.”

Try it out and tell us how you enjoy living as a chicken!

Étienne

Hi! Étienne Carrier, here. I’m the Technical Artist on FATED.

Developing graphics for VR is an awesome challenge! It’s an all new playing field with whole new constraints and rules. I’m learning new tricks every day, and through this blog I aim to share them with you.

ART STYLE

When we started developing the graphic pipeline, it was clear that we needed a visual style that would help us reach our performance target. In VR, there’s no slacking off. If the framerate drops even for a second, you get a hefty dose of simulation sickness. It was therefore a lot more natural to go with a stylized art style that would not only help us with performance, but also look good in VR. Smooth and non-noisy texture feels great in virtual reality, and it helped the 3D stay faithful to the awesome artwork that Marianne Martin (Art Director) and Marie-Hélène Morin-Fafard (Concept Artist) created.

Fated_Screenshot02Fated_Screenshot03

VIEW DISTANCE & NORMAL MAPS

Applying tricks learned while developing mobile games can turn out to be a lifesaver when you have to run at 75+ fps. The art really needs to be planned accordingly. We built our environments in a way that limits the view distance and allows the occlusion culling to work for us. Normal maps only work well for micro details or from far away. We actually want to fade details out with distance, as it tends to get noisy due to the pixels density on VR headsets. The micro details go against our soft texture style, and the distant normal maps go against the fade of details in the distance. So we decided to never use normal maps, which also helps us get more performance.

GAME ENGINE

We chose Unreal Engine 4 to develop our game. We began by setting up our basic scenes, because even the default template scene was not reaching 75 fps on some PCs. We removed most of the post-process, screen space reflection, and anti-aliasing. We also used static directional lights, then built from there while profiling every step of the way.

static directionnal lightspost process

UNREAL TOOLS

Unreal has a great many tools to help us build our environments efficiently. Here is a video showing how we used some of these tools together. Along with Unreal’s landscape and foliage, we built a material that projects a texture on top of props like rocks to help them blend with the terrain. The blueprints allow us to dynamically create a material instance for each static mesh, so they can all have their specific settings. Using the same texture as the landscape helps to blend them seamlessly.

Fated_RockBlueprintParametersFated_RockBlueprintFated_RockShader

That’s it for now, we’ll have more tips & tricks coming up later!

Cheers

Etienne Carrier
Technical Artist

 

Marianne

Hello, my name is Marianne Martin, Art Director on FATED.

I was thrilled when I learned we were going to work on a VR project – and Norse-themed at that! Like many, I wondered if virtual reality was a fad until I tried a few different demos and realized the fantastic potential for immersion. Here was an opportunity to build the most tangible game world ever! With VR able to fool our brains into the truest feeling of presence yet (to the point of vertigo!), visual development is entering a new age.

FatedBlog_Art_IMG1

During the initial concept phases, we were all excited about the epic narrative. This was VR: our Midgard and our characters had to be so… real! Character bios in hand, I started sketching detailed, documented designs. When concept artist Marie-Hélène Morin-Fafard joined the project, she drew some spectacular, moody environments that we hoped would be the perfect grounds on which to stage the FATED adventure. We did what pre-production is intended for: we went big.

FatedBlog_Art_IMG2

When we started getting more specs and technical info, we realized our strategy wasn’t quite right. With the unexpected technical limitations and a small team, we needed to change our academic, too-realistic approach. I took a step back and toyed with the idea of exploiting the limitations as if they were fully intended, rather than produce something that couldn’t be simulated believably on our platform.

FatedBlog_Art_IMG3

The games and animated movies that made a lasting impression on me are not impaired by stylization – they are probably better for being stylized. I thought: if we stylize FATED, focus on expression, create a meaningful color script, and give it distinctive visuals to enhance the experience that makes VR so special, we’ll have something extremely solid. We can make the player believe they are an emotionally and physically-involved part of that universe, and this became our focus.

FatedBlog_Art_IMG4

FatedBlog_Art_IMG5_fixed

So, back to basics we went, designing simpler characters and environments with more striking and memorable silhouettes. We got our 3D pipeline into gear to achieve the vision, and our technical artist, Étienne Carrier, developed clever systems to fully exploit our visuals and create environment variety in Unreal. Before long, we had a successful test scene, a new sense of wonder, and a few of the building blocks for the world of FATED.

We’re not out of the woods yet, so stay tuned…

FatedBlog_Art_IMG6