Archive for the 'Graphics' Category
Lighting: The Rendering Equation
Introduction
Back in 2002, I started my first job in the games industry at Climax Studios in England. I have to admit, I didn’t know very much about game development at the time. Don’t get me wrong, I’d been writing little 2D games, and messing around with rubbish particle systems at home, but it was nothing like what I was about to get involved with. Despite my inexperience, somehow I did enough to pass the interview, and I was offered a job as a junior programmer.
As seemed to be typical for the time, my introduction to the industry was pretty much a trial by fire. I quickly found out that I couldn’t hope to truly understand every single new thing I encountered, so I learned to just accept some things as the truth. For example, I was told that the dot product of two normalized vectors yields the cosine of the angle between them. I just accepted this, and only took the time to find out why later on.
One of the things I accepted at the beginning was the maths used to perform the lighting of our models during rendering. While I could understand how the equations appeared to yield decent looking results, I never understood where they came from, and why they worked.
After a while I began to wonder about this… Where did the equations for diffuse and specular reflections come from? What are the units of brightness we use for lights? What are the units for the pixels that get rendered?
It took lot of reading, re-reading, and re-re-reading, for me to really understand some of these things, so now that I have a blog, I thought I would share what I learned just in case anyone else is wondering about these things too.
The Rendering Equation
When light hits a point on a surface, some of it might get absorbed, reflected or possibly even refracted. Also, there may be additional light being emitted from that point by a power source, or perhaps scattered in from another point on the surface. Things can get complicated pretty quickly!
Luckily for us, some smart guys came up with something called the rendering equation to deal with these factors. The rendering equation can produce incredibly realistic-looking images, but in its original form, it can also be very costly to evaluate.
I’ve included a slightly simplified version of the rendering equation below. If you compare this to the version currently on Wikipedia, you’ll see that I’ve removed a couple of parameters, t and lamba.
Typically if you have something like the power output of a light varying over time, it’s a better to idea to evaluate it before trying to solve the rendering equation. By doing this, you can assume time is constant, and so you can ignore it.
The lambda symbol in the original equation represents a dependency on the wavelength of the light. Without this, everything would be greyscale, so it’s an important property. Rather than dealing with wavelength explicitly, we can just treat the red, green and blue color channels independently and solve the rendering equation once for each channel. Note that in practice we end up using per-component vector mathematics to solve the equation for all three channels at the same time.
Outgoing Light
Emitted Light
This is any light that is being emitted from the point. Most surfaces don’t emit light, so normally you don’t see any contribution here.
Integral
This says that the enclosed functions need to be integrated over all directions w’ in the hemisphere above x. The orientation of the hemisphere is determined by the normal, n.
BRDF
This is the bidirectional reflectance distribution function (BRDF). It’s a fancy name for the ratio of the amount of light reflected in a particular direction w, to the amount received from another direction w’. The BRDF warrants its own discussion, but for now it can just be thought of as the reflection amount.
Incoming Light
This is the incoming light at the point x from the direction w’. Note that the incoming light doesn’t have to come from a light source (direct light). It may have been reflected or refracted from another point in the scene (indirect light).
Normal Attenuation
This attenuates the incoming light at x based on the cosine of the angle between the normal n and the incoming light direction w’.
Making it Real-Time
We use the rendering equation to perform lighting calculations in games, albeit in a simpler form. The most obvious problem for evaluating the rendering equation in a pixel shader is the integral, so we need to find a way to approximate it. One thing we can do is to split the way we deal with direct light and indirect light.
Since the indirect light is the harder of the two to deal with, we can approximate it. There are various different ways you might want to approximate the indirect light, from a simple ambient color, to more complex forms like spherical harmonics. Typically you would modulate your indirect light approximation with the diffuse part of your BRDF, since you don’t have a directional component to use in direction-dependent BRDFs.
By approximating the indirect light, we only have to worry the direct light in the rendering equation, so we can replace the integral with a simple sum over k light sources. Also, we typically model the change in the BRDF over space by using textures mapped over the surface, so x is no longer needed for that function.
I’ve changed up the notation here to match more closely what you see in games literature. The vector v is the normalized direction from the point x to the camera. The vector li is the normalized vector from light source i to the point x.
Units…
Well, there are well defined units at play here, but I’m going to say right now that the units don’t really matter. I don’t know of any game engines where they attribute real-world units to lighting and material values, since those kinds of things tend to be driven more by the appearance than the physical correctness. The relative intensities of, say, a candle and a 100W light bulb are probably more important than the absolute values.
Having said all that, here’s the information anyway: The light is measured in units of radiance (Wsr-1m-2). Looking back at the integral in the rendering equation, you can see that the radiance is multiplied by the differential solid angle. This converts the radiance into irradiance (Wm-2).
Remember that the BRDF is the ratio of light reflected to light received? Well that’s a ratio of radiance to irradiance, so the BRDF has units of sr-1.
One thing to be careful of is that the incoming light in the simplified version of the rendering equation (using the sum over the lights) is measured in units of irradiance (Wm-2). We have to use irradiance directly here since the typical lights we use (point, spot, directional) don’t have any area.
That’s It For Now
I hope what I’ve explained so far was fairly clear, but I know I’ve left some pretty big holes here. I’m going to be looking at some of the following at a later date:
- What is the difference between irradiance and radiance?
- What is the BRDF?
- What is energy conservation, and does it matter for games?
Nothing To Report
Well, there’s not much to report really. I’ve been spreading myself very thin recently, investigating a few different things, so I thought might be nice to post a status update.
Variance Shadow Maps
Yes, I’m jumping on the variance shadow maps boat. They look nice, they’re easy to implement, and have relatively small overhead over traditional shadow maps. I read the article in GPU Gems 3, which is also available here, and it was pretty easy to implement from there. It probably took me about 2 hours to go from no shadows at all to variance shadow maps. After a bit of tomfoolery to get rid of the light-leaking I ended up with the image below. I haven’t implemented any bias yet, so there are some errors there, but the VSMs appear to be working very nicely.
Better Tone Mapping
My current tone mapping algorithm doesn’t look that good, so I’ve been investigating any alternatives. I was initially put off Reinhard tone mapping when I first looked at the DirectX HDRLighting sample a long time ago because it treated each channel completely independently. This is a really nasty idea since it can change the hue of the image during tone mapping as each channel gets compressed by a different amount. I recently re-read a gamedev article and then implemented Reinhard’s tone mapping using based on the pixel luminance, which gave much nicer results, and didn’t change the hue. One note on that article though: It says that to convert to RGB from the final luminance you need to multiply by the luminance, which doesn’t seem right. What I did was to divide by the old luminance and then multiply by the tone mapped luminance. It’s hard to tell right now if it looks better or worse, it’s just different.
The image above is using this new tone mapping, but I’m not doing the dynamic image key calculation at the moment. That basically gives me three variables (image key, midzone luminance, and white luminance) which aren’t very intuitive to tweak. I’ve generally been leaving the midzone gray at 0.18 as suggested, and then tweaking the image key and white luminance to get something that looks OK.
Windows Presentation Foundation
I’ve spent a lot of time at work recently improving one of our C# tools, which is Windows Forms based. There are many things I dislike about Windows Forms, like the lack of built-in support for something useful like the command-pattern, the horrible input handling, menu shortcuts, you name it! For my own education, last weekend I started reading about the latest and greatest from Microsoft: The Windows Presentation Foundation. From my brief foray into WPF, it seems that a lot of the the things in Windows Forms that I’ve spent time trying to fix or circumvent appear to have been adressed.
For example, in Windows Forms, if you don’t handle a key-press event in a control, then no-one else gets to handle it either, unless they have specifically attached an event handler to that controls key-press event. In WPF they have the notion of tunneling and bubbling commands. This basically means that if you don’t handle that event then your parent gets a chance to handle it (it bubbles up), and vice-versa for tunneling events.
There seems to be a much greater emphasis of splitting the View from the Model (in the Model-View-Controller sense that is), with only very loose references (or none at all) required to bind everything together. I’m very interested in finding a good method for splitting the View code from the Controller and Model code, so I’ve been reading a few articles, some of which you can read here: Model-View-Controller, Model-View-ViewModel, Presentation Model. I really liked Martin Fowler’s description of Presentation Model in particular. The Model-View-ViewModel paper was also a good read, but I really dislike the resuse of the words Model and View in the terminology (it’s doubly confusing since I already have C++ classes called Model and View representing a 3D model and a view point respectively). In Windows Forms you seem to be almost forced to keep your View and Controller code tightly bound, but there appears to be much more freedom in WPF.
I also bought a book about WPF by some bloke from Microsoft called Adam Nathan, which so far has been some good reading (despite the liberal sprinkling of exclamation marks on every page). I’d love to see a book that combines WPF and something like Model-View-ViewModel to create a real-world, complicated application rather than the fairly simple examples that appear in books and online articles. If anyone knows of a book like this, then please let me know!
Oh, but it’s not all roses in WPF though. One of the first videos I watched was this one. If you’ve seen it, then you probably have the same same thought running around your head that that I did when I first watched it: WTF? You’re basically allowed to reuse some storage from a dependency property that isn’t used to affect your control. For example, assuming you’re not parented to a DockPanel, you can use something like DockPanel.Width to store an arbitrary number. It’s like the Tag in Windows Forms, but much much worse. Microsoft seem to be touting it like a really cool feature, when it feels more like a dirty hack to me. Using unused properties to store arbitrary data is a really quick way to make your code unmaintainable as far as I’m concerned. I don’t want to have to search the code for where someone put the data into the property in order to work out what it is actually storing… Urgh!
Articles
Yeah, I’m about 80% through writing one about storing function call arguments in MockItNow at the moment. It’ll be up as soon as I finish the last 80%.
That’s all!
Materials
Introduction
A while back, I wanted to improve the material system I was using in my engine at home, since at the time, I had just just one fixed material type. I had to specify three textures for each object I wanted to render: Diffuse, Specular and Normal. If I didn’t want specular, I had to use a black specular texture. This was barely OK for very simple things, but soon I wanted more…
I’m not really going to mention anything about the lighting in this article, since that’s still a work in progress (as can be seen by the lack of shadows below), but I hope this will be interesting nonetheless.
Constraints
Designing a material system can be a bit of a can of worms, so I thought for a while about what features I really wanted.
Only One Shader
This first constraint is really all about being able to maintain the shader code, and reduce the amount of C++ code I need to write in order to use it. I’ve seen a few different engines that piece together different fragments of HLSL to form shaders. This is pretty nice since there’s absolutely no redundant code that gets compiled in, but the problem is that you can end up with lots and lots of shaders. You then need to write code to load the correct shader when you need it, write code to compile the shaders on the fly if they don’t exist etc. Compiling all the permutations is an option, but you can quickly end up having to compile thousands of shaders despite the fact that you may use only a handful. Since I’m just one guy, I want something quick to implement, and quick to maintain.
Optional Material Features
Obviously my first material system sucked because it was hard coded to one specific material type. I had to do all the specular calculations even if I didn’t want specular. Not only was this a waste of precious pixel shader instructions and texture fetches, it was also annoying to have to specifically ‘zero-out’ features that I didn’t want to use. I wanted to be able to just specify the material properties that I do want to use.
Optional Textures
When I was playing around with creating materials, I would quite often find myself wanting to have a solid color. The way I had to do this in the old system was to create a constant-colored texture. I really wanted to just be able to specify a base color for something like diffuse, then if I want to override it, I could specify a texture as well.
Implementation
There were really only two things I had to do inside my shader in order to satisfy my constraints:
- Use branches inside the shader to choose whether or not to execute the code for a particular feature.
- Use the fact that DirectX 10 returns 0 from an unbound sampler to lerp between base colors and texture samples.
Uber-Shaders and Branching
Using branches in a shader isn’t a new idea by any means. You’ve probably heard of uber-shaders before, and that’s exactly what I implemented. An uber-shader is basically a shader that does everything, but you can turn features on and off using branches. One thing you need to beware of though, is that there are two types of branches inside shaders: Static and Dynamic.
When you use static branching, you are using the contents of a constant register to decide which branch to take. As such, the shader knows which side of the branch it is going to take before the shader even executes. This means that the branches that are not taken are skipped completely, and so this is a very efficient form of branching.
Dynamic branching uses the contents of a variable inside the shader to make the branch decision. Microsoft says that “the performance hit is the cost of the branch plus the cost of the instructions on the side of the branch taken”. At first, this seems pretty good, since the performance hit of a branch instruction (if, else, endif) is only two cycles, but this isn’t quite the whole truth. When pixels are being shaded, they are part of a pixel quad. This is just a group of four pixels that are being shaded at the same time (I assume that the reason for this is to allow gradient operations to work correctly or something). All the pixels in the quad get shaded simultaneously, so if one pixel takes one side of the branch, and another takes the other side, the shader must execute both sides for the quad. This is actually worse than if there was no branch at all, since you have paid the cost for both sides of the branch, plus the cost of the branch instructions too!
Given that fact that I know at draw time which branches I want to take before each draw call, I use static branching to turn features on and off.
Sampling From Unbound Textures
There was a change in behavior in Direct3D 10 where reads from unbound samplers now return zero rather than one. This turned out to be pretty convenient, since I use this to sample from textures in a slightly different way. I just wrote a function which samples a texture, then uses the alpha channel to blend between a predetermined value and the sampled value modulated with the predetermined value.
That sentence was probably a bit wordy, here’s some psuedo-code:
float3 LerpSample(float3 baseValue, Texture2D sourceTexture)
{
float4 sample = texture.Sample(texcoord, sampler);
return Lerp(baseValue, baseValue * sampler.rgb, sample.a);
}
So imagine that I have a red base color for my diffuse color. When I call LerpSample on an unbound sampler, the alpha value is zero, so I just get back my original red color. When the sampler is bound, then the alpha value is used to blend between the base value, and the tinted sampled value.
float3 diffuseColor = LerpSample(material.baseDiffuseColor, diffuseTexture, diffuseSampler, texcoord);
For sampling a normal map, I use a slightly different function, since I just want either the base value, or the sampled value, not a blend between them.
Material Descriptions
I store my materials in xml format, and parse them and convert to a runtime format as part of my pipeline. This allows me to leave out whole sections (like diffuse, specular, reflective etc) very easily, and it is also very simple to edit. Take a look at a sample material xml file here.
Conclusion
Well that’s it really. It’s still a rather rudimentary material system, but it’s a big step up from what I used to have. I’d like to add support for multiple textures of the same type at some point, but I don’t really need it right now. Overall I’m pretty satisfied, and so far it has been very easy to use.
Here’s a breakdown the different layers used in the wood material:
Ambient Occlusion:
Indirect lighting approximation:
Direct diffuse lighting:
Direct specular highlights:
Reflections from an environment map:
All together:
And that’s it! I’ll talk a little bit about the lighting models I’ve tried out, and my current lighting solution next time.