Ray-tracing soft shadows in real-time

Shadows are arguably one of the most important graphical effects in order to achieve a realistic look in a game. Traditionally shadows are done using the rasterizer and a shadow mapping technique which stores depth in a texture. There are a few problems doing this, one being that this texture needs to be very large to achieve sharp-looking shadows. Everyone who has implemented shadow mapping also knows about peter panning and shadow acne which are issues caused by the low precision of the depth texture. Current popular ways of rendering good looking shadows are cascading shadow maps and percentage-closer soft shadows. The downside of these is that they are based on estimated calculations and might not look good in all situations. Using raytracing we can simulate how light behaves in reality and create photorealistic shadows but the difficult part is doing this in real-time.

Using ray-tracing we can achieve pixel-perfect hard shadows very cheaply by tracing a ray towards each light and checking if it hit anything or not. Hard shadows never really show up in our real world though since all light sources have an area that creates both an umbra- and a penumbra part of the shadow.

Hard shadows are only caused by light sources that are either infinitely small or infinitely far away. Using ray-traced hard shadows will still look realistic for games that use the sun as the only light source since it can be approximated as an infinitely far away source.

However, most games use way more than just the sun as a light source and in order to shadow the world realistically using many lights requires a method which shows both the umbra and penumbra. This is called soft shadows and can be done in a photorealistic way using real-time ray-tracing. It is definitely not as cheap as hard shadows but there are many ways to speed it up in order to run it in real-time. The following method of implementing ray-traced soft shadows should be seen as simple and about the minimum required to achieve a decent result.

Before we continue, I will assume that you as the reader have a decent understanding of 3D rendering. No knowledge on DXR or Vulkan raytracing APIs is required but still recommended since these are currently the APIs to use if you want to implement what this article teaches. Some code snippets are shown in HLSL but converting to another shading language should not be a problem for the reader.

If you want to learn DXR I can recommend reading Nvidias tutorial to learn the basics. Also, check out the official DXR functional specifications. If you want to learn more about raytracing in general then I highly recommend the book Ray Tracing Gems.

Let's get to it!

The shadow rays

In order to simulate the umbra and penumbra perfectly, we need to cast a ray towards each point on the light. This is practically impossible as there are an infinite amount of points to trace towards and we want to reduce the number of rays/samples per pixel (spp) for performance reasons. In this implementation I used 1spp. This ray is traced towards a random point on the light, the point also changes each frame which gets important later.

To figure out the direction in which to shoot this ray we use some linear algebra and trigonometry. All light sources in this example are spherical for simplicity. The first step is to figure out the angle of the cone which starts at the pixel being shaded and perfectly encapsulates the light source, see figure below.

A vector perpendicular to pl is calculated using cross product with the world up vector. This vector is then used to calculate the vector pe. When pl and pe are known we can calculate the cone angle α using acos of the dot product between the two. The following is an HLSL snippet that does exactly this.

When the angle is known we need a way to get a random direction vector within the cone so that we can sample a point on the light. The following is an HLSL snippet to do this. My method below is based on the answer given here https://math.stackexchange.com/a/205589/81266.

At this point, we have our direction vector which can be used to trace a ray.

The ray starts at the world position of the surface hit by the visibility ray. In other words, the world position of the point which will later be shaded. From this point we trace in the direction calculated from the cone angle. It is important the ray has a maximum distance set to the distance between the world position of the pixel and the position of the light, otherwise you will get false positives as the ray may hit geometry located behind the light. If the ray hits anything then the pixel is in shadow. We save this information to a screen-space texture.

Since we use only 1spp, the resulting texture will be very noisy. A lot of work is still required to get this denoised before applying it to the shading.

Spatiotemporal accumulation

This is the first step to reduce the amount of noise by, in a way, faking using multiple samples per pixel. The name spatiotemporal accumulation hints that spatial and temporal information will be used to achieve this by combining the noisy textures from previous frames with the current one. This is basically a form of Monte Carlo integration, which is why it is important that the point we previously sampled on each light source is randomized every frame.

Storing copies of the shadow textures from multiple previous frames will take up quite a bit of VRAM, which can be a problem. Luckily there is a way to accumulate multiple frames by only storing the shadow textures from the last frame by using what is called an exponential moving average: alpha * shadowThisFrame + (1 — alpha) * shadowLastFrame , where alpha is a factor between 0 and 1 indicating how much temporal stability should be traded for lag. Increasing the number lowers the ghosting/temporal lag but it also increases the amount of noise. I found a value of 0.3 to be a decent tradeoff.

So far we have done temporal accumulation. This will result in a less noisy texture, but there is a big flaw that is very noticeable when the camera or objects visible in the scene move around — the shadows also seem to slightly move which causes visual artifacts.

The problem is that pixels might have moved from the last frame which should also move the shadow, but since the temporal accumulation only cares about screen positions this is ignored causing the artifact. To solve this we need to take each pixel from the shadow texture of the last frame and move them to the location they would be on the current shadow texture by taking object and camera movements into account. This is done using a special motion vector texture that is written by the rasterized G-buffer pass. Normally when doing rasterization you only take into account the current world, view and projection matrices for an object, but in the case of motion vectors we also take the world, view and projection matrices from the last frame into account. After transforming each object into screen-space using both sets of matrices, the difference is used and stored in the motion vector texture which contains two channels of 16bpp floating point precision. The following is a HLSL snippet doing this.

Each pixel in the texture then contains the screen-space offset of how much it has moved from the previous frame. Now we can apply this data in our texture filtering as the spatial part of the spatiotemporal filtering. When sampling InputShadowsLastFrame (seen below), we take the motion vector into account by subtracting the motion vector from the current texture coordinates.

Keep in mind that there exists a possibility of pixels never before seen can enter the screen. This should be handled separately as there does not exist any spatiotemporal information in that area. However, it is not currently handled in this implementation which causes artifacts around the edge of the screen when the camera moves. This is something that can be improved upon in the future.

The end result of adding the spatial aspect gets rid of the issue that made shadows move with the player and further keeps the noise down even when the camera is moved.

Final denoising using bilateral filtering

As seen in the image above, the resulting shadow texture is significantly less noisy than before applying spatiotemporal accumulation. It is however still not at the point of being usable for shading as it still contains noise that will be visible to the player. To try and get rid of this last noise, we run the spatiotemporally filtered output through a bilateral filter. A bilateral filter is similar to gaussian blur in the sense that it replaces the intensity of each pixel with a weighted average of intensity values from nearby pixels, see wiki. The important difference is that a bilateral filter is edge-preserving, meaning that areas of the image with sharp edges will not be blurred, or blurred less than areas with no edges. This is important when filtering soft shadows because the end goal of using soft shadows is to achieve a penumbra and umbra which has an edge between the two and its sharpness will vary depending on the size of the light and the distance between the object casting shadow and the receiver of the shadow.

Using multiple light sources

So far we have only worked with a single light source. When handling multiple light sources it is important to realize that many shading implementations require light-specific shadow information. One pixel on the screen may be shadowed from one light source but not from another light source. It is therefore not possible to render all shadows into one texture. They need to be separated into one texture per light source where each texture then contains shadows for that particular light source. This is different from shadow mapping since these textures only need to be in screen resolution in order to achieve pixel perfection. Each shadow texture needs to be denoised separately, which is one of the most performance heavy operations of my implementation. This is an area where future improvements could substantially improve performance.

Separating shading from ray-tracing

Usually, shading can be done directly in the ray-tracing shaders, but this is not the case for us. The spatiotemporal filtering is done in the ray-tracing pass, and the compute shaders doing the final bilateral filtering runs afterward. The fully filtered shadow textures are required when doing the final shading of the output image, which forces a separation between shading and ray-tracing/denoising. Shading is done in a rasterization pass from a full-screen quad that runs right after the last bilateral compute shader. This separation causes other issues regarding missing information, the main one being extra material information that is required for shading. In my implementation, I solved this by writing the material information to textures during the ray-tracing pass and used the textures in the rasterized shading pass. This approach can use quite a bit of VRAM (depending on how much material information your shading method requires) but the only way to reduce it is to reduce the number of textures which would only be possible if the shading could be done directly in the ray-tracing shaders or if material information could be accessed in the shading pass.

Final shading pipeline

We now have a basic implementation that can render good looking soft shadows from multiple light sources. The implementation could be further improved both in performance and visuals using new research such as varying the sample amount per pixel depending on if the pixel is in a penumbra or not as explained in chapter 13 in the fantastic book Ray Tracing Gems.

Hopefully, this article has given you a better understanding of how shadows work as well as how basic denoising can be done to achieve real-time raytracing.