Video Interface processing (Patreon)

Hi Everyone!

With the VI video processing pipeline being implemented now, I feel ready to tell you about all the features it can do. We will do this today step by step, looking from the big picture and zoom in on every component in detail.

Let's have a look at the overview first:

We have a data flow from the RDRAM holding the framebuffer to the VI processing the data and then outputting it to the Digital-Analog-Converter before it's send to the screen. The CPU can write all kind of settings in the VI which will modify the image that is output.

If we go on step deeper into the VI, there are several modules:

Of course i cannot be sure that this is the logic the real VI uses, as I cannot look into it, but it's the structure that the core uses to fulfill the same tasks, so it shouldn't be too far off.

First step is the line fetch. The N64 has 3 full line buffers of pixel data to work on. Whenever a new line is to be output, new lines maybe have to be fetched. This is relativly complicated, because the N64 can scale the output for x and y axis. In case the output is streched, multiple lines can be needed and it the opposite case, the old data might be output again.

When the current output line, the previous and the next are fetched into the 3 linebuffers, the Filter will start to work on these. Filtering includes Antialiasing, Dedithering and Divot. We will look into these 3 in detail later.

The filter will reduce the 3 input lines to only 1 output line, which is then fed into the Output processing. This contains bilinear scaling, gamma and noise dither functionality.  As the bilinear scaling needs 2 input lines, it has to wait for 2 lines to be filtered before it can work, unless the scaling is configured by the game to be off.

After the bilinear scaling we are only dealing with 1 pixel at a time and this gets gamma and noise dither to be finally finished for sending it to the screen.

The video timing will now take these pixel colors and send them to the DAC...well not so fast! It has multiple options for borders and safeguards so that you don't output color while being in the HSync or Vsync area. It's the part of the core that is currently not yet fully implemented and why you might see the image not being centered correctly or showing garbage next to it.

Coverage

Now that we have an overview of the functionality, we could dive into the fun part of looking how each of those functions work on a pixel level, but we have one more thing to learn before: coverage.

The N64 has a very advanced rendering pipeline compared to lets say the PSX. One of these features is sub pixel rendering:

With the PSX rendering on the left side and N64 on the right, you can see that for each pixel the PSX renders, the N64 has 4x4 subpixels.

Within these subpixels we can determine the coverage, which tells use how many of those 16 subpixels are covered by a given polygon. This can be used for example when two polygons want to render into the same pixel, so they can be blended together with the priority based on their coverage of that pixel.

To do that, we would need to know the amount of subpixels covered from the previous polygon when we render a new one. So we don't only need a framebuffer for color and z-buffer but also for coverage.

This is where the 9 bit memory of the N64 comes into play: typical RAM has 8 bits per byte or address, but the N64 RDRAM had 9 bits. With a 5 bit per color framebuffer, 15 bits are required, but 2x9=18 bits are available and we have 3 bits spare that could be used.

3 bits can save any number from 0 to 7, but with 4x4 subpixels, we would have a coverage of 0 to 16. So here comes another trick by the RDP developers: checkerboard

The RDP only uses every second subpixel for the coverage. This way it only needs to store the numbers 0 to 8.  Well that's not too far off, so they just decided that 7 is close enough to 8 and store 8 as 7. Done.

What does this coverage value mean now?

- Coverage of 7 means the pixel is completly covered, so it's inside of some polygon

- Coverage of 1-6 means that the pixel is on the border of a polygon

- Coverage of 0 means the pixel is not covered at all, so outside of any polygon. But it could also be a rare case of barely covered border due to the checkerboard, that is why it is also handled like a border

Dedither

With that we can finally work, so let's start to dive into Dedithering. Dedithering is needed to remove dithering. Sounds crazy, but why even do it in the first place?

The N64 can render in full 8 bit per color, but to store and read back that from the framebuffer would require more bandwidth of the RDRAM.  With the RDRAM being THE most performance critical ressource of the whole system, this would have slowed down the framerate, so only 5 bit per color are used by most games.

As 5 bit per color means only 32 different intensities per color, the result is color banding. Dithering is adding a pattern to the color to prevent this color banding.

You can see in the left image that the grass has very strong color banding that is reduced a lot with dithering in the middle, even with still using 5 bit per color.

With dedithering on the right side, the dither pattern that was introduced in the middle is gone again.

Why is that possible? Because after dedither we have 8 bit per color again.

(In this and all upcoming cases I will always write about a pixel or a color. But be aware: everything happens for the 3 colors red, green und blue in parallel and independently.)

Dedither will only work on pixels that are inside a polygon, so with coverage = 7. For such a pixel, all neighbor pixels are compared against it, that's why it's called center here. We have 8 comparisons and each time a neighbor color is larger(higher intensity), we will add 1 to the intensity of the center and each time a neighbor color is lower, we will subtract 1 from the center.

As the dithering pattern was still done with 5 bit per color and we are now back to 8 bit per color, we gain additional 3 bits that are still not used and can take up these additional plus/minus up to 8, resulting in a smooth color gradient.

In this process the color of one pixel is kind of smeared to neighbor pixels, so it can create a more blurry look, but keep in mind that each center always compares against the source pixel of a neighbor, not the already dedithered pixel. Therefore, some color information cannot be dragged away further than 1 pixel from the source and even with the extreme case of +8 or -8,  the different is, compared to the source pixel, never higher than 1/32 of the maximum intensity, so you cannot really move away from the original color for each pixel, meaning the added blur is limited.

Anti-aliasing

At the same time Dedither is calculated, also Anti-aliasing is done:

The idea is to smooth edges out by transfering some of the color to nearby pixels.

How does that work?

When a pixel has a coverage of below 7, it cannot be in a polygon, so it must be outside or on the edge and Anti-aliasing can be applied. With that information we can look at the center pixel and neighbor pixels to see if they fall into this category.

For Anti-aliasing, we are looking at pixels on the edge of a polygon, but we want "stable" color from inside a polygon, so we search in a hexagonal pattern of pixels around the center for pixels with a coverage of 7.

Within these 6 pixels that are checked, there can be any number of such fully covered pixels from 0 up to 6.

In case we only find 2 or less, we just keep the color of our center pixel.

If we find more, it gets complicated:

- within all these pixels with coverage of 7 we search for the pixels with the second lowest and second highest color value

- the difference of these two color values against the center is calculated

- finally the center values color is interpolated against this difference, depending on how high the coverage of the center was

Two things are odd: the hexagon is squished with being in a 5x3 area rather then a 5x5 area. This is due to the higher cost of having 5 full framebuffer lines inside the chip, which was too much. The result is still good, but might not be equally good for all degrees of borders.

The second odd choice is using the second highest and lowest value, instead of the highest and lowest. This seems more of a try-and-error optimization choice where different scenarios where compared and it delivered better results, by sorting out more extreme pixels.

The result is quite good, considering that no additional rendering was done for these pixels. If you look the the border between grass and sand, you can see that is it much smoother in the upper image with Anti-Aliasing on.

Divot

Due to the Anti-aliasing acting kind of random, without really knowing which pixels are really part of which polygon, it sometimes happens that it takes colors from nearby pixels that don't really fit in the overall image.

Such stray pixels as marked in the left image can happen. In this case the border of the shadow of the coin received Anti-Aliasing and it didn't work out so well. The wrong pixels of the checkerboard background have been picked to smooth out the border of this shadow.

How can we get rid of them? The algorithm is relativly simple:

- 3 adjacent horizontal pixels are compared. Those are taken from the filtered result

- if any of these 3 pixels has a coverage below 7, Divot is applied

- Divot will pick the median color value of those 3 pixels

Let's do an example of what that does with artifact pixels. We assume all those pixels have a coverage below 7 to keep it simple:

We have an incoming pixel stream with one stray pixel: 9

It doesn't fit in the colors of all the others so we want to remove it.

- In the first step we look at the pixels 5,4,9 and sort them, so we get 4,5,9 and then we pick the middle one: 5

- The same is done for 4,9,3 -> 3,4,9 with the middle 4 being picked

- The last pixel is picked as 4 again from 9,3,4 sorted to 3,4,9.

-> We reached our goal, the 9 is gone.

Now you could say that this might remove important information in the image, but if you think about what antialising did before, it's very unlikely.

The reason is that Antialiasing will smear the border first before it is handled by Divot, so the color information of the border will be in several adjacent pixels, while Divot will only remove border pixels that don't fit to any neighbor.

This completes the filtering steps and we can look at the output stage.

Bilinear Scaling

The N64 has a configurable scaler. You can shrink the image in the framebuffer or enlarge it. It is kind of comparable to what modern consoles or your MiSTer do when outputting via HDMI to for example fixed 1080p while the image has a smaller render resolution.

The big difference is that the N64 has a much lower output resolution with only 640x240 (NTSC) and the scaling is not very advanced, so the quality of the resulting image is lower compared to modern solutions.

Still, for it's time, this offered several possibilities. Some of them you might not like, but it's an older console with older games so we have to take it like it is:

- render FMVs at low resolution, e.g. 160x120 and upscale

- render PAL with the same resolution as NTSC and upscale

- render some odd resolution like 400x300 and either downscale to 240p or upscale to 480i

How does it work?

We have two counters for the x and y position on the screen. With every output pixel x is increased by a fix point number, so e.g. by 1.0 or 2.0 or 0.5 with the same being done with every new line for y.

These x and y positions determines which pixel of the filtered framebuffer is picked to be output. With bilinear scaling turned off and only point sampling being used, the fractional part is just cut off.

For example x = 5.4 and y = 7.8 will pick the pixel from the position 5|7

With bilinear scaling on, it will filter the pixels 5|7, 6|7, 5|8 and 6|8

You may remember this image from a previous article?

That's right, it is how the PSX core does the optionally added bilinear filtering. The method is the same:

Without filtering, the dark green color of the square the red dot is in would be used as color.

With bilinear filtering the 3 closest other texture pixels are also used for the final color calculation. The main color still contributes most to the final color and the other colors contribute less, depending on how far away their center is compared to the sample point, visualized by the thickness of the red lines.

Due to the simple nature of the filtering, there can be a lot of blur, especially when the fractional part is used all the time, e.g. when scaling to 110% size.

You can turn off this feature in the core and use point scaling instead and for some games that only scale 320x240 to 640x240 like Mario 64, this makes sense.

However, for PAL games that do minimal up or down scaling by only some %, this leads to severe issues with text being unreadable, so the filtering with blur is often still the better solution.

Gamma

After we calculated the single output pixel now, we have two more optional steps before it is output. The first is Gamma.

You probably expect some very sophisticated feature now with adjustable curves and possibilites to tweak it, but I have to disappoint you there. All gamma does is a square root of the color value.

As that would make the color value too small, it is shifted by 6 bits before, so multiplied by 64 and the result is also multiplied by 2.

One example of a dark pixel:

- color of 32 -> multiplied by 64 -> 2048

- square root of 2048 -> 45,25 rounded down to 45

- multiply 45 by 2 -> 90

-> The color gets significantly more intense.

One example of a bright pixel:

- color of 250 -> multiplied by 64 -> 16000

- square root of 16000 ->126,49 rounded down to 126

- multiply 126 by 2 -> 252

-> The color is not modified much at all.

That's what it does in the end: boost the intensitiy of darker colors.

You can see it in the left image on the character models.

Not many games use that feature overall, but those that do really depend on it, otherwise you cannot see much in the dark areas.

Gamma Dithering

And we finally reached the last feature. The image can receive additional Noise that is often named as Gamma Dithering, but I don't think the name really fits it well.

This feature has 2 different results when it is turned on:

When gamma is off, it randomly modifies the colors by 1 on a scale of 0-255. You can already guess how weak that effect is. I need to be close up to my development LCD Monitor to even notice it and my TV filters this noise out, so it's completly gone. You can try youself: Mario64 is using this.

When Gamma is on, it behaves different: when the color is multiplied by 64, the lower 6 bits of the color information are empty. With Gamma dithering also on, these 6 bits are filled with random numbers. We have already seen with Gamma that is mostly boosts dark colors, so this effect is very visible on dark areas.

I had to enhance the overall brightness to really show you the difference in a still image, but you can see that the image below has this random noise in the image.

Does the name "Gamma Dithering" make any sense now? I don't think so. For making it more clear to the user, I named it Noisedither in the OSD. For one, because it also has some effect even if gamma is off and also to make more clear this is only noise in the end and not so much dither, as it cannot really reduce any banding at all and it's also not used for that.

That completes all the video processing of the VI. As mentioned above, the output stage is still to be done on the core but also doesn't hold anything special for the N64, it's just your normal video signal generator for NTSC and PAL and mostly work to be done.

if you want to try out the new features, I attached a core where you can test them all. There are toggles for all these features in the OSD which makes it easy to play around and see the differences.

Until next time, have fun!