Perceptual Hashing and Wabbajack 2.5.0.0 (Patreon)

Over the past few weeks I've been developing some new features for Wabbajack that I'm happy to announce today. These features are coming in the next version of the app, 2.5.0.0, and we should see a release soon if our testing goes well.

I've added a feature that allows Wabbajack to detect when a modlist author has optimized some textures. This means we can perform the same resizing/recompression during install instead of patching the files with our binary patching. This process works via a process called "perceptual hashing". Normal hashing, that you see take place in several parts of the app today, works on a binary level. An algorithm reads every byte in a file, and generates a number based on the file. The key here is that if two files have exactly the same contents they will hash to exactly the same value. The algorithm we use today for files is called xxHash64, and the key thing to remember here is that it works at the binary level. A single byte being different in the file will result in a completely different hash.

Perceptual hashing is a bit different. In this case we analyze the pixels in the image and generate a number, a number that can be compared to other perceptual hashes. The result of this comparison is a "similarity" value, where 0 is "the images share nothing in common" and 1 is "the images are exactly the same". So a similarity value of 0.99 means "they're really close, so close a human will never see the difference".

So how is this useful? Well there's something really nasty about texture compression: DDS compression is not deterministic. The same image compressed on two computers, or by two GPUs will have different values inside the DDS file and will have slightly different values when displayed to the screen. When I say "slightly" I mean "really slightly". The variance is somewhere around 0.1% or less, but it's enough that the xxHash of the files will be completely off.

The problem we have in Wabbajack today, when people use something like the Cathedral Asset Optimizer is that some data is lost during compression and resizing. So if someone compresses their textures with BC7, we can't decompress the files, hash them, and match them to the original archives, because the lost data means we end up with completely different hashes. Also our binary patcher prefers files that don't have a lot of noise; the patcher likes to patch files that have a small number of modifications.

This is why some .wabbajack files you see are 2GB or larger. The modlist likely contains a lot of recompressed or optimized textures, and while WJ can handle these cases, it does so by a brute force method of comparing the files and trying to patch them.

In Wabbajack 2.5.0.0 we have a new option: match files via perceptual hashing, then record the changes needed to transform one file to another. We can set the threshold rather high, like 0.90, and only link files that match by file name. What WJ then sees are two files that match each other in name, and appearance, one is decompressed and the other is compressed. It's then simple for WJ to just record that it needs to compress the file during installation. So instead of a 5MB binary patch, we only have to save around 64 bytes of metadata.

In one test I ran, I optimized and resized all the textures in Skyrim's Textures - 1.bsa file. With Wabbajack 2.4 the resulting .wabbajack file was 350MB. With this new method it came in at 5MB.

We're testing this feature now, and hopefully it'll be ready to release in a week or so.