And now we start Dogfooding (Patreon)

In software development there's belief that developers should regularly "eat your own dog food" also known as "Dogfooding". In short it means personally using the product you're building. New features are nice, but until you actually try to use the product, you're never quite sure if it's going to work in the wild.

So this weekend I sat down and started installing the YASH'ed guide ( https://www.nexusmods.com/skyrimspecialedition/mods/23894 ). I chose this pack because I'm familiar with the general concepts as NORDIC used to be my preferred pack, and it's a good mid-range pack. Not quite so involved as Lexy's and yet with quite a few more odd edge cases compared to some of the other SSE packs in existence.

My testing process is fairly simple. After each major section of the guide, I setup Wabbajack and run the build. Every few sections I take the built installer and install the guide again to a different location on my system, then use WinMerge to perform a binary compare between the two files. Aside from a few edge cases, every single file should be exactly the same in both locations down to the byte.

So far this has produced some interesting results. Last night I found out that my BSA builder code had a bug and that Skyrim would crash after Wabbajack installed a BSA that it had built. That ended up being a bug in how I calculated file sizes inside the BSA, that's fixed, and Skyrim now runs no the output of Wabbajack without crashing.

Next I got a bit irritated at how slow the compilation process was, so I dove into how to add a bit more parallelism to the work queue, and implemented something known as "work stealing", where a CPU core that's waiting on another core can help out while it waits. It's a bit like telling management to "go help on the assembly line instead of sitting in your office waiting on people all day".

I also discovered that Wabbajack doesn't handle ESPs that were created by the user during the installation process. This applies to dummy ESPs created by CAO, as well as hand written patch files. Dummy ESPs are now detected by Wabbajack, and if you create an ESP from scratch you can put it in a mod, and put "WABBAJACK_INCLUDE" in the MO2 notes or comments, and Wabbajack will pick that up and include it verbatim as part of the modlist.

Finally, I'm still kicking around switching away from SHA256 hashing of files. SHA256 works great, but it is a cryptographic hash, which means that it's rather slow. During compilation of a modpack we have to hash about 50GB of files, and right now the bottleneck on my box is the CPU. I could speed up the build process by about 5x if I could find a way to make the hashing Disk bound instead of CPU bound. But that's something I'll continue to look into and it can be changed at any time without affecting end users.

So anyways, we can install the first 7 steps of the YASH'ed guide with Wabbajack, only about 25 left to go :D