Wabbajack 3.0 Development Update - Rate Limiters (Patreon)

Well it's been a long week and a half. Turns out my research into how to best implement a rate limiter in Wabbajack resulted in me writing the same code about 4 different ways before finally finding an approach that works.

Wabbajack needs limiters in several places (as mentioned in the previous update), but my mistake was thinking that the best approach would be to limit on a physical resource: Network bandwidth, Disk Bandwidth, CPU threads, etc. The problem with this approach is that some tasks require more than one of these resources, resulting in a deadlock during complex tasks. A deadlock is when task A is waiting on B which is waiting on C which is waiting on A.

Instead the approach I finally landed on was based on a task type: Downloads, archive extraction, HTTP requests, etc. This means that in Wabbajack 3.0 we'll now be able to say "never download more than 2 archives at once, but you can have 10 outstanding requests to HTTP services (like the Nexus) at one time". In the final release of 3.0 you'll be able to set limits on each of these independently, so you'll be able to say "you can download 5 files at once, but don't extract more than one archive at a time".

As of tonight, our new batch based file validation method can successfully validate all the curated Wabbajack modlists. Now I need to clean up that utility a bit, set it up so we can trigger it with GitHub Actions, then update Wabbajack 2.5 to use this new validation service. At that point I can turn off a major portion of the Wabbajack build server and instead run this Github action once every few hours (or whenever a list changes). My goal is to get that all in place by the end of this week.

In the past week and a half I also implemented all the remaining downloaders in use by curated lists and figured out how to get them to pass their test suite on Linux, OSX and Windows. I also got CEF (or in-app web browser) working on Ubuntu and was able to load the new Wabbajack UI app on my Ubuntu laptop and log into a website. This means that after this week I should be able to dive full time into working on porting the Wabbajack UI to the new codebase, and we'll be even closer to our beta release.

Thank you all for your support, and be sure to keep an eye out for future updates.