Progress Report - November 14, 2023 (Patreon)

Hello, everyone!

I apologize for being a day late for this week's Progress Report. I wanted to make sure the system I was working on yesterday actually worked before making the post, and I just wasn't quite able to get it before end-of-day on Monday.

Last week, I described how I had figured out a flexible conversation system in Unreal Engine 5, and how I was in the process of building a pipeline for automatically building up conversations from an abstract graph designer. I had gotten the graph designer working, and the automatic generation of text-to-speech placeholder audio as well.

I had noted that the UE5 importer part of the pipeline was going to be the make-or-break of it all. The "big doozy", as I put it.

Relatively Smooth Waters

As a whole, the importer actually went pretty smoothly. It turns out, UE5 has support for Python when it comes to writing editor tools. You can't use Python during runtime (for some reason), but for writing editor tools, Python is fully supported.

Which is perfect for me, because the Balabolka invoker used to generate the audio was already written in Python, and a requirement of the Balabolka invoker was importing the graph and rebuilding it in memory. Which is exactly what I need to do in UE5.

So I was able to literally just copy-paste the Balabolka invoker over into the UE5 Python script, and then cut out the audio-generation code and replace it with asset-importing code. Literally best-case-scenario, it couldn't have been any easier. Just a drop-in solution, and all the parsing code already worked.

Getting the asset importing working, once I figured out how the UE5 asset system actually works, was pretty straightforward. I already have all of the audio file data saved in the Audio Association Files, so it's just a matter of loading the filenames from that file and importing them as audio assets in Unreal. The subtitle files are named exactly the same as the audio files, just with a .LRC filename rather than .WAV, and so they were trivial to find. The .LRC file format is super simple to read, it's just timestamps and text lines.

I ran into a bit of a snag actually applying the subtitles to the audio files, but that turned out to not be a problem with me (technically), nor a problem with UE5. Instead, it was a problem with Python as a language. I ended up burning over an hour fighting with it, before finally figuring it out.

To keep a long and technical story short and easily accessible, I will simply say that I had made a typo that should have caused the script to crash and throw an error, but because of the way Python handles how variables in loops work, it didn't error. It wasn't until I caught the typo the old-fashioned way of just reading the code line-by-line that I finally fixed the error.

After that, it was more smooth sailing, as I found duplicating the input master sequence and renaming it for all of the individual dialogue nodes to be extremely straightforward. It took a little bit of fancy footwork on my end to only make a single duplicate for choices, but that's a technical issue resulting from the graph not being a one-to-one representation of how the dialogue system actually works (the graph splits out into several nodes, for ease of reading at a glance, while the dialogue system condenses everything down more for ease of management). But I got it in the end, without much issue.

When it came to actually adding the audio to those duplicated sequences, that was a bit rougher. The Level Sequence structure in Unreal is steeped waist-high in abstraction, and navigating the documentation rabbit-hole was disorienting and a bit of a chore due to it being largely automatically generated. In the end, I had to pull out my experience from probing Source Filmmaker's entirely undocumented Python API, and experimentally probe Level Sequences in UE5 to figure out what goes where and how to make it.

In the end, adding audio clips to a Level Sequence in UE5 via Python is actually pretty straightforward. You just need to know the arcane invocation, but once you do know that, it can be done in just a few lines.

Similarly, adding the event triggers is pretty straightforward as well. There is a simple method you call to add a keyframe at a given time on a given track--in this case, the event track--and then you set what that keyframe's value is, which in this is the name of the event you want it to call. Nothing too complicated there.

So then, what all have we accomplished in the UE5 importer? We've got the audio imported, the subtitles set up, the Level Sequences duplicated out, the audio tracks added to the Sequences, the event triggers added to the event track.

All that's left is actually hooking all of the Level Sequences up. The way the system I devised last week did that is pretty straightforward: each individual Sequence has its own unique blueprint, with the sequences hard-coded into them. The first sequence in a dialogue hard-codes locking the player into dialogue, and then the next sequence is hard-coded for non-choices, and the choices hard-coded as appropriate.

Looks a little something like this:

Easy, right?

So Close, Yet So Far

I could write entire theses about how difficult this last step was. Or the several days I spent trying to solve it, then work around it, and finally redesign the whole system with the discovered limitations in mind.

Instead, I will just give a brief overview of what the problem was, and the solution I came up with. I will leave out the days of screaming and swearing in between these two end-points.

For this problem to make sense, I need to make clear a few distinctions. A Level Sequence contains all of the cinematic data, the audio and characters and camera angles and animations and choreography and all that such. It is what is actually played when the cinematic is played back. A Level Sequence Director is a separate object that manages the Level Sequence.

The blueprint shown above, where all of the events need to be defined, exist on the Level Sequence Director, not on the Level Sequence itself. I only have access to the Level Sequence itself. I need access to the Level Sequence Director in order to modify this blueprint and automatically generate the nodes necessary to hook an imported sequence together.

To cut a long, stressful story short: you can't do that in Unreal Python.

So instead, the solution I finally came to, after 3 or 4 days of fighting this, is to move all of the data upstream. The original design has each Level Sequence contain the data as to where it needs to go next, meaning that a given conversation's data is spread across dozens of Level Sequence files, and all of the game's conversation data is spread across hundreds or potentially thousands of them.

The new design takes all of that information and centralizes it. UE5 has the concept of a "data table", which is basically just a big lookup table. If you've ever worked with MySQL, then it's what MySQL refers to as a table. If you've ever worked with a programming language, then it's a dictionary / map / associative array (we sure love renaming the same thing over and over again in programming).

The long and short of it is, you have a big list of entries of data, and you can immediately jump to a specific row by a unique name.

Instead of storing the data for what sequence is next and what choices to display after a given Level Sequence on said Level Sequence, I can instead take that data and just shove it into a single global data table, in an entry named after the Level Sequence instead. It ends up being exactly the same, with the data being defined on a per-Sequence level. It's just, instead of the data being stored across hundreds to thousands of files, it's instead all stored in a single big file.

This ends up having a lot of advantages over the original design. I have to begrudgingly admit that it is a better design, after spending days trying to get the old design to work. I honestly should have thought of it sooner.

The two biggest advantages of this new design are the aforementioned centralization, and code standardization. The advantage of centralization is, if something goes wrong, there is only one file I have to check - the data-table. Of course, this also means that if anything goes wrong, everything goes wrong. But you win some, you lose some. It's still much less files to sort through to find errors.

The code standardization is a more subtle advantage to this new design. In the old design, every Level Sequence had its own unique Blueprint, with its own unique changes in the code. And a lot of hard-coding within those Blueprints, which is generally considered bad practice. In this new design though, all Level Sequences have the same base Blueprint and layout:

The advantage of this is really an extension of the advantage of centralization: by having less unique code scattered around, there are less things I have to worry about breaking. Less moving parts mean less points of failure. With all the Sequences calling the same functions, if there is something I need to change in those underlying functions (like, say, the Play Next Dialogue Sequence needing the ability to consider conditional choices...), then I only need to change the underlying function, and every single dialogue Level Sequence is automatically updated with the new functionality. Before, I would have to go and edit every single Level Sequence by hand with the changes made (or more realistically, just delete all of the imported conversations and re-import them all).

So yeah.

It took a few days to get here, but in the end, not only does this solution actually work with the importer, but it honestly is a better solution than what I had before.

Now when a conversation is imported into UE5, all of its dialogue data will be automatically inserted into the master conversation data table, automatically hooking everything up. Combine with the fact the master sequence has these already pre-declared in it, which is then duplicated and propagated to all of the imported Level Sequences, and it all just Todd Howards.

As of now, I officially have an automated pipeline for taking conversations written in my abstract graph editor tool, and importing them, fully working, into Unreal Engine 5.

The system isn't perfect, there are a few bugs I have to work out that aren't related to the importer but are related to the greater conversation system. Things like the "Set Player In Dialogue" not working correctly, with the player able to still interact with things while in a conversation, which isn't intended behavior.

But the importer itself works. It doesn't have any artistic style to it, of course--it doesn't place camera angles or animate characters. But it was never meant to. It's just automating the boring technical rigmarole, leaving only the art to be done. And the art is the fun part.

So what's next?

Bootstrapping an entire university course

As I alluded to a bit earlier, I need to modify the Play Next Dialogue Sequence function to take into account conditionals. I also need to modify the rest of the pipeline to accommodate that, but I am starting at the UE5 importer end, because it's the part that has to actually evaluate the conditionals.

Just to make sure we're all on the same page here, there are two types of conditionals the system has to account for: conditional dialogue, and conditional choices. Conditional dialogue is the game choosing which dialogue path to take, while conditional choices is deciding what options are available to the player.

Conditional dialogue is things like characters commenting on whether or not you closed a door. If you closed a door, they say "thanks for closing the door behind you." If you didn't close the door, they say "I asked you to close the door behind you, but whatever." That "If" right there marks a conditional statement.

Conditional choices is things like a character saying "you don't listen to anything I ask of you". If you closed the door, then you can say "but I closed the door like you asked". If you didn't close the door, then you don't get to say that - because you didn't. Again, the "If" is the magic sauce that declares a conditional choice.

At first, I was going to just make a really simple conditional system. A simple "if this variable is set to true then". But the more I thought on it, the more I realized how insufficient that is. What if the door needs to be closed and the shutters closed? What if the room door needs to be closed or the front door be closed? What if both the room door and the room shutters need to be closed, or only the front door needs to be closed? What if any door needs to be closed? What if any door and all shutters need to be closed, or all doors and any shutters need to be closed?

As you can see, it's pretty easy to imagine increasingly complex conditions that, in a broader game context, are not that unreasonable to actually expect for a conversation system. And then all of that is just simple true/false statements. What about questions of counting? What if you need to have at least 3 keys? What if the number of keys you need increases with higher levels? Suddenly now you need not only a system capable of arbitrarily complex true/false statements, but a way to actually compute values as well, such as "how many keys I have, plus the number of keys per level times the current level". And then it has to compute inequalities as well, "at least this many keys", "no more than this many guards killed", "exactly this many flying spaghetti monsters."

Enter the image at the top of this post, which I will post again now for posterity because gosh darn did it take a long time to get here.

The title of this chapter, "bootstrapping an entire university course", comes from the fact that what I am looking to create here is a fundamental component of code interpreters and compilers. It's only two steps removed from writing an entire programming language. There are universities and textbooks dedicated solely to this subject. Textbooks like this one:

I love computer science text books. They always have such goofy and whimsical, borderline shitpost, covers.

What I don't love is basically implementing text books cover-to-cover for a fucking porn game.

But here we are.

In all actuality, I am being hyperbolic. Not very hyperbolic, mind you, but still hyperbolic. I don't need to implement a full compiler - but I do need to implement a few full interpreters.

After a lot of digging around for boots-on-the-ground implementations of expression evaluators, I came across this absolutely fantastic piece taken from a David J Eck textbook, Section 9.5: A Simple Recursive Descent Parser. It's a clean, no-nonsense, straightforward implementation of a simple expression evaluator. I was able to fairly-quickly implement it in UE5 Blueprints - amusingly, it took me longer to implement my own string streams system, which isn't strictly-speaking necessary, but makes life a hell of a lot easier. And more importantly, Eck's code here uses a string stream (the getAnyCharacter() and getDouble() methods, which "consume" the values from the string). And I don't want to make any more work for myself than is necessary, so I just want to adapt his code as close to one-to-one as I can.

After butting heads against how UE5 Blueprints handle structs (why can I pass in by reference, but not pass out by reference? Absurd.), I actually managed to get it working!

It was honestly not that difficult to implement in UE5 Blueprints. Most of the headscratching moments were less "why isn't this translating" and more "was I high when I wrote this?"

Despite this "only" being 1 of the 4 evaluators I need to write for this conditional-dialogue-and-choices system, it is far and away the most important and the most complicated of the four.

The Boolean evaluator is actually the exact same implementation of the recursive descent parser as the Expression evaluator. The only difference is, instead of performing mathematical operations like plus and minus, it performs Boolean operations like AND and OR. I can just copy-paste the Expression evaluator, and change the operators. The Boolean evaluator basically drops out for free.

The Replacement Evaluator is a pretty simple parser at a high level. It's just a bunch of lookups and replacements. I'm not too bothered by it.

That leaves the Statement Evaluator, which at least at first glance is just another implementation of the recursive descent parser. But I'm not wholly convinced of that. The big hang-up for me is that it is a change of value-types. The Expression Evaluator is numbers to numbers. The Boolean evaluator is bools to bools. But the Statement Evaluator is numbers to bools. Not sure how that will work. Will have to actually sit down and fiddle with it.

But yeah. That's where my week has gone (most of the week went into trying to solve the fucking Level Sequence Director Blueprint accessibility problem), and where it's going.

I am hoping to have the full conditional system implemented and working by next week.

That's all for now. Before I sign off, I want to say I really appreciate your guys' patience with me putting all this together. I know all this programming hoo-hah isn't that glamorous, and I'm sure my pausing the work on my animation projects to focus on building all this game-dev infrastructure is frustrating for a lot of people.

It means a lot to me that, despite all that, you guys and gals are sticking around and continuing to support my efforts. I really can't overstate how thankful I am to have such an awesome and understanding Patron base as all of you. Every day, I am acutely aware of just how lucky I am.

With that being said, I'm signing off for this week. See you all next Monday!