Progress Report - November 20, 2023 (Patreon)

Hello, everyone!

Finishing Up the Conditional Evaluator

Last week, I regaled you all with the fun and exciting journey that has been my effort to automate the conversation system for my Unreal Engine prototype, which naturally led into the effort of building an entire arbitrary-complexity Boolean interpreter, for the sake of dynamically linking conditional dialogue trees.

That Progress Report ended on my having about two-thirds of the system done, with my major concern being how to convert the numerical statements into Boolean conditions, and with aspirations for having the conditional dialogue system figured out and implemented by the next Progress Report.

Well, it is now the next Progress Report, and I am very excited to announce that I managed to achieve that goal!

Ladies and gentlemen, this is what the past 2ish weeks of work all led to. All that work, all that testing, all that headache and heartache. All for this one little function:

The exact mechanism of the translation from numerical statements to Boolean conditions is rather messy and complicated, but the heart behind the idea is pretty straightforward:

1.) Simply keep track of what characters you've seen in the string, while keeping count of how many unmatched parentheses you've seen, +1 for an open paren, -1 for a closed paren.

2.) As soon as you find a Boolean operator, such as "AND", skip over the unmatched parentheses, evaluate what you've seen so far, and then stick the result into the original string as a replacement for what you just evaluated.

3.) Reset your trackers and keep marching along, evaluating and replacing mathematical expressions as you find them.

By the end, you've replaced all the mathematical expressions with Boolean values, and you have a full Boolean expression ready to be evaluated.

The devil, of course, is in the details. And in this case, specifically the "skip over unmatched parentheses". It turns out, that is a non-trivial ask. Especially when you introduce the idea of negative skipping.

We all know what it means when we're told to skip over 3 parentheses. You just read from left to right, keeping track of how far along the string you are, counting how many parentheses you see. Once you see your third parenthesis, you simply return the string starting from where you are now to the end. Simple.

But what does it mean to skip negative 3 parentheses? Well, thinking back to the idea described above for our translation mechanism, remember that we reset our counter every time we evaluate an expression. Which means, by the time we get to the end of the statement, where all the closing parentheses are, we are counting parentheses that were encapsulating expressions we've already evaluated (but lost track of). Meaning our count goes negative.

As much as I love the phrase "I leave this as an exercise for the reader", I'll do this homework assignment for you: skipping -3 parentheses, in this case, means we have to skip the last 3 parentheses. Simple in theory.

But in practice?

I will spare you all the gory details. I'll just leave this comment I left on the function, after much blood, sweat, and swears were left behind. Your imaginations can fill in the gaps.

Thankfully, that was the last step in this leg of the journey. Once I had the Statement to Boolean converter done, I had all of the pieces necessary. The final function, Evaluate Gameplay Condition, has 4 major steps to it: It finds and replaces all of the variable names with their values; it attempts to immediately evaluate the condition as a Boolean and quickly return if that succeeds (most all conditions in a given conversation will simply be "true", meaning "always show this dialogue"); it converts the condition from Statements to Booleans (the function we just implemented); and finally it evaluates the converted condition as Boolean and returns the final value.

All of this work, for one measly cutesy little function, all to allow me to arbitrarily set and change conditions on the fly without having to dig into the guts of the system and manually rewire everything.

It was a colossal amount of work, but I think it will pay for itself with dividends the first time I sit down and write genuine dialogue trees with gusto.

Eventually I need to go back and add robust error-catching to it all. Right now it just assumes a perfectly-formatted condition string and will happily go off the rails in a colossal trainwreck without a care in the world, excitedly reporting "false" because whatever garbled mess its internal state became is an incomprehensible cosmic horror that isn't the word "true".

Similarly, eventually I need to add some tools for me to quickly make sure the expressions are all formatted correctly. Take at look at this hell of a string and take a guess as to how many times it took me to get all the parentheses right. If your guess is under 20, try again.

(((((1 + 1) + 1) > ((1 + 1) - 1) AND ((1 - 1) + 1) < ((1 + 1) + 1)) OR ((1 + 1) + 1) < (1 + (1 - 1))) XOR (((1 + (1+ 1)) + 1) != ((1 + (1 + (1 + 1))))))

That is an extreme example, but is a demonstration of the arbitrary complexity I wanted this evaluator to be able to handle. In practice, if I ever write a dialogue condition that is this convoluted, I am going to beat myself with a tire iron.

But, all of that is eventually. Right now, it all works. Minus the fact that the Boolean NOT operator isn't supported yet, but I'll get to that... later. I don't need it right now, and can get around it by rewriting expressions.

With all the dust settled, this is what a conversation looks like, in its entirety.

Each segment of the conversation is its own entry in this data table, and the entry keeps track of all the possible directions it can go. Dialogue -> Dialogue keeps track of the possible next-dialogue sequences, and what the conditions needed to navigate there are. And Dialogue -> Choice similarly keeps track of all the possible dialogue choices, the conditions needed to reveal them to the player, and what direction key they are associated with.

The actual logic within any given conversation piece is literally this simple:

The "Play Next Dialogue Sequence" function manages all of the logic of looking up the current dialogue sequence in the data table above, determining if there are any choices to show and if so what ones, and if there are no choices then picking the first dialogue sequence whose conditions are met and playing it. If there are no valid choices to show, and no valid dialogue sequence to go, then it concludes the conversation and tidies everything up, returning control to the player.

The entire mechanism of the dialogue tree is fully automated, with no input from me the user beyond its original design and then running the tools to automatically import, generate, and link all of the assets and data files. All that's left for me to do then is the actual artistry: setting up camera angles, animating characters, lighting the scene. All the fun stuff.

Leave the boring computations to the computer. That is what it's for. Let the artist focus only on the art.

Refactoring the State Manager

Beyond getting the conditional evaluator finally all figured out and working (sans the NOT operator, anyways), I also took some time refactoring the state manager.

Before this refactor, I had a whole bunch of various ambiguous switches that all controlled various different things. For example, the code for pressing E to interact on things would check if you're in a conversation, if you're in a cutscene, or if you're currently interacting with something else - all of which would result in you not being able to interact. Similar logic was done for all sorts of things, like the WASD keys (which are also used for interacting and dialogue choices) and even just looking around.

This was a really convoluted system, and every time I needed to add a choice, it complicated the logic further and further. The entire thing just devolved into spaghetti in no time flat.

So I took a step back, stripped out all of those old checks and ambiguous states that each controlled way too much, and instead sorted the player out into various different explicit modes. I hope each of these are rather self-explanatory:

Of course, the trade-off now is that, instead of these things setting a single state and calling it day, they now have to explicitly set each individual mode they want to manipulate. For example, entering a conversation can no longer just flip the "Is In Dialogue" state - it now has to explicitly flip off the Can Move, Can Look, Can See Object Names, Can See Highlights, Can Interact, and Can See Character modes. And then leaving a dialogue has to explicitly flip those states on.

No free rides, but I would rather have a long chain of explicit statements that I can easily read and figure out what it's doing, rather than a single ambiguous state change that tells me nothing about what it actually does.

Just by moving all of my existing systems over to using this mode system, I actually solved a lot of insidious bugs that had snuck their way into the prototype, entirely for free. For example, it used to be that when a conversation was over, it just... broke. The player didn't regain control of the character, they couldn't move or look, their character model didn't even draw - but they could see highlights and interact with things again!

I'm not sure what exactly the problem there was, other than some obscure interaction of the various different states and the checks involved. But the switch over to using these explicit mods perfectly resolved it. Conversations end, the player regains full control and continue along their merry way, as they should.

There was one tiny hitch I ran across though, which was that the player's camera would snap to the inside of the character's pelvis, rather than properly hanging over their shoulder. I'm not sure what exactly caused it, but I was able to diagnose the conditions it happened, and a way to fix it.

For some reason, jumping directly from playing one Level Sequence to another Level Sequence would cause it. The high-five test animation for example didn't have any issues, because it was just a single Level Sequence. But conversations are a series of several Level Sequences all tied together, with that complex conditional logic I spent so much time on now.

I was able to verify this by intentionally breaking the conversations. If I played the Start sequence by itself, the camera returns fine. If I play the Comment sequence by itself, the camera returns fine. But if I play the Start sequence and then the Comment sequence, the camera snaps to the character's hips.

Again, I don't know why. Thankfully, the solution was simple as using a Set View Target Blend node to move the player's camera from itself... to itself. Yeah. I don't know why that works, but it does, and  I'm not going to complain about it.

Finally Moving On With the Prototype Plan

With all of that said and done, I am finally ready to move on to the prototype plan I laid out in the October 31 Progress Report. For posterity, this is that plan:

Right now, I think my next plans for my UE5 experiments are to figure out building custom levels (complete with lighting and such), transitioning between levels, managing global state (such as keeping track of decisions made, inventory items, equipped outfit, all that fun stuff), saving and loading (kinda import in a game innit?), and then finally building a full conversation system.

At some point I should also finally give Chloe her hair and outfit. Might as well make sure the new generalized outfit system described above works correctly for her, too. Don't see why it shouldn't. Famous last words.

Once I have all the systems built, I intend to build a tiny prototype that brings everything together in an actual meaningful mechanism. I am thinking it will be a simple motel room with Chloe smoking while laying on the bed. You can interact with a dresser to open and closer a drawer, and inside the drawer will be 3 different outfit bundles you can use to change clothes. There will be a few knicknacks in the room you can look at it, maybe a window to open and close as described above, and a door you can exit into the hallway through. The hallway will be mysteriously segmented off with no connecting lobby because that's quickly getting outside the scope of what I want to do.

After that prototype is done, I think that will conclude my UE5 experiments. At that point, I will return to working on Overbreed, and we will revisit the UE5 stuff once the Frozen project is waiting for audio, and again once it's pushed out the door and called done, meaning the on-stream project can officially focus on making a UE5 game.

Of this plan layout, I have the global state manager built (it's what is used in the "resolving variable names in conditions" part of the Gameplay Condition Evaluator function), and, obviously, the dialogue system.

You'll notice that the dialogue system was labeled as "finally", and then I immediately went and did it. What can I say, I like to tackle the interesting challenges first. And fuck me if that dialogue system wasn't a 2+ week "interesting" challenge.

So then, my next goal is building out custom levels with lighting and figuring out a level transition system. After that I'll work on a saving and loading system.

And then I think it's time to build the actual prototype, which brings it all together in a serious (if trivial) way.

I do want to note that there will likely be a slight deviation to that plan though, and that is regarding animations. One of my original aspirations for animating in UE5 is the ability to leverage animation layers and blend animations - of having core character animation on one layer, lip-sync on another layer, emotive animation (happy, sad, angry, horny, etc) on a third layer, and detail animation like breathing and natural character sway on a fourth layer.

And from my limited experiments, those are all available... if you use a Sequence animation in a Level Sequence. I don't want to get too into the weeds here, but basically in UE5 you have two main types of animations: Control Rig animations, and Sequence animations. Control Rig animations allow you to actively animate the character in a frame-by-frame basis, it's the actual building of the animation. Sequence animations are animations you can only play back, but you can't edit.

If you're familiar with Source Filmmaker animations, then Control Rig animations are the animations you build in the Graph Editor; and Sequence animations are the animations you import onto a model with Import Sequence or Import Animation.

Now, in theory this isn't that big of a deal, because UE5 has a straightforward way to convert a Control Rig animation to a Sequence animation. The standing idle animations you see Chloe playing in all of my prototype footage are Control Rig animations that I converted into Sequence animations.

So in theory, I could build the core animation in Control Rig, convert it to a Sequence animation, hide the Control Rig character, bring in a fresh copy of the character without the Control Rig, load the Sequence animation onto the fresh copy, and then layer the other animations on top of that.

That's a clunky solution though. I would like, for example, build the lip-sync animation first, and then animate the Control Rig around the lip-sync. This solution would require me to build the Control Rig animation first, and then apply the lip-sync on top of it.

So I need to spend some time exploring that, and seeing what the best solution forward is.

But that's the only real complication from the current projected plan that I see.

I'd also like to look into the best ways to build outdoor environments in Unreal. They're a complete cunt to make in Source, and I've seen some absolutely beautiful exterior scenes in Unreal. I am hoping there are some tools similar to Hammer's displacement tool that will make it relatively painless to build exterior environments.

That's technically a detour from the level-building, but I don't intend to spend long on it. Just a bit of googling and fiddling around, really.

By next week's Progress Report, I am hoping to have figured out building new levels and getting level transitions between them, and hopefully even saving and loading state as well. I want to be on the cusp of building that formal prototype by next Monday.

Until then, everyone!