Creative AI and Baldur's Gate Modding

So, now that you know all about my modding history and relationship with BG, we can hop back to this last week.

Motivated by my now seedling idea of "What if an AI designed a mod", I began by having some brainstorming chats (triggered by my idea of "what can AI do for me", as well as just wanting to check it out) with text-davinci-003 under the openAi playground (I noticed the chat.openai.com was too restricting/self-censoring after gaining some popularity). This helped me to build up 5 characters for the mod idea, a central story line, and even some statistics for in game items (AD&D 2nd edition was the intent - I wasn't specific enough and they came out a little more like 5th edition, which I'm unfamiliar with).

You can see the thread where I shared this intent with the g3 community here: https://www.gibberlings3.net/forums/topic/36224-the-first-ai-designed-mod-the-heart-of-baldurs-gate/ and I'll share a brief excerpt of the conversation inline (the g3 link has a link to the sprunge containing the full data dump).

Human: I want you to design a new mod for gibberlings3, can you come up with an outline for a new Baldur's Gate mod?

AI: Absolutely! Here's a rough outline for a new Baldur's Gate mod:
1. Establish a detailed story with interesting characters and
   interesting locations.
2. Develop a complete set of new quests and objectives specifically designed for the mod.
3. Create a believable environment, with randomly generated weather and realistic NPCs.
4. Introduce new items, armors, weapons, and spells to make the game more balanced and realistic.
5. Design an original user interface and menu system to help make the game easier to use.
6. Implement a balanced and enjoyable difficulty level that is suitable for all types of players.
7. Test the mod thoroughly and provide comprehensive bug fixes before its release.


Human: Can you give me the storyline of the quest?

AI: Sure! The
storyline of the quest would revolve around a powerful magical
artifact, hidden away in the deep dungeons of Baldur's Gate. The
player's task is to locate and recover the artifact, which will
require them to overcome various obstacles and enemies along the
way. On the journey, they may be joined by allies who can help them
battle particularly difficult foes, or provide crucial knowledge and
wisdom to help them make informed decisions. Ultimately, the player
will be responsible for protecting the artifact from those who seek to
misuse its power and bring peace to Baldur's Gate.


Human: What should I call the artifact?

AI: The artifact could be
called "The Heart of Baldur's Gate".

Human: Tell me about one of the friends, what would their name be, and
what is their biography?

AI: Your friend's name could be Athar, a
curious and helpful warrior of Baldur's Gate who was born into a noble
family. He is knowledgeable and wise, with a great affinity for the
ancient ways of magic and combat. He joined your quest to obtain the
Heart of Baldur's Gate, driven by a sense of adventure and a desire to
protect his home from those who seek to use its power for
ill. Together, you will overcome the dangers of your journey and
ensure the safety of Baldur's Gate.


Human: What class and alignment would Athar be?

AI: Athar would be a
Paladin of Lawful Good alignment. He would be skilled in the ways of
melee combat and have knowledge of ancient magical arts.


Human: Describe the other friend that could come along, what is their
name and biography?

AI: The other friend could be Anari, a wild elf
from the woods of Baldur's Gate. She is a master archer and swordsman,
as well as a powerful spellcaster. Anari is brave and fiercely loyal
to her friends and allies, and will do whatever it takes to protect
them from harm. She too has joined your quest to discover the Heart of
Baldur's Gate, and together you will face the many obstacles and
challenges that await.

Human: Are there any other heroes who wish to join the protagonist on
this adventure in addition to Anari and Athar?

AI: Yes, the
protagonist could be joined by a few additional allies on their
journey. These may include a dwarf berserker, a tiefling rogue, and a
mysterious cleric. Each of these characters would bring their own
unique skills to the team, helping them to overcome challenges and
ultimately succeed in their quest.


Human: Tell me more about the dwarf berserker, what is their history
and name?

AI: The dwarf berserker is named Olrun. He is a fierce
warrior, deeply devoted to the gods of his people, and driven by a
need to prove his strength in battle. Olrun has a penchant for
charging straight into danger, often without much thought or regard
for his own safety. However, his courage and bravery make him an
invaluable ally in difficult situations.


Human: Tell me about the tiefling rogue, what is their name and
history?

AI: The tiefling rogue is named Zariel. She is a cunning and
skilled thief, who grew up on the streets of Baldur's Gate. She is
incredibly agile and stealthy, able to slip through almost any
situation undetected. Zariel is driven by a need to prove her worth,
and is willing to do whatever it takes to get the job done.


Human: What about the mysterious cleric? What is their name?  Why are
they mysterious?

AI: The mysterious cleric is known only as The
Voice. Little is known about them, except that they have a great and
powerful connection to the gods. They are enigmatic and wise, offering
guidance and advice to those who seek it. It is said that The Voice
can be found in the darkest corners of Baldur's Gate, where they
protect the city from its enemies.


Human: Give me some sample quotes of what Anari might say to the protagonist
AI:

"We mustn't give up now—we're so close to achieving our goal!"
"Let's work together, and I'm sure we can make it through this challenge!"
"No matter what obstacles stand in our way, we will prevail!"
"We must stay alert and remain vigilant against those who would seek to use the Heart for ill."
"Let us bravely march forward and protect Baldur's Gate!"

This really got my creative juices going. All of a sudden, I began to understand the attachment some mod authors have to their characters (previously I couldn't empathize, thinking of their creations as nothing more than my standard FOSS utility functions/libraries I would write) despite the initial prompts and concepts being produced by the AI (note: I used this as a kick off point - I am taking them in a guided direction from this, but with much of my own interpretation now).

Now I needed to visualize these friends.

So, I have these characters that are now becoming near and dear to me, yet I need to actually see them. For this, I tried my refreshed 15 dall-e credits - it wasn't terrible, but it didn't get me very far. They ended up looking quite similar to the generations I did a month ago (https://ahungry.com/img/ahungry-portraits/) - an Icewind Dale 2 flavor, but not really what I was looking for - I needed something better, and I was out of Dall-E credits.

I had heard of MidJourney in passing, and decided to give it a go. Turned out signing up was quite easy, and the main interface is via a Discord bot, where you type "/imagine <whatever prompt info here>".

After a couple hours, I had the initial designs, but they were a bit too realistic (BG art is not this realistic):

It was soon after this that I had exhausted my free MidJourney credits (had I been focusing on one character, I could have got it done in the free credits, but I was focusing on 5 new characters and 3 villains).

I took a look at their plans and saw for $30/month, I could get 15 hours of image generation (the free minutes were like half an hour or so). Since the tech is exciting, and this is still much cheaper than commissioned artwork (and less time intensive than drawing it on my own) I went ahead and subscribed, and began to resume my efforts.

With some more prompt tuning, I had what I felt was an acceptable result to work with:

The end result of all the characters:

Having the character's personalities, as well as their visual portrayal worked out, my excitement to glue it all together as a mod was blooming.

However, one thing that often sets a mod apart (or any media, game, video etc.) is having a high quality (or at least, acceptable quality) audio score.

How would I fulfill my needs for audio? Each character needed around 50 voiced lines for the basic in game interaction (clicking on them, commanding them to act etc.), and my own skill at voice acting (even with ffmpeg type filters/modification) is far from adequate - especially considering the roster has such a range of voices required, and I didn't want it to verbally assault the mod user's ears.

This was one where I hadn't heard much in the online space - I knew of AWS/Google/IBM offerings for TTS systems, but they are more integration oriented, for real time support/TTS (calling a customer service line, getting a robot that may fool some less than astute callers into believing it's a real person etc.).

I couldn't even find a basic trial or pricing structure to just run some test lines.

Searching a bit, I found one under a moniker that rhymed with "turf ai", and I'm still not sure if they're legitimate or not, but before trusting something, it's always good to see if "real" people are discussing it, by running a search similar to:

site:reddit.com your phrase here

to get results (via duckduckgo or google) that are limited to the domain following "site:" (in this case reddit).

Unfortunately for "turf"'s potential business, all the results were SEO spam under less than reputable looking subreddits, so I went looking further and eventually came across a reddit post mentioning "Replica Studios".

I found their webite (to my chagrin, they have a notice that their web UI is being deprecated in favor for Windows/Mac binaries - nothing mentioning future GNU/Linux support) and began testing some voices. It turns out they were much better than I had hoped, and I was able to produce all the soundsets (300+ audio clips of ~1 to 10 seconds in length) in the free 30m trial (I'm still trying to get a response from their Discord, if they plan to support GNU/Linux, or leave the web UI up, as I don't want to pay them $20/month and end up being stuck with an unusable tool, if they were to decomm the web UI).

In fact, we can compare my original voice over with one produced by the tool, and it'll be easy to tell this was a worthwhile endeavor.

My voice over (pitch shifted deeper with ffmpeg) for Olrun:

The AI voice of Olrun:

My voice over (pitch shifted higher with ffmpeg) for Anari:

The AI voice of Anari:

and a bonus to show the range (and that she's one of my favorite characters being developed)

The AI voice for Zariel:

So, not only is the voice acting work much simpler (copy/paste lines of dialogue, vs me trying to record it and stress my throat attempting to do a range, and then apply tweaks via audio software).

It's also much cheaper - from talking with the community, some other voice acted mods spent around $100 to get one character voiced - to shell out close to a thousand of my own dollars for a non-commercial community mod that is given out freely is just not an option - but paying $20/month for AI VA is certainly in the realm of "fun cash"/"expendable money" (I ended up using 18 of 30 free minutes - Replica Studios offers a monthly plan for 2 hours of credits for $20).

So, I had almost all the resources necessary to really start glueing things together, yet the best mods I've seen also include a "theme song" of sorts for dialogue that is unvoiced, while the player reads the text.

Initially, I pulled out some old audio files I created using RoseGarden (a software that allows music generation similar to midi files of old, by placing notes and assigning instruments to "play" them in those notes). I had some unique tunes, however I've never had a musical background, and can't quite read sheet music (hmm, perhaps a new goal to focus on?), so I am certainly no composer.

However, trying to find an AI medium to create dynamic music (I know some exist that can produce pretty trippy AI generated videos, with scenes that transform before the eyes) was beyond my interest - I just wanted some CC-BY-0 / Royalty Free songs, and luckily I came across some on SoundCloud. This built out all the necessary resources I required (I did have to tweak a few with ffmpeg filters, as a lot are very recognizable as commonly used YouTube BGM).

After all the components were ready, I got to work putting them together, yet I had a few research tasks to perform - I had no idea how to build a weidu dialogue file from scratch (let alone 5 of them).

I began digging around, and found a nice beginner guide on the SHS Forums (some BG forums that were most popular circa 2010). You can find that here (http://www.shsforums.net/topic/7219-weidu-how-to-create-a-simple-npc/).

This got me to the point of basic understanding, however I hit some snags in following the instruction, as I was trying to develop the mod against (here comes parts only for the BG modding community to understand) an existing EET install. The "script" referred to as the "character DeathVar" kept being overwritten by MULTIG, a generic multi-player dialogue file, instead of my custom written one.

Some g3 discord members tried to help out with advice (which led me to a wonderful interactive mod, with the intent of teaching others how to build a mod - the Branwen mod: https://github.com/Pocket-Plane-Group/Branwen_for_BGII), but the problem persisted.

It turns out that the base game looks into override/pdialog.2da (sort of like a tab separated values file, but any whitespace will work) as a lookup table - it'll find the character entry, in this case "Zariel", with the game asset ID of "uxzar", and from that row, read the dialog file value. BUT the big gotcha here is, in an EET install, there are a multitude of other files used in this process (all in override, bddialog.2da, 25dialog.2da, and bgdialog.2da). The EET tool has a sub-install called "EET_End", that's supposed to run at the very end of your mod setup - it'll propagate the old style entries in pdialog.2da into the new tables (this problem also occurs with in game banters run off the banter engine - in addition to interdia.2da, you need to populate bgbanter.2da, 25banter.2da, and bdbanter.2da).

Now, the astute reader may just ask, "Ok, so why don't you run EET_End after installing your mod?" - to which I would retort, "My mod installs in about 10 seconds, EET_End takes 5 to 10 minutes, and when I'm learning a new programming language or technology, having the time between cause (changing the code) and effect (seeing the code in action) needs to be as short as possible, and 10 seconds is much preferable to 10 minutes".

To automate the solution to this problem, I took my first major divergence from how the Branwen tutorial suggests handling code - I broke off some common segments into a "util.tpa" file, so I could write my own custom functions, including one that populates these tables conditionally based on the environment, and does so for any given character/resource based on parameters (vs duplicating the snippet 5x - not that the Branwen mod did that, it was only for one character, and it also has conditional application, but only between BG2 and BG2+TOB expansion, with a final block for handling some EET things).

Next up was hammering out "brute force" style the following core concepts I wanted in my mod:

• Player Initiated Dialogues (pids)
• NPC to NPC Banters
• NPC to Player Banters
• Custom items
• Custom characters (binding the audio, portraits, game stats, class etc.)
• Some rudimentary custom scripting (Zariel falls asleep, the player can wake her up - I haven't really noticed an event like this in the core game and it reminded me of waking up the warcraft peon)
• Using "best practices" and trying to "traify" (writing weidu dialogue files with string references removed in favor of general ids, so translators can produce alternate "tras" (translations) in the future)

I noticed quite soon in the week that for each minor thing I was doing from above, I was having to propagate it via copy/paste 4 additional times (one for each npc). There are not a lot of meta-programming facilities, and while weidu is very featureful, some things are often done via copy/paste programming, which I'm not a fan of).

Having those things in place, I took a little break to try to make a video of the progress to share with the world mid-week (note - the audio quality is very poor, that was due to my recording setup - I had pulseaudio off, so no sound multiplexing, and the video recorded over ALSA using my computer laptop's default mic - causing an echo and some audible key presses).

Still, the reception was pretty good, and the product was looking quite nice!

Now, I needed to take a step back, and analyze the groundwork - if I plan to extend this to have hundreds of lines of dialogue and a compelling story/character building, I can't spend most my time deciphering obscure code and confusing file layouts.

One neat thing I found is that the .acm files in the music/ directory do not need to be converted into true InterplayACM files (of which there are almost no converters online - ffmpeg/mpv support decoding, but not encoding) other than a very old SND2ACM.exe file from some Russian Fallout forums/message boards. The software didn't quite work - it changes the sample rate, so requires a lot of finnagling, and most audio produced by it would crash due to decoding errors when trying to replay it via ffmpeg after conversion to ACM. On a hunch (having noticed/been told that .ogg files can work if renamed to .wav files, for other BG:EE audio) I thought, "Why not see if this works for acm as well?" - well, it did!

So, now it's quite easy to add new background songs - I have about 5 to 6 new ones being added with the mod, which I think is pretty unique (this is the audio you can call with PlaySong(123), and will run in tandem with any audio that is triggered via bracket calls in the tra files). I'm not sure if I've seen any mods in the past using it at all, likely due to the proprietary acm format, and the lack of mods that would need it.

If you notice the snippet above, I chose Janet, a small/minimal Clojure-like lisp - I wanted to flex my Nim chops, but trying to learn a new language, and get a task done, while a good learning opportunity, can be a bit "too much" all at once. I did get a quick and dirty prototype working with it, but lisp lends itself SO well to structuring code as data, while I was constantly refactoring my Nim code, and updating many different type definitions.

I really like Nim and would recommend it for apps that you want to have a small footprint, in particular ones that you plan to collaborate on others with, but to just whip out a working prototype, it's had to compete with lisps and their REPL features.

The Nim DSL looked like this:

next2(
  Say(
    cond: @[global("ux_prelude_done", 1), not_global("ux_another_check", 3)],
    text: "Hello",
    replies: @[
      Reply(
        text: "Can you help me?",
        next: Say(
          text: "Thank you!",
          replies: @[
            Reply(text: "I was am glad to help!", next: Say(text: "Goodbye!"))
          ]
      )),
      Reply(
        text: "I don't need your help",
        next: Say(text: "Well, thanks anyways...")),
    ]
  )
)

Which is a bit wordier than the aforementioned lisp.