This is part 3 of a 4-part series, mirrored from the Gale Force Games page.
A quick foreword on "un-exporting" projects
The only other engine in ScummVM that really resembles mTropolis in terms of the development style and challenges is Director, and ScummVM's Director engine is able to take advantage of a tool called ProjectorRays that converts exported and protected Director movies back into a form that Director itself can load, including decompiling Lingo bytecode into source code.
Wouldn't something like that for mTropolis be useful too? Unfortunately, no, for multiple reasons:
- Running things in mTropolis is buggy. If you change the state of a scene object while running the game, the state can persist into the editor. The documentation even tells you to do this for list variables because there is no way to edit their contents in the editor for some reason. Scenes may be loaded when playing in the editor when they wouldn't be loaded in-game.
- Setting up test cases to determine its behavior doesn't usually require using the actual title.
- Plug-ins have editor-only files that aren't distributed with titles. For instance, here's the Resource directory of the demo:The files ending with ".ePP" provide editor functionality, only the ".rPP" and ".cPP" files are distributed with the built title. That means if a title is using a non-standard plug-in, the editor won't be able to use it, because the ".ePP" file for it is missing.
- mTropolis has a lot of non-standard plug-ins. Remember what I said earlier about it seeming like mFactory was using a business-to-business model? A lot of plug-ins have shown up in retail titles that seem like pre-release or customized versions of the standard ones. There is very little consistency.
- I made a debugger that is way more informative than the mTropolis editor anyway, which we'll get to very soon!
We can rebuild it. We have the technology.
In the last installment, I covered how MTDisasm came into being and ultimately was able to fully dump Obsidian's game data. At that point, making it work again was clearly an achievable goal, it was just going to take a lot of work to do it.
My initial plan was to convert MTDisasm into a library and use that as the loader for a new program called MTEmu (mUlator was a bit too on-the-nose). Unfortunately, while I had done a bunch of OS interface stuff for my Glider PRO port, very little of it except for the resource loader was going to be useful for Obsidian. Even the StuffIt unpack tool, ported from The Unarchiver, wasn't going to be too useful - the installer format used a different algorithm, so I'd have to port that too. I was basically going to be starting from 2D drawing boilerplate, and writing boilerplate sucks.
Trying to add it to ScummVM was somewhat under consideration, and it already had things I needed: QuickTime parsing and decoding, PE-COFF resource parsing (needed to get cursors out of DLLs), MIDI output, and all of the OS boilerplate. My biggest reservation though was the license. I am not a GPL fan, everything I do is MIT/Apache licensed if I can help it because I think reaching more platforms is more important than contributor reciprocation.
I ultimately decided to try doing it as a ScummVM addition anyway for three reasons:
- It was already in-demand for being added to ScummVM, and doing it as a separate project was probably going to result in it being brought in anyway with a bunch of duplicated work.
- I didn't think my reservations about the licensing were going to matter because I didn't
think anyone was actually going to untangle the IP rights mess needed to put it back into print. Surprise!
- I figured it was probably technically possible to do in some kind of arms-length way that the mTropolis player code could be easily hoisted out if I wanted to do it later. It was, in fact, coded that way.
Obsidian on console when?
I want to keep the possibility open, but bringing it to console would mean re-implementing the things mentioned above (basically going the MTEmu route) and paying for certification. It wouldn't be cheap or easy, and given the level of interest in it, I think it would be difficult to justify.
I didn't think it would be re-released in the first place though, so who knows? If the people with the rights to it want ink on a dotted line, they know where to find me.
Foundational work and design
ScummVM has a "create_engine" tool now, but at the time the mTropolis work was started, the recommendation was to clone the engine used for acclaimed 1993 Game of the Year Plumbers Don't Wear Ties, so that was used to get a simple foundation, but one immediate problem was that the initial plan of bringing MTDisasm in as a library was going to be a no-no with the project's code standards (which, among other things, prefers using its own integer types instead of stdint.h).
Because of this, the MTDisasm data object loaders were mostly brought in by hand. One thing I knew I wanted to do from the start was separate the loading of object data from instantiation of the objects, partly for cleanliness (though at the cost of duplication), partly to deal with disconnects between how data was stored on disk vs. how it should exist in a loaded object.
One of the first technical challenges was planning for linking up object references, which has 2 problematic aspects: The first problem was that if objects are loaded in some order, then it's possible that an object references another object that hasn't been loaded yet, so object link-up has to be its own phase. The second, and much bigger problem, was mTropolis's system of "aliases."
As briefly mentioned in an earlier part, a modifier in mTropolis may be converted into an "alias" that allows the modifier to be reused in multiple different objects. This was especially important in tandem with "behaviors," which are modifiers containing other modifiers. One thing that might not be apparent from that though is that the behaviors have to be cloned when they are brought into the scene as aliases. If a behavior includes a variable inside of the behavior, for instance, then each instance of that behavior has its own copy of the variable - and other things inside of the behavior reference that variable!
On top of that, if a variable is converted to an alias, then the aliased variable has another special behavior: Changes to any instance of the alias apply to all of them, making it effectively a variable reference.
Aliased variables were actually implemented via an incorrect assumption for quite a while. Initially, they were implemented via a somewhat complicated approach where the aliased variable modifier existed globally, but references to it could be put in multiple parts of the project. It later turned out that aliased variable modifiers are actually distinct objects - If you add them to the scene twice, each one has a different GUID. The engine was eventually changed to handle this correctly, cloning variable modifiers and making them reference a shared storage object instead.
As a legacy of that though, the mTropolis engine's loader has several steps to making an object exist:
- The definition of an object is is loaded from disk as a DataObject.
- An object corresponding to the DataObject's type (which may be an alias!) is created.
- The object is initialized by loading the data in the DataObject.
- The object is, at some point, added to the scene.
- After some operation that adds objects to the scene, the object is "materialized."
"Materializing" an object does 3 things:
- Assigns it a new runtime GUID
- Replaces any alias modifiers with clones of the global object referenced by the alias.
- Resolves any references inside of the object. In some cases, this means references with static GUIDs are resolved up the scene hierarchy. In other cases, it means that it detects that a reference is to an object that was cloned, and the reference is replaced with a reference to the clone.
Even now, it's not entirely clear how object references are supposed to work though. Muppet Treasure Island, another mTropolis title, had numerous problems with variables having duplicate names but different contents and behaving in a way where the correct behavior must have involved resolving the references to a different one than what the GUID pointed to, but in other cases, the one pointed to by the GUID was the correct one.
Investing in the development experience
Dealing with a complex thing like this is really hard if you can't actively see what's happening. The first line of defense is logging messages. Since mTropolis depends heavily on message-passing for logic, it's really important to be able to see what messages are being sent and where they are going. If you run the game with debug level 3 or higher (e.g. by putting "-d 3" in the command line), then ScummVM will log all message propagation to the console.
This is extremely important, but printing things to the console doesn't provide a lot of information about the state of the scene. To help with a lot of development problems at once, one feature that went in very early was the debug overlay. It was so high-priority that its position in the to-do list was right after being able to reach the first screen.
If you've paid attention while adding Obsidian to ScummVM (or if you go to the options by hitting Ctrl-F5 in the ZOOM/Steam version to exit to the launcher...), you may notice that there's an option marked "Start with debugger."
This launches the game with a debug overlay and a few buttons on the side, and a display that shows you the name of the active scene and active shared scene.
Unfortunately, the step-through debugger part was planned (and much of the internal architecture built around it), but never actually came to fruition because it turned out to not be the right tool for debugging most of the logic bugs that popped up.
Step-through debugging is useful if you want to be able to analyze the state of a program while it's in the middle of running scripts, for example, but most problems with game logic working properly were not due to scripts executing incorrectly, they were due to problems with messaging - sending messages in the wrong order, sending messages that weren't supposed to be sent, not sending messages that were supposed to be sent, and so on.
Debugging message problems on the other hand didn't benefit a lot from having a step-through debugger, it mostly depended on looking at the disassembled scene to figure out how it was supposed to work, comparing that with the message log to figure out it was actually doing instead, and then setting up test scenarios in mTropolis to see what messages actually get sent - and in what order - in a similar situation.
One example of how messaging went wrong is having to figure out the exact point where queued messages were discharged during a scene transition, something that was causing the wrong music to play in the Statue lower level.
The project viewer and inspector, however, were incredibly helpful:
These let you see all kinds of information in real-time: What is loaded, what the values of variables are set to, what the GUIDs of objects in the scene (which allows them to be cross-referenced with MTDisasm output), and any other information that's been exposed for that object.
There are also some toast notifications that pop up at the bottom:
Warnings are colored yellow and errors are colored red. You won't see many warnings these days, but the main source of warnings is that every modifier and element in the ScummVM mTropolis engine has a "support level," which is either unimplemented, partially-finished, or finished.
Entering a scene with a partially-finished or unimplemented element or modifier results in a warning notification. This is to make it clear that a scene with things in an unfinished state has been entered, and if something isn't working correctly, one of those unfinished things is likely to be the culprit!
(You may be wondering why there's a warning about text labels in a scene with no text labels. That's because Obsidian has its own debug overlay text label, but in the retail version, it's moved off-screen.)
Error popups mostly occur due to Miniscript errors. If you play Obsidian, you'll notice several of these. That's not actually a problem with ScummVM - the game has some scripts that do invalid things, and in all of the cases that I'm aware of, I've confirmed that the proper behavior is in fact to throw an error and stop the script.
Making this also ran into a bit of a problem with ScummVM's internal architecture, it has its own GUI system, which you can see in the in-game settings and launcher, and it's not a great GUI architecture but it's not really bad for what it does either, but it does have one problem: It assumes exclusive control until you leave it, which means it is completely unusable while the game is running. Dealing with that required rolling a new small UI kit to make the windows, scroll bars, hierarchy tree, etc. that you see in the debug overlay.
Having that UI kit also required everything else to be aware of it. For example, if the game is supposed to change your mouse cursor, it actually doesn't, it changes the cursor assignment of the main game "window" so that the mouse is visible, and not the game cursor, when it's over one of the debug overlays. Same goes for detecting mouse movement if the mouse isn't in the game window.
Of course, the big irony of all of this is that these debugging capabilities are considerably more advanced and informative than what mTropolis gives you, so I actually had way more information available to me than the developers of the game did!
I know your deepest, darkest secrets
One fun thing about being able to see everything in the game data is being able to see things that were either obscure or in one case, really not meant to be found at all, and find all kinds of new information about the game that was either not well-known or not known at all.
Realm names
The internal name of the bureau realm is "Labyrinth" and the name of the spider realm is "Abraxas."
Space bar skip
It turns out pressing the space bar skips most cinematics, except for important story cinematics, letting you get through the game much faster. Huh. Doing this causes some bugs sometimes though, like preventing the music from stopping when you beat the Bureau chapter.
I heard you like FMV games so I put an FMV game in your FMV game
You may have noticed that there is a help booth in the first part of the Bureau named "Sources" that shows a falling book, referencing the Myst intro, and if you click the screen, you get a crazed man telling you to bring him "the blue pages," another Myst reference. What you may not know is that all of the booths are reachable from the phone puzzle, and if you call the Sources booth, the phone is answered by Henry Stauf from The 7th Guest.
Speaking of the phone puzzle, I've looked at the code for it, and I still don't understand it. Each of the dials actually represents a positive or negative coordinate, and a coordinate is only valid if either all three are negative or positive (determined by which half of the slider it's on), but that means there are actually 2 valid coordinates for any point you want to reach.
Fireflies in the Bismuth junkyard
There are supposed to be fireflies in the Bismuth junkyard, but you probably won't see them on a normal playthrough because when you first land in it, you're actually in a duplicate of it that belongs to the previous chapter, and the duplicate doesn't have the fireflies. Normally, leaving that landing scene triggers a disk change, which puts you in the actual section. Try going back down and looking around!
The dirtiest dirty secret of them all
Not only was this one completely unknown previously, it was actually discovered by accident due to the unfinished modifier toast popup mentioned earlier. Viewing the zoetrope popped up a warning about an unimplemented "Path Motion Modifier," which was a bit strange because the wobbling frame was working just fine and nothing seemed to be unusual, let alone having problems because of a missing modifier that basically existed to move sprites around.
Looking at the scene in the debugger showed some interesting things.
The path motion modifier responds to a message named B206_Start_Drop. Drop what though? Further down there is a "Click Behavior" but it's actually not a click behavior, it has 5 key detection triggers and a counter that fires when all 5 are reached.
After looking through the logic, it turns out that if you type "Voxel" after beating the puzzle, then a head appears and the animated bird drops a turd on it.
To make matters worse, the path motion modifier is not used for anything
else in the game except for this, which meant I had to implement its
behavior just for this bird poop Easter egg that nobody even knew about.
 |
| You thought nobody would ever find out... |
Spaghetti no-code
No game can survive without reusable systems though, but making systems out of objects in a scene hierarchy is particularly weird.
Much of Obsidian's navigation system is handled by a list of "nodes," and you're usually on one of those nodes. The node lits is usually supplied by compound variable that has to be named "cSR" in the subsection.
This is a fairly common recurring pattern: Behaviors are placed in the hierarchy alongside variables that the behaviors have to consume.
A pretty gnarly piece of "what were they thinking?" here though is how those values get in the list in the first place. The editor lets you set the initial value of variables, so I bet you're thinking that you can edit lists in the editor right? Well you would be wrong, the actual way is that you have a script set the values that you want, run your project, and then when you go back into the editor, the values that you set while running it have persisted into the value. Intentionally leaking play state back into editor-persistent state as intended design? Whee!
Some other mTropolis users would later come up with their own patterns, like Muppet Treasure Island populating lists by broadcasting a message to part of the scene hierarchy and having all of the responders ping back. Why this was better than just giving the scripting language a "for" loop, I have no idea.
You're probably wondering then, if coding was so cumbersome in this, how did the library terminals work? Well, for one thing, the terminals are done via a combination of several different parts that handle specific functions, from text output to actually updating things. It's basically a finite state machine, so most of the logic behavior simply handles all possible states.
The game also includes a custom plug-in called "RSGKit" that includes a few modifiers used to do more complex tasks, like string manipulation (which Miniscript has no built-in support for) and creating WordMixer answers.
Speaking of WordMixer, the dictionary data is also completely stored in the RSGKit plug-in. The ScummVM mTropolis engine has its own internal plug-in system to implement these modifiers, including parsing out the dictionary data from hard-coded offsets in the DLLs, and the dictionaries are used for both WordMixer and the filing cabinets. Naturally, getting the word list exactly correct (including short words) is mandatory, since some important files are referenced by number in the scripts.
Trouble is on the menu
Sometimes, the way the game and its logic system work create problems that clash with how ScummVM is supposed to work in ways that are really hard to do anything about. A big one is the menus.
If you go to the game options in ScummVM and change the sound and music volume, you might notice that it... doesn't really work.
That's because the game logic manages the sound levels, and it kind of has to because there is no global volume, every sound emitter has its own volume and the game logic has to update them itself.
You might be thinking "well so what, just scale the volume of sounds with the SFX level and the MIDI music with the music level!" That's easier said than done. The volume levels are stored in the save files (by the game logic), and syncing them up with ScummVM's settings requires manually hooking into the game logic. Additionally, the sound volume system used in Obsidian was designed to evaluate volume levels when entering a scene - not while you're already in the scene! But one quirk you may not be expecting is that MIDI was used for some non-music sounds, like the keycard sound in the maze, and (surprise!) the vidbot screen on/off sound. (Remember that last one for the next installment!)
Similar problems when trying to load from the in-game menu:
Fortunately, I was able to implement saving from the menus for Obsidian specifically, since whether or not you can press the Esc button to go to the menu is controlled by a single boolean variable, and you can save from anywhere that the menu is available.
Even supporting a menu is more complicated than it may seem. The game has to be able to support transitioning out of and then back into just about any scene, which is a problem if there is game state from puzzles persisting through scenes. That's probably why the piazza puzzle has a blocker that prevents you from saving anywhere in the main puzzle area, and there is at least one bug where reloading the game in a particular scene skips the puzzle (which is not a ScummVM bug - it was present in the retail version).
Room to grow
I think this has covered most of the process involved in getting the ScummVM mTropolis engine into a working state. The next and final installment of this series will cover how it went beyond the original, with better MIDI support, widescreen mode, subtitles, improved color depth, and a tiny little quirk to get the save screenshots right! It'll also have some final thoughts about wrapping this project up, and the future of the mTropolis engine in ScummVM.
No comments:
Post a Comment