This is part 1 of a 4 part series, mirrored from the Gale Force Games page.
I'm putting this together to tell the weird story and technical details of how Obsidian was brought back to life and added to ScummVM. I didn't originally want to do this because I think dev blogs are more interesting when they show the state of the art and help people understand how to solve similar problems, and if there are two things that stand out about porting Obsidian, it's how unusual of a project it was, and how much luck was involved in it being as easy as it turned out to be, and possible at all.
Maybe that weirdness will be an interesting story on its own? Who knows.
A little bit of history (and context)
In 1993, Cyan released Myst, a 3D-rendered surreal point-and-click adventure game heavy on puzzles and atmosphere, it was a massive success. Numerous other games attempted to follow in its footsteps, and most were... not very good. In 1997, Rocket Science Games released an ambitious, high-budget game with a similar gameplay concept and fantastic visuals, set in a series of surreal dream worlds full of creative and similarly-surreal puzzles. Unfortunately, for various reasons, the game didn't sell well, and RSG was forced to close its doors.
With Obsidian relying on QuickTime, the seeds of its demise as a playable piece of software were sown. Apple built an entirely new operating system, leaving the Mac version unplayable when the emulation environment for their old operating system was discontinued. They also discontinued QuickTime for Windows, already a notoriously finicky piece of software, and future updates to Windows made it quite difficult to run.
Fast-forward 25 years later to August 2022, and support for Obsidian was added to ScummVM. Barely over a year later, the rights to the game were acquired by Jordan Freeman Group and it was officially put back into print on ZOOM Platform and Steam.
A quick history of the state of the art of 1990's multimedia
In 1984, Apple released the Macintosh computer, which kicked off the foundation of the user interface used by desktop computers to this day, with later models incorporating high-resolution displays and high-fidelity waveform sound output. It was this design, combined with the release of QuickTime, Apple's video authoring and playback system, that kicked off an era of "multimedia" experiences.
With the release of Windows 3.1 in 1992 and especially Windows 95 in 1995, Windows had largely caught up to Macs in multimedia capability, but much of the multimedia authoring ecosystem was still on Mac, with popular tools such as HyperCard, Authorware, SuperCard, and Director. HyperCard and SuperCard never got Windows support, and Director didn't get it until its fourth major version.
mTropolis entered the market in 1995 with a Mac-only authoring environment and runtime support on Mac and Windows. Unlike Director, a project exported from mTropolis couldn't be packaged into a stand-alone application, it had to be bundled with a player program. This also came with a major caveat (which we will come back to, because it's really important!): Projects exported for Mac and Windows were in different formats.
The life of a program in a multimedia operating system
Before we get into the format differences, I need to explain the nature of the operating systems in this environment. Another way that Macs were a major departure from DOS, and much closer to how things work today, was how application development for it worked. Applications were developed against a large suite of functions provided by the operating system, unlike DOS where your application was mostly shoving information directly into the hardware.
One of Apple's big innovations at the time was that a huge amount of multimedia capability was standard and provided by the operating system. An API called QuickDraw handled drawing, Sound Manager handled sound output, and later QuickTime handled video and audio decompression and playback. It also had its own image format (PICT) that interfaced with so many of its capabilities that it's difficult to open on Windows to this day.
With Windows, Microsoft sought to have similar facilities. In particular, an API called GDI existed to provide much of the functionality that QuickDraw provided on Mac. Naturally, both APIs had different memory structures and formats that they used to provide their data to their drawing systems.
The life of a big number that lives on a circuit
Another concept that will quickly become important here is byte ordering. All numbers on computers are stored as a sequence of binary data, usually chopped up into 8-bit numbers or bytes. Most numbers are either 8-bit, 16-bit, 32-bit, or 64-bit in size. I'll be discussing most numbers in hexadecimal here.
When a number larger than a byte is stored, there is an important question to ask: What order do you store the bytes in? Say you have decimal number 123456789, that is 75bcd15 in hexadecimal. There are two ways of storing this: Least-significant byte first ("little endian"), which produces the byte sequence 15 cd 5b 07, or most-significant byte first ("big endian"), which produces the sequence 07 5b cd 15. Why would you use one versus the other? That's a question for the electrial engineers, but Macs were originally developed for the Motorola 68000 CPU series, which used MSB-first ordering when loading large values from memory, while DOS and Windows were developed for Intel 8086 derivatives which used LSB-first ordering.
How does one make a cross-platform multimedia framework anyway?
Combining all of these things, we start to see one of the timeless dilemmas of portability: A lot of multimedia stuff is being provided by the operating system, but the operating system is determining the format that the data has to be in to use it. How do you deal with this? There are two strategies: Use a common format that you can convert into the format that the operating system expects at runtime, or export to the format that the target operating system expects.
There is a third option, which is to load the OS-native format of one operating system on the other operating system. This turns out to be a pretty bad option for Mac-based authoring for two reasons: First, many of the memory and data formats that have first-class citizenship on MacOS were thinly-documented, and second, Apple liked to have complicated (but capable) file formats, often integrating with other pieces of their multimedia infrastructure, making supporting them robustly on other platforms difficult. (Windows, by comparison, tended to use fairly simple formats.)
One thing that helped with this process though (but ultimately consigned Obsidian to bit-rot) was that Apple had ported QuickTime to Windows. While Microsoft had come up with their own competing technology, Video for Windows (commonly known for playing AVI files), repackaging QuickTime files into AVIs was generally not a simple process due to QuickTime's more advanced features not being supported in VFW. Ultimately, mTropolis could play back AVI files on the Windows version of a project, but it had to be in AVI format ahead of time.
This became a major problem further down the line though, as Apple discontinued support for QuickTime on Windows, and getting QuickTime to work became more and more difficult with later versions.
Coming soon to... nowhere
mTropolis was positioned as an alternative to Macromedia Director, but ultimately failed in the market. mFactory was bought out by Quark, who released mTropolis 2.0 only as an update to existing mTropolis licensees and then closed the company, favoring their own product, Quark Immedia (which also failed). What's not so clear about this is how mTropolis was actually sold to licensees though.
Director was pretty pricey, but could be bought as a through a whole bunch of authorized resellers. mTropolis was considerably more rare, and VERY expensive. The first version was $4,500 per copy, equivalent to $9,200 today, and I've yet to find any evidence of it being sold as a mass-market product. It may have only been sold directly to licensees, and numerous credit lines and mentions suggest that mFactory was more hands-on with their licensees. Was it being offered under a more "premium" business-to-business model where they provided lots of support? Was the plan to go boxed-product after they'd worked out the kinks with early licensees? Who knows. Either way, used copies of it are extremely scarce.
While I was eventually able to obtain a used copy of the full version, the easiest way around this problem is actually the demo version, which was bundled with some reference material (which is also rare) and on a few issues of the UK magazine MacFormat, specifically the February 1997 (MF47) and May 1998 (MF63) issues. While the demo versions don't allow you to save your projects, they do allow you to export them, and since we're only interested in the exported format, that's perfectly fine! Now just need to dust off an old piece of fruit company hardware to run it.
 |
| Sorry about those aftermarket prices |
Asking a toddler to solve a math problem
After finally obtaining a copy, I wasn't really expecting to contribute more than as much information as I could find and hope that someone else who know what they were doing would pick up development of it. Most engines in ScummVM are developed by reverse-engineering the game executable with IDA or Ghidra, tools which can help turn compiled code run by the CPU into a more readable format. This requires a considerable amount of work, since the structure of more complex types and the usage of values have to all be determined by hand, and sometimes two different values wind up being treated as the same value because they were mapped to the same hardware register, and all other kinds of problems.
Doing that is often crucial though, because often you really need to know what the program is doing to handle certain things. File formats and behaviors can be cryptic, logic can be hard-coded, and being unable to look at the actual code doing things is a big problem.
I had no idea how to do any of that (although I've picked up on it somewhat since!), but I did know how to poke around data formats, and I had just come off of porting Glider PRO, which involved doing a lot of research on Mac-native formats documented in the Inside Macintosh reference book series, and getting a copy of mTropolis meant I had room to play with it. At this point, actually making it work again was not even remotely the plan.
What $4,500 (or a magazine subscription) buys you
Let's see what this thing actually looks like! You may have all kinds of ideas in your head of what a good way to write a multimedia authoring suite would look like, and then you have mTropolis, which is a bit... sparse? |
| mTropolis authoring interface |
All elements can have modifiers attached, which is where the real fun is. Variables are modifiers, scripts are modifiers, logical things are modifiers, movement is done via modifiers, basically everything interesting is done by attaching a modifier to an object.
We'll get to how all of this works in a later installment of this series. First, we need to talk about how to analyze this thing. If you go to the "Build" menu and select "Build Title..." then you get this nice dialog.
 |
| Build dialog |
"mFactory Animation?" That could be a serious problem. We'll talk about why later. For now, let's click Build to export our project!
 |
| Prompt for path for startup segment export |
Looks like we forgot something! We need to set the directory to export to. mTropolis was designed from the ground up for multi-CD titles, and the way it does this is by letting you create segment labels, each of which corresponds to a single exported file, and then assign each section in the project to a segment label. We can set these up further in the project config:
 |
| Segment mapping editor |
But anyway, let's set our startup segment to the "Move to PC" directory here and build it with a Macintosh target, and then build it again with a Windows target.
 |
| Exported projects |
Success! Let's move the files to my machine running the best operating system for open source development and everything else (Windows) and pop it open in XVI32.
 |
| Exported project in XVI32 |
There's not a lot that we can tell from the outset, but resizing the window horizontally causes things to line up. That seems to suggest that there's a table with a bunch of fixed-size elements in it. Of what purpose? We'll find out soon, but I'm also going to highlight another pattern that occurs quite often here.
 |
| Aligned header table, length-prefixed strings |
We can tell from this now that this size table takes up 34 bytes per row, but it's also not clear where it starts. But also, look at the boxes highlighted purple and green. Both of them contain strings with a length prefix. While C represents the length of strings by terminating them with zero bytes, MacOS was written mainly in assembly and Pascal, and Pascal uses length-prefixed strings. Even on MacOS, there was an extension to insert length prefixes to strings in C because the MacOS APIs preferred that so much, and that mentality carried over to a lot of Mac software, like this.
However, that also brings us to another problem: How many bytes long is that length prefix? That's actually a recurring problem throughout this kind of thing: Knowing how big a value is, and where it starts and ends, is really important, but most of the time, values are going to be too small to take up their entire capacity, so it's not clear if the zeroes in the other half belong to that value, or represent something else.
The lucky break that made everything else possible
Let's open the Windows version, "Untitled-11.MPL", and compare it side-by-side to the Mac version.
I've highlighted a few blocks here, but really this type of thing is all over the format. Remember what I said earlier about Mac and Windows machines using different byte order? If you compare the files data tends to appear in the same positions, but in reverse byte order. This is immensely helpful, because it means the overwhelming majority of data start positions and sizes can now be determined by just exporting the data into both formats and looking for the byte order flips.
There did turn out to be a few caveats to this though. The developers of this decided to try sticking to platform-native formats whenever they could, so a few things have data that is in completely-different format. One of those things that is the source of much deception are the rectangle and point structures. On MacOS, a point is typically two 16-bit values, with the first representing the vertical position and the second representing the horizontal position. On Windows, it is the reverse. The problem with this is that the member swapping combined with the byte swapping results in a complete 32-bit swap, so it is often difficult to tell if a 4-byte flip is a 32-bit number or a point.
This is still a bunch of salad though, and the earliest stages of doing something like this are the most painful because, while there are numerous concepts used repeatedly throughout this, they haven't been discovered yet. Even today, I haven't actually figured out what all of the values in the file header mean.
We can start with one basic one though: Most of the time, when a file format has a catalog, it has size and position data. In this case, some of the values in the catalog are a bit deceptive: The "sceneStr" text is actually just junk that was left in memory when the catalog data was being written, only the part with the stream type up until the first 0-byte terminator is relevant, and the scene type is a fixed-size 24-byte character buffer.
We can look at some of the values and see what is likely to be catalog positioning. An easy guess is that if two values sum up into a third value, then the value that gets summed into is probably the file position and the other one is the size.
Once we've identified this though, we still need to figure out if that data belongs to the scene type string before or after it. In this case, that can be guessed pretty easily by seeing if the data after the last string makes sense as a catalog entry or not. In this case, it does, so the data probably belongs to the string preceding it.
This can be poked with a little further by adding more assets and scenes to the project, which causes the value at 0x30 to correspondingly increase. Great, we know where the substreams are located!
At this point, I started making a program called MTDisasm to do further analysis, in particular dumping the substreams of a project to separate files. I'll spare most of the details, but the catalog is a series of streams marked by segment number, and the file positions are relative to the start of the segment file. Segment files have no header!
Marco Polo
Thankfully, this byte-swapping thing is unbelievably, helpful, but our adventure is just beginning. We still don't know what all of that data is before the "bootStream" or what the contents of any of them mean. This is where we get into "black box analysis" where we start doing things to our project, exporting it, comparing exports, and using those comparisons to understand how what we do in the editor is reflected in the data.
In theory, if it's possible to completely determine how everything you can do in the editor maps to the file format, then you can basically determine almost all of what the creators had in their project before they exported it.
There are still many issues in front of us though. Knowing what the data says doesn't always tell us how things are supposed to behave, sometimes the exported data is missing important things, and there's still all the work of making it all actually happen.
About those animations...
I'll cover the details of this in a later installment, but when I said this required a lot of luck, the byte-swapping stuff was a massive windfall, but there is one other type of thing that could have easily derailed this, at least to the point of really needing to start reverse-engineering the actual code: Proprietary compression formats!
For the most part, if data is fixed-size, it is relatively easy to load, but proprietary compression formats can be a nightmare. If you look at the deflate compression algorithm used in the ZIP format, for instance, it has LZ77 commands and literals compressed by Huffman trees, and the Huffman trees themselves are compressed, and there's very weak correlation between what the bits in the data are and what values they actually represent.
Even simple compression formats like Cinepak have multiple layers of information processing before you can get the data that you want, and being lossy formats, there isn't a reliable way to put data into it where you get a predictable result out.
mTropolis uses a proprietary compression format for its animations called "mFactory Animation" and fortunately, that format turned out to be relatively simple. This is actually somewhat expected: mTropolis was supposed to run on low-end Macs, at a time when high-end Macs could only decode an MP3 file in real-time if you weren't doing anything else, so compression formats were often relatively simple.
This type of thing was a pretty close call though. If it weren't for that, this might not have happened.
I'll get to how that was reverse-engineered in a later installment of this series.
Coming up next...
Plot twists, surprises, an anamorphic filter created solely for the intro video of a dead company, and of course, jank! But maybe less jank than you expect. In future installments I'll talk about:
- How mTropolis stores things
- How mTropolis handles game logic
- Dealing with platform-specific format differences
- Decompiling Miniscript
- Going from decompiling projects to actually running them
- Figuring out how things are supposed to work
- Debugging
- Game systems built out of spaghetti
- MIDI jank
- Auto-save, widescreen support, and subtitles
- Tales from other titles in ScummVM's mTropolis engine
No comments:
Post a Comment