Restoration of other Wolf3d/Spear EXE versions

Wolf3d Code & Tutorial Development Area

Moderators: Andy_Nonymous, Adam Biser, BrotherTank

NY00123
Can I Play Daddy
Can I Play Daddy
Posts: 47
Joined: Wed May 20, 2015 5:52 pm

Restoration of other Wolf3d/Spear EXE versions

Post by NY00123 »

Hey Chris,

Thanks for your feedback on Super 3-D Noah's Ark.

Before responding to your inquiries, I'm adding to my preceding post that I've also thought about splitting the repository into git submodules (thanks to Blzut3 for reminding me about this option). I'm not entirely sure if I do want to go for this, though.

Advantages:
- If setup the way I think they usually are, then all that someone will need to do in order to work on a specific program (e.g., a game) from my repos, is to recursively clone the sources of the program and any library on which it depends, without obtaining the rest of the sources.
- If I want to inspect the git log of recreated code for just a specific library or program (without any dependent library), it should be straightforward.
- Makes it easier to manage the programs and libraries separately, say if you want to commit changes just for one of these.
- Can link just to the one relevant repository if it's brought up during a discussion.

Disadvantages:
- I have this feeling that maintaining the structure of the submodules, or at least recreating it, is possibly somewhat less simple.
- I'm not sure if I should update a program's repository after updating any of the libraries; I've seen instances of certain git hashes being referenced for submodules' dirs.
- I think that I can't easily make a clone of all code without duplicating a library used with more than one program; That is, unless I just create one root repository with not much more than a README, along with all programs and libraries as submodules of it. Then again, it's somewhat less convenient for other users, and is probably not the intended way of using git submodules.
Chris wrote:On the one hand, it was quite fun comparing Wolf3d to Super Noah's Ark to see what was different in the code (which you could do in w3d_plus) and fun compiling Noah's Ark exe, and being able to easily see it's features with the possibility of porting some things from Noah into Wolf3d (like the midi music, maybe the automap).
I indeed thought more than once about the question if S3DNA should be separate or not.

Usually, I do keep the codebases of differing games separate, unless they were originally in the same codebase again, as done with Duke3D / Enhanced Duke, NAM and WW2GI. Even then, one may think that it's better to keep NAM, WW2GI and Enhanced Duke separate from the earlier Duke3D revisions; Or at least WW2GI and Enhanced Duke, due to additions like the gamevars.

S3DNA is at least quite close to Wolf3D in many ways, and there are a few really minor examples where its code is concidentally matching Wolf3D shareware v1.0 or that March 1992 prototype build. It also has clear differences / code additions, the MIDI playback routines being a notable example. Main reason for separating S3DNA would be if you wanted to keep debugging information intact, at least partially (like line numbers). I actually separated differing versions of SW due to the amount of code changes, as well as line numbers information used for debugging (even though you eventually can't exactly 100% replicate the binary EXEs from the 90s): https://gitlab.com/NY00123/sw-src-and-build-addendum/
On the other hand, I actually kept an older version from the time of the original post (before the new games were added) and use that mostly, to make things easier when working with wolf3d/spear only. It still uses the version names defines too: https://my.pcloud.com/publink/show?code ... xIYREcP6yk
Hmm, I'm not 100% sure if you spotted me suggesting this on Discord, but it probably won't hurt to mention here for other readers, either way:

You can use unifdef to extract a portion of the code based on one macro or more, like code matching Wolf3D shareware v1.1, or alternatively S3DNA.
edit: If you meant just changing the name back to wolf3d I think that sounds good too. The Noah and Spear stuff would just be an added bonus in it. :D
Now, it is interesting to see Spear of Destiny being brought up as well. I've also had it in my mind, main difference being that SOD was internally developed at id, and the Wolf3D sources as released in 1995 also include SOD.
NY00123
Can I Play Daddy
Can I Play Daddy
Posts: 47
Joined: Wed May 20, 2015 5:52 pm

Post by NY00123 »

Hi all,

Good to see that the forums are back. Guess that I can repost the following from the Duke4.net forums, with minor edits.

So, I've gotten an important technical update:
- Following Bitbucket's planned removal of Mercurial support, I've converted the repository to git.
- Additionally, I've then split it into separate submodules, and set up a Bitbucket team for all of them (thanks to Blzut3 for the suggestions).

As a consequence, gamesrc-ver-recreation has gotten a new URL: https://bitbucket.org/gamesrc-ver-recreation/

We'll see if the above migration has worked well, and especially if there's a difference in the convenience of using it, when compared to working with just one repository.

Notes for anybody trying to clone any of the submodules:
► Show Spoiler
================

As for any added contents, I'm afraid that there isn't much to renew in terms of Wolfenstein 3D or related games/programs.

To summarize, there's now the addition of a tool, which is still gaming-related, but isn't exactly a game on its own. Basically, this is the Build Editor for Duke Nukem 3D.

This covers the editor as distributed with registered version 1.3d of Duke Nukem 3D, as well as the editor bundled with Duke Nukem 3D: Atomic Edition v1.4-1.5 (1.4's editor was reused for 1.5).
As a small bonus, the Build Engine objects as used in the editor for Shadow Warrior v1.2 are also covered.

================

What can I say about the code recreation process, for the two Duke3D editor versions? Well, it was quite different for each of them.

For the ones not aware, like the rest of the Build Engine, most of the Build editor code was originally offered to developers in a binary BUILD.OBJ file, with no source code.
However, they had the option to extend its functionality, with the assistance of hooks present in a file named BSTUB.C.

For Duke Nukem 3D, this file was renamed ASTUB.C. The Shadow Warrior sources have FMSTUB.C and JNSTUB.C, although it was eventually the latter which was used for the editor as distributed with SW 1.2.

So basically, there were two components to take care of: ASTUB.C, and the Build Engine code.
After the Duke3D sources were GPLed, a version of ASTUB.C was released under the same terms. I suspected that it matched v1.3d, but I first wanted to recreate 1.4/1.5's editor. As expected, there was a bunch of code to add, rather than delete (dealing with additional enemies introduced in 1.4 was just one example).

- At this point, only the ASTUB.C:Keys3d function is not 100% matching the original in terms of layout, but it should be equivalent in behaviors from what I recall.
- Fun fact: It actually turned out that certain bits of Duke3D's ASTUB.C code were later used in SW's JNSTUB.C. This clearly helped me, especially with the Keys3d function.
- In terms of Build Engine code, I initially assumed that it matched Duke3D v1.4. While A.OBJ and CACHE1D.OBJ are matching, it actually turned out that the editor uses a somewhat earlier revision of ENGINE.OBJ. I also had to update the code for BUILD.OBJ, which was expected (it was originally matching just Ken-Build as released in 2000, including a minor update from 2002).

I then went on to take care of v1.3d's editor. I'm reminding about the guess that the GPLed ASTUB.C file is matching it. As it eventually turned out, except for the version string being "DUKE NUKEM BUILD: V032696" instead of "DUKE NUKEM BUILD: V041996", as well as changes I assume that Charlie did for compatibility with Duke3D v1.5's engine for most, it was indeed perfectly matching v1.3d.

- Unlike v1.4, almost all work done for Duke3D v1.3d's editor was in the engine code, with very little done in ASTUB.C itself.
- Portions of code from a 1995 revision of ENGINE.C released on May 2019 were used (https://forums.duke4.net/topic/10589-re ... d-sources/).
- Thanks again to Nuke.YKT, he already got a great deal (in fact, most?) of the required work on v1.3d's engine done. He was originally doing this for Redneck Rampage, with the initial assumption that its engine was at least earlier than Duke3D v1.4's in some way, before figuring out that it's actually matching 1.5 (also used by me in the same repository later).
- A few ENGINE.OBJ functions still don't perfectly match, but it should be quite close now, and the file mostly matches anyway (with a bit of hacks covering assembly pragmas, so a couple of functions are at least a bit closer in layout, and match in size).

Also, this time, there were instances in which I used a combination of a few files, like a script and a (variant of a) makefile. What for? Well, instead of manually guess, say, what should be the order in which the local variables of a certain function are defined, I could kinda brute-force this via scripting. A few more details:
► Show Spoiler

Before this is forgotten, what about Shadow Warrior v1.2's Build Editor again? Well:
- All of the Build Engine objects are covered. In particular, A.OBJ is the same as used in Duke3D v1.4-1.5 and their editor, CACHE1D.OBJ is the same as in Duke3D v1.5, and BUILD.OBJ is currently identified as the 19961213 revision. The version found in the Ken-Build sources as released on 2000 seems to be virtually identical. It think that the couple of changes from the 10/4/97 entry of BUILD2.TXT are the only differences. This comes to increasing MAXTILES from 6144 to 9216 for Xatrix, as well as adding "#pragma push" and "#pragma pop" to BUILD.H, reason being the default struct packing was changed from 1 to 8 with the move to Watcom 11.0.
- For the SW-specific additions, see the JNSTUB.C file in the SW sources as re-released on Feb 2019 (REGCODE subdir): https://gitlab.com/NY00123/sw-src-and-b ... E/JNSTUB.C
User avatar
Arielus
DieHard Mutant
DieHard Mutant
Posts: 895
Joined: Thu Mar 13, 2003 9:43 pm
Location: Portland, Oregon, USA

Re: Restoration of other Wolf3d/Spear EXE versions

Post by Arielus »

Chris wrote:
► Show Spoiler
► Show Spoiler
Best,

Ariel
Metal lives forever... blazing on and on!!!
NY00123
Can I Play Daddy
Can I Play Daddy
Posts: 47
Joined: Wed May 20, 2015 5:52 pm

Post by NY00123 »

Hi,

I'm having additions to the wolf3d repository after more than 6.5 years, covering two added revisions of code. But there are also a few more general points to add:
- The various repositories aren't submodules of the "gamesrc-ver-recreation" repository itself anymore. Nuke.YKT and I haven't really been using this feature, so I've decided to do this change. The repositories still exist, I just don't use the submodule feature.
- While currently done in the wolf3d tree only, doing so in other repositories isn't necessarily out of the question. So, a significant subset of the file notes-restoration.md was replaced with a new README.md file, aiming to replace it. It should be much smaller, albeit it's still not necessarily a small file. I left various notes under notes-restoration.md but I currently have no guarantee of them being up-to-date. Interestingly, the older incarnation titled "game-srccode-ver-recreation" had a quite small README.md file added to it, so this can be seen in part as going back to the roots.

As for wolf3d, before getting to the aforementioned two code revisions, I also made these changes:
- Output build directories were collapsed. That should be more consistent with other repositories, the Blake Stone repository being just one example. For instance, WL1AP10.PRJ's output dir was changed from "OBJ\WL1AP10" to "WL1AP10".
- One directory was mistakenly named "STATIC\WOLF3D\WL920512". Instead of "WL920512", "WL920312" is used now.

Let's get to the added revisions themselves.
- The first one is what I suspect to be a quite unknown April 16 1992 prototype executable, somewhere in-between WL920312 and shareware v1.0. There wasn't a lot of new code added but I did have to go through existing macros, so it took some time. There were a few exceptions here and there, like placeholder code under the function "Victory", but what was there beforehand could be reused otherwise.
- Secondly, under a new directory, I added a separate modification of the open-source release of Wolf3D into code more-or-less matching a very early ROTT prototype. Known as ROTT0993, it's still essentially a modified Wolf3D. There wasn't a lot of code added and I didn't introduce new game data to the repository. This revision draws textured floors and ceilings and also has additional hotkey checks for debugging. As for the output EXE, expect differences in debugging symbols. I was reducing them, but even after making timestamps match, I still had 29 differing bytes.

To finish, here's one more thing. Getting back to the wolf3d repository after about 6.5 years, short of the few occasional edits done in-between, is not something I recall doing beforehand under gamesrc-ver-recreation right now. One point was known to me earlier, but having another look after a while further clarified it. Basically, the use of pre-processor macros did make it possible to cover many versions under a single codebase - currently 19 in total. On the other hand, over time, it may become difficult to maintain this code and track it with the added pre-processor macro checks. Then again, they did make it quite convenient to see differences between versions under a single window, without using a diff program. But that can be a challenge when it comes to supporting multiple versions in source ports. In ReflectionHLE's case, I mostly bypassed it by repeatedly rebuilding ported Wolf3D sources with a bit different configurations and then linking the builds into a single binary. Definitions of pre-processor macros inherited from gamesrc-ver-recreation were changed across these sets of objects. I still made adjustments, if due to the replacement of the macro UPLOAD with a variable of the same name, or any other technical reason.
User avatar
ReddimusVonAggrevatii
DieHard Guard
DieHard Guard
Posts: 221
Joined: Thu Oct 12, 2017 6:10 am
Location: A Jjaro space station.
Contact:

Post by ReddimusVonAggrevatii »

Out of curiosity, was ROTT0993 ever released in any official capacity?
I regret nothing.
NY00123
Can I Play Daddy
Can I Play Daddy
Posts: 47
Joined: Wed May 20, 2015 5:52 pm

Post by NY00123 »

ReddimusVonAggrevatii wrote:Out of curiosity, was ROTT0993 ever released in any official capacity?
Not that I'm aware of; But let me quote a subset of a Duke4.net forums post of mine from last November, referring to the ROTT prototypes:
NY00123 wrote:I did see it mentioned by a New Blood person in one public Discord server that there were plans to release the nowadays leaked prototypes. There are (or at least were?) still plans to do this, but that's what I know for now.
Hence, I had less of a concern with adding reconstructed sources for ROTT0993. For now they're still in a separate directory (also for debug information), and none of ROTT0993's game data is in the repository.
User avatar
Adam Biser
Utility Developer
Utility Developer
Posts: 2412
Joined: Fri Jun 06, 2003 5:53 pm
Location: USA
Contact:

Post by Adam Biser »

Where do you keep your list of versions?

Wolf3D prototypes I know of:
W3D31492
W3D41392
W3D51492 - This is WL3
W3D53092 - This is WL6

SOD 1992-08-01
SOD 1992-08-18
NY00123
Can I Play Daddy
Can I Play Daddy
Posts: 47
Joined: Wed May 20, 2015 5:52 pm

Post by NY00123 »

Adam Biser wrote:Where do you keep your list of versions?
Under gamesrc-ver-recreation/wolf3d? See its README.md file. There are also details like checksums, something that was originally suggested for Hexen at the time.
User avatar
Adam Biser
Utility Developer
Utility Developer
Posts: 2412
Joined: Fri Jun 06, 2003 5:53 pm
Location: USA
Contact:

Post by Adam Biser »

The Wolf3d prototypes I have seen:
► Show Spoiler
Post Reply