Skip to content

Commit

Permalink
massive refactor of load_bsp for #17 & RavenBsp + RitualBsp update
Browse files Browse the repository at this point in the history
  • Loading branch information
snake-biscuits committed Oct 16, 2021
1 parent 60ea839 commit 85343c2
Show file tree
Hide file tree
Showing 55 changed files with 922 additions and 517 deletions.
12 changes: 11 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@
## v0.4.0 (???)

## New
* Added support for Ritual Entertainment's Ubertools (Quake III Engine Branch)
* If `autoload` cannot find the specified `.bsp` file a UserWarning is issued

## Changed
* Moved physics SpecialLumpClasses to `branches/shared/physics.py`
Expand All @@ -12,16 +14,24 @@
- `from_bytes` method added
- built in asserts to verify accurate definitions (TODO: move to tests)
- `as_bytes` method added
* Completely refactored `branch_script` detection
* `load_bsp` now only accepts `branch_script`s in it's optional argument

### Newly Supported
* Source Engine
- Tactical Intervention
* Ubertools

### Updated Support
* Quake Engine
- Hexen 2
* Id Tech 3
- Quake III Arena
- Raven Software Titles
* Source Engine
* Titanfall Engine


## v0.3.1 (4th October 2021)

### New
Expand All @@ -44,7 +54,7 @@

### New
* Added `load_bsp` function to identify bsp type
* Added `D3DBsp`, `IdTechBsp`, `RespawnBsp` & `ValveBsp` classes
* Added `InfinityWardBsp`, `IdTechBsp`, `RespawnBsp` & `ValveBsp` classes
* Added general support for the PakFile lump
* Added general support for the GameLump lump
* Extension scripts
Expand Down
42 changes: 23 additions & 19 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,13 +62,12 @@ Full documentation: [snake-biscuits.github.io/bsp_tool/](https://snake-biscuits.
* [Arkane Studios](https://github.com/snake-biscuits/bsp_tool/tree/master/bsp_tool/branches/arkane)
- [Dark Messiah of Might & Magic](https://github.com/snake-biscuits/bsp_tool/tree/master/bsp_tool/branches/arkane/dark_messiah.py) :x:
* [Gearbox Software](https://github.com/snake-biscuits/bsp_tool/tree/master/bsp_tool/branches/gearbox)
- [Half-Life: Blue Shift](https://github.com/snake-biscuits/bsp_tool/tree/master/bsp_tool/branches/gearbox/bshift.py)
- [Half-Life: Blue Shift](https://github.com/snake-biscuits/bsp_tool/tree/master/bsp_tool/branches/gearbox/blue_shift.py)
- [Half-Life: Opposing Force](https://github.com/snake-biscuits/bsp_tool/tree/master/bsp_tool/branches/valve/goldsrc.py)
* [Id Software](https://github.com/snake-biscuits/bsp_tool/tree/master/bsp_tool/branches/id_software)
- [Quake](https://github.com/snake-biscuits/bsp_tool/tree/master/bsp_tool/branches/id_software/quake.py) :x:
- [Quake](https://github.com/snake-biscuits/bsp_tool/tree/master/bsp_tool/branches/id_software/quake.py)
- [Quake II](https://github.com/snake-biscuits/bsp_tool/tree/master/bsp_tool/branches/id_software/quake2.py)
- [Quake III Arena](https://github.com/snake-biscuits/bsp_tool/tree/master/bsp_tool/branches/id_software/quake3.py)
- Quake 4 :o:
- Quake Champions :o:
- [Quake Live](https://github.com/snake-biscuits/bsp_tool/tree/master/bsp_tool/branches/id_software/quake3.py)
* [Infinity Ward](https://github.com/snake-biscuits/bsp_tool/tree/master/bsp_tool/branches/infinity_ward)
Expand All @@ -80,25 +79,25 @@ Full documentation: [snake-biscuits.github.io/bsp_tool/](https://snake-biscuits.
- [Counter-Strike: Online 2](https://github.com/snake-biscuits/bsp_tool/tree/master/bsp_tool/branches/nexon/cso2.py) :x:
- [Titanfall: Online](https://github.com/snake-biscuits/bsp_tool/tree/master/bsp_tool/branches/respawn/titanfall.py)
- [Vindictus](https://github.com/snake-biscuits/bsp_tool/tree/master/bsp_tool/branches/nexon/vindictus.py) :x:
* Raven Software
- [Hexen 2](https://github.com/snake-biscuits/bsp_tool/tree/master/bsp_tool/branches/id_software/quake.py) :x:
- [Soldier of Fortune](https://github.com/snake-biscuits/bsp_tool/tree/master/bsp_tool/branches/id_software/quake2.py) :o:
- [Soldier of Fortune II: Double Helix](https://github.com/snake-biscuits/bsp_tool/tree/master/bsp_tool/branches/id_software/quake3.py) :o:
- [Star Trek: Voyager - Elite Force](https://github.com/snake-biscuits/bsp_tool/tree/master/bsp_tool/branches/id_software/quake3.py) :o:
- [Star Wars Jedi Knight II: Jedi Outcast](https://github.com/snake-biscuits/bsp_tool/tree/master/bsp_tool/branches/id_software/quake3.py) :o:
- [Star Wars Jedi Knight: Jedi Academy](https://github.com/snake-biscuits/bsp_tool/tree/master/bsp_tool/branches/id_software/quake3.py) :o:
* [Raven Software](https://github.com/snake-biscuits/bsp_tool/tree/master/bsp_tool/branches/raven)
- Heretic II :o:
- [Hexen 2](https://github.com/snake-biscuits/bsp_tool/tree/master/bsp_tool/branches/raven/hexen2.py)
- [Soldier of Fortune](https://github.com/snake-biscuits/bsp_tool/tree/master/bsp_tool/branches/id_software/quake3.py) :x:
- [Soldier of Fortune II: Double Helix](https://github.com/snake-biscuits/bsp_tool/tree/master/bsp_tool/branches/raven/sin.py)
- [Star Trek: Voyager - Elite Force](https://github.com/snake-biscuits/bsp_tool/tree/master/bsp_tool/branches/id_software/quake3.py)
- [Star Wars Jedi Knight: Jedi Academy](https://github.com/snake-biscuits/bsp_tool/tree/master/bsp_tool/branches/raven/sin.py)
- [Star Wars Jedi Knight II: Jedi Outcast](https://github.com/snake-biscuits/bsp_tool/tree/master/bsp_tool/branches/raven/sin.py)
* [Respawn Entertainment](https://github.com/snake-biscuits/bsp_tool/tree/master/bsp_tool/branches/respawn)
- [Titanfall](https://github.com/snake-biscuits/bsp_tool/tree/master/bsp_tool/branches/respawn/titanfall.py)
- [Titanfall 2](https://github.com/snake-biscuits/bsp_tool/tree/master/bsp_tool/branches/respawn/titanfall2.py)
- [Apex Legends](https://github.com/snake-biscuits/bsp_tool/tree/master/bsp_tool/branches/respawn/apex_legends.py)
* [Ritual Entertainment](https://github.com/snake-biscuits/bsp_tool/tree/master/bsp_tool/branches/ritual)
- [American McGee's Alice](https://github.com/snake-biscuits/bsp_tool/tree/master/bsp_tool/branches/ritualfakk2.py)
- [American McGee's Alice](https://github.com/snake-biscuits/bsp_tool/tree/master/bsp_tool/branches/ritual/fakk2.py)
- [Heavy Metal F.A.K.K. 2](https://github.com/snake-biscuits/bsp_tool/tree/master/bsp_tool/branches/ritual/fakk2.py)
- Medal of Honor: Allied Assault :o:
- SiN :o:
- SiN: Gold :o:
- SiN Episodes: Emergence :o:
- Star Trek: Elite Force II :o:
- [SiN](https://github.com/snake-biscuits/bsp_tool/tree/master/bsp_tool/branches/raven/sin.py) :x:
- [SiN: Gold](https://github.com/snake-biscuits/bsp_tool/tree/master/bsp_tool/branches/raven/sin.py) :x:
- [SiN Episodes: Emergence](https://github.com/snake-biscuits/bsp_tool/tree/master/bsp_tool/branches/valve/source.py) :o:
- [Star Trek: Elite Force II](https://github.com/snake-biscuits/bsp_tool/tree/master/bsp_tool/branches/ritual/star_trek_elite_force2.py)
* [Valve Software](https://github.com/snake-biscuits/bsp_tool/tree/master/bsp_tool/branches/valve)
- [Alien Swarm](https://github.com/snake-biscuits/bsp_tool/tree/master/bsp_tool/branches/valve/alien_swarm.py)
- [Alien Swarm: Reactive Drop](https://github.com/snake-biscuits/bsp_tool/tree/master/bsp_tool/branches/valve/alien_swarm.py)
Expand Down Expand Up @@ -130,32 +129,37 @@ Full documentation: [snake-biscuits.github.io/bsp_tool/](https://snake-biscuits.
* Other
- [Black Mesa](https://github.com/snake-biscuits/bsp_tool/tree/master/bsp_tool/branches/valve/sdk_2013.py)
- [Blade Symphony](https://github.com/snake-biscuits/bsp_tool/tree/master/bsp_tool/branches/valve/sdk_2013.py)
- Brink :o:
- Call of Duty: Black Ops III :o:
- Call of Duty: World at War :o:
- Daikatana :o:
- [Fortress Forever](https://github.com/snake-biscuits/bsp_tool/tree/master/bsp_tool/branches/valve/orange_box.py)
- [G-String](https://github.com/snake-biscuits/bsp_tool/tree/master/bsp_tool/branches/valve/orange_box.py)
- [Garry's Mod](https://github.com/snake-biscuits/bsp_tool/tree/master/bsp_tool/branches/valve/orange_box.py)
- [Halfquake Trilogy](https://github.com/snake-biscuits/bsp_tool/tree/master/bsp_tool/branches/valve/goldsrc.py)
- [Medal of Honor: Allied Assault](https://github.com/snake-biscuits/bsp_tool/tree/master/bsp_tool/branches/ritual) :o:
- [Medal of Honor: Allied Assault](https://github.com/snake-biscuits/bsp_tool/tree/master/bsp_tool/branches/ritual/moh_allied_assault.py) :x:
- [NEOTOKYO](https://github.com/snake-biscuits/bsp_tool/tree/master/bsp_tool/branches/valve/orange_box.py)
- [Sven Co-op](https://github.com/snake-biscuits/bsp_tool/tree/master/bsp_tool/branches/valve/goldsrc.py)
- [Synergy](https://github.com/snake-biscuits/bsp_tool/tree/master/bsp_tool/branches/valve/source.py)
- [Tactical Intervention](https://github.com/snake-biscuits/bsp_tool/tree/master/bsp_tool/branches/valve/orange_box.py)
- [Team Fortress Quake](https://github.com/snake-biscuits/bsp_tool/tree/master/bsp_tool/branches/id_software/quake.py) :x:
- [The Beginner's Guide](https://github.com/snake-biscuits/bsp_tool/tree/master/bsp_tool/branches/valve/sdk_2013.py) :o:
- [The Stanley Parable](https://github.com/snake-biscuits/bsp_tool/tree/master/bsp_tool/branches/valve/sdk_2013.py) :o:
- Vampire: The Masquerade - Bloodlines :o:
- [Vampire: The Masquerade - Bloodlines](https://github.com/snake-biscuits/bsp_tool/tree/master/bsp_tool/branches/valve/source.py)


## Thanks
* [BobTheBob](https://github.com/BobTheBob9)
- Identified loads of Titanfall lumps (90% of static props + more)
* [Chris Strahl](https://github.com/Chrissstrahl)
- Preserving **extensive** documentation, mods & source code for Quake 3 & Ubertools games
* [Ficool2](https://github.com/ficool2)
- Providing lots of current and detailed info on Source & helping track down some rarer titles
* [Maxime Dupuis](https://github.com/maxdup)
- Helping me identify multiple lumps in Source Engine .bsps
* [MobyGames](https://www.mobygames.com/)
- Keeping records of the credits on so many games, helping to pin down engine origins
* [pakextract](https://github.com/yquake2/pakextract)
- Super useful tool for `.pak` files
* [REDxEYE](https://github.com/REDxEYE)
- Being very open and actively collaborating on SourceIO & Titanfall .bsps
* [Taskinoz](https://github.com/taskinoz)
Expand Down
130 changes: 60 additions & 70 deletions bsp_tool/__init__.py
Original file line number Diff line number Diff line change
@@ -1,90 +1,80 @@
"""A library for .bsp file analysis & modification"""
__all__ = ["base", "branches", "load_bsp", "lumps", "tools",
"GoldSrcBsp", "ValveBsp", "QuakeBsp", "IdTechBsp",
"D3DBsp", "RespawnBsp", "UberBsp"]
"GoldSrcBsp", "IdTechBsp", "InfinityWardBsp", "QuakeBsp",
"RavenBsp", "RespawnBsp", "RitualBsp", "ValveBsp"]

import difflib
import os
from types import ModuleType
from typing import Union

from . import base # base.Bsp base class
from . import branches # all known .bsp variant definitions
from . import lumps # handles loading data dynamically
from . import lumps
from .id_software import QuakeBsp, IdTechBsp
from .infinity_ward import D3DBsp
from .infinity_ward import InfinityWardBsp
from .raven import RavenBsp
from .respawn import RespawnBsp
from .ritual import UberBsp
from .ritual import RitualBsp
from .valve import GoldSrcBsp, ValveBsp


# NOTE: Quake Live branch_script should be quake3, but auto-detect defaults to quake2 on BSP_VERSION
# NOTE: CoD1 auto-detect by version defaults to ApexLegends
BspVariant_from_file_magic = {b"2015": RitualBsp,
b"EF2!": RitualBsp,
b"FAKK": RitualBsp,
b"IBSP": IdTechBsp, # or InfinityWardBsp
b"rBSP": RespawnBsp,
b"RBSP": RavenBsp,
b"VBSP": ValveBsp}
# NOTE: if no file_magic is present, options are:
# - GoldSrcBsp
# - QuakeBsp
# - 256-bit XOR encoded Tactical Intervention .bsp

GoldSrc_versions = {*branches.valve.goldsrc.GAME_VERSIONS.values(),
*branches.gearbox.blue_shift.GAME_VERSIONS.values()}
InfinityWard_versions = {v for s in branches.infinity_ward.scripts for v in s.GAME_VERSIONS.values()}
Quake_versions = {*branches.id_software.quake.GAME_VERSIONS.values()}

developers_by_file_magic = {b"FAKK": UberBsp,
b"IBSP": IdTechBsp, # or D3DBsp
b"rBSP": RespawnBsp,
b"VBSP": ValveBsp}
# HACK: GoldSrcBsp has no file-magic, substituting BSP_VERSION
goldsrc_versions = [branches.valve.goldsrc.BSP_VERSION, branches.gearbox.bshift.BSP_VERSION]
developers_by_file_magic.update({v.to_bytes(4, "little"): GoldSrcBsp for v in goldsrc_versions})

developers_by_file_magic.update({branches.id_software.quake.BSP_VERSION.to_bytes(4, "little"): QuakeBsp})

cod_ibsp_versions = [getattr(branches.infinity_ward, b).BSP_VERSION for b in branches.infinity_ward.__all__]


def guess_by_file_magic(filename: str) -> (base.Bsp, int):
"""returns BspVariant & version"""
if os.path.getsize(filename) == 0: # HL2/ d2_coast_02.bsp
raise RuntimeError(f"{filename} is an empty file")
BspVariant = None
if filename.endswith(".d3dbsp"):
BspVariant = D3DBsp
elif filename.endswith(".bsp"):
with open(filename, "rb") as bsp_file:
file_magic = bsp_file.read(4)
if file_magic not in developers_by_file_magic:
raise RuntimeError(f"'{filename}' does not resemble a .bsp file")
bsp_version = int.from_bytes(bsp_file.read(4), "little")
BspVariant = developers_by_file_magic[file_magic]
if BspVariant == GoldSrcBsp:
bsp_version = int.from_bytes(file_magic, "little")
# D3DBsp has b"IBSP" file_magic
if file_magic == b"IBSP" and bsp_version in cod_ibsp_versions:
BspVariant = D3DBsp
else: # invalid extension
raise RuntimeError(f"{filename} is not a .bsp file!")
return BspVariant, bsp_version


def load_bsp(filename: str, branch: Union[str, ModuleType] = "Unknown"):
def load_bsp(filename: str, branch_script: ModuleType = None) -> base.Bsp:
"""Calculate and return the correct base.Bsp sub-class for the given .bsp"""
# TODO: OPTION: use filepath to guess game / branch
# is filename real?
if not os.path.exists(filename):
raise FileNotFoundError(f".bsp file '{filename}' does not exist.")
BspVariant, bsp_version = guess_by_file_magic(filename)
if isinstance(branch, ModuleType):
return BspVariant(branch, filename, autoload=True)
elif isinstance(branch, str):
# TODO: default to other methods on fail
branch: str = branches.simplify_name(branch)
if branch != "unknown": # not default
if branch not in branches.by_name:
close_matches = difflib.get_close_matches(branch, branches.by_name)
if len(close_matches) == 0:
raise NotImplementedError(f"'{branch}' .bsp format is not supported, yet.")
else:
print(f"'{branch}'.bsp format is not supported. Assumed branches:",
"\n".join(close_matches),
f"Trying '{close_matches[0]}'...", sep="\n")
branch: str = close_matches[0]
branch: ModuleType = branches.by_name[branch] # "name" -> branch_script
# guess branch by format version
elif os.path.getsize(filename) == 0: # HL2/ d2_coast_02.bsp
raise RuntimeError(f"{filename} is an empty file")
# parse header
with open(filename, "rb") as bsp_file:
file_magic = bsp_file.read(4)
version = int.from_bytes(bsp_file.read(4), "little")
# NOTE: not every version is this format
# -- DarkMessiahBspHeader { char file_magic[4]; short version[2]; ... };
# identify BspVariant
if filename.lower().endswith(".d3dbsp"): # CoD2
assert file_magic == b"IBSP", "Mystery .d3dbsp!"
assert version in InfinityWard_versions, "Unexpected .d3dbsp format version!"
BspVariant = InfinityWardBsp
elif filename.lower().endswith(".bsp"):
if file_magic not in BspVariant_from_file_magic:
version = int.from_bytes(file_magic, "little")
file_magic = None
if version in Quake_versions:
BspVariant = QuakeBsp
elif version in GoldSrc_versions:
BspVariant = GoldSrcBsp
else:
raise NotImplementedError("TODO: Check if encrypted Tactical Intervention .bsp")
else:
if bsp_version not in branches.by_version:
raise NotImplementedError(f"{BspVariant} v{bsp_version} is not supported!")
branch: ModuleType = branches.by_version[bsp_version]
return BspVariant(branch, filename, autoload=True)
else:
raise TypeError(f"Cannot use branch of type `{branch.__class__.__name__}`")
if file_magic == b"IBSP" and version in InfinityWard_versions:
BspVariant = InfinityWardBsp
else:
BspVariant = BspVariant_from_file_magic[file_magic]
else: # invalid extension
raise RuntimeError(f"{filename} is not a .bsp file!")
# identify branch script
# TODO: ata4's bspsrc uses unique entity classnames to identify branches
# -- need this for identifying variants with overlapping versions
# -- e.g. (b"VBSP", 20) & (b"VBSP", 21)
if branch_script is None:
branch_script = branches.script_from_file_magic_and_version[(file_magic, version)]
return BspVariant(branch_script, filename, autoload=True) # might raise errors
5 changes: 3 additions & 2 deletions bsp_tool/base.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
import struct
from types import MethodType, ModuleType
from typing import Dict, List
import warnings

from . import lumps

Expand All @@ -32,7 +33,7 @@ class Bsp:
# ^ {"LUMP_NAME": Exception encountered}

def __init__(self, branch: ModuleType, filename: str = "untitled.bsp", autoload: bool = True):
if not filename.endswith(".bsp"):
if not filename.lower().endswith(".bsp"):
raise RuntimeError("Not a .bsp")
filename = os.path.realpath(filename)
self.folder, self.filename = os.path.split(filename)
Expand All @@ -42,7 +43,7 @@ def __init__(self, branch: ModuleType, filename: str = "untitled.bsp", autoload:
if os.path.exists(filename):
self._preload()
else:
print(f"{filename} not found, creating a new .bsp")
warnings.warn(UserWarning(f"{filename} not found, creating a new .bsp"))
self.headers = {L.name: LumpHeader(0, 0, 0, 0) for L in self.branch.LUMP}

def __enter__(self):
Expand Down
18 changes: 11 additions & 7 deletions bsp_tool/branches/README.md
Original file line number Diff line number Diff line change
@@ -1,22 +1,26 @@
# How bsp_tool loads .bsps
If you're using `bsp_tool.load_bsp("reallycool.bsp")` to load a `.bsp` a few things happen behind the scenes to figure out the format
If you're using `bsp_tool.load_bsp("reallycoolmap.bsp")` to load a `.bsp` a few things happen behind the scenes to figure out the format
Since bsp_tool supports a range of `.bsp` variants, a single script to handle the rough format wasn't going to cut it
To narrow down the exact format of a bsp file `load_bsp` looks at some key information in each file:


### Developer variants
First, `load_bsp` tries to determine the developer behind the chosen .bsp
If the file extension is `.d3dbsp`, it's a Call of Duty 2 or 4 `D3DBsp`
If the file extension is `.d3dbsp`, it's a Call of Duty 2 `InfinityWardBsp`
Other bsps use the `.bsp` extension (Call of Duty 1 included)
The developer is identified from the "file-magic", the first four bytes of any .bsp are:
- `b"IBSP"` for `IdTechBsp` Id Software
- `b"IBSP"` for `D3DBsp` Infinity Ward
- `b"rBSP"` for `RespawnBsp` Respawn
- `b"VBSP"` for `ValveBsp` Valve
- `b"2015"` for `RitualBsp`
- `b"EF2!"` for `RitualBsp`
- `b"FAKK"` for `RitualBsp`
- `b"IBSP"` for `IdTechBsp`
- `b"IBSP"` for `InfinityWardBsp`
- `b"RBSP"` for `RavenBsp`
- `b"rBSP"` for `RespawnBsp`
- `b"VBSP"` for `ValveBsp`

> This rule isn't perfect! most mods out there are Source Engine forks with b"VBSP"
> GoldSrc .bsp files don't have any file magic!
> Quake & GoldSrc .bsp files don't have any file magic!
Most of the major differences between each developer's format are the number of lumps & bsp header
They also use some lumps which are unique to each developer's Quake based engine
Expand Down
Loading

0 comments on commit 85343c2

Please sign in to comment.