mirror/oot
1
0
Fork 0
mirror of https://github.com/zeldaret/oot.git synced 2025-08-09 08:20:17 +00:00
Code Issues Projects Releases Wiki Activity

New assets system (#2481)

Browse source 
* wip: New assets system tm

Builds gc-eu-mq-dbg OK from clean after
1) make setup
2) python3 -m tools.assets.extract -j
3) replace 0x80A8E610 with sShadowTex in extracted/gc-eu-mq-dbg/assets/overlays/ovl_En_Jsjutan/sShadowMaterialDL.inc.c
4) make various symbols in extracted data like sTex static

* use variables from config.yml for gMtxClear and sShadowTex addresses

* Write source with static for overlays using `HACK_IS_STATIC_ON` hack

* gc-eu-mq-dbg OK from clean with `make setup && make`

* implement more skeleton-related types, cleanups, fixups

* fix extracted data to no longer produce compilation warnings

* implement more of RoomShapeImage types

* yeet XmlPath from ExternalFile usage

* Implement PlayerAnimationDataResource (link_animetion data)

* fix csdis CS_TIME extra arg

* dmadata file names no longer hardcoded for gc-eu-mq-dbg

* ntsc-1.0 OK

* xml fixes

* slightly improve standard output

* rm extract_assets.py

* generate and use Limb enums (TODO: check Skin skels and implement for Curve skels)

* handle dependencies between xmls

* introduce RawPointers xml attribute to ignore specific pointers and keep them raw

* add tools/extract_assets.sh

* fixups

* only extract if xmls changed or if -f (force) is used

* fixups, gc-eu OK

* all versions OK

* check attributes of xml resources elements

* Implement legacy skelanime resources

* fix ASSET_FILES_BIN_EXTRACTED/COMMITTED: look for .u8.bin specifically instead of just .bin

* implement JFIFResource

* fix png/jpg wildcards: look specifically for .u64.png .u32.png .u64.jpg

* Makefile: Add rules to build .png, .bin and .jpg in assets/ too

* start writing actual docs

* extract sTransCircleDL and sTransWipeDL

* misc cleanup/fixes, pygfxd 1.0.3

* refactor CDataExt.set_write callback args to use a dataclass

* Move {} to in-source

* misc

* more progress on spec

* fix missing braces in n64dd_error_textures.c

* finish xml spec doc

* assets xmls fixes

* some cleanup, use `gNameTex_WIDTH/HEIGHT` macros in dlists

* handle hackmode_syotes_room, fix compile

* C build_from_png

* rm tools/assets/bin2c

* rm ZAPD

* format

* remove rule to generate dmadata_table.py

* CC0 license (and some import cleanup)

* dont try to build zapd (rmd)

* simplify palettes with single user (ci images with a non-shared palette)

* add docs on how images are handled

* bss

* allow -j N

* fix n64texconv python bindings memory management

* move -j at the end of calling extraction script

* with -j, update last_extracts.json as each job completes rather than only if all complete

* make interrupting less jank by making child processes ignore sigint

* use enum names in `SCENE_CMD_SKYBOX_SETTINGS`

* `multiprocessing.get_context("fork")`

* import rich, except ImportError s

* fix optional rich usage

* .bss

* .bss

* .bss

* assets extraction: -j -> -j$(N_THREADS)

* .bss

* change LIMB_NONE/MAX defaults to be FILE_OFFSET instead of SKELNAME

* 0XHEX -> 0xHEX

* fix bss

* Proper includes for assets

mostly proper, some includes like dlists resources always causing a sys_matrix.h include (when not every dlist references gIdentityMtx) could be done better

* rm z64.h

* rm z64.h take two

* bss

* Make .u64 suffix for pngs optional

* fixup: rm .u64 suffix from n64dd image paths

* Remove elemtype suffixes from .bin and .jpg files

* Update images.md

* some build_from_png cleanup, more error handling, comments

* Handle skybox textures

Introduce "sub-format" suffix for pngs, with sub-formats split_lo and split_hi being used for skybox textures

* fixup for older python

* improve collision output some

* fully use SURFACETYPE[01] macros in writing extracted surface types

* use WATERBOX_PROPERTIES in extracted waterboxes

* some SceneCommandsResource cleanup

* format EnvLightSettingsList output
This commit is contained in:
Dragorn421 2025-05-18 01:29:09 +02:00 • committed by GitHub
parent 0c6c112cb9
commit 1e556e3a3d
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
460 changed files with 14342 additions and 48656 deletions
Download patch file Download diff file Expand all files Collapse all files

1
tools/assets/descriptor/README.md Normal file
View file

@ -0,0 +1 @@
This package serves as an abstraction layer wrapping assets xml files.

32
tools/assets/descriptor/__main__.py Normal file
View file

@ -0,0 +1,32 @@
# SPDX-FileCopyrightText: © 2025 ZeldaRET
# SPDX-License-Identifier: CC0-1.0
from pprint import pprint as vanilla_pprint
try:
from rich.pretty import pprint
except ImportError:
pprint = vanilla_pprint
from tools import version_config
from . import base
def main():
vc = version_config.load_version_config("gc-eu-mq-dbg")
pools = base.get_resources_desc(vc)
try:
for pool in pools:
if any(coll.out_path.name == "gameplay_keep" for coll in pool.collections):
vanilla_pprint(pool)
else:
pprint(pool)
input("Press enter for next pool")
except KeyboardInterrupt:
print()
main()

318
tools/assets/descriptor/base.py Normal file
View file

@ -0,0 +1,318 @@
# SPDX-FileCopyrightText: © 2025 ZeldaRET
# SPDX-License-Identifier: CC0-1.0
import abc
import dataclasses
from functools import cache
from pathlib import Path
from typing import Callable, Optional
from xml.etree import ElementTree
from tools import version_config
class BackingMemory(abc.ABC):
pass
@dataclasses.dataclass
class BaseromFileBackingMemory(BackingMemory):
name: str
range: Optional[tuple[int, int]]
"""If set, consider file_data[range[0]:range[1]] instead of the full file"""
@dataclasses.dataclass
class NoBackingMemory(BackingMemory):
pass
# eq=False so this uses id-based equality and hashing
# Subclasses must also be made to use id-based equality and hashing
@dataclasses.dataclass(eq=False)
class ResourceDesc(abc.ABC):
"""A resource is a data unit.
For example, a symbol's data such as a DList or a texture."""
symbol_name: str
offset: int
"""How many bytes into the backing memory the resource is located at"""
collection: "ResourcesDescCollection" = dataclasses.field(repr=False)
origin: object
"""opaque object with data about where this resource comes from (for debugging)"""
hack_modes: set[str] = dataclasses.field(init=False, default_factory=set)
class StartAddress(abc.ABC):
pass
@dataclasses.dataclass
class VRAMStartAddress(StartAddress):
vram: int
@dataclasses.dataclass
class SegmentStartAddress(StartAddress):
segment: int
@dataclasses.dataclass(eq=False)
class ResourcesDescCollection:
"""A collection is a list of resources backed by the same memory."""
out_path: Path
backing_memory: BackingMemory
start_address: Optional[StartAddress]
resources: list[ResourceDesc]
last_modified_time: float
depends: list["ResourcesDescCollection"]
@dataclasses.dataclass(eq=False)
class ResourcesDescCollectionsPool:
"""A pool contains a minimal set of interconnected collections.
For example, gkeep and all files using gkeep,
or more simply a single collection with no connection."""
collections: list[ResourcesDescCollection]
ResourceHandlerPass2Callback = Callable[[ResourcesDescCollectionsPool], None]
@dataclasses.dataclass
class ResourceHandlerNeedsPass2Exception(Exception):
resource: ResourceDesc
pass2_callback: ResourceHandlerPass2Callback
# eq=False so this uses id-based equality and hashing
@dataclasses.dataclass(eq=False)
class AssetConfigPiece:
ac: version_config.AssetConfig
last_modified_time: float = None
etree: ElementTree.ElementTree = None
depends: list["AssetConfigPiece"] = dataclasses.field(default_factory=list)
"""The AssetConfigPiece s this instance depends on"""
collections: list[ResourcesDescCollection] = dataclasses.field(default_factory=list)
def get_resources_desc(vc: version_config.VersionConfig):
# Wrap AssetConfig objects in AssetConfigPiece for hashability and to collect data
acps = [AssetConfigPiece(ac) for ac in vc.assets]
# Parse xmls
for acp in acps:
acp.last_modified_time = acp.ac.xml_path.stat().st_mtime
try:
with acp.ac.xml_path.open(encoding="UTF-8") as f:
etree = ElementTree.parse(f)
acp.etree = etree
except Exception as e:
raise Exception(f"Error when parsing XML for {acp}") from e
# Resolve pools
acp_by_name = {acp.ac.name: acp for acp in acps}
pools = {acp: {acp} for acp in acps}
for acp in acps:
try:
rootelem = acp.etree.getroot()
assert rootelem.tag == "Root", rootelem.tag
for fileelem in rootelem:
assert fileelem.tag in {"ExternalFile", "File"}, fileelem.tag
if fileelem.tag == "ExternalFile":
externalfile_name = str(
Path(fileelem.attrib["OutPath"]).relative_to("assets")
)
assert externalfile_name in acp_by_name, externalfile_name
externalfile_acp = acp_by_name[externalfile_name]
acp.depends.append(externalfile_acp)
acp_pool = pools[acp]
externalfile_acp_pool = pools[externalfile_acp]
merged_pool = acp_pool | externalfile_acp_pool
for merged_pool_acp in merged_pool:
pools[merged_pool_acp] = merged_pool
except Exception as e:
raise Exception(f"Error while resolving pools with {acp}") from e
# List unique pools
pools_unique: list[set[AssetConfigPiece]] = []
while pools:
pool = next(iter(pools.values()))
pools_unique.append(pool)
for acp in pool:
del pools[acp]
# Build resources for all pools
pools: list[ResourcesDescCollectionsPool] = []
for pool in pools_unique:
try:
all_needs_pass2_exceptions: list[ResourceHandlerNeedsPass2Exception] = []
rescolls: list[ResourcesDescCollection] = []
# Pass 1: create Resource objects
for acp in pool:
try:
rootelem = acp.etree.getroot()
for fileelem in rootelem:
if fileelem.tag == "File":
rc, needs_pass2_exceptions = (
_get_resources_fileelem_to_resourcescollection_pass1(
vc, pool, acp, fileelem
)
)
acp.collections.append(rc)
rescolls.append(rc)
all_needs_pass2_exceptions.extend(needs_pass2_exceptions)
except Exception as e:
raise Exception(f"Error with {acp}") from e
rcpool = ResourcesDescCollectionsPool(rescolls)
#
for acp in pool:
for acp_coll in acp.collections:
acp_coll.depends.extend(
(_coll for _coll in acp.collections if _coll != acp_coll)
)
for acp_dep in acp.depends:
acp_coll.depends.extend(acp_dep.collections)
# Pass 2: execute callbacks
for needs_pass2_exc in all_needs_pass2_exceptions:
try:
needs_pass2_exc.pass2_callback(rcpool)
except Exception as e:
raise Exception(
f"Error with pass 2 callback for {needs_pass2_exc.resource}"
) from e
pools.append(rcpool)
except Exception as e:
raise Exception(f"Error with pool {pool}") from e
return pools
def _get_resources_fileelem_to_resourcescollection_pass1(
vc: version_config.VersionConfig,
pool: list[AssetConfigPiece],
acp: AssetConfigPiece,
fileelem: ElementTree.Element,
):
# Determine backing_memory
if acp.ac.start_offset is None:
assert acp.ac.end_offset is None
baserom_file_range = None
else:
assert acp.ac.end_offset is not None
baserom_file_range = (acp.ac.start_offset, acp.ac.end_offset)
backing_memory = BaseromFileBackingMemory(
name=fileelem.attrib["Name"],
range=baserom_file_range,
)
# Determine start_address
if any(
acp.ac.name.startswith(_prefix) for _prefix in ("code/", "n64dd/", "overlays/")
):
# File start address is vram
assert "Segment" not in fileelem.attrib
assert acp.ac.start_offset is not None and acp.ac.end_offset is not None, (
"Unsupported combination: "
f"start/end offset not in config for vram asset {acp.ac.name}"
)
if acp.ac.name.startswith("overlays/"):
overlay_name = acp.ac.name.split("/")[1]
start_address = VRAMStartAddress(
vc.dmadata_segments[overlay_name].vram + acp.ac.start_offset
)
else:
file_name = acp.ac.name.split("/")[0] # "code" or "n64dd"
start_address = VRAMStartAddress(
vc.dmadata_segments[file_name].vram + acp.ac.start_offset
)
elif "Segment" in fileelem.attrib:
# File start address is a segmented address
assert acp.ac.start_offset is None and acp.ac.end_offset is None, (
"Unsupported combination: "
"start/end offset in config and file starts at a segmented address"
)
start_address = SegmentStartAddress(int(fileelem.attrib["Segment"]))
else:
# File does not have a start address
start_address = None
# resources
resources: list[ResourceDesc] = []
collection = ResourcesDescCollection(
Path(acp.ac.name),
backing_memory,
start_address,
resources,
acp.last_modified_time,
[],
)
needs_pass2_exceptions: list[ResourceHandlerNeedsPass2Exception] = []
for reselem in fileelem:
try:
symbol_name = reselem.attrib["Name"]
offset = int(reselem.attrib["Offset"], 16)
res_handler = _get_resource_handler(reselem.tag)
try:
res = res_handler(symbol_name, offset, collection, reselem)
except ResourceHandlerNeedsPass2Exception as needs_pass2_exc:
res = needs_pass2_exc.resource
needs_pass2_exceptions.append(needs_pass2_exc)
assert isinstance(res, ResourceDesc)
resources.append(res)
except Exception as e:
raise Exception(
"Error with resource element:\n"
+ ElementTree.tostring(reselem, encoding="unicode")
) from e
return collection, needs_pass2_exceptions
ResourceHandler = Callable[
[str, int, ResourcesDescCollection, ElementTree.Element],
ResourceDesc,
]
@cache
def _get_resource_handler(tag: str) -> ResourceHandler:
from . import n64resources
from . import z64resources
resource_handlers = {
"DList": n64resources.handler_DList,
"Blob": n64resources.handler_Blob,
"Mtx": n64resources.handler_Mtx,
"Array": n64resources.handler_Array,
"Texture": n64resources.handler_Texture,
"Collision": z64resources.handler_Collision,
"Animation": z64resources.handler_Animation,
"PlayerAnimation": z64resources.handler_PlayerAnimation,
"LegacyAnimation": z64resources.handler_LegacyAnimation,
"Cutscene": z64resources.handler_Cutscene,
"Scene": z64resources.handler_Scene,
"Room": z64resources.handler_Room,
"PlayerAnimationData": z64resources.handler_PlayerAnimationData,
"Path": z64resources.handler_PathList,
"Skeleton": z64resources.handler_Skeleton,
"Limb": z64resources.handler_Limb,
"CurveAnimation": z64resources.handler_CurveAnimation,
"LimbTable": z64resources.handler_LimbTable,
}
rh = resource_handlers.get(tag)
if rh is None:
raise Exception(f"Unknown resource tag {tag}")
else:
return rh

241
tools/assets/descriptor/n64resources.py Normal file
View file

@ -0,0 +1,241 @@
# SPDX-FileCopyrightText: © 2025 ZeldaRET
# SPDX-License-Identifier: CC0-1.0
import dataclasses
import enum
from xml.etree.ElementTree import Element
from ..n64 import G_IM_FMT, G_IM_SIZ
from .base import (
ResourceDesc,
ResourcesDescCollection,
ResourcesDescCollectionsPool,
ResourceHandlerNeedsPass2Exception,
BaseromFileBackingMemory,
)
from . import xml_errors
# TODO remove
STATIC_ATTRIB = {"Static"}
class GfxMicroCode(enum.Enum):
F3DEX = enum.auto()
F3DEX2 = enum.auto()
@dataclasses.dataclass(eq=False)
class DListResourceDesc(ResourceDesc):
ucode: GfxMicroCode
raw_pointers: set[int] = dataclasses.field(default_factory=set)
"""Pointers in the dlist that are fine to keep raw ("in hex") instead of using symbols"""
def handler_DList(symbol_name, offset, collection, reselem: Element):
xml_errors.check_attrib(
reselem, {"Name", "Offset"}, {"Ucode", "RawPointers"} | STATIC_ATTRIB
)
if "Ucode" in reselem.attrib:
ucode = GfxMicroCode[reselem.attrib["Ucode"].upper()]
else:
ucode = GfxMicroCode.F3DEX2
res = DListResourceDesc(symbol_name, offset, collection, reselem, ucode)
raw_pointers_str = reselem.attrib.get("RawPointers")
if raw_pointers_str:
for rp_str in raw_pointers_str.split(","):
res.raw_pointers.add(int(rp_str, 16))
return res
@dataclasses.dataclass(eq=False)
class BlobResourceDesc(ResourceDesc):
size: int
def handler_Blob(symbol_name, offset, collection, reselem: Element):
xml_errors.check_attrib(reselem, {"Name", "Offset", "Size"}, STATIC_ATTRIB)
size = int(reselem.attrib["Size"], 16)
return BlobResourceDesc(symbol_name, offset, collection, reselem, size)
@dataclasses.dataclass(eq=False)
class MtxResourceDesc(ResourceDesc):
pass
def handler_Mtx(symbol_name, offset, collection, reselem: Element):
xml_errors.check_attrib(reselem, {"Name", "Offset"}, STATIC_ATTRIB)
return MtxResourceDesc(symbol_name, offset, collection, reselem)
@dataclasses.dataclass(eq=False)
class S16ArrayResourceDesc(ResourceDesc):
count: int
@dataclasses.dataclass(eq=False)
class Vec3sArrayResourceDesc(ResourceDesc):
count: int
@dataclasses.dataclass(eq=False)
class VtxArrayResourceDesc(ResourceDesc):
count: int
def handler_Array(symbol_name, offset, collection, reselem: Element):
xml_errors.check_attrib(reselem, {"Name", "Offset", "Count"}, STATIC_ATTRIB)
count = int(reselem.attrib["Count"])
assert len(reselem) == 1, "Expected exactly one child of Array node"
array_elem = reselem[0]
if array_elem.tag == "Vtx":
array_resource_type = VtxArrayResourceDesc
elif (
array_elem.tag == "Vector"
and array_elem.attrib["Type"] == "s16"
and int(array_elem.attrib["Dimensions"]) == 3
):
array_resource_type = Vec3sArrayResourceDesc
elif array_elem.tag == "Scalar" and array_elem.attrib["Type"] == "s16":
array_resource_type = S16ArrayResourceDesc
else:
raise NotImplementedError(f"Array of {array_elem.tag}")
return array_resource_type(symbol_name, offset, collection, reselem, count)
class TextureFormat(enum.Enum):
RGBA16 = (G_IM_FMT.RGBA, G_IM_SIZ._16b)
RGBA32 = (G_IM_FMT.RGBA, G_IM_SIZ._32b)
CI4 = (G_IM_FMT.CI, G_IM_SIZ._4b)
CI8 = (G_IM_FMT.CI, G_IM_SIZ._8b)
I4 = (G_IM_FMT.I, G_IM_SIZ._4b)
I8 = (G_IM_FMT.I, G_IM_SIZ._8b)
IA4 = (G_IM_FMT.IA, G_IM_SIZ._4b)
IA8 = (G_IM_FMT.IA, G_IM_SIZ._8b)
IA16 = (G_IM_FMT.IA, G_IM_SIZ._16b)
def __init__(self, fmt: G_IM_FMT, siz: G_IM_SIZ):
self.fmt = fmt
self.siz = siz
@dataclasses.dataclass(eq=False)
class TextureResourceDesc(ResourceDesc):
format: TextureFormat
width: int
height: int
@dataclasses.dataclass(eq=False)
class CITextureResourceDesc(TextureResourceDesc):
tlut: TextureResourceDesc
def handler_Texture(
symbol_name, offset, collection: ResourcesDescCollection, reselem: Element
):
xml_errors.check_attrib(
reselem,
{"Name", "Offset", "Format", "Width", "Height"},
# TODO remove OutName, SplitTlut
{
"OutName",
"SplitTlut",
"TlutOffset",
"ExternalTlut",
"ExternalTlutOffset",
"HackMode",
}
| STATIC_ATTRIB,
)
format = TextureFormat[reselem.attrib["Format"].upper()]
width = int(reselem.attrib["Width"])
height = int(reselem.attrib["Height"])
if format.fmt == G_IM_FMT.CI:
res = CITextureResourceDesc(
symbol_name, offset, collection, reselem, format, width, height, None
)
if reselem.attrib.get("SplitTlut") == "true":
res.hack_modes.add("hackmode_split_tlut_true")
if reselem.attrib.get("SplitTlut") == "false":
res.hack_modes.add("hackmode_split_tlut_false")
assert (
"TlutOffset" in reselem.attrib or "ExternalTlutOffset" in reselem.attrib
), f"CI texture {symbol_name} is missing a tlut offset"
if "TlutOffset" in reselem.attrib:
xml_errors.check_attrib(
reselem,
{"Name", "Offset", "Format", "Width", "Height", "TlutOffset"},
# TODO remove OutName, SplitTlut
{"OutName", "SplitTlut", "HackMode"} | STATIC_ATTRIB,
)
tlut_offset = int(reselem.attrib["TlutOffset"], 16)
def pass2_callback(pool: ResourcesDescCollectionsPool):
matching_tlut_resources = [
res for res in collection.resources if res.offset == tlut_offset
]
assert len(matching_tlut_resources) == 1, (
f"Found {len(matching_tlut_resources)} resources at TlutOffset "
f"0x{tlut_offset:X} instead of exactly one"
)
assert isinstance(
matching_tlut_resources[0], TextureResourceDesc
), matching_tlut_resources[0]
res.tlut = matching_tlut_resources[0]
else:
xml_errors.check_attrib(
reselem,
{
"Name",
"Offset",
"Format",
"Width",
"Height",
"ExternalTlut",
"ExternalTlutOffset",
},
# TODO remove OutName, SplitTlut
{"OutName", "SplitTlut", "HackMode"} | STATIC_ATTRIB,
)
external_tlut_file = reselem.attrib["ExternalTlut"]
external_tlut_offset = int(reselem.attrib["ExternalTlutOffset"], 16)
def pass2_callback(pool: ResourcesDescCollectionsPool):
matching_collections = [
coll
for coll in pool.collections
if isinstance(coll.backing_memory, BaseromFileBackingMemory)
and coll.backing_memory.name == external_tlut_file
]
assert len(matching_collections) == 1
matching_tlut_resources = [
res
for res in matching_collections[0].resources
if res.offset == external_tlut_offset
]
assert len(matching_tlut_resources) == 1, matching_tlut_resources
assert isinstance(
matching_tlut_resources[0], TextureResourceDesc
), matching_tlut_resources[0]
res.tlut = matching_tlut_resources[0]
raise ResourceHandlerNeedsPass2Exception(res, pass2_callback)
else:
xml_errors.check_attrib(
reselem,
{"Name", "Offset", "Format", "Width", "Height"},
# TODO remove OutName
{"OutName", "HackMode"} | STATIC_ATTRIB,
)
res = TextureResourceDesc(
symbol_name, offset, collection, reselem, format, width, height
)
if reselem.attrib.get("HackMode") == "ignore_orphaned_tlut":
res.hack_modes.add("hackmode_ignore_orphaned_tlut")
return res

291
tools/assets/descriptor/spec.md Normal file
View file

@ -0,0 +1,291 @@
This document describes the expected structure of xml files describing assets.
# Top elements
## `Root`
```xml
<Root>
...
</Root>
```
This is the root element in the file, containing exclusively `<File>` and `<ExternalFile>` elements as direct children.
## `File`
```xml
<File Name="baserom_file" Segment="11">
...
</File>
```
A `<File>` contains resources elements as children.
- Required attributes: `Name`
- Optional attributes: `Segment`
`Name` is the name of the baserom file from which the data is to be extracted.
`Segment` (decimal) is the segment number for the file.
## `ExternalFile`
```xml
<ExternalFile OutPath="assets/name"/>
```
Declare the `<File>`s in the xml may reference symbols from an external file.
The external file is located by matching its name against the list of assets in the version's `config.yml`.
For example, `baseroms/gc-eu/config.yml` contains
```yml
assets:
- name: objects/gameplay_keep
xml_path: assets/xml/objects/gameplay_keep_pal.xml
```
then `<ExternalFile OutPath="assets/objects/gameplay_keep/"/>` refers to that gameplay_keep entry, which uses the `gameplay_keep_pal.xml` xml file when extracting assets for version gc-eu.
# Resource elements
Resource elements describe resources. Resources are pieces of data corresponding to a symbol each.
Two attributes are required on all resource elements: `Name` and `Offset`.
- `Name` is the name of the symbol associated to the resource.
- `Offset` is the location in bytes from the start of the file data.
## `Blob`
```xml
<Blob Name="gNameBlob" Size="0x421" Offset="0x1230">
```
Unstructured binary data.
- Required attributes: `Size`
`Size` is the size of the binary blob in bytes.
## `DList`
```xml
<DList Name="gNameDL" Offset="0x1230" Ucode="f3dex2" RawPointers="0x08000000,0x09000000"/>
```
A display list.
- Optional attributes: `Ucode`, `RawPointers`
`Ucode` (defaults to `f3dex2`) picks the graphics microcode for which to disassemble the dlist. It may be `f3dex` or `f3dex2`.
`RawPointers` (defaults to an empty value) is a comma-separated list of values the display list uses as raw pointers ("hex" instead of a symbol). The purpose of this attribute is to silence extraction warnings.
## `Mtx`
```xml
<Mtx Name="gNameMtx" Offset="0x1230"/>
```
A fixed-point matrix.
## `Texture`
```xml
<Texture Name="gNameTex" Format="rgba16" Width="16" Height="16" Offset="0x1230"/>
<Texture Name="gNameTex" Format="ci8" Width="16" Height="16" Offset="0x1230" TlutOffset="0x2340"/>
<Texture Name="gNameTex" Format="ci8" Width="16" Height="16" Offset="0x1230" ExternalTlut="baserom_file" ExternalTlutOffset="0x2340"/>
```
A texture, an image in one of the native N64 formats.
- Required attributes for all formats: `Format`, `Width`, `Height`
- Required attributes for CI formats (`ci4`, `ci8`): `TlutOffset`, or `ExternalTlut` and `ExternalTlutOffset`
`Format` is the format of the texture, one of `rgba32`, `rgba16`, `i4`, `i8`, `ia4`, `ia8`, `ia16`, `ci4` or `ci8`.
`Width` and `Height` specify the dimensions of the texture.
For CI formats, the TLUT (Texture Look Up Table, or palette) must be specified with either `TlutOffset` if the TLUT is in the same file as the texture, or both of `ExternalTlut` and `ExternalTlutOffset` if the TLUT is in a different file. `ExternalTlut` is the name of the baserom file where the TLUT is. In both cases, the TLUT must also be declared as a resource.
## `Array`
```xml
<Array Name="gNameVtx" Count="42" Offset="0x1230">
<Vtx/>
</Array>
<Array Name="gNameVec3sArray" Count="42" Offset="0x1230">
<Vector Type="s16" Dimensions="3"/>
</Array>
<Array Name="gNameS16Array" Count="42" Offset="0x1230">
<Scalar Type="s16"/>
</Array>
```
An array of vertices, vectors or scalars. The child element determines the array's element type.
- Required attributes: `Count`
`Count` is the length of the array.
The child element may be one of `<Vtx/>` (for `Vtx[]`), `<Vector Type="s16" Dimensions="3"/>` (for `Vec3s[]`) or `<Scalar Type="s16">` (for `s16[]`).
## `Scene`
```xml
<Scene Name="name" Offset="0x1230"/>
```
Scene commands.
## `Room`
```xml
<Room Name="name" Offset="0x1230"/>
```
Room commands.
## `Collision`
```xml
<Collision Name="gNameCol" Offset="0x1230"/>
```
Collision header.
## `Cutscene`
```xml
<Cutscene Name="gNameCs" Offset="0x1230"/>
```
Cutscene script.
## `Path`
```xml
<Path Name="gNamePathList" Offset="0x1230" NumPaths="12"/>
```
Path list.
- Required attributes: `NumPaths`
`NumPaths` is the length of the path list.
## `Skeleton`
```xml
<Skeleton Name="gNameSkel" Type="Normal" LimbType="Standard" LimbNone="NAME_LIMB_NONE" LimbMax="NAME_LIMB_MAX" EnumName="NameLimb" Offset="0x1230"/>
```
Skeleton header.
- Required attributes: `Type`, `LimbType`
- Optional attributes: `LimbNone`, `LimbMax`, `EnumName`
`Type` is the type of the skeleton, one of `Normal`, `Flex` or `Curve`.
`LimbType` is the type of limb used in the skeleton, one of `Standard`, `LOD`, `Skin`, `Curve` or `Legacy`.
Not all skeleton types are compatible with all limb types:
`LimbType` | Compatible skeleton `Type`
-----------|---------------------------
`Standard` | `Normal`, `Flex`
`LOD` | `Normal`, `Flex`
`Skin` | `Normal`
`Curve` | `Curve`
`Legacy` | none
`LimbNone`, `LimbMax`, `EnumName` can be set to override the corresponding names in the generated limb enum:
```c
typedef enum NameLimb {
NAME_LIMB_NONE,
...
NAME_LIMB_MAX
} NameLimb;
```
## `LimbTable`
```xml
<LimbTable Name="gNameLimbs" LimbType="Standard" Count="12" Offset="0x1230"/>
```
Limb table.
- Required attributes: `LimbType`, `Count`
`LimbType` is one of `Standard`, `LOD`, `Skin`, `Curve` or `Legacy`.
`Count` is the amount of limbs.
## `Limb`
```xml
<Limb Name="gNameLimb" LimbType="Standard" EnumName="NAME_LIMB_NAME" Offset="0x1230"/>
```
Limb of a skeleton.
- Required attributes: `LimbType`
- Optional attributes: `EnumName`
`LimbType` is one of `Standard`, `LOD`, `Skin`, `Curve` or `Legacy`.
`EnumName` can be set to override the limb name in the generated limb enum.
## `Animation`
```xml
<Animation Name="gNameAnim" Offset="0x1230"/>
```
Animation header.
## `CurveAnimation`
```xml
<CurveAnimation Name="gNameAnim" SkelOffset="0x120" Offset="0x1230"/>
```
Curve animation header.
- Required attributes: `SkelOffset`
`SkelOffset` is the offset of the skeleton which uses this animation. The skeleton must also be declared as a resource.
## `LegacyAnimation`
```xml
<LegacyAnimation Name="gNameAnim" Offset="0x1230"/>
```
Legacy animation header.
## `PlayerAnimation`
```xml
<PlayerAnimation Name="gNamePlayerAnim" Offset="0x1230"/>
```
Player animation header.
## `PlayerAnimationData`
```xml
<PlayerAnimationData Name="gNamePlayerAnimData" FrameCount="20" Offset="0x1230"/>
```
Player animation data.
- Required attributes: `FrameCount`
`FrameCount` is the amount of frames in the animation.

31
tools/assets/descriptor/xml_errors.py Normal file
View file

@ -0,0 +1,31 @@
# SPDX-FileCopyrightText: © 2025 ZeldaRET
# SPDX-License-Identifier: CC0-1.0
from xml.etree import ElementTree
class XMLDescError(Exception):
pass
def check_attrib(
elem: ElementTree.Element,
required: set[str],
optional: set[str] = set(),
):
required_and_missing = required - elem.attrib.keys()
if required_and_missing:
raise XMLDescError(
"Missing attributes on "
+ ElementTree.tostring(elem, encoding="unicode")
+ ": "
+ ", ".join(required_and_missing)
)
unknown = elem.attrib.keys() - required - optional
if unknown:
raise XMLDescError(
"Unknown attributes on "
+ ElementTree.tostring(elem, encoding="unicode")
+ ": "
+ ", ".join(unknown)
)

218
tools/assets/descriptor/z64resources.py Normal file
View file

@ -0,0 +1,218 @@
# SPDX-FileCopyrightText: © 2025 ZeldaRET
# SPDX-License-Identifier: CC0-1.0
import dataclasses
import enum
from typing import Optional
from xml.etree.ElementTree import Element
from .base import (
ResourceDesc,
ResourcesDescCollection,
ResourceHandlerNeedsPass2Exception,
)
from . import xml_errors
@dataclasses.dataclass(eq=False)
class CollisionResourceDesc(ResourceDesc):
pass
def handler_Collision(symbol_name, offset, collection, reselem: Element):
xml_errors.check_attrib(reselem, {"Name", "Offset"})
return CollisionResourceDesc(symbol_name, offset, collection, reselem)
@dataclasses.dataclass(eq=False)
class AnimationResourceDesc(ResourceDesc):
pass
def handler_Animation(symbol_name, offset, collection, reselem: Element):
xml_errors.check_attrib(reselem, {"Name", "Offset"})
return AnimationResourceDesc(symbol_name, offset, collection, reselem)
@dataclasses.dataclass(eq=False)
class PlayerAnimationResourceDesc(ResourceDesc):
pass
def handler_PlayerAnimation(symbol_name, offset, collection, reselem: Element):
xml_errors.check_attrib(reselem, {"Name", "Offset"})
return PlayerAnimationResourceDesc(symbol_name, offset, collection, reselem)
@dataclasses.dataclass(eq=False)
class LegacyAnimationResourceDesc(ResourceDesc):
pass
def handler_LegacyAnimation(symbol_name, offset, collection, reselem: Element):
xml_errors.check_attrib(reselem, {"Name", "Offset"})
return LegacyAnimationResourceDesc(symbol_name, offset, collection, reselem)
@dataclasses.dataclass(eq=False)
class CutsceneResourceDesc(ResourceDesc):
pass
def handler_Cutscene(symbol_name, offset, collection, reselem: Element):
xml_errors.check_attrib(reselem, {"Name", "Offset"})
return CutsceneResourceDesc(symbol_name, offset, collection, reselem)
@dataclasses.dataclass(eq=False)
class SceneResourceDesc(ResourceDesc):
pass
def handler_Scene(symbol_name, offset, collection, reselem: Element):
xml_errors.check_attrib(reselem, {"Name", "Offset"})
return SceneResourceDesc(symbol_name, offset, collection, reselem)
@dataclasses.dataclass(eq=False)
class RoomResourceDesc(ResourceDesc):
pass
def handler_Room(symbol_name, offset, collection, reselem: Element):
xml_errors.check_attrib(reselem, {"Name", "Offset"}, {"HackMode"})
res = RoomResourceDesc(symbol_name, offset, collection, reselem)
if reselem.attrib.get("HackMode") == "syotes_room":
res.hack_modes.add("hackmode_syotes_room")
return res
@dataclasses.dataclass(eq=False)
class PlayerAnimationDataResourceDesc(ResourceDesc):
frame_count: int
def handler_PlayerAnimationData(symbol_name, offset, collection, reselem: Element):
xml_errors.check_attrib(reselem, {"Name", "Offset", "FrameCount"})
frame_count = int(reselem.attrib["FrameCount"])
return PlayerAnimationDataResourceDesc(
symbol_name, offset, collection, reselem, frame_count
)
@dataclasses.dataclass(eq=False)
class PathListResourceDesc(ResourceDesc):
num_paths: int
def handler_PathList(symbol_name, offset, collection, reselem: Element):
xml_errors.check_attrib(reselem, {"Name", "Offset", "NumPaths"})
num_paths = int(reselem.attrib["NumPaths"])
return PathListResourceDesc(symbol_name, offset, collection, reselem, num_paths)
class SkeletonType(enum.Enum):
NORMAL = enum.auto()
FLEX = enum.auto()
CURVE = enum.auto()
class LimbType(enum.Enum):
STANDARD = enum.auto()
LOD = enum.auto()
SKIN = enum.auto()
CURVE = enum.auto()
LEGACY = enum.auto()
@dataclasses.dataclass(eq=False)
class SkeletonResourceDesc(ResourceDesc):
type: SkeletonType
limb_type: LimbType
limb_enum_name: Optional[str]
limb_enum_none_member_name: Optional[str]
limb_enum_max_member_name: Optional[str]
def handler_Skeleton(symbol_name, offset, collection, reselem: Element):
xml_errors.check_attrib(
reselem,
{"Name", "Offset", "Type", "LimbType"},
{"EnumName", "LimbNone", "LimbMax"},
)
skel_type = SkeletonType[reselem.attrib["Type"].upper()]
limb_type = LimbType[reselem.attrib["LimbType"].upper()]
return SkeletonResourceDesc(
symbol_name,
offset,
collection,
reselem,
skel_type,
limb_type,
reselem.attrib.get("EnumName"),
reselem.attrib.get("LimbNone"),
reselem.attrib.get("LimbMax"),
)
@dataclasses.dataclass(eq=False)
class LimbResourceDesc(ResourceDesc):
limb_type: LimbType
limb_enum_member_name: Optional[str]
def handler_Limb(symbol_name, offset, collection, reselem: Element):
xml_errors.check_attrib(reselem, {"Name", "Offset", "LimbType"}, {"EnumName"})
limb_type = LimbType[reselem.attrib["LimbType"].upper()]
return LimbResourceDesc(
symbol_name,
offset,
collection,
reselem,
limb_type,
reselem.attrib.get("EnumName"),
)
@dataclasses.dataclass(eq=False)
class LimbTableResourceDesc(ResourceDesc):
limb_type: LimbType
count: int
def handler_LimbTable(symbol_name, offset, collection, reselem: Element):
xml_errors.check_attrib(reselem, {"Name", "Offset", "LimbType", "Count"})
limb_type = LimbType[reselem.attrib["LimbType"].upper()]
count = int(reselem.attrib["Count"])
return LimbTableResourceDesc(
symbol_name, offset, collection, reselem, limb_type, count
)
@dataclasses.dataclass(eq=False)
class CurveAnimationResourceDesc(ResourceDesc):
skeleton: SkeletonResourceDesc
def handler_CurveAnimation(
symbol_name, offset, collection: ResourcesDescCollection, reselem: Element
):
xml_errors.check_attrib(reselem, {"Name", "Offset", "SkelOffset"})
res = CurveAnimationResourceDesc(symbol_name, offset, collection, reselem, None)
skel_offset = int(reselem.attrib["SkelOffset"], 16)
def pass2_callback(pool):
matching_tlut_resources = [
res for res in collection.resources if res.offset == skel_offset
]
assert len(matching_tlut_resources) == 1, (
f"Found {len(matching_tlut_resources)} resources at SkelOffset "
f"0x{skel_offset:X} instead of exactly one"
)
assert isinstance(
matching_tlut_resources[0], SkeletonResourceDesc
), matching_tlut_resources[0]
res.skeleton = matching_tlut_resources[0]
raise ResourceHandlerNeedsPass2Exception(res, pass2_callback)