New assets system (#2481)

* wip: New assets system tm

Builds gc-eu-mq-dbg OK from clean after
1) make setup
2) python3 -m tools.assets.extract -j
3) replace 0x80A8E610 with sShadowTex in extracted/gc-eu-mq-dbg/assets/overlays/ovl_En_Jsjutan/sShadowMaterialDL.inc.c
4) make various symbols in extracted data like sTex static

* use variables from config.yml for gMtxClear and sShadowTex addresses

* Write source with static for overlays using `HACK_IS_STATIC_ON` hack

* gc-eu-mq-dbg OK from clean with `make setup && make`

* implement more skeleton-related types, cleanups, fixups

* fix extracted data to no longer produce compilation warnings

* implement more of RoomShapeImage types

* yeet XmlPath from ExternalFile usage

* Implement PlayerAnimationDataResource (link_animetion data)

* fix csdis CS_TIME extra arg

* dmadata file names no longer hardcoded for gc-eu-mq-dbg

* ntsc-1.0 OK

* xml fixes

* slightly improve standard output

* rm extract_assets.py

* generate and use Limb enums (TODO: check Skin skels and implement for Curve skels)

* handle dependencies between xmls

* introduce RawPointers xml attribute to ignore specific pointers and keep them raw

* add tools/extract_assets.sh

* fixups

* only extract if xmls changed or if -f (force) is used

* fixups, gc-eu OK

* all versions OK

* check attributes of xml resources elements

* Implement legacy skelanime resources

* fix ASSET_FILES_BIN_EXTRACTED/COMMITTED: look for .u8.bin specifically instead of just .bin

* implement JFIFResource

* fix png/jpg wildcards: look specifically for .u64.png .u32.png .u64.jpg

* Makefile: Add rules to build .png, .bin and .jpg in assets/ too

* start writing actual docs

* extract sTransCircleDL and sTransWipeDL

* misc cleanup/fixes, pygfxd 1.0.3

* refactor CDataExt.set_write callback args to use a dataclass

* Move {} to in-source

* misc

* more progress on spec

* fix missing braces in n64dd_error_textures.c

* finish xml spec doc

* assets xmls fixes

* some cleanup, use `gNameTex_WIDTH/HEIGHT` macros in dlists

* handle hackmode_syotes_room, fix compile

* C build_from_png

* rm tools/assets/bin2c

* rm ZAPD

* format

* remove rule to generate dmadata_table.py

* CC0 license (and some import cleanup)

* dont try to build zapd (rmd)

* simplify palettes with single user (ci images with a non-shared palette)

* add docs on how images are handled

* bss

* allow -j N

* fix n64texconv python bindings memory management

* move -j at the end of calling extraction script

* with -j, update last_extracts.json as each job completes rather than only if all complete

* make interrupting less jank by making child processes ignore sigint

* use enum names in `SCENE_CMD_SKYBOX_SETTINGS`

* `multiprocessing.get_context("fork")`

* import rich, except ImportError s

* fix optional rich usage

* .bss

* .bss

* .bss

* assets extraction: -j -> -j$(N_THREADS)

* .bss

* change LIMB_NONE/MAX defaults to be FILE_OFFSET instead of SKELNAME

* 0XHEX -> 0xHEX

* fix bss

* Proper includes for assets

mostly proper, some includes like dlists resources always causing a sys_matrix.h include (when not every dlist references gIdentityMtx) could be done better

* rm z64.h

* rm z64.h take two

* bss

* Make .u64 suffix for pngs optional

* fixup: rm .u64 suffix from n64dd image paths

* Remove elemtype suffixes from .bin and .jpg files

* Update images.md

* some build_from_png cleanup, more error handling, comments

* Handle skybox textures

Introduce "sub-format" suffix for pngs, with sub-formats split_lo and split_hi being used for skybox textures

* fixup for older python

* improve collision output some

* fully use SURFACETYPE[01] macros in writing extracted surface types

* use WATERBOX_PROPERTIES in extracted waterboxes

* some SceneCommandsResource cleanup

* format EnvLightSettingsList output

tools/assets/extract/extase_oot64/animation_resources.py

@@ -0,0 +1,205 @@
+# SPDX-FileCopyrightText: © 2025 ZeldaRET
+# SPDX-License-Identifier: CC0-1.0
+
+from typing import TYPE_CHECKING
+
+if TYPE_CHECKING:
+    from ..extase.memorymap import MemoryContext
+
+from ..extase import (
+    RESOURCE_PARSE_SUCCESS,
+    ResourceParseWaiting,
+    File,
+)
+from ..extase.cdata_resources import (
+    CDataResource,
+    CDataExt_Value,
+    CDataExt_Struct,
+    CDataExt_Array,
+    CDataExtWriteContext,
+)
+
+
+class AnimationFrameDataResource(CDataResource, can_size_be_unknown=True):
+    def write_binang(resource, memory_context, v, wctx: CDataExtWriteContext):
+        wctx.f.write(wctx.line_prefix)
+        wctx.f.write(f" 0x{v:04X}" if v >= 0 else "-0x" + f"{v:04X}".removeprefix("-"))
+        return True
+
+    elem_cdata_ext = CDataExt_Value("h").set_write(write_binang)
+
+    def __init__(self, file: File, range_start: int, name: str):
+        super().__init__(file, range_start, name)
+        self.length = None
+
+    def try_parse_data(self, memory_context):
+        if self.length is not None:
+            self.cdata_ext = CDataExt_Array(self.elem_cdata_ext, self.length)
+            self.range_end = self.range_start + self.cdata_ext.size
+            return super().try_parse_data(memory_context)
+        else:
+            raise ResourceParseWaiting(waiting_for=["self.length"])
+
+    def get_c_declaration_base(self):
+        return f"s16 {self.symbol_name}[]"
+
+    def get_c_reference(self, resource_offset: int):
+        if resource_offset == 0:
+            return self.symbol_name
+        else:
+            raise ValueError()
+
+    def get_h_includes(self):
+        return ("ultra64.h",)
+
+
+class AnimationJointIndicesResource(CDataResource, can_size_be_unknown=True):
+    elem_cdata_ext = CDataExt_Struct(
+        (
+            ("x", CDataExt_Value.u16),
+            ("y", CDataExt_Value.u16),
+            ("z", CDataExt_Value.u16),
+        )
+    )
+
+    def __init__(self, file: File, range_start: int, name: str):
+        super().__init__(file, range_start, name)
+        self.length = None
+
+    def try_parse_data(self, memory_context):
+        if self.length is not None:
+            self.cdata_ext = CDataExt_Array(self.elem_cdata_ext, self.length)
+            self.range_end = self.range_start + self.cdata_ext.size
+            return super().try_parse_data(memory_context)
+        else:
+            raise ResourceParseWaiting(waiting_for=["self.length"])
+
+    def get_c_declaration_base(self):
+        return f"JointIndex {self.symbol_name}[]"
+
+    def get_c_reference(self, resource_offset: int):
+        if resource_offset == 0:
+            return self.symbol_name
+        else:
+            raise ValueError()
+
+    def get_h_includes(self):
+        return ("z64animation.h",)
+
+
+class AnimationResource(CDataResource):
+    def write_frameData(
+        resource, memory_context: "MemoryContext", v, wctx: CDataExtWriteContext
+    ):
+        assert isinstance(v, int)
+        address = v
+        wctx.f.write(wctx.line_prefix)
+        wctx.f.write(memory_context.get_c_reference_at_segmented(address))
+        return True
+
+    def write_jointIndices(
+        resource, memory_context: "MemoryContext", v, wctx: CDataExtWriteContext
+    ):
+        assert isinstance(v, int)
+        address = v
+        wctx.f.write(wctx.line_prefix)
+        wctx.f.write(memory_context.get_c_reference_at_segmented(address))
+        return True
+
+    cdata_ext = CDataExt_Struct(
+        (
+            (
+                "common",
+                CDataExt_Struct((("frameCount", CDataExt_Value.s16),)),
+            ),
+            ("pad2", CDataExt_Value.pad16),
+            (
+                "frameData",
+                CDataExt_Value("I").set_write(write_frameData),
+            ),
+            (
+                "jointIndices",
+                CDataExt_Value("I").set_write(write_jointIndices),
+            ),
+            ("staticIndexMax", CDataExt_Value.u16),
+            ("padE", CDataExt_Value.pad16),
+        )
+    )
+
+    def try_parse_data(self, memory_context):
+        super().try_parse_data(memory_context)
+
+        frameData_address = self.cdata_unpacked["frameData"]
+        assert isinstance(frameData_address, int)
+        resource_frameData = memory_context.report_resource_at_segmented(
+            self,
+            frameData_address,
+            AnimationFrameDataResource,
+            lambda file, offset: AnimationFrameDataResource(
+                file,
+                offset,
+                f"{self.name}_{frameData_address:08X}_FrameData",
+            ),
+        )
+
+        jointIndices_address = self.cdata_unpacked["jointIndices"]
+        assert isinstance(jointIndices_address, int)
+        resource_jointIndices = memory_context.report_resource_at_segmented(
+            self,
+            jointIndices_address,
+            AnimationJointIndicesResource,
+            lambda file, offset: AnimationJointIndicesResource(
+                file,
+                offset,
+                f"{self.name}_{jointIndices_address:08X}_JointIndices",
+            ),
+        )
+
+        # The length of the frameData and jointIndices arrays is
+        # for now assumed to fill the space to the animation,
+        # at the very least before subtracting the offsets check that
+        # the offsets belong to the same file
+        # TODO better idea for computing this data's size
+
+        if not (resource_frameData.file == resource_jointIndices.file == self.file):
+            raise NotImplementedError(
+                "Expected frameData and jointIndices to be in the same file as the animation",
+                self.cdata_unpacked,
+                resource_frameData.file,
+                resource_jointIndices.file,
+                self.file,
+            )
+
+        if (
+            resource_frameData.range_start
+            < resource_jointIndices.range_start
+            < self.range_start
+        ):
+            resource_frameData.length = (
+                resource_jointIndices.range_start - resource_frameData.range_start
+            ) // AnimationFrameDataResource.elem_cdata_ext.size
+            resource_jointIndices.length = (
+                self.range_start - resource_jointIndices.range_start
+            ) // AnimationJointIndicesResource.elem_cdata_ext.size
+        else:
+            raise NotImplementedError(
+                "Expected offsets of frameData, jointIndices, animation to be in order",
+                self.cdata_unpacked,
+                hex(resource_frameData.range_start),
+                hex(resource_jointIndices.range_start),
+                hex(self.range_start),
+            )
+
+        return RESOURCE_PARSE_SUCCESS
+
+    def get_c_reference(self, resource_offset: int):
+        if resource_offset == 0:
+            return f"&{self.symbol_name}"
+        else:
+            raise ValueError()
+
+    def get_c_declaration_base(self):
+        return f"AnimationHeader {self.symbol_name}"
+
+    def get_h_includes(self):
+        return ("z64animation.h",)