mirror/oot
1
0
Fork 0
mirror of https://github.com/zeldaret/oot.git synced 2025-08-08 15:30:14 +00:00
Code Issues Projects Releases Wiki Activity

New assets system (#2481)

Browse source 
* wip: New assets system tm

Builds gc-eu-mq-dbg OK from clean after
1) make setup
2) python3 -m tools.assets.extract -j
3) replace 0x80A8E610 with sShadowTex in extracted/gc-eu-mq-dbg/assets/overlays/ovl_En_Jsjutan/sShadowMaterialDL.inc.c
4) make various symbols in extracted data like sTex static

* use variables from config.yml for gMtxClear and sShadowTex addresses

* Write source with static for overlays using `HACK_IS_STATIC_ON` hack

* gc-eu-mq-dbg OK from clean with `make setup && make`

* implement more skeleton-related types, cleanups, fixups

* fix extracted data to no longer produce compilation warnings

* implement more of RoomShapeImage types

* yeet XmlPath from ExternalFile usage

* Implement PlayerAnimationDataResource (link_animetion data)

* fix csdis CS_TIME extra arg

* dmadata file names no longer hardcoded for gc-eu-mq-dbg

* ntsc-1.0 OK

* xml fixes

* slightly improve standard output

* rm extract_assets.py

* generate and use Limb enums (TODO: check Skin skels and implement for Curve skels)

* handle dependencies between xmls

* introduce RawPointers xml attribute to ignore specific pointers and keep them raw

* add tools/extract_assets.sh

* fixups

* only extract if xmls changed or if -f (force) is used

* fixups, gc-eu OK

* all versions OK

* check attributes of xml resources elements

* Implement legacy skelanime resources

* fix ASSET_FILES_BIN_EXTRACTED/COMMITTED: look for .u8.bin specifically instead of just .bin

* implement JFIFResource

* fix png/jpg wildcards: look specifically for .u64.png .u32.png .u64.jpg

* Makefile: Add rules to build .png, .bin and .jpg in assets/ too

* start writing actual docs

* extract sTransCircleDL and sTransWipeDL

* misc cleanup/fixes, pygfxd 1.0.3

* refactor CDataExt.set_write callback args to use a dataclass

* Move {} to in-source

* misc

* more progress on spec

* fix missing braces in n64dd_error_textures.c

* finish xml spec doc

* assets xmls fixes

* some cleanup, use `gNameTex_WIDTH/HEIGHT` macros in dlists

* handle hackmode_syotes_room, fix compile

* C build_from_png

* rm tools/assets/bin2c

* rm ZAPD

* format

* remove rule to generate dmadata_table.py

* CC0 license (and some import cleanup)

* dont try to build zapd (rmd)

* simplify palettes with single user (ci images with a non-shared palette)

* add docs on how images are handled

* bss

* allow -j N

* fix n64texconv python bindings memory management

* move -j at the end of calling extraction script

* with -j, update last_extracts.json as each job completes rather than only if all complete

* make interrupting less jank by making child processes ignore sigint

* use enum names in `SCENE_CMD_SKYBOX_SETTINGS`

* `multiprocessing.get_context("fork")`

* import rich, except ImportError s

* fix optional rich usage

* .bss

* .bss

* .bss

* assets extraction: -j -> -j$(N_THREADS)

* .bss

* change LIMB_NONE/MAX defaults to be FILE_OFFSET instead of SKELNAME

* 0XHEX -> 0xHEX

* fix bss

* Proper includes for assets

mostly proper, some includes like dlists resources always causing a sys_matrix.h include (when not every dlist references gIdentityMtx) could be done better

* rm z64.h

* rm z64.h take two

* bss

* Make .u64 suffix for pngs optional

* fixup: rm .u64 suffix from n64dd image paths

* Remove elemtype suffixes from .bin and .jpg files

* Update images.md

* some build_from_png cleanup, more error handling, comments

* Handle skybox textures

Introduce "sub-format" suffix for pngs, with sub-formats split_lo and split_hi being used for skybox textures

* fixup for older python

* improve collision output some

* fully use SURFACETYPE[01] macros in writing extracted surface types

* use WATERBOX_PROPERTIES in extracted waterboxes

* some SceneCommandsResource cleanup

* format EnvLightSettingsList output
This commit is contained in:
Dragorn421 2025-05-18 01:29:09 +02:00 • committed by GitHub
parent 0c6c112cb9
commit 1e556e3a3d
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
460 changed files with 14342 additions and 48656 deletions
Download patch file Download diff file Expand all files Collapse all files

378
tools/assets/extract/extase_oot64/skelcurve_resources.py Normal file
View file

@ -0,0 +1,378 @@
# SPDX-FileCopyrightText: © 2025 ZeldaRET
# SPDX-License-Identifier: CC0-1.0
from typing import TYPE_CHECKING
if TYPE_CHECKING:
from ..extase.memorymap import MemoryContext
from ..extase import (
File,
RESOURCE_PARSE_SUCCESS,
ResourceParseWaiting,
)
from ..extase.cdata_resources import (
CDataResource,
CDataExt_Struct,
CDataExt_Value,
CDataExt_Array,
CDataExtWriteContext,
)
from . import dlist_resources
class KnotCountsArrayResource(CDataResource, can_size_be_unknown=True):
elem_cdata_ext = CDataExt_Value.u8
def __init__(self, file: File, range_start: int, name: str):
super().__init__(file, range_start, name)
self.length = None
def try_parse_data(self, memory_context: "MemoryContext"):
if self.length is not None:
self.cdata_ext = CDataExt_Array(self.elem_cdata_ext, self.length)
self.range_end = self.range_start + self.cdata_ext.size
return super().try_parse_data(memory_context)
else:
raise ResourceParseWaiting(waiting_for=["self.length"])
def get_c_declaration_base(self):
return f"u8 {self.symbol_name}[]"
def get_c_reference(self, resource_offset: int):
if resource_offset == 0:
return self.symbol_name
else:
raise ValueError()
def get_h_includes(self):
return ("ultra64.h",)
class CurveInterpKnotArrayResource(CDataResource, can_size_be_unknown=True):
elem_cdata_ext = CDataExt_Struct(
(
("flags", CDataExt_Value.u16),
("abscissa", CDataExt_Value.s16),
("leftGradient", CDataExt_Value.s16),
("rightGradient", CDataExt_Value.s16),
("ordinate", CDataExt_Value.f32),
)
)
def __init__(self, file: File, range_start: int, name: str):
super().__init__(file, range_start, name)
self.length = None
def try_parse_data(self, memory_context: "MemoryContext"):
if self.length is not None:
self.cdata_ext = CDataExt_Array(self.elem_cdata_ext, self.length)
self.range_end = self.range_start + self.cdata_ext.size
return super().try_parse_data(memory_context)
else:
raise ResourceParseWaiting(waiting_for=["self.length"])
def get_c_declaration_base(self):
return f"CurveInterpKnot {self.symbol_name}[]"
def get_c_reference(self, resource_offset: int):
if resource_offset == 0:
return self.symbol_name
else:
raise ValueError()
def get_h_includes(self):
return ("z64curve.h",)
class ConstantDataArrayResource(CDataResource, can_size_be_unknown=True):
elem_cdata_ext = CDataExt_Value.s16
def __init__(self, file: File, range_start: int, name: str):
super().__init__(file, range_start, name)
self.length = None
def try_parse_data(self, memory_context: "MemoryContext"):
if self.length is not None:
self.cdata_ext = CDataExt_Array(self.elem_cdata_ext, self.length)
self.range_end = self.range_start + self.cdata_ext.size
return super().try_parse_data(memory_context)
else:
raise ResourceParseWaiting(waiting_for=["self.length"])
def get_c_declaration_base(self):
return f"s16 {self.symbol_name}[]"
def get_c_reference(self, resource_offset: int):
if resource_offset == 0:
return self.symbol_name
else:
raise ValueError()
def get_h_includes(self):
return ("ultra64.h",)
class CurveAnimationHeaderResource(CDataResource):
def report_knotCounts(resource, memory_context: "MemoryContext", v):
assert isinstance(v, int)
address = v
memory_context.report_resource_at_segmented(
resource,
address,
KnotCountsArrayResource,
lambda file, offset: KnotCountsArrayResource(
file, offset, f"{resource.name}_{address:08X}_KnotCounts"
),
)
def write_knotCounts(
resource, memory_context: "MemoryContext", v, wctx: CDataExtWriteContext
):
assert isinstance(v, int)
address = v
wctx.f.write(wctx.line_prefix)
wctx.f.write(memory_context.get_c_reference_at_segmented(address))
return True
def report_interpolationData(resource, memory_context: "MemoryContext", v):
assert isinstance(v, int)
address = v
memory_context.report_resource_at_segmented(
resource,
address,
CurveInterpKnotArrayResource,
lambda file, offset: CurveInterpKnotArrayResource(
file, offset, f"{resource.name}_{address:08X}_InterpolationData"
),
)
def write_interpolationData(
resource, memory_context: "MemoryContext", v, wctx: CDataExtWriteContext
):
assert isinstance(v, int)
address = v
wctx.f.write(wctx.line_prefix)
wctx.f.write(memory_context.get_c_reference_at_segmented(address))
return True
def report_constantData(resource, memory_context: "MemoryContext", v):
assert isinstance(v, int)
address = v
memory_context.report_resource_at_segmented(
resource,
address,
ConstantDataArrayResource,
lambda file, offset: ConstantDataArrayResource(
file, offset, f"{resource.name}_{address:08X}_ConstantData"
),
)
def write_constantData(
resource, memory_context: "MemoryContext", v, wctx: CDataExtWriteContext
):
assert isinstance(v, int)
address = v
wctx.f.write(wctx.line_prefix)
wctx.f.write(memory_context.get_c_reference_at_segmented(address))
return True
cdata_ext = CDataExt_Struct(
(
(
"knotCounts",
CDataExt_Value("I")
.set_report(report_knotCounts)
.set_write(write_knotCounts),
), # u8*
(
"interpolationData",
CDataExt_Value("I")
.set_report(report_interpolationData)
.set_write(write_interpolationData),
), # CurveInterpKnot*
(
"constantData",
CDataExt_Value("I")
.set_report(report_constantData)
.set_write(write_constantData),
), # s16*
("unk_0C", CDataExt_Value.s16),
("frameCount", CDataExt_Value.s16),
)
)
def try_parse_data(self, memory_context):
super().try_parse_data(memory_context)
knotCounts = self.cdata_unpacked["knotCounts"]
interpolationData = self.cdata_unpacked["interpolationData"]
constantData = self.cdata_unpacked["constantData"]
resource_knotCounts = memory_context.resolve_segmented(knotCounts).get_resource(
KnotCountsArrayResource
)
resource_interpolationData = memory_context.resolve_segmented(
interpolationData
).get_resource(CurveInterpKnotArrayResource)
resource_constantData = memory_context.resolve_segmented(
constantData
).get_resource(ConstantDataArrayResource)
if (
resource_knotCounts.file
== resource_interpolationData.file
== resource_constantData.file
== self.file
):
knotCounts_offset = resource_knotCounts.range_start
constantData_offset = resource_constantData.range_start
interpolationData_offset = resource_interpolationData.range_start
animHeader_offset = self.range_start
assert (
knotCounts_offset
< constantData_offset
< interpolationData_offset
< animHeader_offset
)
resource_knotCounts.length = (
constantData_offset - knotCounts_offset
) // resource_knotCounts.elem_cdata_ext.size
resource_constantData.length = (
interpolationData_offset - constantData_offset
) // resource_constantData.elem_cdata_ext.size
resource_interpolationData.length = (
animHeader_offset - interpolationData_offset
) // resource_interpolationData.elem_cdata_ext.size
return RESOURCE_PARSE_SUCCESS
else:
raise NotImplementedError
def get_c_declaration_base(self):
return f"CurveAnimationHeader {self.symbol_name}"
def get_c_reference(self, resource_offset: int):
raise ValueError()
def get_h_includes(self):
return ("z64curve.h",)
class SkelCurveLimbResource(CDataResource):
cdata_ext = CDataExt_Struct(
(
("child", CDataExt_Value.u8),
("sibling", CDataExt_Value.u8),
("pad2", CDataExt_Value.pad16),
(
"dList",
CDataExt_Array(
dlist_resources.cdata_ext_gfx_segmented, # Gfx*
2,
),
),
)
)
def get_c_declaration_base(self):
return f"SkelCurveLimb {self.symbol_name}"
def get_c_reference(self, resource_offset: int):
if resource_offset == 0:
return f"&{self.symbol_name}"
else:
raise ValueError()
def get_h_includes(self):
return ("z64curve.h",)
class SkelCurveLimbArrayResource(CDataResource):
def report_limb_element(resource, memory_context: "MemoryContext", v):
assert isinstance(v, int)
address = v
memory_context.report_resource_at_segmented(
resource,
address,
SkelCurveLimbResource,
lambda file, offset: SkelCurveLimbResource(
file, offset, f"{resource.name}_{address:08X}"
),
)
def write_limb_element(
resource, memory_context: "MemoryContext", v, wctx: CDataExtWriteContext
):
assert isinstance(v, int)
address = v
wctx.f.write(wctx.line_prefix)
wctx.f.write(memory_context.get_c_reference_at_segmented(address))
return True
elem_cdata_ext = (
CDataExt_Value("I")
.set_report(report_limb_element)
.set_write(write_limb_element)
)
def __init__(self, file: File, range_start: int, name: str, length: int):
self.cdata_ext = CDataExt_Array(self.elem_cdata_ext, length)
super().__init__(file, range_start, name)
def get_c_declaration_base(self):
return f"SkelCurveLimb* {self.symbol_name}[]"
def get_c_reference(self, resource_offset: int):
if resource_offset == 0:
return self.symbol_name
else:
raise ValueError()
def get_h_includes(self):
return ("z64curve.h",)
class CurveSkeletonHeaderResource(CDataResource):
def report_limbs(resource, memory_context: "MemoryContext", v):
assert isinstance(v, int)
address = v
memory_context.report_resource_at_segmented(
resource,
address,
SkelCurveLimbArrayResource,
lambda file, offset: SkelCurveLimbArrayResource(
file,
offset,
f"{resource.name}_Limbs_",
resource.cdata_unpacked["limbCount"],
),
)
def write_limbs(
resource, memory_context: "MemoryContext", v, wctx: CDataExtWriteContext
):
assert isinstance(v, int)
address = v
wctx.f.write(wctx.line_prefix)
wctx.f.write(memory_context.get_c_reference_at_segmented(address))
return True
cdata_ext = CDataExt_Struct(
(
(
"limbs",
CDataExt_Value("I").set_report(report_limbs).set_write(write_limbs),
), # SkelCurveLimb**
("limbCount", CDataExt_Value.u8),
("pad5", CDataExt_Value.pad8),
("pad6", CDataExt_Value.pad16),
)
)
def get_c_declaration_base(self):
return f"CurveSkeletonHeader {self.symbol_name}"
def get_c_reference(self, resource_offset: int):
raise ValueError()
def get_h_includes(self):
return ("z64curve.h",)