1
0
Fork 0
mirror of https://github.com/zeldaret/oot.git synced 2024-12-05 01:06:37 +00:00
oot/extract_assets.py

209 lines
8.1 KiB
Python
Raw Normal View History

#!/usr/bin/env python3
import argparse
import json
import os
import signal
import time
import multiprocessing
2023-05-06 21:31:30 +00:00
from pathlib import Path
Building on Macs (#1086) * git subrepo pull (merge) tools/ZAPD subrepo: subdir: "tools/ZAPD" merged: "945e6ca1a" upstream: origin: "https://github.com/zeldaret/ZAPD.git" branch: "master" commit: "50242eca9" git-subrepo: version: "0.4.3" origin: "https://github.com/ingydotnet/git-subrepo.git" commit: "2f68596" * Fix extract_assets.py multithreading * Update binutils doc a bit * Remove * import, add multiprocessing option and way to pass arguments to ZAPD * Update format.sh to be more platform-independent * git subrepo pull --force tools/ZAPD subrepo: subdir: "tools/ZAPD" merged: "fd5a7f434" upstream: origin: "https://github.com/zeldaret/ZAPD.git" branch: "master" commit: "fd5a7f434" git-subrepo: version: "0.4.3" origin: "https://github.com/ingydotnet/git-subrepo.git" commit: "2f68596" * Remove ; * Update formatting script to not just use 11 * Add Python requirements, move the Mac stuff in the README into its own doc * Fix readme link * Minor format thing * . * Move ZAPDArgs into its own function * Update readme and remove requirements.txt * Dragorn-inspired rewrite of processZAPDArgs * git subrepo pull --force tools/ZAPD subrepo: subdir: "tools/ZAPD" merged: "a0d3f7b68" upstream: origin: "https://github.com/zeldaret/ZAPD.git" branch: "master" commit: "a0d3f7b68" git-subrepo: version: "0.4.3" origin: "https://github.com/ingydotnet/git-subrepo.git" commit: "2f68596" * Fix function definition * Building docs overhaul * Add Python packages to Mac and Cygwin * Heading number * format format.sh (!) * Replace "currently" * Remove Debian * git subrepo pull (merge) --force tools/ZAPD subrepo: subdir: "tools/ZAPD" merged: "0ba781304" upstream: origin: "https://github.com/zeldaret/ZAPD.git" branch: "master" commit: "0ba781304" git-subrepo: version: "0.4.3" origin: "https://github.com/ingydotnet/git-subrepo.git" commit: "2f68596"
2022-01-17 00:43:07 +00:00
from tools import version_config
Extract the assets from most of the overlays (#985) * remove fake match * Fire and Ice arrows * Light arrows * Ganon otyuka and end title * Oceff_Spot * Add missing Vtx to arrows * Update assets/xml/overlays/ovl_Arrow_Ice.xml Co-authored-by: Anghelo Carvajal <angheloalf95@gmail.com> * Update assets/xml/overlays/ovl_Arrow_Light.xml Co-authored-by: Anghelo Carvajal <angheloalf95@gmail.com> * Fix fire arrows * Update assets/xml/overlays/ovl_Arrow_Fire.xml Co-authored-by: Dragorn421 <Dragorn421@users.noreply.github.com> * Update ovl_Bg_Ganon_Otyuka.xml * Update src/overlays/actors/ovl_Bg_Ganon_Otyuka/z_bg_ganon_otyuka.c Co-authored-by: Dragorn421 <Dragorn421@users.noreply.github.com> * Change a comment and rename a DList * rename DList * Jya cobra * shd and elf_msg * elf msg 2 * En Bili * Remove extern form ice piece * remove some externs * en holl * kanban * make * sda * ssh and some cleanup * replace undefined sym in en_kusa * st * oceff wipe 3 * oceff wipe * oceff spot * oceff spot * oceff wipe 4 * Update VTXDIS * update VTXDIS again * fix sun song * magic dark * magic wind and oceff wipe 2 * magic fire and oceff storm * Update src/overlays/actors/ovl_Eff_Dust/z_eff_dust.c Co-authored-by: Dragorn421 <Dragorn421@users.noreply.github.com> * Update assets/xml/overlays/ovl_En_Sth.xml.ignore Co-authored-by: Dragorn421 <Dragorn421@users.noreply.github.com> * Update assets/xml/overlays/ovl_Elf_Msg.xml Co-authored-by: Dragorn421 <Dragorn421@users.noreply.github.com> * Update assets/xml/overlays/ovl_Magic_Wind.xml Co-authored-by: Dragorn421 <Dragorn421@users.noreply.github.com> * fix emptyDL, run formatter on en_ssh * run formatter * Update format.sh * update to VTXDIS * small change * textureDL -> materialDL, vtxDL -> modelDL * upper CASE * setupDL -> materialDL * GeometryDL->ModelDL and VisualDL->MaterialDL * revert rcp.c * revert holl.c * another revert in rcp.c * some u32 DL -> Gfx* DL * the rest of dragorns comments * git subrepo pull --force tools/ZAPD subrepo: subdir: "tools/ZAPD" merged: "3e9ed72e2" upstream: origin: "https://github.com/zeldaret/ZAPD.git" branch: "master" commit: "3e9ed72e2" git-subrepo: version: "0.4.3" origin: "https://github.com/ingydotnet/git-subrepo.git" commit: "2f68596" * start updating overlays with static * en sth OK * git subrepo pull --force tools/ZAPD subrepo: subdir: "tools/ZAPD" merged: "d0cd6b397" upstream: origin: "https://github.com/zeldaret/ZAPD.git" branch: "master" commit: "d0cd6b397" git-subrepo: version: "0.4.3" origin: "https://github.com/ingydotnet/git-subrepo.git" commit: "2f68596" * fix names of the overlays * remove segment 128, add gameplay keep to global config file * Fix end title and remove hex mode from VTXdis.c * add code assets Co-authored-by: Anghelo Carvajal <angheloalf95@gmail.com> Co-authored-by: Dragorn421 <Dragorn421@users.noreply.github.com>
2021-11-09 01:51:45 +00:00
def SignalHandler(sig, frame):
print(f'Signal {sig} received. Aborting...')
mainAbort.set()
# Don't exit immediately to update the extracted assets file.
2020-03-17 04:31:30 +00:00
def ExtractFile(assetConfig: version_config.AssetConfig, outputPath: Path, outputSourcePath: Path):
name = assetConfig.name
xmlPath = assetConfig.xml_path
version = globalVersionConfig.version
if globalAbort.is_set():
# Don't extract if another file wasn't extracted properly.
return
2023-05-06 21:31:30 +00:00
zapdPath = Path("tools") / "ZAPD" / "ZAPD.out"
configPath = Path("tools") / "ZAPDConfigs" / version / "Config.xml"
outputPath.mkdir(parents=True, exist_ok=True)
outputSourcePath.mkdir(parents=True, exist_ok=True)
execStr = f"{zapdPath} e -eh -i {xmlPath} -b {globalBaseromSegmentsDir} -o {outputPath} -osf {outputSourcePath} -gsf 1 -rconf {configPath} --cs-float both {ZAPDArgs}"
2023-05-06 21:31:30 +00:00
if name.startswith("code/") or name.startswith("n64dd/") or name.startswith("overlays/"):
assert assetConfig.start_offset is not None
assert assetConfig.end_offset is not None
2023-05-06 21:31:30 +00:00
execStr += f" --start-offset 0x{assetConfig.start_offset:X}"
execStr += f" --end-offset 0x{assetConfig.end_offset:X}"
Building on Macs (#1086) * git subrepo pull (merge) tools/ZAPD subrepo: subdir: "tools/ZAPD" merged: "945e6ca1a" upstream: origin: "https://github.com/zeldaret/ZAPD.git" branch: "master" commit: "50242eca9" git-subrepo: version: "0.4.3" origin: "https://github.com/ingydotnet/git-subrepo.git" commit: "2f68596" * Fix extract_assets.py multithreading * Update binutils doc a bit * Remove * import, add multiprocessing option and way to pass arguments to ZAPD * Update format.sh to be more platform-independent * git subrepo pull --force tools/ZAPD subrepo: subdir: "tools/ZAPD" merged: "fd5a7f434" upstream: origin: "https://github.com/zeldaret/ZAPD.git" branch: "master" commit: "fd5a7f434" git-subrepo: version: "0.4.3" origin: "https://github.com/ingydotnet/git-subrepo.git" commit: "2f68596" * Remove ; * Update formatting script to not just use 11 * Add Python requirements, move the Mac stuff in the README into its own doc * Fix readme link * Minor format thing * . * Move ZAPDArgs into its own function * Update readme and remove requirements.txt * Dragorn-inspired rewrite of processZAPDArgs * git subrepo pull --force tools/ZAPD subrepo: subdir: "tools/ZAPD" merged: "a0d3f7b68" upstream: origin: "https://github.com/zeldaret/ZAPD.git" branch: "master" commit: "a0d3f7b68" git-subrepo: version: "0.4.3" origin: "https://github.com/ingydotnet/git-subrepo.git" commit: "2f68596" * Fix function definition * Building docs overhaul * Add Python packages to Mac and Cygwin * Heading number * format format.sh (!) * Replace "currently" * Remove Debian * git subrepo pull (merge) --force tools/ZAPD subrepo: subdir: "tools/ZAPD" merged: "0ba781304" upstream: origin: "https://github.com/zeldaret/ZAPD.git" branch: "master" commit: "0ba781304" git-subrepo: version: "0.4.3" origin: "https://github.com/ingydotnet/git-subrepo.git" commit: "2f68596"
2022-01-17 00:43:07 +00:00
if name.startswith("overlays/"):
overlayName = name.split("/")[1]
baseAddress = globalVersionConfig.dmadata_segments[overlayName].vram + assetConfig.start_offset
execStr += f" --base-address 0x{baseAddress:X}"
execStr += " --static"
if globalUnaccounted:
Building on Macs (#1086) * git subrepo pull (merge) tools/ZAPD subrepo: subdir: "tools/ZAPD" merged: "945e6ca1a" upstream: origin: "https://github.com/zeldaret/ZAPD.git" branch: "master" commit: "50242eca9" git-subrepo: version: "0.4.3" origin: "https://github.com/ingydotnet/git-subrepo.git" commit: "2f68596" * Fix extract_assets.py multithreading * Update binutils doc a bit * Remove * import, add multiprocessing option and way to pass arguments to ZAPD * Update format.sh to be more platform-independent * git subrepo pull --force tools/ZAPD subrepo: subdir: "tools/ZAPD" merged: "fd5a7f434" upstream: origin: "https://github.com/zeldaret/ZAPD.git" branch: "master" commit: "fd5a7f434" git-subrepo: version: "0.4.3" origin: "https://github.com/ingydotnet/git-subrepo.git" commit: "2f68596" * Remove ; * Update formatting script to not just use 11 * Add Python requirements, move the Mac stuff in the README into its own doc * Fix readme link * Minor format thing * . * Move ZAPDArgs into its own function * Update readme and remove requirements.txt * Dragorn-inspired rewrite of processZAPDArgs * git subrepo pull --force tools/ZAPD subrepo: subdir: "tools/ZAPD" merged: "a0d3f7b68" upstream: origin: "https://github.com/zeldaret/ZAPD.git" branch: "master" commit: "a0d3f7b68" git-subrepo: version: "0.4.3" origin: "https://github.com/ingydotnet/git-subrepo.git" commit: "2f68596" * Fix function definition * Building docs overhaul * Add Python packages to Mac and Cygwin * Heading number * format format.sh (!) * Replace "currently" * Remove Debian * git subrepo pull (merge) --force tools/ZAPD subrepo: subdir: "tools/ZAPD" merged: "0ba781304" upstream: origin: "https://github.com/zeldaret/ZAPD.git" branch: "master" commit: "0ba781304" git-subrepo: version: "0.4.3" origin: "https://github.com/ingydotnet/git-subrepo.git" commit: "2f68596"
2022-01-17 00:43:07 +00:00
execStr += " -Wunaccounted"
2020-03-17 04:31:30 +00:00
print(execStr)
exitValue = os.system(execStr)
if exitValue != 0:
globalAbort.set()
print("\n")
print(f"Error when extracting from file {xmlPath}", file=os.sys.stderr)
print("Aborting...", file=os.sys.stderr)
print("\n")
2020-03-17 04:31:30 +00:00
def ExtractFunc(assetConfig: version_config.AssetConfig):
objectName = assetConfig.name
xml_path = assetConfig.xml_path
xml_path_str = str(xml_path)
outPath = globalOutputDir / objectName
outSourcePath = outPath
2020-03-17 04:31:30 +00:00
if xml_path_str in globalExtractedAssetsTracker:
timestamp = globalExtractedAssetsTracker[xml_path_str]["timestamp"]
modificationTime = int(os.path.getmtime(xml_path))
if modificationTime < timestamp:
# XML has not been modified since last extraction.
return
currentTimeStamp = int(time.time())
ExtractFile(assetConfig, outPath, outSourcePath)
if not globalAbort.is_set():
2024-09-26 04:21:00 +00:00
# Only update timestamp on successful extractions
if xml_path_str not in globalExtractedAssetsTracker:
globalExtractedAssetsTracker[xml_path_str] = globalManager.dict()
globalExtractedAssetsTracker[xml_path_str]["timestamp"] = currentTimeStamp
def initializeWorker(versionConfig: version_config.VersionConfig, abort, unaccounted: bool, extractedAssetsTracker: dict, manager, baseromSegmentsDir: Path, outputDir: Path):
global globalVersionConfig
global globalAbort
global globalUnaccounted
global globalExtractedAssetsTracker
global globalManager
global globalBaseromSegmentsDir
global globalOutputDir
globalVersionConfig = versionConfig
globalAbort = abort
globalUnaccounted = unaccounted
globalExtractedAssetsTracker = extractedAssetsTracker
globalManager = manager
globalBaseromSegmentsDir = baseromSegmentsDir
globalOutputDir = outputDir
Building on Macs (#1086) * git subrepo pull (merge) tools/ZAPD subrepo: subdir: "tools/ZAPD" merged: "945e6ca1a" upstream: origin: "https://github.com/zeldaret/ZAPD.git" branch: "master" commit: "50242eca9" git-subrepo: version: "0.4.3" origin: "https://github.com/ingydotnet/git-subrepo.git" commit: "2f68596" * Fix extract_assets.py multithreading * Update binutils doc a bit * Remove * import, add multiprocessing option and way to pass arguments to ZAPD * Update format.sh to be more platform-independent * git subrepo pull --force tools/ZAPD subrepo: subdir: "tools/ZAPD" merged: "fd5a7f434" upstream: origin: "https://github.com/zeldaret/ZAPD.git" branch: "master" commit: "fd5a7f434" git-subrepo: version: "0.4.3" origin: "https://github.com/ingydotnet/git-subrepo.git" commit: "2f68596" * Remove ; * Update formatting script to not just use 11 * Add Python requirements, move the Mac stuff in the README into its own doc * Fix readme link * Minor format thing * . * Move ZAPDArgs into its own function * Update readme and remove requirements.txt * Dragorn-inspired rewrite of processZAPDArgs * git subrepo pull --force tools/ZAPD subrepo: subdir: "tools/ZAPD" merged: "a0d3f7b68" upstream: origin: "https://github.com/zeldaret/ZAPD.git" branch: "master" commit: "a0d3f7b68" git-subrepo: version: "0.4.3" origin: "https://github.com/ingydotnet/git-subrepo.git" commit: "2f68596" * Fix function definition * Building docs overhaul * Add Python packages to Mac and Cygwin * Heading number * format format.sh (!) * Replace "currently" * Remove Debian * git subrepo pull (merge) --force tools/ZAPD subrepo: subdir: "tools/ZAPD" merged: "0ba781304" upstream: origin: "https://github.com/zeldaret/ZAPD.git" branch: "master" commit: "0ba781304" git-subrepo: version: "0.4.3" origin: "https://github.com/ingydotnet/git-subrepo.git" commit: "2f68596"
2022-01-17 00:43:07 +00:00
def processZAPDArgs(argsZ):
badZAPDArg = False
for z in argsZ:
if z[0] == '-':
print(f'error: argument "{z}" starts with "-", which is not supported.', file=os.sys.stderr)
Building on Macs (#1086) * git subrepo pull (merge) tools/ZAPD subrepo: subdir: "tools/ZAPD" merged: "945e6ca1a" upstream: origin: "https://github.com/zeldaret/ZAPD.git" branch: "master" commit: "50242eca9" git-subrepo: version: "0.4.3" origin: "https://github.com/ingydotnet/git-subrepo.git" commit: "2f68596" * Fix extract_assets.py multithreading * Update binutils doc a bit * Remove * import, add multiprocessing option and way to pass arguments to ZAPD * Update format.sh to be more platform-independent * git subrepo pull --force tools/ZAPD subrepo: subdir: "tools/ZAPD" merged: "fd5a7f434" upstream: origin: "https://github.com/zeldaret/ZAPD.git" branch: "master" commit: "fd5a7f434" git-subrepo: version: "0.4.3" origin: "https://github.com/ingydotnet/git-subrepo.git" commit: "2f68596" * Remove ; * Update formatting script to not just use 11 * Add Python requirements, move the Mac stuff in the README into its own doc * Fix readme link * Minor format thing * . * Move ZAPDArgs into its own function * Update readme and remove requirements.txt * Dragorn-inspired rewrite of processZAPDArgs * git subrepo pull --force tools/ZAPD subrepo: subdir: "tools/ZAPD" merged: "a0d3f7b68" upstream: origin: "https://github.com/zeldaret/ZAPD.git" branch: "master" commit: "a0d3f7b68" git-subrepo: version: "0.4.3" origin: "https://github.com/ingydotnet/git-subrepo.git" commit: "2f68596" * Fix function definition * Building docs overhaul * Add Python packages to Mac and Cygwin * Heading number * format format.sh (!) * Replace "currently" * Remove Debian * git subrepo pull (merge) --force tools/ZAPD subrepo: subdir: "tools/ZAPD" merged: "0ba781304" upstream: origin: "https://github.com/zeldaret/ZAPD.git" branch: "master" commit: "0ba781304" git-subrepo: version: "0.4.3" origin: "https://github.com/ingydotnet/git-subrepo.git" commit: "2f68596"
2022-01-17 00:43:07 +00:00
badZAPDArg = True
if badZAPDArg:
exit(1)
ZAPDArgs = " ".join(f"-{z}" for z in argsZ)
print("Using extra ZAPD arguments: " + ZAPDArgs)
return ZAPDArgs
def main():
parser = argparse.ArgumentParser(description="baserom asset extractor")
parser.add_argument(
"baserom_segments_dir",
type=Path,
help="Directory of uncompressed ROM segments",
)
parser.add_argument(
"output_dir",
type=Path,
help="Output directory to place files in",
)
parser.add_argument("-v", "--version", dest="oot_version", help="OOT game version", default="gc-eu-mq-dbg")
parser.add_argument("-s", "--single", help="Extract a single asset by name, e.g. objects/gameplay_keep")
parser.add_argument("-f", "--force", help="Force the extraction of every xml instead of checking the touched ones (overwriting current files).", action="store_true")
Building on Macs (#1086) * git subrepo pull (merge) tools/ZAPD subrepo: subdir: "tools/ZAPD" merged: "945e6ca1a" upstream: origin: "https://github.com/zeldaret/ZAPD.git" branch: "master" commit: "50242eca9" git-subrepo: version: "0.4.3" origin: "https://github.com/ingydotnet/git-subrepo.git" commit: "2f68596" * Fix extract_assets.py multithreading * Update binutils doc a bit * Remove * import, add multiprocessing option and way to pass arguments to ZAPD * Update format.sh to be more platform-independent * git subrepo pull --force tools/ZAPD subrepo: subdir: "tools/ZAPD" merged: "fd5a7f434" upstream: origin: "https://github.com/zeldaret/ZAPD.git" branch: "master" commit: "fd5a7f434" git-subrepo: version: "0.4.3" origin: "https://github.com/ingydotnet/git-subrepo.git" commit: "2f68596" * Remove ; * Update formatting script to not just use 11 * Add Python requirements, move the Mac stuff in the README into its own doc * Fix readme link * Minor format thing * . * Move ZAPDArgs into its own function * Update readme and remove requirements.txt * Dragorn-inspired rewrite of processZAPDArgs * git subrepo pull --force tools/ZAPD subrepo: subdir: "tools/ZAPD" merged: "a0d3f7b68" upstream: origin: "https://github.com/zeldaret/ZAPD.git" branch: "master" commit: "a0d3f7b68" git-subrepo: version: "0.4.3" origin: "https://github.com/ingydotnet/git-subrepo.git" commit: "2f68596" * Fix function definition * Building docs overhaul * Add Python packages to Mac and Cygwin * Heading number * format format.sh (!) * Replace "currently" * Remove Debian * git subrepo pull (merge) --force tools/ZAPD subrepo: subdir: "tools/ZAPD" merged: "0ba781304" upstream: origin: "https://github.com/zeldaret/ZAPD.git" branch: "master" commit: "0ba781304" git-subrepo: version: "0.4.3" origin: "https://github.com/ingydotnet/git-subrepo.git" commit: "2f68596"
2022-01-17 00:43:07 +00:00
parser.add_argument("-j", "--jobs", help="Number of cpu cores to extract with.")
parser.add_argument("-u", "--unaccounted", help="Enables ZAPD unaccounted detector warning system.", action="store_true")
Building on Macs (#1086) * git subrepo pull (merge) tools/ZAPD subrepo: subdir: "tools/ZAPD" merged: "945e6ca1a" upstream: origin: "https://github.com/zeldaret/ZAPD.git" branch: "master" commit: "50242eca9" git-subrepo: version: "0.4.3" origin: "https://github.com/ingydotnet/git-subrepo.git" commit: "2f68596" * Fix extract_assets.py multithreading * Update binutils doc a bit * Remove * import, add multiprocessing option and way to pass arguments to ZAPD * Update format.sh to be more platform-independent * git subrepo pull --force tools/ZAPD subrepo: subdir: "tools/ZAPD" merged: "fd5a7f434" upstream: origin: "https://github.com/zeldaret/ZAPD.git" branch: "master" commit: "fd5a7f434" git-subrepo: version: "0.4.3" origin: "https://github.com/ingydotnet/git-subrepo.git" commit: "2f68596" * Remove ; * Update formatting script to not just use 11 * Add Python requirements, move the Mac stuff in the README into its own doc * Fix readme link * Minor format thing * . * Move ZAPDArgs into its own function * Update readme and remove requirements.txt * Dragorn-inspired rewrite of processZAPDArgs * git subrepo pull --force tools/ZAPD subrepo: subdir: "tools/ZAPD" merged: "a0d3f7b68" upstream: origin: "https://github.com/zeldaret/ZAPD.git" branch: "master" commit: "a0d3f7b68" git-subrepo: version: "0.4.3" origin: "https://github.com/ingydotnet/git-subrepo.git" commit: "2f68596" * Fix function definition * Building docs overhaul * Add Python packages to Mac and Cygwin * Heading number * format format.sh (!) * Replace "currently" * Remove Debian * git subrepo pull (merge) --force tools/ZAPD subrepo: subdir: "tools/ZAPD" merged: "0ba781304" upstream: origin: "https://github.com/zeldaret/ZAPD.git" branch: "master" commit: "0ba781304" git-subrepo: version: "0.4.3" origin: "https://github.com/ingydotnet/git-subrepo.git" commit: "2f68596"
2022-01-17 00:43:07 +00:00
parser.add_argument("-Z", help="Pass the argument on to ZAPD, e.g. `-ZWunaccounted` to warn about unaccounted blocks in XMLs. Each argument should be passed separately, *without* the leading dash.", metavar="ZAPD_ARG", action="append")
args = parser.parse_args()
baseromSegmentsDir: Path = args.baserom_segments_dir
version: str = args.oot_version
outputDir: Path = args.output_dir
args.output_dir.mkdir(parents=True, exist_ok=True)
versionConfig = version_config.load_version_config(version)
Building on Macs (#1086) * git subrepo pull (merge) tools/ZAPD subrepo: subdir: "tools/ZAPD" merged: "945e6ca1a" upstream: origin: "https://github.com/zeldaret/ZAPD.git" branch: "master" commit: "50242eca9" git-subrepo: version: "0.4.3" origin: "https://github.com/ingydotnet/git-subrepo.git" commit: "2f68596" * Fix extract_assets.py multithreading * Update binutils doc a bit * Remove * import, add multiprocessing option and way to pass arguments to ZAPD * Update format.sh to be more platform-independent * git subrepo pull --force tools/ZAPD subrepo: subdir: "tools/ZAPD" merged: "fd5a7f434" upstream: origin: "https://github.com/zeldaret/ZAPD.git" branch: "master" commit: "fd5a7f434" git-subrepo: version: "0.4.3" origin: "https://github.com/ingydotnet/git-subrepo.git" commit: "2f68596" * Remove ; * Update formatting script to not just use 11 * Add Python requirements, move the Mac stuff in the README into its own doc * Fix readme link * Minor format thing * . * Move ZAPDArgs into its own function * Update readme and remove requirements.txt * Dragorn-inspired rewrite of processZAPDArgs * git subrepo pull --force tools/ZAPD subrepo: subdir: "tools/ZAPD" merged: "a0d3f7b68" upstream: origin: "https://github.com/zeldaret/ZAPD.git" branch: "master" commit: "a0d3f7b68" git-subrepo: version: "0.4.3" origin: "https://github.com/ingydotnet/git-subrepo.git" commit: "2f68596" * Fix function definition * Building docs overhaul * Add Python packages to Mac and Cygwin * Heading number * format format.sh (!) * Replace "currently" * Remove Debian * git subrepo pull (merge) --force tools/ZAPD subrepo: subdir: "tools/ZAPD" merged: "0ba781304" upstream: origin: "https://github.com/zeldaret/ZAPD.git" branch: "master" commit: "0ba781304" git-subrepo: version: "0.4.3" origin: "https://github.com/ingydotnet/git-subrepo.git" commit: "2f68596"
2022-01-17 00:43:07 +00:00
global ZAPDArgs
ZAPDArgs = processZAPDArgs(args.Z) if args.Z else ""
global mainAbort
Building on Macs (#1086) * git subrepo pull (merge) tools/ZAPD subrepo: subdir: "tools/ZAPD" merged: "945e6ca1a" upstream: origin: "https://github.com/zeldaret/ZAPD.git" branch: "master" commit: "50242eca9" git-subrepo: version: "0.4.3" origin: "https://github.com/ingydotnet/git-subrepo.git" commit: "2f68596" * Fix extract_assets.py multithreading * Update binutils doc a bit * Remove * import, add multiprocessing option and way to pass arguments to ZAPD * Update format.sh to be more platform-independent * git subrepo pull --force tools/ZAPD subrepo: subdir: "tools/ZAPD" merged: "fd5a7f434" upstream: origin: "https://github.com/zeldaret/ZAPD.git" branch: "master" commit: "fd5a7f434" git-subrepo: version: "0.4.3" origin: "https://github.com/ingydotnet/git-subrepo.git" commit: "2f68596" * Remove ; * Update formatting script to not just use 11 * Add Python requirements, move the Mac stuff in the README into its own doc * Fix readme link * Minor format thing * . * Move ZAPDArgs into its own function * Update readme and remove requirements.txt * Dragorn-inspired rewrite of processZAPDArgs * git subrepo pull --force tools/ZAPD subrepo: subdir: "tools/ZAPD" merged: "a0d3f7b68" upstream: origin: "https://github.com/zeldaret/ZAPD.git" branch: "master" commit: "a0d3f7b68" git-subrepo: version: "0.4.3" origin: "https://github.com/ingydotnet/git-subrepo.git" commit: "2f68596" * Fix function definition * Building docs overhaul * Add Python packages to Mac and Cygwin * Heading number * format format.sh (!) * Replace "currently" * Remove Debian * git subrepo pull (merge) --force tools/ZAPD subrepo: subdir: "tools/ZAPD" merged: "0ba781304" upstream: origin: "https://github.com/zeldaret/ZAPD.git" branch: "master" commit: "0ba781304" git-subrepo: version: "0.4.3" origin: "https://github.com/ingydotnet/git-subrepo.git" commit: "2f68596"
2022-01-17 00:43:07 +00:00
mainAbort = multiprocessing.Event()
manager = multiprocessing.Manager()
signal.signal(signal.SIGINT, SignalHandler)
extraction_times_p = outputDir / "assets_extraction_times.json"
extractedAssetsTracker = manager.dict()
if extraction_times_p.exists() and not args.force:
with extraction_times_p.open(encoding='utf-8') as f:
extractedAssetsTracker.update(json.load(f, object_hook=manager.dict))
singleAssetName = args.single
if singleAssetName is not None:
assetConfig = None
for asset in versionConfig.assets:
if asset.name == singleAssetName:
assetConfig = asset
break
else:
print(f"Error. Asset {singleAssetName} not found in config.", file=os.sys.stderr)
exit(1)
initializeWorker(versionConfig, mainAbort, args.unaccounted, extractedAssetsTracker, manager, baseromSegmentsDir, outputDir)
# Always extract if -s is used.
xml_path_str = str(assetConfig.xml_path)
if xml_path_str in extractedAssetsTracker:
del extractedAssetsTracker[xml_path_str]
ExtractFunc(assetConfig)
else:
2023-05-06 21:31:30 +00:00
class CannotMultiprocessError(Exception):
pass
try:
Building on Macs (#1086) * git subrepo pull (merge) tools/ZAPD subrepo: subdir: "tools/ZAPD" merged: "945e6ca1a" upstream: origin: "https://github.com/zeldaret/ZAPD.git" branch: "master" commit: "50242eca9" git-subrepo: version: "0.4.3" origin: "https://github.com/ingydotnet/git-subrepo.git" commit: "2f68596" * Fix extract_assets.py multithreading * Update binutils doc a bit * Remove * import, add multiprocessing option and way to pass arguments to ZAPD * Update format.sh to be more platform-independent * git subrepo pull --force tools/ZAPD subrepo: subdir: "tools/ZAPD" merged: "fd5a7f434" upstream: origin: "https://github.com/zeldaret/ZAPD.git" branch: "master" commit: "fd5a7f434" git-subrepo: version: "0.4.3" origin: "https://github.com/ingydotnet/git-subrepo.git" commit: "2f68596" * Remove ; * Update formatting script to not just use 11 * Add Python requirements, move the Mac stuff in the README into its own doc * Fix readme link * Minor format thing * . * Move ZAPDArgs into its own function * Update readme and remove requirements.txt * Dragorn-inspired rewrite of processZAPDArgs * git subrepo pull --force tools/ZAPD subrepo: subdir: "tools/ZAPD" merged: "a0d3f7b68" upstream: origin: "https://github.com/zeldaret/ZAPD.git" branch: "master" commit: "a0d3f7b68" git-subrepo: version: "0.4.3" origin: "https://github.com/ingydotnet/git-subrepo.git" commit: "2f68596" * Fix function definition * Building docs overhaul * Add Python packages to Mac and Cygwin * Heading number * format format.sh (!) * Replace "currently" * Remove Debian * git subrepo pull (merge) --force tools/ZAPD subrepo: subdir: "tools/ZAPD" merged: "0ba781304" upstream: origin: "https://github.com/zeldaret/ZAPD.git" branch: "master" commit: "0ba781304" git-subrepo: version: "0.4.3" origin: "https://github.com/ingydotnet/git-subrepo.git" commit: "2f68596"
2022-01-17 00:43:07 +00:00
numCores = int(args.jobs or 0)
if numCores <= 0:
numCores = 1
print("Extracting assets with " + str(numCores) + " CPU core" + ("s" if numCores > 1 else "") + ".")
2023-05-06 21:31:30 +00:00
try:
mp_context = multiprocessing.get_context("fork")
except ValueError as e:
raise CannotMultiprocessError() from e
with mp_context.Pool(numCores, initializer=initializeWorker, initargs=(versionConfig, mainAbort, args.unaccounted, extractedAssetsTracker, manager, baseromSegmentsDir, outputDir)) as p:
p.map(ExtractFunc, versionConfig.assets)
2023-05-06 21:31:30 +00:00
except (multiprocessing.ProcessError, TypeError, CannotMultiprocessError):
2024-09-26 04:21:00 +00:00
print("Warning: Multiprocessing exception occurred.", file=os.sys.stderr)
print("Disabling mutliprocessing.", file=os.sys.stderr)
initializeWorker(versionConfig, mainAbort, args.unaccounted, extractedAssetsTracker, manager, baseromSegmentsDir, outputDir)
for assetConfig in versionConfig.assets:
ExtractFunc(assetConfig)
with extraction_times_p.open('w', encoding='utf-8') as f:
serializableDict = dict()
for xml, data in extractedAssetsTracker.items():
serializableDict[xml] = dict(data)
json.dump(dict(serializableDict), f, ensure_ascii=False, indent=4)
if mainAbort.is_set():
exit(1)
if __name__ == "__main__":
Building on Macs (#1086) * git subrepo pull (merge) tools/ZAPD subrepo: subdir: "tools/ZAPD" merged: "945e6ca1a" upstream: origin: "https://github.com/zeldaret/ZAPD.git" branch: "master" commit: "50242eca9" git-subrepo: version: "0.4.3" origin: "https://github.com/ingydotnet/git-subrepo.git" commit: "2f68596" * Fix extract_assets.py multithreading * Update binutils doc a bit * Remove * import, add multiprocessing option and way to pass arguments to ZAPD * Update format.sh to be more platform-independent * git subrepo pull --force tools/ZAPD subrepo: subdir: "tools/ZAPD" merged: "fd5a7f434" upstream: origin: "https://github.com/zeldaret/ZAPD.git" branch: "master" commit: "fd5a7f434" git-subrepo: version: "0.4.3" origin: "https://github.com/ingydotnet/git-subrepo.git" commit: "2f68596" * Remove ; * Update formatting script to not just use 11 * Add Python requirements, move the Mac stuff in the README into its own doc * Fix readme link * Minor format thing * . * Move ZAPDArgs into its own function * Update readme and remove requirements.txt * Dragorn-inspired rewrite of processZAPDArgs * git subrepo pull --force tools/ZAPD subrepo: subdir: "tools/ZAPD" merged: "a0d3f7b68" upstream: origin: "https://github.com/zeldaret/ZAPD.git" branch: "master" commit: "a0d3f7b68" git-subrepo: version: "0.4.3" origin: "https://github.com/ingydotnet/git-subrepo.git" commit: "2f68596" * Fix function definition * Building docs overhaul * Add Python packages to Mac and Cygwin * Heading number * format format.sh (!) * Replace "currently" * Remove Debian * git subrepo pull (merge) --force tools/ZAPD subrepo: subdir: "tools/ZAPD" merged: "0ba781304" upstream: origin: "https://github.com/zeldaret/ZAPD.git" branch: "master" commit: "0ba781304" git-subrepo: version: "0.4.3" origin: "https://github.com/ingydotnet/git-subrepo.git" commit: "2f68596"
2022-01-17 00:43:07 +00:00
main()