Repository navigation
feat(calidad): banco de libros de Patchouli con traducciones humanas #30
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,133 @@ | ||
| """Arma un banco de libros (Patchouli) con traducciones humanas, para medir la calidad en textos largos. | ||
|
|
||
| Usa los libros que bajó `scripts/descargar_lang.py` de mods de licencia libre (.descargas/<pack>/langs/<jar>/…): | ||
| cada archivo en inglés se empareja con su versión en español del mismo mod, campo por campo | ||
| (`paquete.emparejar_libro`), y se descartan las referencias a medias (inglés y español mezclados). El banco va a | ||
| `.descargas/banco/libros.json`, solo para uso local (ver docs/datos-de-terceros.md). | ||
|
|
||
| Uso: uv run python scripts/armar_banco_libros.py [--textos 150] [--salida RUTA] | ||
| Y luego: uv run python scripts/medir_calidad.py --banco .descargas/banco/libros.json --vanilla .descargas/vanilla | ||
| """ | ||
|
|
||
| from __future__ import annotations | ||
|
|
||
| import argparse | ||
| import json | ||
| import random | ||
| import re | ||
| import sys | ||
| from dataclasses import dataclass | ||
| from pathlib import Path | ||
| from typing import Any | ||
|
|
||
| import armar_banco_local as abl | ||
| import descargar_corpus as dc | ||
|
|
||
| from traductor_es.salida.paquete import emparejar_libro | ||
|
|
||
| #: …/langs/<jar>/assets/<espacio>/patchouli_books/<libro>/en_us/<resto>.json | ||
| _EN_INGLES = re.compile(r"langs/([^/]+)/assets/([^/]+)/patchouli_books/([^/]+)/en_us/(.+\.json)$") | ||
| #: Variantes de español en el orden en que se prefieren como referencia. | ||
| _PREFERIDAS = ("es_mx", "es_es") | ||
| #: Desde Patchouli 1.20 casi todos los libros usan claves de idioma (`"i18n": true`): su texto vive en `lang/`. | ||
| #: Una clave de libro tiene uno de estos tramos («mysticalagriculture.guide.entry…», «occultism.book.…»). | ||
| _CLAVE_DE_LIBRO = re.compile(r"(^|[._])(book|books|guide|guidebook|lexicon|codex|journal|manual)([._]|$)", re.I) | ||
| _NO_ES_LIBRO = ("options.", "config.", "modernfix.") | ||
| #: Largo mínimo de un texto de libro sacado de `lang/` (los títulos cortos ya los mide el banco de interfaz). | ||
| _MINIMO_LANG = 60 | ||
| #: Tope de textos por mod, para que un mod con un libro enorme no ocupe medio banco. | ||
| _MAXIMO_POR_MOD = 25 | ||
|
|
||
|
|
||
| @dataclass(frozen=True, slots=True) | ||
| class ParLibro: | ||
| jar: str | ||
| libro: str | ||
| ingles: str | ||
| castellano: str | ||
|
|
||
|
|
||
| def pares_de_libros(raiz: Path) -> list[ParLibro]: | ||
| """Los textos de libros que existen en inglés y en español en el mismo mod, sin referencias a medias.""" | ||
| salida: list[ParLibro] = [] | ||
| for ingles in sorted(raiz.glob("*/langs/*/assets/*/patchouli_books/*/en_us/**/*.json")): | ||
| m = _EN_INGLES.search(ingles.as_posix()) | ||
| if m is None: | ||
| continue | ||
| jar, espacio, libro, resto = m.groups() | ||
| base = ingles.parents[len(Path(resto).parts) - 1] # …/patchouli_books/<libro>/en_us | ||
| variantes = sorted(p.name for p in base.parent.iterdir() if p.is_dir() and p.name.startswith("es_")) | ||
| orden = [v for v in _PREFERIDAS if v in variantes] + [v for v in variantes if v not in _PREFERIDAS] | ||
| espanol = next((base.parent / v / resto for v in orden if (base.parent / v / resto).is_file()), None) | ||
| if espanol is None: | ||
| continue | ||
| for en, es in emparejar_libro(ingles.read_bytes(), espanol.read_bytes()).items(): | ||
| if abl.referencia_limpia(en, es): | ||
| salida.append(ParLibro(jar, f"{espacio}:{libro}", en, es)) | ||
| return salida | ||
|
|
||
|
|
||
| def pares_de_libros_en_lang(pares: list[abl.Par]) -> list[ParLibro]: | ||
| """Textos largos de libros que vienen en los archivos de idioma, por su clave.""" | ||
| return [ | ||
| ParLibro(p.jar, f"{p.namespace}:lang", p.ingles, p.castellano) | ||
| for p in pares | ||
| if _CLAVE_DE_LIBRO.search(p.clave) | ||
| and not p.clave.startswith(_NO_ES_LIBRO) | ||
| and len(p.ingles) >= _MINIMO_LANG | ||
| and abl.referencia_limpia(p.ingles, p.castellano) | ||
| ] | ||
|
|
||
|
|
||
| def armar(raiz: Path, textos: int, semilla: int = 7, pares_lang: list[abl.Par] | None = None) -> dict[str, Any]: | ||
| """El banco: una muestra fija de textos de libros, con el glosario que sacaría la app de los demás mods. | ||
|
|
||
| Junta los libros en JSON y los que vienen en `lang/`, con un tope por mod. | ||
| """ | ||
| lang = abl.pares() if pares_lang is None else pares_lang | ||
| todos = sorted( | ||
| set(pares_de_libros(raiz)) | set(pares_de_libros_en_lang(lang)), key=lambda p: (p.jar, p.libro, p.ingles) | ||
| ) | ||
| unicos = list({p.ingles: p for p in todos}.values()) # un texto repetido cuenta una vez | ||
| azar = random.Random(semilla) | ||
| azar.shuffle(unicos) | ||
| por_mod: dict[str, int] = {} | ||
| elegidos: list[ParLibro] = [] | ||
| for p in unicos: | ||
| if len(elegidos) >= textos or por_mod.get(p.jar, 0) >= _MAXIMO_POR_MOD: | ||
| continue | ||
| por_mod[p.jar] = por_mod.get(p.jar, 0) + 1 | ||
| elegidos.append(p) | ||
| medir = {p.jar for p in elegidos} | ||
| glosario = abl.glosario_del_banco(lang, medir, ingenuo=False) if (raiz / "vanilla").is_dir() else {} | ||
| return { | ||
| "descripcion": "Textos reales de libros (Patchouli) de mods libres, en inglés y español. Solo uso local.", | ||
| "dialecto": "es_mx", | ||
| "glosario": glosario, | ||
| "textos": [ | ||
| {"ambito": "libro", "original": p.ingles, "referencia": p.castellano, "jar": p.jar, "pista": "libro: text"} | ||
| for p in elegidos | ||
| ], | ||
| } | ||
|
|
||
|
|
||
| def main(argv: list[str] | None = None) -> int: | ||
| sys.stdout.reconfigure(encoding="utf-8", errors="replace") # pyright: ignore[reportAttributeAccessIssue] | ||
| ap = argparse.ArgumentParser(description=__doc__) | ||
| ap.add_argument("--textos", type=int, default=150) | ||
| ap.add_argument("--salida", type=Path, default=None, help="por defecto .descargas/banco/libros.json") | ||
| args = ap.parse_args(argv) | ||
| banco = armar(dc.RAIZ, args.textos) | ||
| if not banco["textos"]: | ||
| print("No hay libros con traducción al español: baja antes el corpus con scripts/descargar_lang.py.") | ||
| return 1 | ||
| destino = args.salida or dc.RAIZ / "banco" / "libros.json" | ||
| destino.parent.mkdir(parents=True, exist_ok=True) | ||
| destino.write_text(json.dumps(banco, indent=1, ensure_ascii=False), encoding="utf-8") | ||
| libros = {t["jar"] for t in banco["textos"]} | ||
| print(f"Banco de libros: {len(banco['textos'])} textos de {len(libros)} mods → {destino}") | ||
| return 0 | ||
|
|
||
|
|
||
| if __name__ == "__main__": | ||
| raise SystemExit(main()) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,74 @@ | ||
| """Banco de libros: pares inglés/español de libros de Patchouli, campo por campo y sin referencias a medias.""" | ||
|
|
||
| from __future__ import annotations | ||
|
|
||
| import json | ||
| import sys | ||
| from pathlib import Path | ||
| from types import ModuleType | ||
|
|
||
| import pytest | ||
|
|
||
|
|
||
| @pytest.fixture | ||
| def armar_banco_libros() -> ModuleType: | ||
| sys.path.insert(0, str(Path(__file__).parent.parent / "scripts")) | ||
| try: | ||
| import armar_banco_libros as modulo # pyright: ignore[reportMissingImports] | ||
| finally: | ||
| sys.path.pop(0) | ||
| return modulo # pyright: ignore[reportUnknownVariableType] | ||
|
|
||
|
|
||
| def _libro(raiz: Path, jar: str, idioma: str, datos: dict[str, object], archivo: str = "entries/a.json") -> None: | ||
| ruta = raiz / "pack" / "langs" / jar / "assets" / "m" / "patchouli_books" / "guia" / idioma / archivo | ||
| ruta.parent.mkdir(parents=True, exist_ok=True) | ||
| ruta.write_text(json.dumps(datos), encoding="utf-8") | ||
|
|
||
|
|
||
| def test_arma_el_banco_con_los_libros_traducidos(tmp_path: Path, armar_banco_libros: ModuleType) -> None: | ||
| _libro( | ||
| tmp_path, | ||
| "m.jar", | ||
| "en_us", | ||
| {"name": "Getting Started", "pages": [{"text": "Welcome to this guide about machines."}]}, | ||
| ) | ||
| _libro( | ||
| tmp_path, | ||
| "m.jar", | ||
| "es_es", | ||
| {"name": "Primeros pasos", "pages": [{"text": "Te damos la bienvenida a esta guía sobre máquinas."}]}, | ||
| ) | ||
| # Una «traducción» a medias (mezcla inglés y español) no sirve de referencia | ||
| _libro(tmp_path, "m.jar", "en_us", {"name": "Power the Grinder now"}, "entries/b.json") | ||
| _libro(tmp_path, "m.jar", "es_es", {"name": "Power the Grinder ahora"}, "entries/b.json") | ||
| # Un libro sin español no aporta nada | ||
| _libro(tmp_path, "otro.jar", "en_us", {"name": "Lonely"}) | ||
| banco = armar_banco_libros.armar(tmp_path, textos=10, pares_lang=[]) # pyright: ignore[reportUnknownMemberType] | ||
| pares = {t["original"]: t["referencia"] for t in banco["textos"]} | ||
| assert pares == { | ||
| "Getting Started": "Primeros pasos", | ||
| "Welcome to this guide about machines.": "Te damos la bienvenida a esta guía sobre máquinas.", | ||
| } | ||
| assert {t["ambito"] for t in banco["textos"]} == {"libro"} and banco["dialecto"] == "es_mx" | ||
|
|
||
|
|
||
| def test_prefiere_es_mx_como_referencia(tmp_path: Path, armar_banco_libros: ModuleType) -> None: | ||
| _libro(tmp_path, "m.jar", "en_us", {"name": "Getting Started"}) | ||
| _libro(tmp_path, "m.jar", "es_es", {"name": "Primeros pasos"}) | ||
| _libro(tmp_path, "m.jar", "es_mx", {"name": "Para empezar"}) | ||
| banco = armar_banco_libros.armar(tmp_path, textos=10, pares_lang=[]) # pyright: ignore[reportUnknownMemberType] | ||
| assert [t["referencia"] for t in banco["textos"]] == ["Para empezar"] | ||
|
|
||
|
|
||
| def test_suma_los_libros_que_vienen_en_lang(tmp_path: Path, armar_banco_libros: ModuleType) -> None: | ||
| par = armar_banco_libros.abl.Par # pyright: ignore[reportUnknownMemberType] | ||
| largo = "Place the main ingredient on the central pedestal and the catalysts around it." | ||
| traduccion = "Coloca el ingrediente principal sobre el pedestal central y los catalizadores alrededor." | ||
| lang = [ | ||
| par("ma.jar", "ma", "ma.guide.entry.infusion.page.1", largo, traduccion), | ||
| par("ma.jar", "ma", "ma.guide.title", "Guide", "Guía"), # corto: ya lo mide el banco de interfaz | ||
| par("x.jar", "x", "options.book.render", largo, traduccion), # una opción, no un libro | ||
| ] | ||
| banco = armar_banco_libros.armar(tmp_path, textos=10, pares_lang=lang) # pyright: ignore[reportUnknownMemberType] | ||
| assert [t["original"] for t in banco["textos"]] == [largo] |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If a downloaded jar contains a Patchouli JSON entry with
..segments in the remainder of its path, this regex accepts it;bajar_langsthen joins that entry name to the download directory without checking containment, so path normalization can escapelangs/<jar>and overwrite writable.jsonfiles elsewhere. Reject traversal components or verify the resolved destination stays within the intended directory before writing.Useful? React with 👍 / 👎.