diff --git a/TTS-DESIGN.md b/TTS-DESIGN.md index e752a70..99fe98f 100644 --- a/TTS-DESIGN.md +++ b/TTS-DESIGN.md @@ -335,6 +335,111 @@ How to apply a change: 5. Runtime `ITTSAttributes` pitch/speed/volume layer on top of the profile: the profile defines the natural values, the attributes shift them per request. +### VOX mode (NetTTS-inspired, Flextalk only) + +`VESKSLOP_TTS_VOX=1` routes the text through a rule-based prosody encoder before +synthesis. `setup/vox_parser.py` is an independent reimplementation of the VOX parser +from [h4rm0n1c/NetTTS](https://github.com/h4rm0n1c/NetTTS), which models Half-Life's +Black Mesa announcement delivery. It was verified byte-identical to the NetTTS parser +on a 31-line corpus (the reference was built locally with g++ for the comparison). + +What it does: sentence splitting; `the` -> `thee` before prepositions, proper nouns, +single letters or numbers; lead-in breaks ("Now...", "Please...", "A reminder..."); +time/number/degree normalization; token weighting (light/medium/heavy) with break +insertion at linkers, title-case runs, heavy gates and a 3-beat limiter; letter +tokens (`A` -> `Ay:`, `C` -> `C:`); `\!sf500 \!br` at sentence ends; and the +`\!wH1 ... \!wH0` staccato wrapper (disable with `VESKSLOP_TTS_VOX_CLEAN=1`). + +Requirements and scope: + +- Flextalk voices only: `\!br`, `\!sfNN` and `\!wH1/\!wH0` are FlexTalk vendor tags. + The host passes `TTSDATAFLAG_TAGGED`, so the engine honors them. Other voices ignore + `VESKSLOP_TTS_VOX`. +- Needs `python3` at runtime; the wrapper fails loudly when it is missing. +- Mutually exclusive with `VESKSLOP_TTS_WORDS` (VOX wins). +- Measured on the dev host for "Now approaching Sector C test lab. Please stand clear + of the test chamber.": plain 4.70 s with 1 pause, VOX 7.81 s with 8 pauses. + +Other NetTTS features and their relevance: + +- Inline `/rate N`, `/pitch N`, `[[pause N]]`: not ported. Veskslop has speed/pitch + controls and the profile window; `[[pause N]]` could be added as a wrapper-level + text splitter later (NetTTS supports it only outside VOX mode). +- TCP server with `START`/`STOP` status sockets: not needed; Veskslop owns playback + and knows when a line ends. +- Winetricks prefix automation (`winxp`, `vcrun6`, `mfc42`, `riched20`): not needed; + our prefix runs the engines with builtin Wine libraries. +- Win32 GUI and preset sliders: Veskslop provides the UI. + +### Inline rate/pitch controls (engine-side) + +The wrapper consumes NetTTS-style, line-leading controls, so Veskslop and the WebUI do +not need to parse or implement them: + + /rate 150 Attention all personnel. # 150% rate, then the line + /pitch 90 Attention all personnel. # 90% pitch + /rate 120 /pitch 110 Inbound tram. # both, in any order + +- `N` is a percent (100 = normal, clamped to 0-200); values `<= 2.0` are multipliers + (`/rate 1.5` = 150%). +- They override `VESKSLOP_TTS_SPEED` / `VESKSLOP_TTS_PITCH` for that line and are not + spoken. +- A line that contains only controls yields 100 ms of silence, so the daemon always + receives a valid WAV. +- Rate maps through the normal speed scale; pitch percent maps to semitones + (`st = 12*log2(N/100)`) and then through the per-voice natural pitch. +- Verified on the dev host: `/rate 50` doubles the line duration; `/pitch 150` raises + the median F0 from 122 Hz to 177 Hz. + +### Paths and configuration (standalone) + +The engine project is standalone and archival: it has no path dependency on Veskslop, +and every non-installable dependency is vendored under `third_party/`, so nothing is +fetched from the network at build or run time: + +- `third_party/speech.h` — SAPI4 SDK header used by the build. +- `third_party/sapi4/` — SAPI4 runtime (`SPEECH.DLL` + INF), MSTTS engine + voices, + and the original installers. +- `third_party/flextalk/data1.cab` — AT&T Flextalk installer (Keith Bell / Kerry Watson). +- `third_party/nettss-reference/` — NetTTS VOX parser source, kept to validate + `setup/vox_parser.py`. + +Only toolchain packages are installed by `setup/install-deps.sh` (Wine/Winelib, ffmpeg, +python3, unshield, mingw). Everything else lives in the repository. + +`setup/install.sh` writes `setup/microslopsam.env` with absolute paths; environment +variables take precedence over that file, so anything can be relocated without editing +scripts: + +| Variable | Default | Meaning | +| --- | --- | --- | +| `MICROSLOP_ROOT` | repo root | project root | +| `MICROSLOP_BUILD` | `$MICROSLOP_ROOT/build` | host binaries, samples | +| `MICROSLOP_PREFIX` | `~/.wine-microslopsam` | wine prefix (exported as `WINEPREFIX`) | +| `MICROSLOP_ASSETS` | `third_party/sapi4` | SAPI4/MSTTS payloads (vendored) | +| `MICROSLOP_FLEXTALK_DIR` | `third_party/flextalk` | Flextalk cabinet/extraction (vendored) | +| `MICROSLOP_SPEECH_H_DIR` | `$MICROSLOP_ROOT/third_party` | `speech.h` include dir (vendored) | +| `MICROSLOP_DISPLAY` | `:77` | Xvfb display used for Wine | +| `MICROSLOP_WINE` / `_XVFB` / `_PYTHON` / `_FFMPEG` | tool names | tool overrides | + +The wrapper needs only `VESKSLOP_TTS_CMD` to point at its absolute path; it derives +everything else from its own location plus the env file. + +### Merging into Veskslop + +This tree is written to survive a copy into the Veskslop repository next to its daemon: + +1. Copy `microslopsam/` (or replace the existing copy). The vendored assets under + `third_party/` come along, so the merged tree stays self-contained. Then refresh the + configuration: `MICROSLOP_UPDATE_ENV=1 microslopsam/setup/install.sh`. Re-running + install is idempotent; it rewrites `setup/microslopsam.env` for the new absolute paths. +2. No daemon changes are required. The env/placeholder contract is unchanged + (`VESKSLOP_TTS_*`, `{out}`, `{text_file}`, voice/speed/pitch/expression/profile/ + highpass/echo/words/word_gap_ms). New features are additive and default off. +3. Optional Veskslop-side follow-up: a `tts.vox` preference that exports + `VESKSLOP_TTS_VOX` (like `words`) and a line in the TTS panel help. The inline + `/rate` and `/pitch` controls need no UI work at all. + ### Transport and latency - Installation is one command: `setup/install.sh` (system dependencies with sudo, wine diff --git a/microslopsam/.gitignore b/microslopsam/.gitignore index 735b6e4..03b4271 100644 --- a/microslopsam/.gitignore +++ b/microslopsam/.gitignore @@ -6,3 +6,7 @@ build/ *.dll setup/out-*.reg *.wav +# Generated per-checkout configuration. +setup/microslopsam.env +# Engine DLLs and voice data are proprietary; never commit them. +third_party/ diff --git a/microslopsam/README.md b/microslopsam/README.md index b1ef54d..2ea4a0e 100644 --- a/microslopsam/README.md +++ b/microslopsam/README.md @@ -35,10 +35,35 @@ Manual steps, if needed: setup/setup-prefix.sh # prefix + SAPI4 runtime + MSTTS voices setup/setup-flextalk.sh # Flextalk extraction + registry # host build: - winegcc -m32 -O2 -o build/sapi4host-winlib host/sapi4host.c -I../reference \ + winegcc -m32 -O2 -o build/sapi4host-winlib host/sapi4host.c -Ithird_party \ -lole32 -luser32 -ladvapi32 i686-w64-mingw32-gcc -O2 -static -o build/sapi4host.exe host/sapi4host.c \ - -I../reference -lole32 -luser32 -ladvapi32 -luuid + -Ithird_party -lole32 -luser32 -ladvapi32 -luuid + +## Configuration + +`setup/install.sh` writes `setup/microslopsam.env` (generated, gitignored) with absolute +paths. Environment variables take precedence over the file, so anything can be relocated +without editing scripts: + +| Variable | Default | Meaning | +| --- | --- | --- | +| `MICROSLOP_ROOT` | repo root | project root | +| `MICROSLOP_BUILD` | `$MICROSLOP_ROOT/build` | host binaries, samples | +| `MICROSLOP_PREFIX` | `~/.wine-microslopsam` | wine prefix (exported as `WINEPREFIX`) | +| `MICROSLOP_ASSETS` | `third_party/sapi4` | SAPI4/MSTTS payloads (vendored) | +| `MICROSLOP_FLEXTALK_DIR` | `third_party/flextalk` | Flextalk cabinet/extraction (vendored) | +| `MICROSLOP_SPEECH_H_DIR` | `$MICROSLOP_ROOT/third_party` | `speech.h` include dir (vendored) | +| `MICROSLOP_DISPLAY` | `:77` | Xvfb display used for Wine | +| `MICROSLOP_WINE`, `MICROSLOP_XVFB`, `MICROSLOP_PYTHON`, `MICROSLOP_FFMPEG` | tool names | tool overrides | + +Refresh the file after moving the tree: `MICROSLOP_UPDATE_ENV=1 setup/install.sh`. + +In the Veskslop tree the `third_party/` assets are **not committed** (licensing; see +`docs/TTS-DESIGN.md`). This checkout has a local `setup/microslopsam.env` (gitignored) that +points `MICROSLOP_ASSETS`, `MICROSLOP_FLEXTALK_DIR`, and `MICROSLOP_SPEECH_H_DIR` at the +MicroslopSam development checkout. A manual host build needs the same override, or a local +`third_party/` with `speech.h`. ## Usage @@ -65,6 +90,11 @@ load: the wrapper writes it to the Wine registry before synthesis. `--voices` ma voices with `expression=NN`, where NN is the registry default. Mapping details are in `docs/TTS-DESIGN.md`. +**VOX prosody** (NetTTS-style, Flextalk voices): set `VESKSLOP_TTS_VOX=1` in the +command environment (`VESKSLOP_TTS_VOX_CLEAN=1` omits the `\!wH1/\!wH0` stutter +wrapper). Line-leading `/rate N` and `/pitch N` controls are also handled by the +wrapper, so the WebUI does not need to implement them. See `docs/TTS-DESIGN.md`. + ## Layout - `host/sapi4host.c` — host: voice select, pitch/speed, text -> WAV @@ -75,6 +105,7 @@ voices with `expression=NN`, where NN is the registry default. Mapping details a - `setup/inf2reg.py` — INF AddReg -> `.reg` converter - `setup/run.sh` — run helper (prefix + Xvfb) - `setup/veskslop-tts.sh` — Veskslop command-provider wrapper (post-processing included) +- `setup/vox_parser.py` — NetTTS-style VOX prosody encoder (Flextalk voices) - `docs/NOTES.md` — registry/CLSID facts, runtime RE notes, gotchas - `docs/TTS-DESIGN.md` — integration contract with Veskslop plus engine clarifications @@ -88,10 +119,9 @@ voices with `expression=NN`, where NN is the registry default. Mapping details a - The Flextalk archive lacks `data2.cab`; `unshield -O` extracts `data1.cab` but one tail file (`WatsonUNInstall.dll`) is incomplete. -## Reference material (not in this repo) +## Reference material -- `../reference/speech.h` — SAPI4 SDK header -- `../reference/SAPI4-CLI`, `../reference/TETYYS-SAPI4` — host examples -- `../Microsoft-Sam-Mary-Mike-TruVoice-WSAPI4` — installers/payloads -- `../flextalk` — Flextalk installer archive + extraction -- `/mnt/ghidra/decomp/{msttssyn.dll,speech.dll}/` — decompiled corpora +- The MicroslopSam development checkout (`third_party/`): SAPI4 runtime, MSTTS voices, + Flextalk cabinet, `speech.h`, NetTTS parser source. Referenced through + `setup/microslopsam.env`; not committed here. +- `/mnt/ghidra/decomp/{msttssyn.dll,speech.dll}/` — decompiled corpora (external, optional) diff --git a/microslopsam/docs/NOTES.md b/microslopsam/docs/NOTES.md index de90d3b..754921a 100644 --- a/microslopsam/docs/NOTES.md +++ b/microslopsam/docs/NOTES.md @@ -93,6 +93,22 @@ final file `WatsonUNInstall.dll` is incomplete). verified audible (0/59/100 produce different waveforms at equal duration). Changing them requires a registry edit and re-selecting the voice. +## VOX parser + +- `setup/vox_parser.py` reimplements the NetTTS VOX prosody encoder (vendor tags + `\!wH1/\!wH0`, `\!br`, `\!sfNN`); the host already passes `TTSDATAFLAG_TAGGED`, so + the engine honors them. +- Validation: the reference `vox_parser.cpp` from a local clone of + `h4rm0n1c/NetTTS` was compiled with g++ in a throwaway harness and diffed against + the Python port; outputs are identical on the 31-line corpus. Re-check after any + parser change while the clone is available. +- VOX mode is Flextalk-only. The wrapper enables it with `VESKSLOP_TTS_VOX=1` + (`VESKSLOP_TTS_VOX_CLEAN=1` drops the stutter wrapper) and it takes precedence + over `VESKSLOP_TTS_WORDS`. +- Line-leading `/rate N` and `/pitch N` controls are consumed by the wrapper + (percent, 100 = normal; `<= 2.0` means multiplier). They override the env + speed/pitch and are not spoken; a controls-only line yields 100 ms of silence. + ## Decompiled corpora (Ghidra) - `/mnt/ghidra/decomp/msttssyn.dll/` (786 functions, engine) diff --git a/microslopsam/docs/TTS-DESIGN.md b/microslopsam/docs/TTS-DESIGN.md index e752a70..99fe98f 100644 --- a/microslopsam/docs/TTS-DESIGN.md +++ b/microslopsam/docs/TTS-DESIGN.md @@ -335,6 +335,111 @@ How to apply a change: 5. Runtime `ITTSAttributes` pitch/speed/volume layer on top of the profile: the profile defines the natural values, the attributes shift them per request. +### VOX mode (NetTTS-inspired, Flextalk only) + +`VESKSLOP_TTS_VOX=1` routes the text through a rule-based prosody encoder before +synthesis. `setup/vox_parser.py` is an independent reimplementation of the VOX parser +from [h4rm0n1c/NetTTS](https://github.com/h4rm0n1c/NetTTS), which models Half-Life's +Black Mesa announcement delivery. It was verified byte-identical to the NetTTS parser +on a 31-line corpus (the reference was built locally with g++ for the comparison). + +What it does: sentence splitting; `the` -> `thee` before prepositions, proper nouns, +single letters or numbers; lead-in breaks ("Now...", "Please...", "A reminder..."); +time/number/degree normalization; token weighting (light/medium/heavy) with break +insertion at linkers, title-case runs, heavy gates and a 3-beat limiter; letter +tokens (`A` -> `Ay:`, `C` -> `C:`); `\!sf500 \!br` at sentence ends; and the +`\!wH1 ... \!wH0` staccato wrapper (disable with `VESKSLOP_TTS_VOX_CLEAN=1`). + +Requirements and scope: + +- Flextalk voices only: `\!br`, `\!sfNN` and `\!wH1/\!wH0` are FlexTalk vendor tags. + The host passes `TTSDATAFLAG_TAGGED`, so the engine honors them. Other voices ignore + `VESKSLOP_TTS_VOX`. +- Needs `python3` at runtime; the wrapper fails loudly when it is missing. +- Mutually exclusive with `VESKSLOP_TTS_WORDS` (VOX wins). +- Measured on the dev host for "Now approaching Sector C test lab. Please stand clear + of the test chamber.": plain 4.70 s with 1 pause, VOX 7.81 s with 8 pauses. + +Other NetTTS features and their relevance: + +- Inline `/rate N`, `/pitch N`, `[[pause N]]`: not ported. Veskslop has speed/pitch + controls and the profile window; `[[pause N]]` could be added as a wrapper-level + text splitter later (NetTTS supports it only outside VOX mode). +- TCP server with `START`/`STOP` status sockets: not needed; Veskslop owns playback + and knows when a line ends. +- Winetricks prefix automation (`winxp`, `vcrun6`, `mfc42`, `riched20`): not needed; + our prefix runs the engines with builtin Wine libraries. +- Win32 GUI and preset sliders: Veskslop provides the UI. + +### Inline rate/pitch controls (engine-side) + +The wrapper consumes NetTTS-style, line-leading controls, so Veskslop and the WebUI do +not need to parse or implement them: + + /rate 150 Attention all personnel. # 150% rate, then the line + /pitch 90 Attention all personnel. # 90% pitch + /rate 120 /pitch 110 Inbound tram. # both, in any order + +- `N` is a percent (100 = normal, clamped to 0-200); values `<= 2.0` are multipliers + (`/rate 1.5` = 150%). +- They override `VESKSLOP_TTS_SPEED` / `VESKSLOP_TTS_PITCH` for that line and are not + spoken. +- A line that contains only controls yields 100 ms of silence, so the daemon always + receives a valid WAV. +- Rate maps through the normal speed scale; pitch percent maps to semitones + (`st = 12*log2(N/100)`) and then through the per-voice natural pitch. +- Verified on the dev host: `/rate 50` doubles the line duration; `/pitch 150` raises + the median F0 from 122 Hz to 177 Hz. + +### Paths and configuration (standalone) + +The engine project is standalone and archival: it has no path dependency on Veskslop, +and every non-installable dependency is vendored under `third_party/`, so nothing is +fetched from the network at build or run time: + +- `third_party/speech.h` — SAPI4 SDK header used by the build. +- `third_party/sapi4/` — SAPI4 runtime (`SPEECH.DLL` + INF), MSTTS engine + voices, + and the original installers. +- `third_party/flextalk/data1.cab` — AT&T Flextalk installer (Keith Bell / Kerry Watson). +- `third_party/nettss-reference/` — NetTTS VOX parser source, kept to validate + `setup/vox_parser.py`. + +Only toolchain packages are installed by `setup/install-deps.sh` (Wine/Winelib, ffmpeg, +python3, unshield, mingw). Everything else lives in the repository. + +`setup/install.sh` writes `setup/microslopsam.env` with absolute paths; environment +variables take precedence over that file, so anything can be relocated without editing +scripts: + +| Variable | Default | Meaning | +| --- | --- | --- | +| `MICROSLOP_ROOT` | repo root | project root | +| `MICROSLOP_BUILD` | `$MICROSLOP_ROOT/build` | host binaries, samples | +| `MICROSLOP_PREFIX` | `~/.wine-microslopsam` | wine prefix (exported as `WINEPREFIX`) | +| `MICROSLOP_ASSETS` | `third_party/sapi4` | SAPI4/MSTTS payloads (vendored) | +| `MICROSLOP_FLEXTALK_DIR` | `third_party/flextalk` | Flextalk cabinet/extraction (vendored) | +| `MICROSLOP_SPEECH_H_DIR` | `$MICROSLOP_ROOT/third_party` | `speech.h` include dir (vendored) | +| `MICROSLOP_DISPLAY` | `:77` | Xvfb display used for Wine | +| `MICROSLOP_WINE` / `_XVFB` / `_PYTHON` / `_FFMPEG` | tool names | tool overrides | + +The wrapper needs only `VESKSLOP_TTS_CMD` to point at its absolute path; it derives +everything else from its own location plus the env file. + +### Merging into Veskslop + +This tree is written to survive a copy into the Veskslop repository next to its daemon: + +1. Copy `microslopsam/` (or replace the existing copy). The vendored assets under + `third_party/` come along, so the merged tree stays self-contained. Then refresh the + configuration: `MICROSLOP_UPDATE_ENV=1 microslopsam/setup/install.sh`. Re-running + install is idempotent; it rewrites `setup/microslopsam.env` for the new absolute paths. +2. No daemon changes are required. The env/placeholder contract is unchanged + (`VESKSLOP_TTS_*`, `{out}`, `{text_file}`, voice/speed/pitch/expression/profile/ + highpass/echo/words/word_gap_ms). New features are additive and default off. +3. Optional Veskslop-side follow-up: a `tts.vox` preference that exports + `VESKSLOP_TTS_VOX` (like `words`) and a line in the TTS panel help. The inline + `/rate` and `/pitch` controls need no UI work at all. + ### Transport and latency - Installation is one command: `setup/install.sh` (system dependencies with sudo, wine diff --git a/microslopsam/setup/config.sh b/microslopsam/setup/config.sh new file mode 100644 index 0000000..5af5637 --- /dev/null +++ b/microslopsam/setup/config.sh @@ -0,0 +1,50 @@ +# shellcheck shell=sh +# Shared configuration for MicroslopSam. +# +# Precedence: environment variables > setup/microslopsam.env (generated by +# setup/install.sh) > built-in defaults. +# +# Source this file near the top of every setup script: +# . "$HERE/config.sh" +# +# All values are absolute paths or tool names. Nothing here depends on the +# project living next to other checkouts, with one exception: the asset repo +# and the Flextalk archive default to sibling directories, which is how the +# development tree is laid out. Set MICROSLOP_ASSETS / MICROSLOP_FLEXTALK_DIR +# to point anywhere else. + +: "${HERE:?config.sh: set HERE to the setup directory before sourcing}" + +if [ -f "$HERE/microslopsam.env" ]; then + # The generated file uses ': ${VAR:=default}' lines, so values already + # present in the environment win over the file. + . "$HERE/microslopsam.env" +fi + +MICROSLOP_ROOT=${MICROSLOP_ROOT:-$(CDPATH= cd -- "$HERE/.." && pwd)} +MICROSLOP_BUILD=${MICROSLOP_BUILD:-$MICROSLOP_ROOT/build} +MICROSLOP_PREFIX=${MICROSLOP_PREFIX:-$HOME/.wine-microslopsam} +MICROSLOP_ASSETS=${MICROSLOP_ASSETS:-$MICROSLOP_ROOT/third_party/sapi4} +MICROSLOP_FLEXTALK_DIR=${MICROSLOP_FLEXTALK_DIR:-$MICROSLOP_ROOT/third_party/flextalk} +MICROSLOP_SPEECH_H_DIR=${MICROSLOP_SPEECH_H_DIR:-$MICROSLOP_ROOT/third_party} +MICROSLOP_DISPLAY=${MICROSLOP_DISPLAY:-:77} +MICROSLOP_XVFB=${MICROSLOP_XVFB:-Xvfb} +MICROSLOP_WINE=${MICROSLOP_WINE:-wine} +MICROSLOP_PYTHON=${MICROSLOP_PYTHON:-python3} +MICROSLOP_FFMPEG=${MICROSLOP_FFMPEG:-ffmpeg} + +export MICROSLOP_ROOT MICROSLOP_BUILD MICROSLOP_PREFIX MICROSLOP_ASSETS +export MICROSLOP_FLEXTALK_DIR MICROSLOP_SPEECH_H_DIR MICROSLOP_DISPLAY +export MICROSLOP_XVFB MICROSLOP_WINE MICROSLOP_PYTHON MICROSLOP_FFMPEG + +# The wine prefix and arch are what the scripts actually need; keep the +# standard names so callers can override them directly too. +export WINEPREFIX=${WINEPREFIX:-$MICROSLOP_PREFIX} +export WINEARCH=${WINEARCH:-win32} + +# Convenience alias for scripts that use $ROOT for repo-relative files. +ROOT=$MICROSLOP_ROOT + +# Derived helpers. +MICROSLOP_XSOCK="/tmp/.X11-unix/X${MICROSLOP_DISPLAY#:}" +export MICROSLOP_XSOCK diff --git a/microslopsam/setup/install.sh b/microslopsam/setup/install.sh index 2c06cdd..77f55d7 100755 --- a/microslopsam/setup/install.sh +++ b/microslopsam/setup/install.sh @@ -1,22 +1,41 @@ #!/bin/sh # One-shot installer for MicroslopSam. Runs all setup steps, idempotently: -# 1. system dependencies (Wine, Winelib, helpers; needs sudo) -# 2. wine prefix + SAPI4 runtime + MSTTS voices + registration -# 3. Flextalk extraction + voices + registration -# 4. host build (Winelib and PE) -# 5. quick voice check +# 1. record resolved paths in setup/microslopsam.env +# 2. system dependencies (Wine, Winelib, helpers; needs sudo) +# 3. wine prefix + SAPI4 runtime + MSTTS voices + registration +# 4. Flextalk extraction + voices + registration +# 5. host build (Winelib and PE) +# 6. quick voice check # -# Usage: setup/install.sh -# Re-running is safe. +# Paths and tools come from setup/config.sh. Override anything with environment +# variables (they win over setup/microslopsam.env), for example: +# MICROSLOP_ASSETS=/srv/assets MICROSLOP_PREFIX=~/.wine-sam setup/install.sh +# Set MICROSLOP_UPDATE_ENV=1 to rewrite an existing microslopsam.env. set -eu HERE=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd) -ROOT=$(CDPATH= cd -- "$HERE/.." && pwd) -ASSETS=${ASSETS:-$ROOT/../Microsoft-Sam-Mary-Mike-TruVoice-WSAPI4} -FLEXTALK_DIR=${FLEXTALK_DIR:-$ROOT/../flextalk} +. "$HERE/config.sh" log() { printf '\n== %s\n' "$*"; } +# 0) record the resolved configuration ---------------------------------------- +ENVFILE="$HERE/microslopsam.env" +if [ ! -f "$ENVFILE" ] || [ "${MICROSLOP_UPDATE_ENV:-0}" = "1" ]; then + { + echo "# Generated by setup/install.sh -- safe to edit or delete." + echo "# Values are defaults; environment variables take precedence." + printf ': "${MICROSLOP_ROOT:=%s}"\n' "$MICROSLOP_ROOT" + printf ': "${MICROSLOP_BUILD:=%s}"\n' "$MICROSLOP_BUILD" + printf ': "${MICROSLOP_PREFIX:=%s}"\n' "$MICROSLOP_PREFIX" + printf ': "${MICROSLOP_ASSETS:=%s}"\n' "$MICROSLOP_ASSETS" + printf ': "${MICROSLOP_FLEXTALK_DIR:=%s}"\n' "$MICROSLOP_FLEXTALK_DIR" + printf ': "${MICROSLOP_SPEECH_H_DIR:=%s}"\n' "$MICROSLOP_SPEECH_H_DIR" + printf ': "${MICROSLOP_DISPLAY:=%s}"\n' "$MICROSLOP_DISPLAY" + } > "$ENVFILE" + chmod 644 "$ENVFILE" 2>/dev/null || true + log "configuration written to $ENVFILE" +fi + # 1) system dependencies ----------------------------------------------------- if [ "$(id -u)" = "0" ]; then log "installing system dependencies" @@ -28,18 +47,19 @@ fi # 2) SAPI4 runtime + MSTTS voices -------------------------------------------- log "setting up wine prefix, SAPI4 runtime and MSTTS voices" -"$HERE/setup-prefix.sh" "$ASSETS" +"$HERE/setup-prefix.sh" "$MICROSLOP_ASSETS" # 3) Flextalk voices ---------------------------------------------------------- -EXTRACTED="$FLEXTALK_DIR/extracted/TTS_Executables" +EXTRACTED="$MICROSLOP_FLEXTALK_DIR/extracted/TTS_Executables" if [ ! -d "$EXTRACTED/EngineDLLs" ]; then CAB="" - for c in "$FLEXTALK_DIR/installer/flextalk/data1.cab" \ - "$ASSETS/flextalk/data1.cab"; do + for c in "$MICROSLOP_FLEXTALK_DIR/data1.cab" \ + "$MICROSLOP_FLEXTALK_DIR/installer/flextalk/data1.cab" \ + "$MICROSLOP_ASSETS/flextalk/data1.cab"; do if [ -f "$c" ]; then CAB="$c"; break; fi done if [ -z "$CAB" ]; then - echo "Flextalk data1.cab not found; set FLEXTALK_DIR to the archive dir" >&2 + echo "Flextalk data1.cab not found; set MICROSLOP_FLEXTALK_DIR" >&2 exit 1 fi command -v unshield >/dev/null 2>&1 || { @@ -47,20 +67,24 @@ if [ ! -d "$EXTRACTED/EngineDLLs" ]; then exit 1 } log "extracting Flextalk installer ($CAB)" - rm -rf "$FLEXTALK_DIR/extracted" - unshield -O -d "$FLEXTALK_DIR/extracted" x "$CAB" >/dev/null + rm -rf "$MICROSLOP_FLEXTALK_DIR/extracted" + unshield -O -d "$MICROSLOP_FLEXTALK_DIR/extracted" x "$CAB" >/dev/null fi log "installing Flextalk voices" "$HERE/setup-flextalk.sh" "$EXTRACTED" # 4) host build --------------------------------------------------------------- log "building the host" -mkdir -p "$ROOT/build" -winegcc -m32 -O2 -o "$ROOT/build/sapi4host-winlib" "$ROOT/host/sapi4host.c" \ - -I"$ROOT/../reference" -lole32 -luser32 -ladvapi32 +[ -f "$MICROSLOP_SPEECH_H_DIR/speech.h" ] || { + echo "speech.h not found in $MICROSLOP_SPEECH_H_DIR" >&2 + exit 1 +} +mkdir -p "$MICROSLOP_BUILD" +winegcc -m32 -O2 -o "$MICROSLOP_BUILD/sapi4host-winlib" "$MICROSLOP_ROOT/host/sapi4host.c" \ + -I"$MICROSLOP_SPEECH_H_DIR" -lole32 -luser32 -ladvapi32 if command -v i686-w64-mingw32-gcc >/dev/null 2>&1; then - i686-w64-mingw32-gcc -O2 -static -o "$ROOT/build/sapi4host.exe" \ - "$ROOT/host/sapi4host.c" -I"$ROOT/../reference" \ + i686-w64-mingw32-gcc -O2 -static -o "$MICROSLOP_BUILD/sapi4host.exe" \ + "$MICROSLOP_ROOT/host/sapi4host.c" -I"$MICROSLOP_SPEECH_H_DIR" \ -lole32 -luser32 -ladvapi32 -luuid fi @@ -72,6 +96,12 @@ cat </dev/null 2>&1 & + if ! [ -S "$MICROSLOP_XSOCK" ]; then + "$MICROSLOP_XVFB" "$MICROSLOP_DISPLAY" -screen 0 1024x768x24 >/dev/null 2>&1 & sleep 1 fi - export DISPLAY=:77 + export DISPLAY=$MICROSLOP_DISPLAY fi -HOST="$ROOT/build/sapi4host-winlib.exe" +HOST="$MICROSLOP_BUILD/sapi4host-winlib.exe" if [ ! -x "$HOST" ]; then - HOST="$ROOT/build/sapi4host.exe" + HOST="$MICROSLOP_BUILD/sapi4host.exe" fi if [ ! -x "$HOST" ]; then - echo "no host binary; build first (see README)" >&2 + echo "no host binary; run setup/install.sh first" >&2 exit 1 fi diff --git a/microslopsam/setup/setup-flextalk.sh b/microslopsam/setup/setup-flextalk.sh index 16dc6bc..c4c4b5c 100755 --- a/microslopsam/setup/setup-flextalk.sh +++ b/microslopsam/setup/setup-flextalk.sh @@ -7,18 +7,15 @@ set -eu HERE=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd) -ROOT=$(CDPATH= cd -- "$HERE/.." && pwd) -SRC=${1:-$ROOT/../flextalk/extracted/TTS_Executables} +. "$HERE/config.sh" +SRC=${1:-$MICROSLOP_FLEXTALK_DIR/extracted/TTS_Executables} [ -d "$SRC/EngineDLLs" ] || { echo "source not found: $SRC" >&2; exit 1; } -export WINEPREFIX=${WINEPREFIX:-$HOME/.wine-microslopsam} -export WINEARCH=win32 - if [ -z "${DISPLAY:-}" ]; then - WRUN="xvfb-run -a wine" + WRUN="xvfb-run -a $MICROSLOP_WINE" else - WRUN="wine" + WRUN="$MICROSLOP_WINE" fi DEST="$WINEPREFIX/drive_c/FlexTalk" @@ -92,4 +89,4 @@ echo "== registry" $WRUN regedit /S "$REG" 2>/dev/null || true echo "== done; list voices:" -echo " $WRUN $ROOT/host/sapi4host.exe -l" +echo " $WRUN $MICROSLOP_BUILD/sapi4host.exe -l" diff --git a/microslopsam/setup/setup-prefix.sh b/microslopsam/setup/setup-prefix.sh index 9729a62..e54db37 100755 --- a/microslopsam/setup/setup-prefix.sh +++ b/microslopsam/setup/setup-prefix.sh @@ -7,18 +7,15 @@ set -eu HERE=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd) -ROOT=$(CDPATH= cd -- "$HERE/.." && pwd) -ASSETS=${1:-$ROOT/../Microsoft-Sam-Mary-Mike-TruVoice-WSAPI4} +. "$HERE/config.sh" +ASSETS=${1:-$MICROSLOP_ASSETS} [ -d "$ASSETS/spchapi" ] || { echo "asset dir not found: $ASSETS" >&2; exit 1; } -export WINEPREFIX=${WINEPREFIX:-$HOME/.wine-microslopsam} -export WINEARCH=win32 - if [ -z "${DISPLAY:-}" ]; then - WRUN="xvfb-run -a wine" + WRUN="xvfb-run -a $MICROSLOP_WINE" else - WRUN="wine" + WRUN="$MICROSLOP_WINE" fi echo "== prefix: $WINEPREFIX" @@ -48,4 +45,4 @@ $WRUN regedit /S "$HERE/out-spchapi.reg" $WRUN regedit /S "$HERE/out-mstts.reg" echo "== done" -echo "list voices: $WRUN $ROOT/host/sapi4host.exe -l" +echo "list voices: $WRUN $MICROSLOP_BUILD/sapi4host.exe -l" diff --git a/microslopsam/setup/veskslop-tts.sh b/microslopsam/setup/veskslop-tts.sh index 6b96170..44fec36 100755 --- a/microslopsam/setup/veskslop-tts.sh +++ b/microslopsam/setup/veskslop-tts.sh @@ -18,14 +18,15 @@ # VESKSLOP_TTS_ECHO mono echo amount 0-100 (default 0) # VESKSLOP_TTS_WORDS 1 = synthesize each word, merge (Flextalk) # VESKSLOP_TTS_WORD_GAP_MS silence between words in ms (0-500) +# VESKSLOP_TTS_VOX 1 = VOX prosody, NetTTS-style (Flextalk) +# VESKSLOP_TTS_VOX_CLEAN 1 = VOX without the \!wH1/\!wH0 stutter # # Output: 48 kHz mono s16le WAV, trimmed, faded, loudness-normalized. set -eu HERE=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd) -ROOT=$(CDPATH= cd -- "$HERE/.." && pwd) -export WINEPREFIX=${WINEPREFIX:-$HOME/.wine-microslopsam} -export WINEARCH=win32 +. "$HERE/config.sh" +ROOT=$MICROSLOP_ROOT # VESKSLOP_TTS_VOICES entries: id|Name|gender|language[|flags] # `expression=NN` marks a Flextalk voice with an Expression profile parameter and @@ -49,6 +50,8 @@ HIGHPASS=${VESKSLOP_TTS_HIGHPASS:-40} ECHO=${VESKSLOP_TTS_ECHO:-0} WORDS=${VESKSLOP_TTS_WORDS:-0} WORD_GAP=${VESKSLOP_TTS_WORD_GAP_MS:-120} +VOX=${VESKSLOP_TTS_VOX:-0} +VOX_CLEAN=${VESKSLOP_TTS_VOX_CLEAN:-0} case "$VOICE" in sam) ENGVOICE="Sam" ;; @@ -83,6 +86,66 @@ case "$ENGVOICE" in "Kerry Watson") SMAX=390; PMAX=250; NAT_PITCH=186 ;; *) SMAX=450; PMAX=200; NAT_PITCH=100 ;; esac +# The daemon does not export DISPLAY; keep a private Xvfb for the engine. +start_xvfb() { + if ! [ -S "$MICROSLOP_XSOCK" ]; then + "$MICROSLOP_XVFB" "$MICROSLOP_DISPLAY" -screen 0 1024x768x24 >/dev/null 2>&1 & + sleep 1 + fi +} +if [ -z "${DISPLAY:-}" ]; then + start_xvfb + export DISPLAY=$MICROSLOP_DISPLAY +fi + +# NetTTS-style line controls, handled here so no caller/WebUI work is needed: +# /rate N and/or /pitch N at the start of the line. +# N is a percent (100 = normal); values <= 2.0 are multipliers (/rate 1.5 = 150%). +# They override the env speed/pitch for this line and are not spoken. A line that +# contains only commands yields 100 ms of silence, so the daemon still gets a WAV. +TMP=$(mktemp -d) +trap 'rm -rf "$TMP"' EXIT +RAW="$TMP/raw.wav" +TXT="$TMP/text.cp1252" +TEXT_IN="$TEXTFILE" +if [ -f "$TEXTFILE" ]; then + set -f + # shellcheck disable=SC2046 + set -- $(cat "$TEXTFILE") + set +f + CMD_SPEED="" + CMD_PITCH="" + while [ "$#" -ge 2 ]; do + case "$1" in + /rate|/pitch) + case "$2" in + ''|*[!0-9.]*) break ;; + esac + if [ "$1" = "/rate" ]; then CMD_SPEED=$2; else CMD_PITCH=$2; fi + shift 2 + ;; + *) break ;; + esac + done + if [ -n "$CMD_SPEED" ] || [ -n "$CMD_PITCH" ]; then + if [ -n "$CMD_SPEED" ]; then + SPEED=$(awk -v v="$CMD_SPEED" 'BEGIN{ if (v<=2.0) v*=100; if (v<0) v=0; if (v>200) v=200; printf "%.4f", v/100 }') + fi + if [ -n "$CMD_PITCH" ]; then + PITCH=$(awk -v v="$CMD_PITCH" 'BEGIN{ if (v<=2.0) v*=100; if (v<0) v=0; if (v>200) v=200; printf "%.4f", 12*log(v/100)/log(2) }') + fi + if [ -z "$*" ]; then + "$MICROSLOP_FFMPEG" -hide_banner -loglevel error -f lavfi -i anullsrc=r=48000:cl=mono \ + -t 0.1 -c:a pcm_s16le -y "$OUT" >/dev/null 2>&1 \ + || { echo "could not write silence" >&2; exit 1; } + exit 0 + fi + printf '%s\n' "$*" > "$TMP/input.txt" + TEXT_IN="$TMP/input.txt" + fi +fi + +# Build the engine arguments from the (possibly command-overridden) speed/pitch. SET_ARGS="" if [ -n "$SPEED" ]; then S=$(awk -v m="$SPEED" 'BEGIN{printf "%d", 150*m+0.5}') @@ -97,23 +160,6 @@ if [ -n "$PITCH" ]; then SET_ARGS="$SET_ARGS -p $P" fi -# The daemon does not export DISPLAY; keep a private Xvfb for the engine. -start_xvfb() { - if ! [ -S /tmp/.X11-unix/X77 ]; then - Xvfb :77 -screen 0 1024x768x24 >/dev/null 2>&1 & - sleep 1 - fi -} -if [ -z "${DISPLAY:-}" ]; then - start_xvfb - export DISPLAY=:77 -fi - -TMP=$(mktemp -d) -trap 'rm -rf "$TMP"' EXIT -RAW="$TMP/raw.wav" -TXT="$TMP/text.cp1252" - # Apply the Flextalk profile before synthesis. Profiles are read at voice load and # the host loads the engine per request, so the values are written every call: the # engine defaults plus any VESKSLOP_TTS_PROFILE overrides. A missing value therefore @@ -187,28 +233,46 @@ if [ -n "$SPEAKER_GUID" ]; then exit 1 fi - if ! command -v wine >/dev/null 2>&1; then + if ! command -v "$MICROSLOP_WINE" >/dev/null 2>&1; then echo "wine is required to set the Flextalk profile" >&2 exit 1 fi - if ! wine reg import "$TMP/profile.reg" >/dev/null 2>&1; then + if ! "$MICROSLOP_WINE" reg import "$TMP/profile.reg" >/dev/null 2>&1; then echo "could not write the Flextalk profile" >&2 exit 1 fi fi -# SAPI4 is ANSI/CP1252; transliterate UTF-8 (emoji become '?'). -if command -v iconv >/dev/null 2>&1; then - iconv -f UTF-8 -t WINDOWS-1252//TRANSLIT "$TEXTFILE" > "$TXT" 2>/dev/null \ - || cp "$TEXTFILE" "$TXT" -else - cp "$TEXTFILE" "$TXT" +# SAPI4 is ANSI/CP1252. VOX mode (Flextalk only) first transforms the text into +# vendor tags (\!wH1/\!wH0, \!br, \!sfNN) with setup/vox_parser.py; the parser +# works on UTF-8, so it runs before the CP1252 transliteration. +TEXT_SRC="$TEXT_IN" +if [ "$VOX" = "1" ] && [ -n "$SPEAKER_GUID" ]; then + if command -v "$MICROSLOP_PYTHON" >/dev/null 2>&1 && [ -f "$MICROSLOP_ROOT/setup/vox_parser.py" ]; then + if [ "$VOX_CLEAN" = "1" ]; then + "$MICROSLOP_PYTHON" "$MICROSLOP_ROOT/setup/vox_parser.py" --clean < "$TEXT_IN" > "$TMP/vox.txt" + else + "$MICROSLOP_PYTHON" "$MICROSLOP_ROOT/setup/vox_parser.py" < "$TEXT_IN" > "$TMP/vox.txt" + fi + TEXT_SRC="$TMP/vox.txt" + else + echo "vox mode needs python3 and setup/vox_parser.py" >&2 + exit 1 + fi fi -HOST="$ROOT/build/sapi4host-winlib.exe" -if [ ! -x "$HOST" ]; then HOST="$ROOT/build/sapi4host.exe"; fi +# transliterate UTF-8 (emoji become '?') +if command -v iconv >/dev/null 2>&1; then + iconv -f UTF-8 -t WINDOWS-1252//TRANSLIT "$TEXT_SRC" > "$TXT" 2>/dev/null \ + || cp "$TEXT_SRC" "$TXT" +else + cp "$TEXT_SRC" "$TXT" +fi + +HOST="$MICROSLOP_BUILD/sapi4host-winlib.exe" +if [ ! -x "$HOST" ]; then HOST="$MICROSLOP_BUILD/sapi4host.exe"; fi if [ ! -x "$HOST" ]; then echo "engine host not built" >&2; exit 1; fi -if ! command -v ffmpeg >/dev/null 2>&1; then echo "ffmpeg not found" >&2; exit 1; fi +if ! command -v "$MICROSLOP_FFMPEG" >/dev/null 2>&1; then echo "ffmpeg not found" >&2; exit 1; fi run_host() { # shellcheck disable=SC2086 @@ -219,9 +283,9 @@ run_host() { try_run_host() { if run_host "$1" "$2"; then return 0; fi - if [ "${DISPLAY:-}" = ":77" ]; then - pkill -f "Xvfb :77" 2>/dev/null || true - rm -f /tmp/.X11-unix/X77 + if [ "${DISPLAY:-}" = "$MICROSLOP_DISPLAY" ]; then + pkill -f "Xvfb $MICROSLOP_DISPLAY" 2>/dev/null || true + rm -f "$MICROSLOP_XSOCK" start_xvfb if run_host "$1" "$2"; then return 0; fi fi @@ -232,7 +296,8 @@ try_run_host() { # Word mode (Flextalk): synthesize each word and merge with silence gaps, which # gives the staccato radio cadence the Half-Life VOX/HEV voices are known for. -if [ "$WORDS" = "1" ] && [ -n "$SPEAKER_GUID" ]; then +# VOX mode takes precedence: it produces its own cadence with vendor tags. +if [ "$WORDS" = "1" ] && [ -n "$SPEAKER_GUID" ] && [ "$VOX" != "1" ]; then GAP=$WORD_GAP case "$GAP" in *[!0-9]*|"") GAP=120 ;; esac [ "$GAP" -gt 500 ] && GAP=500 @@ -245,7 +310,7 @@ if [ "$WORDS" = "1" ] && [ -n "$SPEAKER_GUID" ]; then echo "file '$TMP/w$INDEX.wav'" >> "$TMP/list.txt" if [ "$GAP" -gt 0 ]; then - ffmpeg -hide_banner -loglevel error -f lavfi -i "anullsrc=r=11025:cl=mono" \ + "$MICROSLOP_FFMPEG" -hide_banner -loglevel error -f lavfi -i "anullsrc=r=11025:cl=mono" \ -t "$(awk -v g="$GAP" 'BEGIN{printf "%.3f", g/1000}')" -c:a pcm_s16le -y "$TMP/sil.wav" \ >/dev/null 2>&1 echo "file '$TMP/sil.wav'" >> "$TMP/list.txt" @@ -253,7 +318,7 @@ if [ "$WORDS" = "1" ] && [ -n "$SPEAKER_GUID" ]; then done [ "$INDEX" -gt 0 ] || { echo "no words to speak" >&2; exit 1; } - if ! ffmpeg -hide_banner -loglevel error -f concat -safe 0 -i "$TMP/list.txt" -c copy -y "$RAW" \ + if ! "$MICROSLOP_FFMPEG" -hide_banner -loglevel error -f concat -safe 0 -i "$TMP/list.txt" -c copy -y "$RAW" \ >/dev/null 2>&1; then echo "could not merge the words" >&2 exit 1 @@ -281,7 +346,7 @@ fi # Fixed-gain normalization: single-pass loudnorm misbehaves on short lines # (LRA=0 causes heavy attenuation), so measure EBU R128 integrated loudness and # true peak, then apply one gain clamped by the -3 dBFS peak target. -M=$(ffmpeg -hide_banner -nostats -i "$RAW" -af "$PRE,ebur128=peak=true" \ +M=$("$MICROSLOP_FFMPEG" -hide_banner -nostats -i "$RAW" -af "$PRE,ebur128=peak=true" \ -f null - 2>&1 || true) I=$(printf '%s' "$M" | sed -n 's/ *I: *\(-\{0,1\}[0-9.]*\) *LUFS.*/\1/p' | tail -1) TP=$(printf '%s' "$M" | sed -n 's/ *Peak: *\(-\{0,1\}[0-9.]*\) *dBFS.*/\1/p' | tail -1) @@ -299,7 +364,7 @@ else fi TMPOUT="$TMP/out.wav" -if ! ffmpeg -hide_banner -nostats -y -i "$RAW" -af "$PRE,volume=${GAIN}dB" \ +if ! "$MICROSLOP_FFMPEG" -hide_banner -nostats -y -i "$RAW" -af "$PRE,volume=${GAIN}dB" \ -ar 48000 -ac 1 -c:a pcm_s16le "$TMPOUT" >/dev/null 2>"$TMP/ff"; then head -c 400 "$TMP/ff" >&2 exit 1 diff --git a/microslopsam/setup/vox_parser.py b/microslopsam/setup/vox_parser.py new file mode 100644 index 0000000..879c014 --- /dev/null +++ b/microslopsam/setup/vox_parser.py @@ -0,0 +1,541 @@ +#!/usr/bin/env python3 +"""VOX prosody encoder for FlexTalk, ported from h4rm0n1c/NetTTS. + +Transforms plain text into FlexTalk vendor-tagged text: + + \\!wH1 ... \\!wH0 staccato word-cadence wrapper + \\!br beat boundary + \\!sfNN final pause (centiseconds) + +Our host passes TTSDATAFLAG_TAGGED, so the engine honors these tags. +Rule set studied from https://github.com/h4rm0n1c/NetTTS (vox_parser.cpp, +no license declared; this is an independent reimplementation). + +Usage: + vox_parser.py [--clean] [text] + echo "text" | vox_parser.py + +--clean omits the \\!wH1/\\!wH0 wrapper (plain cadence, no word stutter). +""" +import re +import sys + +BR = r'\!br' +WH1 = r'\!wH1' +WH0 = r'\!wH0' +TAGS = (BR, WH1, WH0) + +RE_LEADIN = re.compile( + r'^\s*((?:Now|Please|A\s+reminder|On\s+behalf\s+of|Good\s+(?:morning|evening)' + r'|This\s+(?:automated\s+train|tram)))\b', re.IGNORECASE | re.ASCII) +RE_TIME12 = re.compile( + r'\b(\d{1,2})[:.](\d{2})\s*((?:A\.?M\.?)|(?:P\.?M\.?))(?:\.)?\b', + re.IGNORECASE | re.ASCII) +RE_TIME_IS = re.compile(r'(\b[Tt]he\s+time\s+is)\s+(\d+)', re.ASCII) +RE_24H = re.compile(r'\b(\d{2})00\s+hours\b', re.IGNORECASE | re.ASCII) +RE_N3 = re.compile(r'\b([1-9])(\d)(\d)\b', re.ASCII) +RE_DEGREES = re.compile(r'\b(\d+)\s+degrees\b', re.IGNORECASE | re.ASCII) + +LIGHT, MEDIUM, HEAVY = 0, 1, 2 + +STOP_WORDS = { + 'the', 'thee', 'a', 'an', 'to', 'for', 'of', 'in', 'on', 'at', 'by', 'with', + 'from', 'as', 'is', 'are', 'was', 'were', 'this', 'that', +} +LIGHT_VERBS = { + 'welcome', 'wait', 'stand', 'provide', 'provided', 'return', 'remain', + 'maintain', 'maintained', 'verify', 'contact', 'board', 'arriving', + 'inbound', 'bound', 'commence', 'commences', +} +UNITS = {'degrees', 'hours', 'percent', '%'} +LINKERS = {'at', 'in', 'on', 'to', 'of', 'for', 'from', 'by'} +DETERMINERS = {'the', 'thee', 'a', 'an', 'this', 'that', 'these', 'those', + 'some', 'any', 'each', 'every', 'no'} +PUNCT = '.,;:!?\'"' + + +def trim(s): + return s.strip() + + +def lower(s): + return s.lower() + + +def bare(tok): + return strip_trailing_punct(tok) + + +def is_all_digits(t): + return bool(t) and all('0' <= c <= '9' for c in t) + + +def is_titlecase_word(t): + return bool(t) and t[0].isalpha() and t[0].isupper() + + +def strip_trailing_punct(t): + while t and t[-1] in PUNCT: + t = t[:-1] + return t + + +def split_tail(tok): + """Return (core, trailing punctuation).""" + core, tail = tok, '' + while core and core[-1] in PUNCT: + tail = core[-1] + tail + core = core[:-1] + return core, tail + + +def ends_with_br(s): + return s.endswith(' ' + BR) + + +def is_tag(tok): + return tok in TAGS + + +def split_sentences(text): + out, cur = [], '' + i = 0 + n = len(text) + while i < n: + c = text[i] + cur += c + if c in '.!?': + j = i + 1 + while j < n and text[j] in '"\'': + cur += text[j] + j += 1 + if trim(cur): + out.append(trim(cur)) + cur = '' + i = j - 1 + i += 1 + tail = trim(cur) + if tail: + out.append(tail) + return out + + +def apply_leadin_break(s): + m = RE_LEADIN.search(s) + if not m: + return s + head = trim(m.group(0)) + rest = trim(s[m.end():]) + return head + ' ' + BR + ' ' + rest + + +def is_single_letter(tok): + b = bare(tok) + if len(b) == 1 and b[0].isalpha() and b[0].isupper(): + return True + return len(b) == 2 and b[0].isalpha() and b[0].isupper() and b[1] == ':' + + +def is_number(tok): + b = bare(tok) + return is_all_digits(b) + + +def apply_thee_rule(s): + tok = s.split() + if not tok: + return s + + def prev_word(i, nth): + for k in range(i - 1, -1, -1): + if is_tag(tok[k]): + continue + nth -= 1 + if nth == 0: + return lower(split_tail(tok[k])[0]) + return '' + + def next_word(i, nth): + for k in range(i + 1, len(tok)): + if is_tag(tok[k]): + continue + nth -= 1 + if nth == 0: + return tok[k] + return '' + + for i in range(len(tok)): + if is_tag(tok[i]): + continue + core, tail = split_tail(tok[i]) + if lower(core) != 'the': + tok[i] = core + tail + continue + make_thee = False + p1 = prev_word(i, 1) + if p1: + if p1 in LINKERS: + make_thee = True + elif p1 == 'of': + p2, p3 = prev_word(i, 2), prev_word(i, 3) + if p2 == 'behalf' and p3 == 'on': + make_thee = True + if not make_thee: + n1 = next_word(i, 1) + if n1 and (is_titlecase_word(bare(n1)) or is_single_letter(n1) + or is_number(n1)): + make_thee = True + tok[i] = ('thee' if make_thee else core) + tail + return ' '.join(tok) + + +def apply_time_numbers_degrees(s): + def time12(m): + hh, mm, ap = m.group(1), m.group(2), lower(m.group(3)) + ap1 = 'P:' if ap.startswith('p') else 'Ay:' + return hh + ' ' + BR + ' ' + mm + ' ' + BR + ' ' + ap1 + ' ' + BR + ' M:' + + s = RE_TIME12.sub(time12, s) + s = RE_TIME_IS.sub(r'\1 ' + BR + r' \2', s) + s = RE_24H.sub(r'\1 ' + BR + ' hundred ' + BR + ' hours', s) + + def n3(m): + yz = m.group(2) + m.group(3) + if yz == '00': + return m.group(0) + rem = yz.lstrip('0') or yz + return m.group(1) + '00 ' + BR + ' and ' + BR + ' ' + rem + + s = RE_N3.sub(n3, s) + s = RE_DEGREES.sub(r'\1 ' + BR + ' degrees ' + BR, s) + s = re.sub(r'M:\s*\.', 'M:', s) + return s + + +def syllables(word): + if not word: + return 0 + w = lower(word) + vowels = set('aeiouy') + count = 0 + in_v = False + for c in w: + if c in vowels: + if not in_v: + count += 1 + in_v = True + else: + in_v = False + if count > 1 and len(w) > 1 and w[-1] == 'e' and not (len(w) > 2 and w[-2] == 'l'): + count -= 1 + if len(w) > 2 and w[-1] == 'e' and w[-2] == 'l': + count += 1 + return max(1, count) + + +def weight(tok): + if not tok: + return LIGHT + pfree = strip_trailing_punct(tok) + tl = lower(pfree) + if tl in STOP_WORDS or tl in LIGHT_VERBS: + return LIGHT + if is_all_digits(pfree): + return HEAVY + if '-' in pfree: + return HEAVY + if is_titlecase_word(pfree): + return HEAVY + if tl in UNITS: + return HEAVY + if syllables(pfree) >= 3 or len(pfree) >= 8: + return HEAVY + return MEDIUM + + +def is_and(tok): + return lower(strip_trailing_punct(tok)) == 'and' + + +def starts_with_punct(tok): + return bool(tok) and tok[0] in '.,;:!?)' + + +def is_open_punct(tok): + return bool(tok) and tok[0] in '("\'' + + +def emit_titlecase_run(v, i, j, out): + if out and not ends_with_br(out): + out += ' ' + BR + for k in range(i, j): + out += ' ' + v[k] + ' ' + BR + return out + + +def build_beats(sentence): + v = sentence.split() + if not v: + return sentence + + out = [''] + state = {'since_break_all_light': True, 'prev_w': LIGHT, 'content_run': 0} + + def reset(): + state['since_break_all_light'] = True + state['content_run'] = 0 + state['prev_w'] = LIGHT + + def ensure_break(): + if not ends_with_br(out[0]): + out[0] += ' ' + BR + + def append_token(tok): + if out[0] and not starts_with_punct(tok) and not is_open_punct(tok): + out[0] += ' ' + out[0] += tok + + def emit_dash_split(tok): + core, tail = split_tail(tok) + cuts = [] + for p, c in enumerate(core): + if c in ('-', '\u2013', '\u2014'): + if 0 < p < len(core) - 1 and core[p - 1].isalpha() and core[p + 1].isalpha(): + cuts.append(p) + else: + return False + if not cuts: + return False + start = 0 + for idx in range(len(cuts) + 1): + end = cuts[idx] if idx < len(cuts) else len(core) + part = core[start:end] + if out[0] and not ends_with_br(out[0]) and not starts_with_punct(part) and not is_open_punct(part): + out[0] += ' ' + if idx == len(cuts): + part += tail + out[0] += part + ensure_break() + reset() + start = end + (1 if idx < len(cuts) else 0) + return True + + i = 0 + while i < len(v): + # Proper-noun blocks + if bare(v[i]) in ('Area', 'Level', 'Sector'): + j = i + 1 + while j < len(v): + wj = bare(v[j]) + if is_all_digits(wj) or is_titlecase_word(wj): + j += 1 + else: + break + if j > i + 1: + out[0] = emit_titlecase_run(v, i, j, out[0]) + i = j + reset() + continue + if is_titlecase_word(bare(v[i])): + j = i + while j < len(v) and is_titlecase_word(bare(v[j])): + j += 1 + if j >= i + 2: + out[0] = emit_titlecase_run(v, i, j, out[0]) + i = j + reset() + continue + + # "and" between heavier items + if is_and(v[i]) and 0 < i < len(v) - 1: + if weight(v[i - 1]) >= MEDIUM and weight(v[i + 1]) >= MEDIUM: + ensure_break() + out[0] += ' and ' + BR + i += 1 + reset() + continue + + # Standalone dash + if v[i] in ('-', '\u2013', '\u2014'): + ensure_break() + reset() + i += 1 + continue + + # Symmetric linker isolation + tl_tok = lower(bare(v[i])) + if i >= 2 and tl_tok in LINKERS: + ensure_break() + reset() + append_token(v[i]) + ensure_break() + reset() + i += 1 + continue + + # Pre-copula break + if i >= 2 and lower(bare(v[i])) == 'is': + k = i + 1 + while k < len(v) and is_tag(v[k]): + k += 1 + trigger = False + if k < len(v): + n1 = v[k] + if (lower(strip_trailing_punct(n1)) in DETERMINERS + or is_titlecase_word(strip_trailing_punct(n1)) + or is_all_digits(strip_trailing_punct(n1)) + or is_single_letter(n1) + or weight(n1) == HEAVY): + trigger = True + if trigger and not ends_with_br(out[0]): + out[0] += ' ' + BR + reset() + + # Hyphenated compounds + if emit_dash_split(v[i]): + i += 1 + continue + + # Generic token logic + tok = v[i] + w_eff = weight(tok) + in_head_span = i < 2 + if i == 0 and is_titlecase_word(bare(tok)): + next_is_title = (i + 1 < len(v)) and is_titlecase_word(bare(v[i + 1])) + if not next_is_title and w_eff == HEAVY: + w_eff = MEDIUM + + if w_eff == HEAVY and state['content_run'] >= 1: + k = i - 1 + while k >= 0 and is_tag(v[k]): + k -= 1 + exempt = False + if k >= 0: + prev = v[k] + if (is_titlecase_word(bare(prev)) or is_all_digits(bare(prev)) + or is_single_letter(prev)): + exempt = True + if not exempt: + ensure_break() + reset() + + if w_eff == HEAVY and state['content_run'] >= 2: + ensure_break() + reset() + + if w_eff == HEAVY and state['since_break_all_light'] and not in_head_span: + if out[0] and not ends_with_br(out[0]): + out[0] += ' ' + BR + reset() + + if i > 0 and state['prev_w'] == HEAVY and w_eff == HEAVY and not in_head_span: + ensure_break() + reset() + + append_token(tok) + + if tok and tok[-1] in ',;' and not ends_with_br(out[0]): + out[0] += ' ' + BR + reset() + i += 1 + continue + + if w_eff == LIGHT: + pass + else: + state['content_run'] += 1 + state['since_break_all_light'] = False + state['prev_w'] = w_eff + i += 1 + + return trim(out[0]) + + +def normalize_letter_tokens(s): + tok = s.split() + if not tok: + return s + + def is_boundary(idx): + return idx < 0 or idx >= len(tok) or is_tag(tok[idx]) + + for i in range(len(tok)): + if is_tag(tok[i]): + continue + core, tail = tok[i], '' + had_colon = False + while core: + c = core[-1] + if c == ':' and not had_colon: + had_colon = True + core = core[:-1] + continue + if c in '.,;!?\'"': + tail = c + tail + core = core[:-1] + continue + break + near_bound = is_boundary(i - 1) or is_boundary(i + 1) + if core in ('A', 'a') and (near_bound or had_colon): + core = 'Ay' + had_colon = True + elif len(core) == 1 and core.isalpha() and core.isupper() and near_bound: + had_colon = True + tok[i] = core + (':' if had_colon else '') + tail + return ' '.join(tok) + + +def tidy(s): + t = re.sub(r'\s+', ' ', s) + t = re.sub(r'\s*' + re.escape(BR) + r'\s*', ' ' + BR + ' ', t) + t = re.sub(r'(\s*' + re.escape(BR) + r'\s*){2,}', ' ' + BR + ' ', t) + t = re.sub(r'\s+([.,!?;:])', r'\1', t) + return t.strip() + + +def vox_process(text, wrap_vox_tags=True): + out = '' + for sent in split_sentences(text): + if not sent: + continue + sent = apply_thee_rule(sent) + sent = apply_leadin_break(sent) + sent = apply_time_numbers_degrees(sent) + beats = build_beats(sent) + beats = normalize_letter_tokens(beats) + beats = tidy(beats) + if beats and not beats.endswith(' '): + beats += ' ' + beats += r'\!sf500 ' + BR + if out: + out += ' ' + out += beats + out = trim(out) + out = re.sub(r'(\s*' + re.escape(BR) + r'\s*){2,}', ' ' + BR + ' ', out) + out = re.sub(re.escape(BR) + r'\s*([,;:])', r'\1 ' + BR, out) + if out: + if wrap_vox_tags: + out = WH1 + ' ' + out + ' ' + WH0 + ' ' + else: + out = out + ' ' + return out + + +def main(): + args = sys.argv[1:] + clean = False + if '--clean' in args: + clean = True + args.remove('--clean') + if args: + lines = [' '.join(args)] + else: + lines = sys.stdin.read().splitlines() + out_lines = [vox_process(line, wrap_vox_tags=not clean) for line in lines if line.strip()] + sys.stdout.write('\n'.join(out_lines)) + if out_lines: + sys.stdout.write('\n') + + +if __name__ == '__main__': + main()