30 Commits

Author SHA1 Message Date
Shivam Mehta
b756809a32 Merge pull request #13 from shivammehta25/dev
Merging dev to main | adding ONNX support
2023-09-29 16:54:09 +02:00
Shivam Mehta
1ead4303f3 Version Bump 2023-09-29 14:50:46 +00:00
Shivam Mehta
7a29fef719 Merge pull request #12 from shivammehta25/dependabot/pip/dev/diffusers-0.21.3
Bump diffusers from 0.21.2 to 0.21.3
2023-09-29 16:48:13 +02:00
Shivam Mehta
9ace522249 Update README.md 2023-09-29 16:46:38 +02:00
Shivam Mehta
ed6e6bbf6c Merge branch 'ONNX_BRANCH' into dev 2023-09-29 14:43:52 +00:00
Shivam Mehta
51ea36d271 Merge pull request #8 from mush42/onnx
ONNX export and inference
2023-09-29 16:43:19 +02:00
Shivam Mehta
269609003b Adding onnx installation command in the README 2023-09-29 14:38:57 +00:00
dependabot[bot]
2a81800825 Bump diffusers from 0.21.2 to 0.21.3
Bumps [diffusers](https://github.com/huggingface/diffusers) from 0.21.2 to 0.21.3.
- [Release notes](https://github.com/huggingface/diffusers/releases)
- [Commits](https://github.com/huggingface/diffusers/compare/v0.21.2...v0.21.3)

---
updated-dependencies:
- dependency-name: diffusers
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-09-28 13:23:02 +00:00
mush42
336dd20d5b Use torch.onnx.is_in_onnx_export() instead of torch.jit.is_scripting() since the former is dedicated to this use case. 2023-09-26 15:28:15 +02:00
mush42
01c99161c4 - Fixed several bugs. Thanks @shivammehta25 for the suggestions 2023-09-26 14:21:17 +02:00
mush42
2c21a0edac Fixed an error encountered when loading the vocoder during export. 2023-09-24 20:28:59 +02:00
mush42
25767f76a8 Readme: added a note about GPU inference with onnxruntime. 2023-09-24 02:13:27 +02:00
mush42
1b204ed42c ONNX export and inference. Complete and tested implmentation. 2023-09-24 01:57:35 +02:00
Shivam Mehta
2cd057187b Update README.md
Add information about installation and compilation of monotonic alignment
2023-09-23 17:39:36 +02:00
Shivam Mehta
d373e9a5b1 Bumping it to an increased version 2023-09-21 13:43:20 +00:00
Shivam Mehta
f12be190a4 ADding video teaser to readme 2023-09-21 13:41:21 +00:00
Shivam Mehta
be998ae31f Merge pull request #4 from shivammehta25/dev
Another version bump because I am new to twine
2023-09-21 15:26:54 +02:00
Shivam Mehta
a49af19f48 Another version bump because I am new to twine 2023-09-21 13:25:40 +00:00
Shivam Mehta
7850aa0910 Merge pull request #3 from shivammehta25/dev
Adding multispeaker 🍵 Matcha-TTS
2023-09-21 15:23:15 +02:00
Shivam Mehta
309739b7cc Version bump 2023-09-21 13:20:17 +00:00
Shivam Mehta
0aaabf90c4 Minor UI fixes 2023-09-21 13:18:32 +00:00
Shivam Mehta
c5dab67d9f Adding teaser url 2023-09-21 13:15:36 +00:00
Shivam Mehta
d098f32730 bumping diffusers, changing depandabot to open PR to dev branch, adding url for multispeaker matcha checkpoint 2023-09-21 12:32:27 +00:00
Shivam Mehta
281a098337 better default speaking rate 2023-09-20 15:23:46 +00:00
Shivam Mehta
db95158043 Will have to load all models to enable multiple synthesis at the same time 2023-09-20 10:40:01 +00:00
Shivam Mehta
267bf96651 Adding multispeaker model in UI 2023-09-20 10:28:48 +00:00
Shivam Mehta
72635012b0 Adding matcha vctk 2023-09-20 07:08:11 +00:00
Gustav Eje Henter
9ceee279f0 Minor improvements to README.md 2023-09-18 18:44:13 +02:00
Shivam Mehta
d7b9a37359 adding more validation to multispeaker CLI 2023-09-18 11:37:58 +00:00
Shivam Mehta
ec43ef0732 Keeping ODE step slider to be greater than 0 always in gradio 2023-09-17 22:08:29 +00:00
14 changed files with 717 additions and 94 deletions

View File

@@ -7,6 +7,7 @@ version: 2
updates: updates:
- package-ecosystem: "pip" # See documentation for possible values - package-ecosystem: "pip" # See documentation for possible values
directory: "/" # Location of package manifests directory: "/" # Location of package manifests
target-branch: "dev"
schedule: schedule:
interval: "daily" interval: "daily"
ignore: ignore:

View File

@@ -17,7 +17,7 @@ create-package: ## Create wheel and tar gz
rm -rf dist/ rm -rf dist/
python setup.py bdist_wheel --plat-name=manylinux1_x86_64 python setup.py bdist_wheel --plat-name=manylinux1_x86_64
python setup.py sdist python setup.py sdist
python -m twine upload dist/* --verbose python -m twine upload dist/* --verbose --skip-existing
format: ## Run pre-commit hooks format: ## Run pre-commit hooks
pre-commit run -a pre-commit run -a

124
README.md
View File

@@ -19,20 +19,22 @@
> This is the official code implementation of 🍵 Matcha-TTS. > This is the official code implementation of 🍵 Matcha-TTS.
We propose 🍵 Matcha-TTS, a new approach to non-autoregressive neural TTS, that uses conditional flow matching (similar to rectified flows) to speed up ODE-based speech synthesis. Our method: We propose 🍵 Matcha-TTS, a new approach to non-autoregressive neural TTS, that uses [conditional flow matching](https://arxiv.org/abs/2210.02747) (similar to [rectified flows](https://arxiv.org/abs/2209.03003)) to speed up ODE-based speech synthesis. Our method:
- Is probabilistic - Is probabilistic
- Has compact memory footprint - Has compact memory footprint
- Sounds highly natural - Sounds highly natural
- Is very fast to synthesise from - Is very fast to synthesise from
Check out our [demo page](https://shivammehta25.github.io/Matcha-TTS). Read our [arXiv preprint for more details](https://arxiv.org/abs/2309.03199). Check out our [demo page](https://shivammehta25.github.io/Matcha-TTS) and read [our arXiv preprint](https://arxiv.org/abs/2309.03199) for more details.
[Pretrained models](https://drive.google.com/drive/folders/17C_gYgEHOxI5ZypcfE_k1piKCtyR0isJ?usp=sharing) will be auto downloaded with the CLI or gradio interface. [Pre-trained models](https://drive.google.com/drive/folders/17C_gYgEHOxI5ZypcfE_k1piKCtyR0isJ?usp=sharing) will be automatically downloaded with the CLI or gradio interface.
[Try 🍵 Matcha-TTS on HuggingFace 🤗 spaces!](https://huggingface.co/spaces/shivammehta25/Matcha-TTS) [Try 🍵 Matcha-TTS on HuggingFace 🤗 spaces!](https://huggingface.co/spaces/shivammehta25/Matcha-TTS)
<br> ## Watch the teaser
[![Watch the video](https://img.youtube.com/vi/xmvJkz3bqw0/hqdefault.jpg)](https://youtu.be/xmvJkz3bqw0)
## Installation ## Installation
@@ -43,7 +45,7 @@ conda create -n matcha-tts python=3.10 -y
conda activate matcha-tts conda activate matcha-tts
``` ```
2. Install Matcha TTS using pip or from source 2. Install Matcha TTS using pip or from source
```bash ```bash
pip install matcha-tts pip install matcha-tts
@@ -53,6 +55,8 @@ from source
```bash ```bash
pip install git+https://github.com/shivammehta25/Matcha-TTS.git pip install git+https://github.com/shivammehta25/Matcha-TTS.git
cd Matcha-TTS
pip install -e .
``` ```
3. Run CLI / gradio app / jupyter notebook 3. Run CLI / gradio app / jupyter notebook
@@ -110,26 +114,13 @@ matcha-tts --text "<INPUT TEXT>" --temperature 0.667
matcha-tts --text "<INPUT TEXT>" --steps 10 matcha-tts --text "<INPUT TEXT>" --steps 10
``` ```
## Citation information
If you find this work useful, please cite our paper:
```text
@article{mehta2023matcha,
title={Matcha-TTS: A fast TTS architecture with conditional flow matching},
author={Mehta, Shivam and Tu, Ruibo and Beskow, Jonas and Sz{\'e}kely, {\'E}va and Henter, Gustav Eje},
journal={arXiv preprint arXiv:2309.03199},
year={2023}
}
```
## Train with your own dataset ## Train with your own dataset
Let's assume we are training with LJSpeech Let's assume we are training with LJ Speech
1. Download the dataset from [here](https://keithito.com/LJ-Speech-Dataset/), extract it to `data/LJSpeech-1.1`, and prepare the filelists to point to the extracted data like the [5th point of setup in Tacotron2 repo](https://github.com/NVIDIA/tacotron2#setup). 1. Download the dataset from [here](https://keithito.com/LJ-Speech-Dataset/), extract it to `data/LJSpeech-1.1`, and prepare the file lists to point to the extracted data like for [item 5 in the setup of the NVIDIA Tacotron 2 repo](https://github.com/NVIDIA/tacotron2#setup).
2. Clone and enter this repository 2. Clone and enter the Matcha-TTS repository
```bash ```bash
git clone https://github.com/shivammehta25/Matcha-TTS.git git clone https://github.com/shivammehta25/Matcha-TTS.git
@@ -167,7 +158,7 @@ data_statistics: # Computed for ljspeech dataset
to the paths of your train and validation filelists. to the paths of your train and validation filelists.
5. Run the training script 6. Run the training script
```bash ```bash
make train-ljspeech make train-ljspeech
@@ -191,20 +182,97 @@ python matcha/train.py experiment=ljspeech_min_memory
python matcha/train.py experiment=ljspeech trainer.devices=[0,1] python matcha/train.py experiment=ljspeech trainer.devices=[0,1]
``` ```
6. Synthesise from the custom trained model 7. Synthesise from the custom trained model
```bash ```bash
matcha-tts --text "<INPUT TEXT>" --checkpoint_path <PATH TO CHECKPOINT> matcha-tts --text "<INPUT TEXT>" --checkpoint_path <PATH TO CHECKPOINT>
``` ```
## ONNX support
> Special thanks to [@mush42](https://github.com/mush42) for implementing ONNX export and inference support.
It is possible to export Matcha checkpoints to [ONNX](https://onnx.ai/), and run inference on the exported ONNX graph.
### ONNX export
To export a checkpoint to ONNX, first install ONNX with
```bash
pip install onnx
```
then run the following:
```bash
python3 -m matcha.onnx.export matcha.ckpt model.onnx --n-timesteps 5
```
Optionally, the ONNX exporter accepts **vocoder-name** and **vocoder-checkpoint** arguments. This enables you to embed the vocoder in the exported graph and generate waveforms in a single run (similar to end-to-end TTS systems).
**Note** that `n_timesteps` is treated as a hyper-parameter rather than a model input. This means you should specify it during export (not during inference). If not specified, `n_timesteps` is set to **5**.
**Important**: for now, torch>=2.1.0 is needed for export since the `scaled_product_attention` operator is not exportable in older versions. Until the final version is released, those who want to export their models must install torch>=2.1.0 manually as a pre-release.
### ONNX Inference
To run inference on the exported model, first install `onnxruntime` using
```bash
pip install onnxruntime
pip install onnxruntime-gpu # for GPU inference
```
then use the following:
```bash
python3 -m matcha.onnx.infer model.onnx --text "hey" --output-dir ./outputs
```
You can also control synthesis parameters:
```bash
python3 -m matcha.onnx.infer model.onnx --text "hey" --output-dir ./outputs --temperature 0.4 --speaking_rate 0.9 --spk 0
```
To run inference on **GPU**, make sure to install **onnxruntime-gpu** package, and then pass `--gpu` to the inference command:
```bash
python3 -m matcha.onnx.infer model.onnx --text "hey" --output-dir ./outputs --gpu
```
If you exported only Matcha to ONNX, this will write mel-spectrogram as graphs and `numpy` arrays to the output directory.
If you embedded the vocoder in the exported graph, this will write `.wav` audio files to the output directory.
If you exported only Matcha to ONNX, and you want to run a full TTS pipeline, you can pass a path to a vocoder model in `ONNX` format:
```bash
python3 -m matcha.onnx.infer model.onnx --text "hey" --output-dir ./outputs --vocoder hifigan.small.onnx
```
This will write `.wav` audio files to the output directory.
## Citation information
If you use our code or otherwise find this work useful, please cite our paper:
```text
@article{mehta2023matcha,
title={Matcha-TTS: A fast TTS architecture with conditional flow matching},
author={Mehta, Shivam and Tu, Ruibo and Beskow, Jonas and Sz{\'e}kely, {\'E}va and Henter, Gustav Eje},
journal={arXiv preprint arXiv:2309.03199},
year={2023}
}
```
## Acknowledgements ## Acknowledgements
Since this code uses: [Lightning-Hydra-Template](https://github.com/ashleve/lightning-hydra-template), you have all the powers that comes with it. Since this code uses [Lightning-Hydra-Template](https://github.com/ashleve/lightning-hydra-template), you have all the powers that come with it.
Other source codes I would like to acknowledge: Other source code I would like to acknowledge:
- [Coqui-TTS](https://github.com/coqui-ai/TTS/tree/dev) :For helping me figure out how to make cython binaries pip installable and encouragement - [Coqui-TTS](https://github.com/coqui-ai/TTS/tree/dev): For helping me figure out how to make cython binaries pip installable and encouragement
- [Hugging Face Diffusers](https://huggingface.co/): For their awesome diffusers library and its components - [Hugging Face Diffusers](https://huggingface.co/): For their awesome diffusers library and its components
- [Grad-TTS](https://github.com/huawei-noah/Speech-Backbones/tree/main/Grad-TTS): For source code of MAS - [Grad-TTS](https://github.com/huawei-noah/Speech-Backbones/tree/main/Grad-TTS): For the monotonic alignment search source code
- [torchdyn](https://github.com/DiffEqML/torchdyn): Useful for trying other ODE solvers during research and development - [torchdyn](https://github.com/DiffEqML/torchdyn): Useful for trying other ODE solvers during research and development
- [labml.ai](https://nn.labml.ai/transformers/rope/index.html): For RoPE implementation - [labml.ai](https://nn.labml.ai/transformers/rope/index.html): For the RoPE implementation

View File

@@ -7,8 +7,8 @@
task_name: "debug" task_name: "debug"
# disable callbacks and loggers during debugging # disable callbacks and loggers during debugging
callbacks: null # callbacks: null
logger: null # logger: null
extras: extras:
ignore_warnings: False ignore_warnings: False

View File

@@ -7,6 +7,9 @@ defaults:
trainer: trainer:
max_epochs: 1 max_epochs: 1
profiler: "simple" # profiler: "simple"
# profiler: "advanced" profiler: "advanced"
# profiler: "pytorch" # profiler: "pytorch"
accelerator: gpu
limit_train_batches: 0.02

View File

@@ -1 +1 @@
0.0.1.dev4 0.0.4

View File

@@ -8,7 +8,7 @@ import torch
from matcha.cli import ( from matcha.cli import (
MATCHA_URLS, MATCHA_URLS,
VOCODER_URL, VOCODER_URLS,
assert_model_downloaded, assert_model_downloaded,
get_device, get_device,
load_matcha, load_matcha,
@@ -22,20 +22,73 @@ LOCATION = Path(get_user_data_dir())
args = Namespace( args = Namespace(
cpu=False, cpu=False,
model="matcha_ljspeech", model="matcha_vctk",
vocoder="hifigan_T2_v1", vocoder="hifigan_univ_v1",
spk=None, spk=0,
) )
MATCHA_TTS_LOC = LOCATION / f"{args.model}.ckpt" CURRENTLY_LOADED_MODEL = args.model
VOCODER_LOC = LOCATION / f"{args.vocoder}"
MATCHA_TTS_LOC = lambda x: LOCATION / f"{x}.ckpt" # noqa: E731
VOCODER_LOC = lambda x: LOCATION / f"{x}" # noqa: E731
LOGO_URL = "https://shivammehta25.github.io/Matcha-TTS/images/logo.png" LOGO_URL = "https://shivammehta25.github.io/Matcha-TTS/images/logo.png"
assert_model_downloaded(MATCHA_TTS_LOC, MATCHA_URLS[args.model]) RADIO_OPTIONS = {
assert_model_downloaded(VOCODER_LOC, VOCODER_URL[args.vocoder]) "Multi Speaker (VCTK)": {
"model": "matcha_vctk",
"vocoder": "hifigan_univ_v1",
},
"Single Speaker (LJ Speech)": {
"model": "matcha_ljspeech",
"vocoder": "hifigan_T2_v1",
},
}
# Ensure all the required models are downloaded
assert_model_downloaded(MATCHA_TTS_LOC("matcha_ljspeech"), MATCHA_URLS["matcha_ljspeech"])
assert_model_downloaded(VOCODER_LOC("hifigan_T2_v1"), VOCODER_URLS["hifigan_T2_v1"])
assert_model_downloaded(MATCHA_TTS_LOC("matcha_vctk"), MATCHA_URLS["matcha_vctk"])
assert_model_downloaded(VOCODER_LOC("hifigan_univ_v1"), VOCODER_URLS["hifigan_univ_v1"])
device = get_device(args) device = get_device(args)
model = load_matcha(args.model, MATCHA_TTS_LOC, device) # Load default model
vocoder, denoiser = load_vocoder(args.vocoder, VOCODER_LOC, device) model = load_matcha(args.model, MATCHA_TTS_LOC(args.model), device)
vocoder, denoiser = load_vocoder(args.vocoder, VOCODER_LOC(args.vocoder), device)
def load_model(model_name, vocoder_name):
model = load_matcha(model_name, MATCHA_TTS_LOC(model_name), device)
vocoder, denoiser = load_vocoder(vocoder_name, VOCODER_LOC(vocoder_name), device)
return model, vocoder, denoiser
def load_model_ui(model_type, textbox):
model_name, vocoder_name = RADIO_OPTIONS[model_type]["model"], RADIO_OPTIONS[model_type]["vocoder"]
global model, vocoder, denoiser, CURRENTLY_LOADED_MODEL # pylint: disable=global-statement
if CURRENTLY_LOADED_MODEL != model_name:
model, vocoder, denoiser = load_model(model_name, vocoder_name)
CURRENTLY_LOADED_MODEL = model_name
if model_name == "matcha_ljspeech":
spk_slider = gr.update(visible=False, value=-1)
single_speaker_examples = gr.update(visible=True)
multi_speaker_examples = gr.update(visible=False)
length_scale = gr.update(value=0.95)
else:
spk_slider = gr.update(visible=True, value=0)
single_speaker_examples = gr.update(visible=False)
multi_speaker_examples = gr.update(visible=True)
length_scale = gr.update(value=0.85)
return (
textbox,
gr.update(interactive=True),
spk_slider,
single_speaker_examples,
multi_speaker_examples,
length_scale,
)
@torch.inference_mode() @torch.inference_mode()
@@ -45,13 +98,14 @@ def process_text_gradio(text):
@torch.inference_mode() @torch.inference_mode()
def synthesise_mel(text, text_length, n_timesteps, temperature, length_scale): def synthesise_mel(text, text_length, n_timesteps, temperature, length_scale, spk):
spk = torch.tensor([spk], device=device, dtype=torch.long) if spk >= 0 else None
output = model.synthesise( output = model.synthesise(
text, text,
text_length, text_length,
n_timesteps=n_timesteps, n_timesteps=n_timesteps,
temperature=temperature, temperature=temperature,
spks=args.spk, spks=spk,
length_scale=length_scale, length_scale=length_scale,
) )
output["waveform"] = to_waveform(output["mel"], vocoder, denoiser) output["waveform"] = to_waveform(output["mel"], vocoder, denoiser)
@@ -61,9 +115,27 @@ def synthesise_mel(text, text_length, n_timesteps, temperature, length_scale):
return fp.name, plot_tensor(output["mel"].squeeze().cpu().numpy()) return fp.name, plot_tensor(output["mel"].squeeze().cpu().numpy())
def run_full_synthesis(text, n_timesteps, mel_temp, length_scale): def multispeaker_example_cacher(text, n_timesteps, mel_temp, length_scale, spk):
global CURRENTLY_LOADED_MODEL # pylint: disable=global-statement
if CURRENTLY_LOADED_MODEL != "matcha_vctk":
global model, vocoder, denoiser # pylint: disable=global-statement
model, vocoder, denoiser = load_model("matcha_vctk", "hifigan_univ_v1")
CURRENTLY_LOADED_MODEL = "matcha_vctk"
phones, text, text_lengths = process_text_gradio(text) phones, text, text_lengths = process_text_gradio(text)
audio, mel_spectrogram = synthesise_mel(text, text_lengths, n_timesteps, mel_temp, length_scale) audio, mel_spectrogram = synthesise_mel(text, text_lengths, n_timesteps, mel_temp, length_scale, spk)
return phones, audio, mel_spectrogram
def ljspeech_example_cacher(text, n_timesteps, mel_temp, length_scale, spk=-1):
global CURRENTLY_LOADED_MODEL # pylint: disable=global-statement
if CURRENTLY_LOADED_MODEL != "matcha_ljspeech":
global model, vocoder, denoiser # pylint: disable=global-statement
model, vocoder, denoiser = load_model("matcha_ljspeech", "hifigan_T2_v1")
CURRENTLY_LOADED_MODEL = "matcha_ljspeech"
phones, text, text_lengths = process_text_gradio(text)
audio, mel_spectrogram = synthesise_mel(text, text_lengths, n_timesteps, mel_temp, length_scale, spk)
return phones, audio, mel_spectrogram return phones, audio, mel_spectrogram
@@ -92,20 +164,31 @@ def main():
with gr.Box(): with gr.Box():
with gr.Row(): with gr.Row():
gr.Markdown(description, scale=3) gr.Markdown(description, scale=3)
gr.Image(LOGO_URL, label="Matcha-TTS logo", height=150, width=150, scale=1, show_label=False) with gr.Column():
gr.Image(LOGO_URL, label="Matcha-TTS logo", height=50, width=50, scale=1, show_label=False)
html = '<br><iframe width="560" height="315" src="https://www.youtube.com/embed/xmvJkz3bqw0?si=jN7ILyDsbPwJCGoa" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>'
gr.HTML(html)
with gr.Box(): with gr.Box():
radio_options = list(RADIO_OPTIONS.keys())
model_type = gr.Radio(
radio_options, value=radio_options[0], label="Choose a Model", interactive=True, container=False
)
with gr.Row(): with gr.Row():
gr.Markdown("# Text Input") gr.Markdown("# Text Input")
with gr.Row(): with gr.Row():
text = gr.Textbox(value="", lines=2, label="Text to synthesise") text = gr.Textbox(value="", lines=2, label="Text to synthesise", scale=3)
spk_slider = gr.Slider(
minimum=0, maximum=107, step=1, value=args.spk, label="Speaker ID", interactive=True, scale=1
)
with gr.Row(): with gr.Row():
gr.Markdown("### Hyper parameters") gr.Markdown("### Hyper parameters")
with gr.Row(): with gr.Row():
n_timesteps = gr.Slider( n_timesteps = gr.Slider(
label="Number of ODE steps", label="Number of ODE steps",
minimum=0, minimum=1,
maximum=100, maximum=100,
step=1, step=1,
value=10, value=10,
@@ -142,58 +225,110 @@ def main():
# with gr.Row(): # with gr.Row():
audio = gr.Audio(interactive=False, label="Audio") audio = gr.Audio(interactive=False, label="Audio")
with gr.Row(): with gr.Row(visible=False) as example_row_lj_speech:
examples = gr.Examples( # pylint: disable=unused-variable examples = gr.Examples( # pylint: disable=unused-variable
examples=[ examples=[
[ [
"We propose Matcha-TTS, a new approach to non-autoregressive neural TTS, that uses conditional flow matching (similar to rectified flows) to speed up O D E-based speech synthesis.", "We propose Matcha-TTS, a new approach to non-autoregressive neural TTS, that uses conditional flow matching (similar to rectified flows) to speed up O D E-based speech synthesis.",
50, 50,
0.677, 0.677,
1.0, 0.95,
], ],
[ [
"The Secret Service believed that it was very doubtful that any President would ride regularly in a vehicle with a fixed top, even though transparent.", "The Secret Service believed that it was very doubtful that any President would ride regularly in a vehicle with a fixed top, even though transparent.",
2, 2,
0.677, 0.677,
1.0, 0.95,
], ],
[ [
"The Secret Service believed that it was very doubtful that any President would ride regularly in a vehicle with a fixed top, even though transparent.", "The Secret Service believed that it was very doubtful that any President would ride regularly in a vehicle with a fixed top, even though transparent.",
4, 4,
0.677, 0.677,
1.0, 0.95,
], ],
[ [
"The Secret Service believed that it was very doubtful that any President would ride regularly in a vehicle with a fixed top, even though transparent.", "The Secret Service believed that it was very doubtful that any President would ride regularly in a vehicle with a fixed top, even though transparent.",
10, 10,
0.677, 0.677,
1.0, 0.95,
], ],
[ [
"The Secret Service believed that it was very doubtful that any President would ride regularly in a vehicle with a fixed top, even though transparent.", "The Secret Service believed that it was very doubtful that any President would ride regularly in a vehicle with a fixed top, even though transparent.",
50, 50,
0.677, 0.677,
1.0, 0.95,
], ],
[ [
"The narrative of these events is based largely on the recollections of the participants.", "The narrative of these events is based largely on the recollections of the participants.",
10, 10,
0.677, 0.677,
1.0, 0.95,
], ],
[ [
"The jury did not believe him, and the verdict was for the defendants.", "The jury did not believe him, and the verdict was for the defendants.",
10, 10,
0.677, 0.677,
1.0, 0.95,
], ],
], ],
fn=run_full_synthesis, fn=ljspeech_example_cacher,
inputs=[text, n_timesteps, mel_temp, length_scale], inputs=[text, n_timesteps, mel_temp, length_scale],
outputs=[phonetised_text, audio, mel_spectrogram], outputs=[phonetised_text, audio, mel_spectrogram],
cache_examples=True, cache_examples=True,
) )
with gr.Row() as example_row_multispeaker:
multi_speaker_examples = gr.Examples( # pylint: disable=unused-variable
examples=[
[
"Hello everyone! I am speaker 0 and I am here to tell you that Matcha-TTS is amazing!",
10,
0.677,
0.85,
0,
],
[
"Hello everyone! I am speaker 16 and I am here to tell you that Matcha-TTS is amazing!",
10,
0.677,
0.85,
16,
],
[
"Hello everyone! I am speaker 44 and I am here to tell you that Matcha-TTS is amazing!",
50,
0.677,
0.85,
44,
],
[
"Hello everyone! I am speaker 45 and I am here to tell you that Matcha-TTS is amazing!",
50,
0.677,
0.85,
45,
],
[
"Hello everyone! I am speaker 58 and I am here to tell you that Matcha-TTS is amazing!",
4,
0.677,
0.85,
58,
],
],
fn=multispeaker_example_cacher,
inputs=[text, n_timesteps, mel_temp, length_scale, spk_slider],
outputs=[phonetised_text, audio, mel_spectrogram],
cache_examples=True,
label="Multi Speaker Examples",
)
model_type.change(lambda x: gr.update(interactive=False), inputs=[synth_btn], outputs=[synth_btn]).then(
load_model_ui,
inputs=[model_type, text],
outputs=[text, synth_btn, spk_slider, example_row_lj_speech, example_row_multispeaker, length_scale],
)
synth_btn.click( synth_btn.click(
fn=process_text_gradio, fn=process_text_gradio,
inputs=[ inputs=[
@@ -204,11 +339,11 @@ def main():
queue=True, queue=True,
).then( ).then(
fn=synthesise_mel, fn=synthesise_mel,
inputs=[processed_text, processed_text_len, n_timesteps, mel_temp, length_scale], inputs=[processed_text, processed_text_len, n_timesteps, mel_temp, length_scale, spk_slider],
outputs=[audio, mel_spectrogram], outputs=[audio, mel_spectrogram],
) )
demo.queue(concurrency_count=5).launch(share=True) demo.queue().launch(share=True)
if __name__ == "__main__": if __name__ == "__main__":

View File

@@ -1,6 +1,7 @@
import argparse import argparse
import datetime as dt import datetime as dt
import os import os
import warnings
from pathlib import Path from pathlib import Path
import matplotlib.pyplot as plt import matplotlib.pyplot as plt
@@ -17,13 +18,20 @@ from matcha.text import sequence_to_text, text_to_sequence
from matcha.utils.utils import assert_model_downloaded, get_user_data_dir, intersperse from matcha.utils.utils import assert_model_downloaded, get_user_data_dir, intersperse
MATCHA_URLS = { MATCHA_URLS = {
"matcha_ljspeech": "https://drive.google.com/file/d/1BBzmMU7k3a_WetDfaFblMoN18GqQeHCg/view?usp=drive_link" "matcha_ljspeech": "https://drive.google.com/file/d/1BBzmMU7k3a_WetDfaFblMoN18GqQeHCg/view?usp=drive_link",
} # , "matcha_vctk": ""} # Coming soon "matcha_vctk": "https://drive.google.com/file/d/1enuxmfslZciWGAl63WGh2ekVo00FYuQ9/view?usp=drive_link",
}
MULTISPEAKER_MODEL = {"matcha_vctk"} VOCODER_URLS = {
SINGLESPEAKER_MODEL = {"matcha_ljspeech"} "hifigan_T2_v1": "https://drive.google.com/file/d/14NENd4equCBLyyCSke114Mv6YR_j_uFs/view?usp=drive_link",
"hifigan_univ_v1": "https://drive.google.com/file/d/1qpgI41wNXFcH-iKq1Y42JlBC9j0je8PW/view?usp=drive_link",
}
VOCODER_URL = {"hifigan_T2_v1": "https://drive.google.com/file/d/14NENd4equCBLyyCSke114Mv6YR_j_uFs/view?usp=drive_link"} MULTISPEAKER_MODEL = {
"matcha_vctk": {"vocoder": "hifigan_univ_v1", "speaking_rate": 0.85, "spk": 0, "spk_range": (0, 107)}
}
SINGLESPEAKER_MODEL = {"matcha_ljspeech": {"vocoder": "hifigan_T2_v1", "speaking_rate": 0.95, "spk": None}}
def plot_spectrogram_to_numpy(spectrogram, filename): def plot_spectrogram_to_numpy(spectrogram, filename):
@@ -62,10 +70,14 @@ def get_texts(args):
def assert_required_models_available(args): def assert_required_models_available(args):
save_dir = get_user_data_dir() save_dir = get_user_data_dir()
model_path = save_dir / f"{args.model}.ckpt" if not hasattr(args, "checkpoint_path") and args.checkpoint_path is None:
model_path = args.checkpoint_path
else:
model_path = save_dir / f"{args.model}.ckpt"
assert_model_downloaded(model_path, MATCHA_URLS[args.model])
vocoder_path = save_dir / f"{args.vocoder}" vocoder_path = save_dir / f"{args.vocoder}"
assert_model_downloaded(model_path, MATCHA_URLS[args.model]) assert_model_downloaded(vocoder_path, VOCODER_URLS[args.vocoder])
assert_model_downloaded(vocoder_path, VOCODER_URL[args.vocoder])
return {"matcha": model_path, "vocoder": vocoder_path} return {"matcha": model_path, "vocoder": vocoder_path}
@@ -81,7 +93,7 @@ def load_hifigan(checkpoint_path, device):
def load_vocoder(vocoder_name, checkpoint_path, device): def load_vocoder(vocoder_name, checkpoint_path, device):
print(f"[!] Loading {vocoder_name}!") print(f"[!] Loading {vocoder_name}!")
vocoder = None vocoder = None
if vocoder_name == "hifigan_T2_v1": if vocoder_name in ("hifigan_T2_v1", "hifigan_univ_v1"):
vocoder = load_hifigan(checkpoint_path, device) vocoder = load_hifigan(checkpoint_path, device)
else: else:
raise NotImplementedError( raise NotImplementedError(
@@ -124,21 +136,70 @@ def validate_args(args):
args.text or args.file args.text or args.file
), "Either text or file must be provided Matcha-T(ea)TTS need sometext to whisk the waveforms." ), "Either text or file must be provided Matcha-T(ea)TTS need sometext to whisk the waveforms."
assert args.temperature >= 0, "Sampling temperature cannot be negative" assert args.temperature >= 0, "Sampling temperature cannot be negative"
assert args.speaking_rate > 0, "Speaking rate must be greater than 0"
assert args.steps > 0, "Number of ODE steps must be greater than 0" assert args.steps > 0, "Number of ODE steps must be greater than 0"
if args.model in SINGLESPEAKER_MODEL:
assert args.spk is None, f"Speaker ID is not supported for {args.model}"
if args.spk is not None:
assert args.spk >= 0 and args.spk < 109, "Speaker ID must be between 0 and 108"
assert args.model in MULTISPEAKER_MODEL, "Speaker ID is only supported for multispeaker model"
if args.model in MULTISPEAKER_MODEL: if args.checkpoint_path is None:
if args.spk is None: # When using pretrained models
print("[!] Speaker ID not provided! Using speaker ID 0") if args.model in SINGLESPEAKER_MODEL.keys():
args.spk = 0 args = validate_args_for_single_speaker_model(args)
if args.model in MULTISPEAKER_MODEL:
args = validate_args_for_multispeaker_model(args)
else:
# When using a custom model
if args.vocoder != "hifigan_univ_v1":
warn_ = "[-] Using custom model checkpoint! I would suggest passing --vocoder hifigan_univ_v1, unless the custom model is trained on LJ Speech."
warnings.warn(warn_, UserWarning)
if args.speaking_rate is None:
args.speaking_rate = 1.0
if args.batched: if args.batched:
assert args.batch_size > 0, "Batch size must be greater than 0" assert args.batch_size > 0, "Batch size must be greater than 0"
assert args.speaking_rate > 0, "Speaking rate must be greater than 0"
return args
def validate_args_for_multispeaker_model(args):
if args.vocoder is not None:
if args.vocoder != MULTISPEAKER_MODEL[args.model]["vocoder"]:
warn_ = f"[-] Using {args.model} model! I would suggest passing --vocoder {MULTISPEAKER_MODEL[args.model]['vocoder']}"
warnings.warn(warn_, UserWarning)
else:
args.vocoder = MULTISPEAKER_MODEL[args.model]["vocoder"]
if args.speaking_rate is None:
args.speaking_rate = MULTISPEAKER_MODEL[args.model]["speaking_rate"]
spk_range = MULTISPEAKER_MODEL[args.model]["spk_range"]
if args.spk is not None:
assert (
args.spk >= spk_range[0] and args.spk <= spk_range[-1]
), f"Speaker ID must be between {spk_range} for this model."
else:
available_spk_id = MULTISPEAKER_MODEL[args.model]["spk"]
warn_ = f"[!] Speaker ID not provided! Using speaker ID {available_spk_id}"
warnings.warn(warn_, UserWarning)
args.spk = available_spk_id
return args
def validate_args_for_single_speaker_model(args):
if args.vocoder is not None:
if args.vocoder != SINGLESPEAKER_MODEL[args.model]["vocoder"]:
warn_ = f"[-] Using {args.model} model! I would suggest passing --vocoder {SINGLESPEAKER_MODEL[args.model]['vocoder']}"
warnings.warn(warn_, UserWarning)
else:
args.vocoder = SINGLESPEAKER_MODEL[args.model]["vocoder"]
if args.speaking_rate is None:
args.speaking_rate = SINGLESPEAKER_MODEL[args.model]["speaking_rate"]
if args.spk != SINGLESPEAKER_MODEL[args.model]["spk"]:
warn_ = f"[-] Ignoring speaker id {args.spk} for {args.model}"
warnings.warn(warn_, UserWarning)
args.spk = SINGLESPEAKER_MODEL[args.model]["spk"]
return args return args
@@ -166,9 +227,9 @@ def cli():
parser.add_argument( parser.add_argument(
"--vocoder", "--vocoder",
type=str, type=str,
default="hifigan_T2_v1", default=None,
help="Vocoder to use", help="Vocoder to use (default: will use the one suggested with the pretrained model))",
choices=VOCODER_URL.keys(), choices=VOCODER_URLS.keys(),
) )
parser.add_argument("--text", type=str, default=None, help="Text to synthesize") parser.add_argument("--text", type=str, default=None, help="Text to synthesize")
parser.add_argument("--file", type=str, default=None, help="Text file to synthesize") parser.add_argument("--file", type=str, default=None, help="Text file to synthesize")
@@ -182,7 +243,7 @@ def cli():
parser.add_argument( parser.add_argument(
"--speaking_rate", "--speaking_rate",
type=float, type=float,
default=1.0, default=None,
help="change the speaking rate, a higher value means slower speaking rate (default: 1.0)", help="change the speaking rate, a higher value means slower speaking rate (default: 1.0)",
) )
parser.add_argument("--steps", type=int, default=10, help="Number of ODE steps (default: 10)") parser.add_argument("--steps", type=int, default=10, help="Number of ODE steps (default: 10)")
@@ -199,8 +260,10 @@ def cli():
default=os.getcwd(), default=os.getcwd(),
help="Output folder to save results (default: current dir)", help="Output folder to save results (default: current dir)",
) )
parser.add_argument("--batched", action="store_true") parser.add_argument("--batched", action="store_true", help="Batched inference (default: False)")
parser.add_argument("--batch_size", type=int, default=32) parser.add_argument(
"--batch_size", type=int, default=32, help="Batch size only useful when --batched (default: 32)"
)
args = parser.parse_args() args = parser.parse_args()
@@ -333,6 +396,8 @@ def unbatched_synthesis(args, device, model, vocoder, denoiser, texts, spk):
def print_config(args): def print_config(args):
print("[!] Configurations: ") print("[!] Configurations: ")
print(f"\t- Model: {args.model}")
print(f"\t- Vocoder: {args.vocoder}")
print(f"\t- Temperature: {args.temperature}") print(f"\t- Temperature: {args.temperature}")
print(f"\t- Speaking rate: {args.speaking_rate}") print(f"\t- Speaking rate: {args.speaking_rate}")
print(f"\t- Number of ODE steps: {args.steps}") print(f"\t- Number of ODE steps: {args.steps}")

View File

@@ -116,7 +116,7 @@ class MatchaTTS(BaseLightningClass): # 🍵
w = torch.exp(logw) * x_mask w = torch.exp(logw) * x_mask
w_ceil = torch.ceil(w) * length_scale w_ceil = torch.ceil(w) * length_scale
y_lengths = torch.clamp_min(torch.sum(w_ceil, [1, 2]), 1).long() y_lengths = torch.clamp_min(torch.sum(w_ceil, [1, 2]), 1).long()
y_max_length = int(y_lengths.max()) y_max_length = y_lengths.max()
y_max_length_ = fix_len_compatibility(y_max_length) y_max_length_ = fix_len_compatibility(y_max_length)
# Using obtained durations `w` construct alignment map `attn` # Using obtained durations `w` construct alignment map `attn`

0
matcha/onnx/__init__.py Normal file
View File

181
matcha/onnx/export.py Normal file
View File

@@ -0,0 +1,181 @@
import argparse
import random
from pathlib import Path
import numpy as np
import torch
from lightning import LightningModule
from matcha.cli import VOCODER_URLS, load_matcha, load_vocoder
DEFAULT_OPSET = 15
SEED = 1234
random.seed(SEED)
np.random.seed(SEED)
torch.manual_seed(SEED)
torch.cuda.manual_seed(SEED)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
class MatchaWithVocoder(LightningModule):
def __init__(self, matcha, vocoder):
super().__init__()
self.matcha = matcha
self.vocoder = vocoder
def forward(self, x, x_lengths, scales, spks=None):
mel, mel_lengths = self.matcha(x, x_lengths, scales, spks)
wavs = self.vocoder(mel).clamp(-1, 1)
lengths = mel_lengths * 256
return wavs.squeeze(1), lengths
def get_exportable_module(matcha, vocoder, n_timesteps):
"""
Return an appropriate `LighteningModule` and output-node names
based on whether the vocoder is embedded in the final graph
"""
def onnx_forward_func(x, x_lengths, scales, spks=None):
"""
Custom forward function for accepting
scaler parameters as tensors
"""
# Extract scaler parameters from tensors
temperature = scales[0]
length_scale = scales[1]
output = matcha.synthesise(x, x_lengths, n_timesteps, temperature, spks, length_scale)
return output["mel"], output["mel_lengths"]
# Monkey-patch Matcha's forward function
matcha.forward = onnx_forward_func
if vocoder is None:
model, output_names = matcha, ["mel", "mel_lengths"]
else:
model = MatchaWithVocoder(matcha, vocoder)
output_names = ["wav", "wav_lengths"]
return model, output_names
def get_inputs(is_multi_speaker):
"""
Create dummy inputs for tracing
"""
dummy_input_length = 50
x = torch.randint(low=0, high=20, size=(1, dummy_input_length), dtype=torch.long)
x_lengths = torch.LongTensor([dummy_input_length])
# Scales
temperature = 0.667
length_scale = 1.0
scales = torch.Tensor([temperature, length_scale])
model_inputs = [x, x_lengths, scales]
input_names = [
"x",
"x_lengths",
"scales",
]
if is_multi_speaker:
spks = torch.LongTensor([1])
model_inputs.append(spks)
input_names.append("spks")
return tuple(model_inputs), input_names
def main():
parser = argparse.ArgumentParser(description="Export 🍵 Matcha-TTS to ONNX")
parser.add_argument(
"checkpoint_path",
type=str,
help="Path to the model checkpoint",
)
parser.add_argument("output", type=str, help="Path to output `.onnx` file")
parser.add_argument(
"--n-timesteps", type=int, default=5, help="Number of steps to use for reverse diffusion in decoder (default 5)"
)
parser.add_argument(
"--vocoder-name",
type=str,
choices=list(VOCODER_URLS.keys()),
default=None,
help="Name of the vocoder to embed in the ONNX graph",
)
parser.add_argument(
"--vocoder-checkpoint-path",
type=str,
default=None,
help="Vocoder checkpoint to embed in the ONNX graph for an `e2e` like experience",
)
parser.add_argument("--opset", type=int, default=DEFAULT_OPSET, help="ONNX opset version to use (default 15")
args = parser.parse_args()
print(f"[🍵] Loading Matcha checkpoint from {args.checkpoint_path}")
print(f"Setting n_timesteps to {args.n_timesteps}")
checkpoint_path = Path(args.checkpoint_path)
matcha = load_matcha(checkpoint_path.stem, checkpoint_path, "cpu")
if args.vocoder_name or args.vocoder_checkpoint_path:
assert (
args.vocoder_name and args.vocoder_checkpoint_path
), "Both vocoder_name and vocoder-checkpoint are required when embedding the vocoder in the ONNX graph."
vocoder, _ = load_vocoder(args.vocoder_name, args.vocoder_checkpoint_path, "cpu")
else:
vocoder = None
is_multi_speaker = matcha.n_spks > 1
dummy_input, input_names = get_inputs(is_multi_speaker)
model, output_names = get_exportable_module(matcha, vocoder, args.n_timesteps)
# Set dynamic shape for inputs/outputs
dynamic_axes = {
"x": {0: "batch_size", 1: "time"},
"x_lengths": {0: "batch_size"},
}
if vocoder is None:
dynamic_axes.update(
{
"mel": {0: "batch_size", 2: "time"},
"mel_lengths": {0: "batch_size"},
}
)
else:
print("Embedding the vocoder in the ONNX graph")
dynamic_axes.update(
{
"wav": {0: "batch_size", 1: "time"},
"wav_lengths": {0: "batch_size"},
}
)
if is_multi_speaker:
dynamic_axes["spks"] = {0: "batch_size"}
# Create the output directory (if not exists)
Path(args.output).parent.mkdir(parents=True, exist_ok=True)
model.to_onnx(
args.output,
dummy_input,
input_names=input_names,
output_names=output_names,
dynamic_axes=dynamic_axes,
opset_version=args.opset,
export_params=True,
do_constant_folding=True,
)
print(f"[🍵] ONNX model exported to {args.output}")
if __name__ == "__main__":
main()

168
matcha/onnx/infer.py Normal file
View File

@@ -0,0 +1,168 @@
import argparse
import os
import warnings
from pathlib import Path
from time import perf_counter
import numpy as np
import onnxruntime as ort
import soundfile as sf
import torch
from matcha.cli import plot_spectrogram_to_numpy, process_text
def validate_args(args):
assert (
args.text or args.file
), "Either text or file must be provided Matcha-T(ea)TTS need sometext to whisk the waveforms."
assert args.temperature >= 0, "Sampling temperature cannot be negative"
assert args.speaking_rate >= 0, "Speaking rate must be greater than 0"
return args
def write_wavs(model, inputs, output_dir, external_vocoder=None):
if external_vocoder is None:
print("The provided model has the vocoder embedded in the graph.\nGenerating waveform directly")
t0 = perf_counter()
wavs, wav_lengths = model.run(None, inputs)
infer_secs = perf_counter() - t0
mel_infer_secs = vocoder_infer_secs = None
else:
print("[🍵] Generating mel using Matcha")
mel_t0 = perf_counter()
mels, mel_lengths = model.run(None, inputs)
mel_infer_secs = perf_counter() - mel_t0
print("Generating waveform from mel using external vocoder")
vocoder_inputs = {external_vocoder.get_inputs()[0].name: mels}
vocoder_t0 = perf_counter()
wavs = external_vocoder.run(None, vocoder_inputs)[0]
vocoder_infer_secs = perf_counter() - vocoder_t0
wavs = wavs.squeeze(1)
wav_lengths = mel_lengths * 256
infer_secs = mel_infer_secs + vocoder_infer_secs
output_dir = Path(output_dir)
output_dir.mkdir(parents=True, exist_ok=True)
for i, (wav, wav_length) in enumerate(zip(wavs, wav_lengths)):
output_filename = output_dir.joinpath(f"output_{i + 1}.wav")
audio = wav[:wav_length]
print(f"Writing audio to {output_filename}")
sf.write(output_filename, audio, 22050, "PCM_24")
wav_secs = wav_lengths.sum() / 22050
print(f"Inference seconds: {infer_secs}")
print(f"Generated wav seconds: {wav_secs}")
rtf = infer_secs / wav_secs
if mel_infer_secs is not None:
mel_rtf = mel_infer_secs / wav_secs
print(f"Matcha RTF: {mel_rtf}")
if vocoder_infer_secs is not None:
vocoder_rtf = vocoder_infer_secs / wav_secs
print(f"Vocoder RTF: {vocoder_rtf}")
print(f"Overall RTF: {rtf}")
def write_mels(model, inputs, output_dir):
t0 = perf_counter()
mels, mel_lengths = model.run(None, inputs)
infer_secs = perf_counter() - t0
output_dir = Path(output_dir)
output_dir.mkdir(parents=True, exist_ok=True)
for i, mel in enumerate(mels):
output_stem = output_dir.joinpath(f"output_{i + 1}")
plot_spectrogram_to_numpy(mel.squeeze(), output_stem.with_suffix(".png"))
np.save(output_stem.with_suffix(".numpy"), mel)
wav_secs = (mel_lengths * 256).sum() / 22050
print(f"Inference seconds: {infer_secs}")
print(f"Generated wav seconds: {wav_secs}")
rtf = infer_secs / wav_secs
print(f"RTF: {rtf}")
def main():
parser = argparse.ArgumentParser(
description=" 🍵 Matcha-TTS: A fast TTS architecture with conditional flow matching"
)
parser.add_argument(
"model",
type=str,
help="ONNX model to use",
)
parser.add_argument("--vocoder", type=str, default=None, help="Vocoder to use (defaults to None)")
parser.add_argument("--text", type=str, default=None, help="Text to synthesize")
parser.add_argument("--file", type=str, default=None, help="Text file to synthesize")
parser.add_argument("--spk", type=int, default=None, help="Speaker ID")
parser.add_argument(
"--temperature",
type=float,
default=0.667,
help="Variance of the x0 noise (default: 0.667)",
)
parser.add_argument(
"--speaking-rate",
type=float,
default=1.0,
help="change the speaking rate, a higher value means slower speaking rate (default: 1.0)",
)
parser.add_argument("--gpu", action="store_true", help="Use CPU for inference (default: use GPU if available)")
parser.add_argument(
"--output-dir",
type=str,
default=os.getcwd(),
help="Output folder to save results (default: current dir)",
)
args = parser.parse_args()
args = validate_args(args)
if args.gpu:
providers = ["GPUExecutionProvider"]
else:
providers = ["CPUExecutionProvider"]
model = ort.InferenceSession(args.model, providers=providers)
model_inputs = model.get_inputs()
model_outputs = list(model.get_outputs())
if args.text:
text_lines = args.text.splitlines()
else:
with open(args.file, encoding="utf-8") as file:
text_lines = file.read().splitlines()
processed_lines = [process_text(0, line, "cpu") for line in text_lines]
x = [line["x"].squeeze() for line in processed_lines]
# Pad
x = torch.nn.utils.rnn.pad_sequence(x, batch_first=True)
x = x.detach().cpu().numpy()
x_lengths = np.array([line["x_lengths"].item() for line in processed_lines], dtype=np.int64)
inputs = {
"x": x,
"x_lengths": x_lengths,
"scales": np.array([args.temperature, args.speaking_rate], dtype=np.float32),
}
is_multi_speaker = len(model_inputs) == 4
if is_multi_speaker:
if args.spk is None:
args.spk = 0
warn = "[!] Speaker ID not provided! Using speaker ID 0"
warnings.warn(warn, UserWarning)
inputs["spks"] = np.repeat(args.spk, x.shape[0]).astype(np.int64)
has_vocoder_embedded = model_outputs[0].name == "wav"
if has_vocoder_embedded:
write_wavs(model, inputs, args.output_dir)
elif args.vocoder:
external_vocoder = ort.InferenceSession(args.vocoder, providers=providers)
write_wavs(model, inputs, args.output_dir, external_vocoder=external_vocoder)
else:
warn = "[!] A vocoder is not embedded in the graph nor an external vocoder is provided. The mel output will be written as numpy arrays to `*.npy` files in the output directory"
warnings.warn(warn, UserWarning)
write_mels(model, inputs, args.output_dir)
if __name__ == "__main__":
main()

View File

@@ -7,15 +7,17 @@ import torch
def sequence_mask(length, max_length=None): def sequence_mask(length, max_length=None):
if max_length is None: if max_length is None:
max_length = length.max() max_length = length.max()
x = torch.arange(int(max_length), dtype=length.dtype, device=length.device) x = torch.arange(max_length, dtype=length.dtype, device=length.device)
return x.unsqueeze(0) < length.unsqueeze(1) return x.unsqueeze(0) < length.unsqueeze(1)
def fix_len_compatibility(length, num_downsamplings_in_unet=2): def fix_len_compatibility(length, num_downsamplings_in_unet=2):
while True: factor = torch.scalar_tensor(2).pow(num_downsamplings_in_unet)
if length % (2**num_downsamplings_in_unet) == 0: length = (length / factor).ceil() * factor
return length if not torch.onnx.is_in_onnx_export():
length += 1 return length.int().item()
else:
return length
def convert_pad_shape(pad_shape): def convert_pad_shape(pad_shape):

View File

@@ -35,7 +35,7 @@ torchaudio
matplotlib matplotlib
pandas pandas
conformer==0.3.2 conformer==0.3.2
diffusers==0.21.1 diffusers==0.21.3
notebook notebook
ipywidgets ipywidgets
gradio gradio