DEX-TTS: Diffusion-based EXpressive Text-to-Speech with Style Modeling on Time Variability

🌈 Abstract

The paper presents Diffusion-based EXpressive TTS (DEX-TTS), an acoustic model designed for reference-based speech synthesis with enhanced style representations. The key innovations include differentiating styles into time-invariant and time-variant categories for effective style extraction, and designing encoders and adapters with high generalization ability. The paper also introduces overlapping patchify and convolution-frequency patch embedding strategies to improve DiT-based diffusion networks for TTS. DEX-TTS outperforms previous expressive TTS methods in terms of objective and subjective evaluation on English multi-speaker and emotional multi-speaker datasets, without relying on pre-training strategies. The paper also demonstrates the effectiveness of the proposed diffusion network improvements in general TTS tasks.

🙋 Q&A

[01] Overall Architecture

1. What are the main components of the DEX-TTS architecture? The main components of DEX-TTS are:

• Text Encoder: Extracts text representation
• Aligner: Predicts duration mapping to mel frames
• Diffusion Decoder: Synthesizes denoised mel-spectrogram using diffusion process
• Time-Invariant (T-IV) Encoder and Adapter: Extracts and incorporates time-invariant styles
• Time-Variant (T-V) Encoder and Adapter: Extracts and incorporates time-variant styles

2. How does DEX-TTS handle time-invariant and time-variant styles? DEX-TTS separates styles into time-invariant (T-IV) and time-variant (T-V) styles. The T-IV encoder uses multi-level feature maps and the T-IV adapter uses adaptive instance normalization (AdaIN) to extract and incorporate T-IV styles. The T-V encoder uses vector quantization to extract refined T-V styles, and the T-V adapter uses cross-attention to incorporate T-V styles while preserving temporal information.

3. What strategies does DEX-TTS use to improve the diffusion-based TTS backbone? DEX-TTS uses overlapping patchify and convolution-frequency patch embedding strategies to better leverage the DiT architecture for the diffusion-based TTS backbone. This allows for more effective extraction of detailed representations compared to previous diffusion-based TTS models.

[02] Experimental Results

1. How does DEX-TTS perform compared to previous expressive TTS methods? DEX-TTS outperforms previous expressive TTS methods in terms of objective metrics (WER, COS) and subjective metrics (MOS-N, MOS-S) on both the VCTK and ESD datasets, including in zero-shot scenarios. This indicates DEX-TTS can effectively capture and reflect rich styles from reference speech while maintaining high speech quality.

2. How does the improved diffusion backbone in GeDEX-TTS perform compared to previous diffusion-based TTS models? GeDEX-TTS, the general TTS version of DEX-TTS, achieves the best performance compared to previous diffusion-based TTS models on the LJSpeech dataset in terms of both objective and subjective metrics. This demonstrates the effectiveness of the proposed patchify and embedding strategies in improving the diffusion-based TTS backbone.

3. What are the model complexities of DEX-TTS and GeDEX-TTS? DEX-TTS has the smallest number of parameters among the expressive TTS methods, showing superior efficiency. However, it has a higher real-time factor (RTF) compared to previous expressive TTS methods due to the iterative denoising process of diffusion-based TTS. GeDEX-TTS achieves a lower RTF among diffusion-based TTS models with similar parameter sizes, indicating the effectiveness of the proposed network design.

[03] Ablation Studies

1. What is the importance of the T-IV and T-V style modeling in DEX-TTS? The ablation studies show that both T-IV and T-V styles significantly impact the performance of DEX-TTS. Removing either T-IV or T-V styles results in considerable degradation in WER and COS, indicating the effectiveness of the proposed style modeling approach.

2. How does the VQ layer in the T-V encoder affect the performance? Removing the VQ layer in the T-V encoder leads to a significant decrease in WER, despite an improvement in COS in unseen scenarios. This suggests that the VQ layer helps obtain well-refined time-variant styles, enabling effective style reflection while preserving temporal details.

3. What is the importance of time step conditioning in the adapters of the diffusion decoder? Removing the time step conditioning from the adapters in the diffusion decoder results in an overall performance decrease, highlighting the necessity of adaptive style incorporation during the iterative denoising process of the diffusion network.

</output_format>