基于多任务损失附加语言模型的语音识别方法

doi:10.3969/j.issn.1671-7775.2023.05.010

摘要
图/表
参考文献(0)
相关文章 (15)

全文: PDF (4840 KB) HTML (1 KB)
输出: BibTeX | EndNote (RIS)

摘要针对Attention过于灵活的对齐方式在复杂环境中适应性差、简单端到端模型对语言特征利用不充分的问题，研究了基于多任务损失附加语言模型的语音识别方法.通过分析语音信号特征，训练中选用包含更多信息的特征.以基于Attention的Conformer端到端模型为基础，采用CTC损失辅助纯Conformer (Attention)的多任务损失训练模型，得到ConformerCTC语音识别模型.在ConformerCTC模型基础上，通过分析对比部分语言模型的特点与效果，将Transformer语言模型通过重打分机制附加至上述模型的训练中，最终得到ConformerCTCTransformer语音识别模型.在AISHELL1数据集上对上述模型进行了试验.结果表明：ConformerCTC模型相对于纯Conformer (Attention)模型在测试集上的字错率（character error rate，CER）降低了0.49%，而ConformerCTCTransformer模型相对于ConformerCTC模型在测试集上的CER又降低了0.79%.CTC损失可以改善Attention对齐方式在复杂环境中的适应性，并且对ConformerCTC模型附加Transformer语言模型重打分后能再次提升0.30%的识别准确率.相较于现有的部分端到端模型，ConformerCTCTransformer模型识别效果较好，说明该模型具有一定的有效性.

	服务

	把本文推荐给朋友
	加入我的书架
	加入引用管理器
	E-mail Alert
	RSS
	作者相关文章
	柳永利
	张绍阳
	王裕恒
	解熠

关键词 ：语音识别, 深度学习, 语言模型, 多任务损失, Conformer, Transformer, CTC

Abstract：To solve the problems that the Attention′s overly flexible alignment was poorly adaptable in complex environments and the language features were not fully utilized by simple endtoend models, a speech recognition method was investigated based on multitask loss with additional language model. By analyzing the characteristics of the speech signal, the features containing more information were selected in the training. Based on the Attentionbased Conformer endtoend model, the model was trained using multitask loss of CTC loss assisted pure Conformer (Attention), and the ConformerCTC speech recognition model was obtained. Based on the ConformerCTC model, by analyzing and comparing the characteristics and effects of some language models, the Transformer language model was added to the training of the above model through rescoring mechanism, and the ConformerCTCTransformer speech recognition model was obtained. The experiments on the above model were completed on the AISHELL1 data set. The results show that compared with the pure Conformer (Attention) model, the character error rate (CER) of the ConformerCTC model on the test set is reduced by 0.49%, and the CER of the ConformerCTCTransformer model on the test set is reduced by 079% compared with the ConformerCTC model. The adaptability of Attention alignment in complex environments can be improved by CTC loss, and after rescoring the TransformerCTC model with the Transformer language model, the recognition accuracy can be increased by 030% again. Compared with some existing endtoend models, the recognition effect of the ConformerCTCTransformer model is better, indicating that the model has certain effectiveness.

Key words： speech recognition deep learning language model multitask loss Conformer Transformer CTC

基金资助:陕西省重点产业创新链（群）项目(2021ZDLGY07-06)

作者简介: 柳永利(1997—),男,甘肃平凉人,硕士研究生(liuyongli@chd.edu.cn),主要从事人工智能、大数据研究. 张绍阳(1971—),男,山西襄汾人,教授(zhsy@chd.edu.cn),主要从事智能交通、大数据研究.

引用本文:

柳永利，张绍阳，王裕恒，解熠. 基于多任务损失附加语言模型的语音识别方法[J]. 江苏大学学报（自然科学版）, 2023, 44(5): 564-569. LIU Yongli, ZHANG Shaoyang, WANG Yuheng, XIE Yi. Speech recognition method based on multitask loss with additional language model[J]. Journal of Jiangsu University(Natural Science Eidtion) , 2023, 44(5): 564-569.

链接本文:

http://zzs.ujs.edu.cn/xbzkb/CN/10.3969/j.issn.1671-7775.2023.05.010 或 http://zzs.ujs.edu.cn/xbzkb/CN/Y2023/V44/I5/564

［1］	DOKUZ Y, TUFEKCI Z. Minibatch sample selection strategies for deep learning based speech recognition [J]. Applied Acoustics, DOI: 10.1016/j.apacoust.2020.107573.
［2］	鱼昆,张绍阳,侯佳正,等. 语音识别及端到端技术现状及展望[J].计算机系统应用, 2021, 30(3): 14-23.
	YU K, ZHANG S Y, HOU J Z, et al. Survey of speech recognition and endtoend techniques[J]. Computer Systems & Applications, 2021,30(3):14-23. (in Chinese)
［3］	邓慧珍.基于局部自注意力CTC的语音识别[D]. 哈尔滨: 黑龙江大学, 2021.
［4］	DAS A, LI J Y, ZHAO R, et al. Advancing connectionist temporal classification with attention modeling[C]∥Proceedings of the 2018 IEEE International Conference on Acoustics, Speech and Signal Processing. Piscataway：IEEE，2018:4769-4773.
［5］	杨威,胡燕.混合CTC/attention架构端到端带口音普通话识别[J].计算机应用研究, 2021, 38(3): 755-759.
	YANG W, HU Y. Hybrid CTC/attention architecture for endtoend multiaccent Mandarin speech recognition[J]. Application Research of Computers, 2021, 38(3):755-759. (in Chinese)
［6］	谢旭康,陈戈,孙俊,等.TCNTransformerCTC的端到端语音识别[J].计算机应用研究, 2022, 39(3):699-703.
	XIE X K, CHEN G, SUN J, et al. TCNTransformerCTC for endtoend speech recognition[J]. Application Research of Computers, 2022, 39(3):699-703. (in Chinese)
［7］	GULATI A, QIN J, CHIU C C, et al. Conformer: convolutionaugmented transformer for speech recognition[C]∥Proceedings of the Annual Conference of the International Speech Communication Association. [S.l.]：International Speech Communication Association，2020:5036-5040.
［8］	BAHDANAU D, CHOROWSKI J, SERDYUK D, et al. Endtoend attentionbased large vocabulary speech recognition[C]∥Proceedings of the 2016 IEEE International Conference on Acoustics, Speech and Signal Processing. Piscataway：IEEE，2016:4945-4949.
［9］	BIADSY F, WEISS R J, MORENO P J, et al. Parrotron: an endtoend speechtospeech conversion model and its applications to hearingimpaired speech and speech separation[C]∥Proceedings of the Annual Conference of the International Speech Communication Association. Lous Tourils, Baixas, France：International Speech Communication Association, 2019:4115-4119.
[10]	MA R, LIU Q, YU K. Highly efficient neural network language model compression using soft binarization training[C]∥Proceedings of the 2019 IEEE Automatic Speech Recognition and Understanding Workshop. Piscataway：IEEE，2019:62-69.
[11]	葛轶洲,许翔,杨锁荣,等.序列数据的数据增强方法综述[J].计算机科学与探索, 2021, 15(7): 1207-1219.
	GE Y Z, XU X, YANG S R, et al. Survey on sequence data augmentation[J]. Journal of Frontiers of Computer Science and Technology, 2021, 15(7):1207-1219. (in Chinese)
[12]	YAO Z Y, WU D, WANG X, et al. WeNet: production oriented streaming and nonstreaming endtoend speech recognition toolkit[C]∥Proceedings of the 22nd Annual Conference of the International Speech Communication Association. [S.l.]: International Speech Communication Association, 2021：2093-2097.
[13]	朱学超,张飞,高鹭,等.基于残差网络和门控卷积网络的语音识别研究[J].计算机工程与应用, 2022, 58(7):185-191.
	ZHU X C, ZHANG F, GAO L, et al. Research on speech recognition based on residual network and gated convolution network[J]. Computer Engineering and Applications, 2022, 58(7):185-191. (in Chinese)
[14]	LIANG C D, XU M L, ZHANG X L. Transformerbased endtoend speech recognition with residual Gaussianbased selfattention[C]∥Proceedings of the 22nd Annual Conference of the International Speech Communication Association. [S.l.]: International Speech Communication Association,2021:1495-1499.