Hello, I’m Seungtaek Choi, the lead of the R.content team at Riiid, where we conduct “multimodal AI research for education.”
Our team is carrying out a variety of research projects, and among them, we had the opportunity to present our research成果 on “Spoken and Written Language Assessment” technology at INTERSPEECH 2023. I wrote this post to share my reflections afterward.
INTERSPEECH 2023 Conference
About the INTERSPEECH Conference
‘INTERSPEECH 2023’ is a world-renowned speech and language processing conference now celebrating its 24th anniversary, and this year it was held in Dublin, Ireland for four days starting on August 20. The INTERSPEECH conference covers a wide range of topics, from traditional speech signal processing, speech analysis, and speech interfaces to speech recognition, speech synthesis, and dialogue systems (chatbots), which are now familiar even to the general public, as well as other topics currently important in industry research. In addition to academic presentations, it also offers technology exhibitions and networking opportunities, making it a collaborative venue where experts from both academia and industry gather to happily share knowledge.
INTERSPEECH 2023 Attendees and Conference Atmosphere
At INTERSPEECH 2023, poster presentation sessions on various topics were held in the main hall, while oral presentation sessions took place in parallel in other halls. As you can see in the photos, far more researchers in the speech field participated than expected, and it was strongly apparent that everyone was eager to share their research成果 and domain expertise while learning from other researchers.
With growing interest in AI recently, and because the speech field has traditionally been closely tied to AI—from Speech-to-Text (STT) to AI speakers—the participation of various global companies such as Google, Amazon, and Meta was especially noticeable at INTERSPEECH 2023. Usually, when you attend a conference, company booths hand out booklets showcasing their research成果 and sometimes give visitors branded goods as gifts.
On the first evening, a Welcome Reception was held. The conference organizers provided simple bite-sized foods along with wine, beer, and more, and you could see attendees gathering in small groups, chatting in high spirits. Since Dublin is home to Guinness, one of the most famous beer companies in the world, a company called Dataporce even set up a booth at the conference serving Guinness draft beer. Perhaps because everyone knew of Guinness’s reputation in Dublin, a crowd gathered there unlike anything seen at any other session.
Riiid’s poster presentation on the first day of the conference
Addressing Cold-start Problem for End-to-end Automatic Speech Scoring
The title of the paper presented by Riiid this time was Addressing Cold-start Problem for End-to-end Automatic Speech Scoring, and it is the result of the AI research project on TOEIC Speaking score prediction, which plays the most important role in Riiid’s TOEIC Speaking scoring service. (The detailed research content is summarized on my personal blog, so if you are interested, I recommend taking a look. In this article, I will focus more on the conference itself.)
The attached photo was taken while I was waiting after setting up for the poster presentation. Since I attended the conference alone from Riiid, I couldn’t take any photos of myself actively communicating with overseas researchers, which is a bit regrettable. In reality, there were too many participants who stopped by to view Riiid’s poster to count, but I had in-depth conversations with about 20 of them. If I had to pick the top 3 most frequently asked questions, they were as follows.
Depending on the researcher, some asked whether the data would be publicly available or requested more detailed model architecture information, while others provided feedback with additional insights based on the characteristics of each model in our result analysis. Through the interest and questions of these researchers, I was once again able to confirm that what we need to focus on in solving the problem is, after all, the model and the data. It was also a valuable experience to showcase both Riiid’s AI capabilities and its data while receiving feedback at the academic level.
Insights gained from exploring the conference and networking
After the presentation session, which was my biggest goal in attending the conference, ended, I walked around the venue eagerly absorbing knowledge in the speech field. There were many different areas, but as a member of a team researching multimodal AI at an education company, the areas that interested me most were broadly 1) speech assessment & pronunciation assessment, 2) acoustic models & language models, and 3) data scarcity & self-supervised learning.
First, speech assessment & pronunciation assessment (Speech Assessment & Pronunciation Assessment)
I think our paper also falls into this area, and perhaps because this is the speech field, I got the impression that pronunciation issues are treated as extremely important. There were attempts to assess pronunciation from various angles, ranging from research that considers the differences between native English speakers and ESL learners, to studies that diagnose pronunciation using information at the phoneme/word/sentence level, and even research suggesting that the characteristics of young children should be taken into account. There were also papers that leveraged information from other tasks together, or approached imbalance in speech assessment data from the perspective of objective engineering. Given the nature of education, Riiid always has to assume imperfect input data, and I was able to gain many important insights that can be applied even in such situations.
Second, acoustic models & language models (Acoustic Model & Language Model)
These days in the AI field, the trend is to take one good model and really dig into it from every angle. INTERSPEECH 2023, as a speech conference, was no exception, and there was a great deal of research on speech models or acoustic models. There were various analytical studies, and one of the most interesting ones I saw was a paper that experimentally demonstrated that representations from speech models also contain grammatical information. Since speech models usually take waveform data as input, one might think they cannot reflect linguistic characteristics. However, this paper showed that, much like language models such as BERT in the NLP field, speech models also contain grammatical information. At Riiid, we were also researching AI that can diagnose not only pronunciation but also content and grammatical aspects through students’ speech, so this was an especially welcome paper in that it suggested that simply making good use of a strong speech model could enable diagnosis of grammatical aspects as well.
There were also papers analyzing ChatGPT from the perspective of INTERSPEECH’s interests. One of the important issues in language research within the speech domain is that errors can be introduced in the process of converting speech into text (Automatic Speech Recognition; ASR). These errors often cause language models to misunderstand what the user actually said. This paper experimentally showed that ChatGPT, a large language model, can to some extent correct and understand ASR errors on its own. Put simply, it means ChatGPT can smoothly make sense of text that contains ASR errors. In addition, there were studies on using language models together with speech models, as well as research on intent detection modules, which are commonly included in AI speakers.
Recently, there was also active discussion around benchmarks for evaluating speech models, similar to the well-known Open LLM Leaderboard. Some papers argued that the current benchmark system is not effective, while others proposed new benchmark systems to evaluate the linguistic knowledge of self-supervised speech models. Google also presented a methodology for measuring and comparing the fine-tuning performance of various speech models without additional training. Meanwhile, a reflective study pointing out that papers at the INTERSPEECH conference often do not release their code together was also notable. In the NLP field, the community began to grow rapidly as various benchmarks such as GLUE were proposed, and it felt like similar movements are gradually increasing in the field of speech and language as well.
Third, Data Scarcity & Self-Supervised Learning (Low Resource & Self-Supervised Learning)
Although we live in an age overflowing with data, paradoxically, every AI researcher is probably dealing with the problem of data scarcity. In environments where high-quality data is lacking, improving model performance through self-supervised learning is essential. At this conference as well, there were quite a few papers presenting ways to improve performance across various tasks through similar approaches.
A wide range of methodologies were discussed, from transfer learning approaches that improve performance by bringing in well-trained models in data-scarce situations to pre-training methodologies based on self-supervised learning. Methodologies already studied in the natural language processing field were being newly explored in speech and language processing with their own distinctive characteristics, and in some cases, entirely new model training methodologies were proposed that had not existed in other fields.
Data scarcity is always a problem in the field of education as well. Through the INTERSPEECH 2023 conference, I was able to learn about the data shortage issues faced by both industry and academia, along with simple yet effective ways to address them.
Wrapping Up
Through participating in INTERSPEECH 2023, I was able to see and feel many things as follows.
First, I was able to learn a great deal about AI in the speech domain. I do feel some regret that, if I had had a bit more expertise in speech, I might have been able to see more and gain even more from the experience. Still, I was able to learn technologies that could be applied immediately in practice, and it was also a great opportunity to think about the direction our company should pursue in the long term.
Second, I was reminded once again of how important data is. I especially felt this in NLP: once data (or a benchmark) is created in the AI field, people hungry for data quickly gather around it and rapidly improve performance, share those methodologies with one another, spread their influence, and ultimately expand the market itself. At INTERSPEECH 2023, it was visibly clear that far more data was being gathered than I had previously felt, and it was an experience that made me look forward to AI advancing even faster.
Lastly, unsurprisingly, I felt that conferences held offline are still better than online conferences for absorbing knowledge quickly. Unless there is a presentation I am especially interested in, I tend to prefer going through all the posters and getting one-on-one crash courses from the authors, and it was a valuable opportunity to ask experts about the fine details that do not show up in papers or presentations alone. At the same time, while researching educational AI at Riiid, it made me feel that we need to work even harder to provide personalized education at a level equal to—or even beyond—offline learning.
This concludes my review of participating in and presenting a poster at INTERSPEECH 2023. Thank you for reading.
