Presently reported is an automatic surtitle generation scheme for live Cantonese opera based on beat tracking. The individual Chinese characters in the surtitles are animated in a karaoke style in sync with the performance. The scheme requires a music-vocal score of the opera but no human intervention during the live performance. A major advantage of the method is that the next Chinese character can be predictively highlighted at the onset of its vocalization with almost zero latency after compensating for the digitizing and processing time of the streaming audio. Specifically, a regularly spaced pulse train registering the onset position of every monosyllabic Chinese character to be sung or spoken is initially prepared from the music-vocal score. On-going synchronization of the pulse train to the live performance is obtained by monitoring the beats in the audio. Assuming beat tracking is accurate, the alignment of the surtitle characters should be 100% correct. An exception is speech that does not have a beat structure for which only an estimated speech rate can be established. The objective of this research is to promote Cantonese opera to those who do not speak Cantonese with the hope that the present surtitle generation
(See document for exact affiliation information.)
https://aes2.org/publications/elibrary-page/?id=22558
(661KB)
Click to purchase paper as a non-member or login as an AES member. If your company or school subscribes to the E-Library then switch to the institutional version. If you are not an AES member Join the AES. If you need to check your member status, login to the Member Portal.
