RSHMM++ for extractive lecture speech summarization

We propose an enhanced Rhetorical-State Hidden Markov Model (RSHMM++) for extracting hierarchical structural summaries from lecture speech. One of the most underutilized information in extractive summarization is rhetorical structure hidden in speech data. RSHMM++ automatically decodes this underlying information in order to provide better summaries. We show that RSHMM++ gives a 72.01% ROUGE-L F-measure, a 9.78% absolute increase in lecture speech summarization performance compared to the baseline system without using rhetorical information. We also propose Relaxed DTW for compiling reference summaries.