Punctuation

[Marijn-van-der-Steen]

I was wondering if it is possible to add punctuation to the generated transcript.
In the documentation of the SpeechRecognition Web API I can see there is an onspeechend event, so I think if this gets exposed in this library then maybe that could be used to add punctuation.
It probably is only a period behind the sentence, since I doubt that the API returns anything useful the determine where a comma or question mark should be placed.
Any ideas if this is somehow possible?

[JamesBrill]

Hi @Marijn-van-der-Steen Some thoughts on this:

• If you want to display a "corrected" transcript (i.e. with punctuation added) to the user, you will need to store this in a separate state. You can then update this state in a useEffect callback whenever the transcript property changes. Though this might be computationally (or financially, if you use an API to fix the punctuation) expensive, so I'd suggest limiting that transform to when the final transcript is updated (i.e. the user has finished speaking and the algorithm has concluded what was said). This library does not currently expose state for "new speech" that gets appended to the final transcript (maybe it should!). However, it is possible to derive this in a slightly hacky way by referencing the previous state of finalTranscript and computing the difference. Here's an example, using a dummy correctPunctuation function to demonstrate how you might do this:

import React, { useState, useEffect, useRef } from 'react'
import { useSpeechRecognition } from 'react-speech-recognition'

const correctPunctuation = (transcript) => `${transcript}...`

const Dictaphone = () => {
  const [correctedTranscript, setCorrectedTranscript] = useState('')
  const { finalTranscript } = useSpeechRecognition()

  const prevFinalTranscriptRef = useRef()
  useEffect(() => {
    prevFinalTranscriptRef.current = finalTranscript;
  })
  const prevFinalTranscript = prevFinalTranscriptRef.current

  useEffect(() => {
    if (finalTranscript !== '') {
      const newSpeech = finalTranscript.substr(prevFinalTranscript.length).trim()
      setCorrectedTranscript(`${correctedTranscript} ${correctPunctuation(newSpeech)}`)
    }
  }, [finalTranscript])

  return <span>{correctedTranscript}</span>
}

export default Dictaphone

• You may find that a paid speech recognition service does a better job at punctuating speech than Google's in-built implementation of the Web Speech API. This library supports a polyfill for Azure Cognitive Services - see here for details. That way, you may not need to modify the punctuation at all.
• If you can't use Azure, I had a quick Google around for "grammar APIs" and this one might be an option for fixing up the punctuation using the code above: https://www.perfecttense.com/api

[daveydee33]

Hi @Marijn-van-der-Steen, did you end up with a useful solution to this? I was testing out the Google and AWS Transcribe services in parallel on some pre-recorded dialogues, and was very much surprised how well AWS did at picking up correct grammar (commas, exclamation points, question marks), and at the same time, I was totally disappointed at how poorly Google's did in that area. They both got about ~95% accurate on the words, but the grammar was unusable with Google's and pretty good with AWS. I'm curious if you guys came up with any nice solutions too. I haven't tried Microsoft or other solutions yet, and was currently focused on working with non-English languages too.

[JamesBrill]

Yes, punctuation is really hit-or-miss across the different speech APIs. I recall the Microsoft Azure one (which has a polyfill) doing a slightly better job than Google, so might be worth a try. I'm currently chasing the AWS polyfill here - if I get time, I may be able to contribute the finishing pieces to it.