EPUB 3 Media Overlays as Audiobooks

[wordskill]

I am suggesting this as a feature worth aspiring towards in the future.

"Read along" or "Read aloud" or "Immersion Reading" functionality, whereby the user can listen to an audiobook while reading the matching text, is a terrific experience and one that I believe could become very popular. It is particularly useful for people learning a language, or wishing to absorb the text at a deeper level.

My own main usage would be to occasionally, while listening to an audiobook, whip out my phone and look at the words during parts, such as statistics, that are more effectively conveyed by text than audio.

Aesthetically, I also like the idea of having my phone or iPad propped up, displaying the scrolling text, while I am working at my laptop and listening to an audiobook on the side.

The EPUB 3 specification allows for audio files to be included in the bundle. By including a SMIL file (an XML-based text file) specifying the exact timing of each sentence, sentences in the ebook can be highlighted as the matching audio plays, and the text display can scroll or turn pages accordingly.

In existing implementations, such as Apple Books or Thorium Reader, or Web apps using Readium, clicking or touching a specific sentence skips the audio to that position.

The SMIL files are easily created by using Open Source projects such as the Python-based Aeneas (https://github.com/readbeyond/aeneas) which uses speech recognition and is available on Linux, Windows, and MacOS.

You simply provide the text and audio files and it creates the EPUB bundle. I also suspect that the current explosion in AI services will result in ways to do the same thing entirely online, without having to install an app.

Another, similar Python script is Syncabook (https://github.com/r4victor/syncabook).

I believe that, once people become aware that "Read along" or "Read aloud" functionality is possible, demand will materialize. Apps such as Aeneas or AI equivalents will make it relatively easy to satisfy that demand. Once EPUB 3 audiobooks are more widely available, it is inevitable that users will ask for that functionality in their favorite audiobook apps.

The Open Source Readium project (https://readium.org/) provides code that allows EPUB 3 functionality to be included in mobile apps (Swift for iOS, Kotlin for Android), desktop apps (node.js and Electron.js), and Web apps (Typescript).

I have no idea if such code is in any way useful to you, but it would be good to see BookPlayer become an early pioneer in this exciting new frontier for audiobooks.