Audiobook Creation Made Easy 🎧

Hey, welcome to a presentation on Emotia where we are making creating audiobooks cheap, automated, extremely easy.

Imagine you have a chapter, you have a chapter of a book that you would like to convert into an audiobook.

You have the text available.

If you want to do it cheaply by yourself, you'll go to a tool like this which will produce an output like that.

Same narrator, robotic wise.

Not a lot of emotion, not a lot of engagement with the person who's listening to it.

What we have created is an agentic pipeline that goes from this to something significantly better than that.

the first version of that that we are making available are ready to be uploaded into an audiobook platform like Audible, audio chapters.

let's listen to an example of what this output looks like.

Knowledge you're working on.

It's different tool.

It's a little bit, it's very different.

You talk to accents that match the characters based on the story, sound effects that are contextual and helps the listener transported to the scene where the story is being told.

The third thing, it's just very immersive.

like this entire experience.

we can stop here.

But we didn't, we went one step further.

Let's look at what would this experience look like if we had another modality in images or sort of a scene to accompany this.

The background music, caffeine noises in the background products.

So what are we trying to solve?

Most of the books are not available as audiobooks because it's extremely pricey to convert one into an audiobook.

You can see the numbers based on our industry research.

We think with Boson AI and our project on the application layer built on top of Bosson AI can get the cost down to less than $50 per chapter, which is incredibly cheap and will make the process of generating audiobooks and making it available to listeners much more cheaper and easier for indie publishers, indie authors, and eventually for publishing houses as well.

what should Poisson care about this?

Well, it's a massive market which, and the impact can be made exponential by branching out to other languages all over the world.

you have like a group of customers who'll be very interested in trying out this product because of the cost savings that it brings and the kind of output we can create with barely any inputs from their end, if that's what they want.

And, and yeah, there are a lot of listeners of audiobooks.

Everybody likes to switch one on when they watch their utensils or their gardening.

I would know.

I work for A company which does audiobooks at scale, amongst other things.

so, how do we go about doing this?

Well a few things.

we first take the text of the chapter, run it through a few agentic workflows to be able to extract important information about the characters, about the scene, convert the text into like a multi person sort of dialogue just so that we can vocalize it correctly, as well as some other processing to make the downstream process of audio generation a little better and easier for us.

the next step is of course generating the audio.

In this case we leverage ElevenLabs voice IDs to be able to generate reference voices that are associated with different characters based on the characteristics.

and we use these reference voices to be able to ground output from boson AI, for the dialogues of these different characters in a given chapter.

We also use elevenlabs, Sound Effect Generation API to be able to generate very specific sound effects to help set the scene for a given chapter.

And the output is like a full chapter's audio stitched together that we heard before.

The third stage of course is the multi agent director where we take the modality of text dialogues, we take in the generated audio files and we take in the entire story to pass it to an LLM director whose job is to suggest more audio effects to be added to the background, suggest how things should be stitched together, suggest timestamps when one image generated using AI should switch to the next based on the scene.

so we basically have a workflow that goes from what was produced before and was taken as input into the kind of emotional sort of experience that you saw.

so yeah, that is sort of what we have built so far.

The code has been provided in the GitHub repo that we have attached.

some of the things that we were quite proud of that we sort of stumbled with a little bit was a blocker.

But then we solved it.

The first one being found it a little hard to get consistent accents and voice types from boson AI realized a few things about it.

The first one was having longer reference text usually led to more consistent results.

not having short dialogues for a given character led to more consistent results.

We do some processing for this upstream in the process.

And the third one being generating the reference text related to the chapter that we want to get vocalized usually also produce better outputs from us in our experimentation.

the last thing I Also want to mention on this is that we are extensively using AI evals, sort of in this entire pipeline where every single time we generate a dialogue or let's say sound effect, we send it to the Audio Understanding API from Boson AI to be able to, and ask it like very specific questions relating to the quality of output.

if the quality is unsatisfactory, we do up to three retries to be able to produce a better quality output.

And the next step here would be to do this agentically where we can change the prompt for generation of voice to sort of address the issue that we are seeing based on what the Audio Understanding API is telling us.

some other things that we dealt with.

It's quite hard to generate the same type of character, across a chapter, across different dialogue groups.

we found it quite hard to do this consistently but realized that providing very, very expressive and rich character descriptions helped us with this a lot still, not completely solved but 90% there.

And the next thing of course is being able to sort of layer different audios, layer like audio and images and then do transitions all using large language models without any viral intervention was very hard.

but my teammates are amazing and we figured it out.

where do we go from here?

Multiple language support of course.

that would be like a stretch goal for tomorrow and personalization at scale.

So we want to give more power to our end users to define the narration type, the voices they want to use, etc.

Etc.

so that seemed like a logical next step, where we would like to take this project.

thank you for your time and attention.

please do listen to the audio.

samples we have shared with you for different types of of text or books.

In this case, we are very excited and looking forward to being included in top 16 to sort of discuss this idea more and flesh it out a little bit more.

Cheers.