Google’s NotebookLM Podcast Generator Gets It Wrong

two tiny robots making a podcastAfter Andrew Byrne’s LinkedIn post about him generating a podcast about his career using Google’s NotebookLM, I wondered if I could generate a podcast based on my first novel, Hell on $5 A Day.

The errors and hallucinations are strong with this one

Being the author and knowing the story intimately may have made me too sensitive to the errors, but there were some serious errors and hallucinations. For example, the hosts keep saying Vietnam when what they were talking about occurred in the forests of France. In fact, NONE of the story takes place in Vietnam. The word “Vietnam” is not in the book.

They call Mammon “the devil” when he’s the demon of greed and we do *eventually* encounter Lucifer (my version, not Neil Gaiman’s). They also credit him with a reaction that happens in a different encounter with a different demon.

They say Albert was a hit man in his life, but he wasn’t.

Getting into more errors would be spoilers.

It wasn’t bad, though

That said, I did enjoy the overall podcast and the takes on the things it got right were interesting. For something entirely generated by AI, I wouldn’t have known it wasn’t real if I just heard it on the radio.

I tried to find out how I could give feedback on the audio overview’s errors and then regenerate it, but couldn’t find a way to add details, reactions, and feedback that would be incorporated into a regeneration of the audio. If you’re trying to use NotebookLM, I’d say it may be good at summarizing shorter, more straightforward content, but it can go off the rails with a novel and get MANY bits wrong.

I’d suggest in future updates to allow people to download a transcript of the audio overview and manually edit it, or provide a detailed critique. But as with my previous experiments, Gemini is the most lying-ass AI on the market… after Llama3 (which would make up a different fictional bio every time I asked it “who is Greg Bulmash?”).

Anyhoo… If you haven’t read my novel, it’s free with Kindle Unlimited, $4.99 on Kindle, and $11.99 for the trade paperback. If Google makes improvements that allow me to generate an accurate “audio overview,” I’ll post it.

Tags:,

Add a Comment

Your email address will not be published. Required fields are marked *