LLM brain fatigue
If you’ve done much work with LLMs you’ve likely experienced this “cognitive swiss cheese” feeling - the mental tax of wading through paragraphs of LLM-generated text. This thread went viral in the development community a few days ago with hundreds of people piling on expressing this frustration.
LLM’s are great token predictors for generating textual responses. But there are two main contributors to that “brain damage” cognitive fatigue everyone is experiencing: 1) LLM’s don’t fundamentally “understand” concepts the way humans do - they don’t maintain mental models of the world and perform assimilation/accommodation on their schemas, they are just predicting the next most logical token in a sentence. This means they can’t explain phenomena like physics the way Richard Feynman could explain physics because they lack a ground up mental latticework of the concepts. 2) Even if they were to gain this capability, they are inherently text-generating factories that return paragraphs of verbiage as responses. Not everyone learns best from reading - some of us are auditory and/or spacial/visual learners and we need a multi-modal experience.
#1 is a deficiency of the current state of the tech. This is gradually getting solved by stronger models, tool use and deep reasoning chain of thought iterative calls. Almost at some point with enough parameters and stackable model tooling it will arrive at something approximating cognition and understanding. Until that happens we can’t solve deficiency #1. #2 however is fully-solvable today and I’ll show you how.
A multi-modal tutor that meets you where you’re at
What if instead of receiving blocks of text responses to your questions you could get back animated explainer videos that meet you precisely where you’re at? What if the “tutor” could first do an archeological expedition to unearth your current level of understanding of a subject to know exactly where your knowledge gaps are and then fill those in via custom-built animated explainer videos?
The Video Explainer skill
I use Loom a lot for making narrated screencasts that capture requirements with me explaining something to the LLM and being able to talk through it on my screen. I open sourced this plugin back in July to make it so my bot could ingest my Loom videos and immediately have the narrated transcript to correlate with the video frames and be able to make sense of my videos. This has been a game-changer for communicating complex requirements, reproducing bugs, or even just talking through my todo list and priorities to communicate complex info quickly and with high fidelity.
This video explainer skill I just released is the other side of that coin giving any agent the ability to respond with its own equivalent of a narrated Loom video. This too has been an absolute game-changer as an alternative to having to wade through paragraphs of explanation for understanding what it’s trying to convey. We can essentially have high-bandwidth inbound/outbound comms with our agents now the way we would communicate by trading Loom videos.
The irony here of me explaining all this via paragraphs of text is not lost on me… so here’s a real example of how I used this tandem set of skills today:
I’m currently learning how to solve a type of number puzzle called “KenKen” (it’s like a variant of sudoku with these caged regions in which the contents have to obey certain math). Here is the Loom video I made today for my bot Jax when I got stuck:
KenKen Strategy for Spotting Logic Errors - Watch Video
And below is the animated explainer video I got back from Jax as a response:
What’s great about this is I’m able to very quickly give it my current understanding and express what I’m struggling with and why. It can then process that and instead of giving me a wall of text to decipher and understand, it can just talk me through it the way a tutor would explain something at the whiteboard.
The implications
I showed this to my buddy Marc and immediately he said “Khan Academy should be all over this.” 100% agree with him. This would definitely amplify the Khanmigo AI tutor to be more useful to audio/visual learners. But think about the other implications this has:
To content marketers: this is a factory for lead magnets that show concise solutions to customer problems.
To site owners: imagine the evolution of context-sensitive help widgets that rather than giving boilerplate tutorials could ingest your visitor’s session through something like Hotjar and then give return a custom explainer vid based on their session and their inferred intent.
To educators: your students who are falling behind can now get calibrated help to meet them exactly where they’re at and clear up their misunderstandings. Conversely your star students can get stretch tutoring to push them beyond the standard book material with more sophisticated concepts.
These are just a few of the use cases that come to mind. Auditory & visual learners are finally on equal footing with reading-centric learners in being able to get bespoke tutoring from the LLM’s via the formats at which they excel.
How to use it
These are two separate skills that can work independently but are more powerful together. If you’re using something like Claude Code, easiest is to point it to this github repo and tell it to install the loom-vision and video-explainer skills.
How it works under the hood
This post explains the Loom Vision skill.
The Video Explainer skill basically has two generation modes:
Simple: uses ffmpeg + whisper to make a basic narrated explainer vid with just frames that change.
Advanced: uses Remotion + whisper to make a narrated explainer vid that can do fades and animation and save the template so that it can be used to change elements dynamically next time.
The simple mode is faster and uses less tokens whereas the advanced mode uses more tokens and requires extra time to render the full video.
Where to go for more
There are a handful of other useful skills in that github repo above. These and more come standard in Behalf.bot which is itself open source. I’m also now teaching others my methodology for using SiYuan as a 2nd brain and building up a bespoke knowledge base of interlaced learning videos for being able to learn anything. You can learn more about that program here. I hope this pair of skills helps you in having a more audio/visual way of communicating with your agents.