Expensive evolutionary search does not always need an exact fitness estimate for every candidate. It often needs a reliable answer to a simpler question: which candidate is better? We address this need through Teacher-Guided Learning NSGA-II (TGL-NSGA-II), a low-fidelity framework for constrained T…

We present \textbf{ENAS}, a hardware-aware Neural Architecture Search (NAS) framework that combines a static feasibility check, a cell-based search space supporting standard, depthwise-separable, and bottleneck blocks with optional skip connections, and a three-stage hybrid search strategy (random $

Enjoy the videos and music that you love, upload original content and share it all with friends, family and the world on YouTube.

A TinyML Object Detection model that identifies clothing colors in real time — trained and deployed entirely through Edge Impulse Studio.

Pete Warden has spent his career on the frontier of small, local AI, first as one of deep learning’s earliest engineers (he coined the term “TinyML”) and now as founder of Useful Sensors and Moonshine AI, where he builds voice models that run entirely on-device. Pete joined Ben to make the case that local AI no longer has to be a compromise. They get into what it actually takes to run a capable model on a laptop today; why the voice interface’s bad reputation is a consequence of rough, early implementations rather than a reflection of current capabilities; and where he stands in the ongoing debate between general “end-to-end” models and the compound AI approach of chaining specialized models together. Pete also explains why he thinks browser-based inference could be an “iPhone moment” for local AI and why more and more enterprises are considering self-hosted local models over commercial options. “The shape of [LLMs] is perfect for running locally,” Pete says, and local models could be a boon to enterprises worried about cost, privacy, and stability. About the Generative AI in the Real World podcast: In 2023, ChatGPT put AI on everyone’s agenda. In 2026, the challenge will be turning those agendas into reality. In Generative AI in the Real World, Ben Lorica interviews leaders who are building with AI. Learn from their experience to help put AI to work in your enterprise. Check out other episodes of this podcast on the O’Reilly learning platform or follow us on YouTube, Spotify, Apple, or wherever you get your podcasts. Takeaways 01.26 The usability gap is smaller than the marketing gap. The capabilities of local models are only a few months behind those from the big commercial companies, but because there’s no subscription revenue model behind local models, they often go unpromoted. “It’s very hard to make money off local models,” Pete explains, so the big companies aren’t focused on selling them. “Every company is going to go for the [product] that has an easy subscription revenue model. And that means you have a massive ton of marketing around all of these tools that are kind of like, ‘Oh, let’s have a little text box on a website.’ And so it means mostly that people have never heard of these local models.” 04.20 Local models are already good enough for most use cases. Pete compares the moment to the early web, when free alternatives like Apache eventually overtook expensive commercial servers. “All of these alternatives, once people actually had time to look around and they had a little bit of time to improve, they just wiped the floor with the commercial [offerings],” he points out. “I don’t know if we’re going to quite get there, but that’s the kind of pattern that I’m seeing.” 07.26 “The hardware barriers are a lot lower than people think.” Ben and Pete discuss what hardware you actually need to get up and running, from parameter counts, quantization (Q4, 8-bit), and VRAM requirements to the new Apple M5 Studio’s unified memory as a way to run very large models locally at usable speed. “The key thing is whether you can fit [your model] into your graphics card’s memory,” Pete says. “So with weight quantization, 9 billion [parameters] if it was 8 bits is like 9 GB. A lot of mid-end decent laptops that are shipping now have more than that.” 18.33 “It’s not that people don’t like voice interfaces. It’s that people don’t like bad voice interfaces.” We’ve solved most of the big problems, like dealing with background noise, phrasing, and speech in a range of accents—or at least have improved tools’ capabilities. However, “there’s no commercial incentive to kind of pull them all together,” Pete says. Most tools feel like they haven’t caught up to the LLM era, but “open source can be a really strong lever” to updating them, argues Pete. 28.26 We’re navigating the split between “LLM maximalist” end-to-end models (favored by big AI companies with the most capital) and the “compound AI” approach of chaining together specialized models from different sources. “If the future is end-to-end models, then only the people with the most money can actually build and train them,” Pete notes. Compound AI lets you “actually train all of the models independently” to accomplish your particular goals. While the performance of end-to-end models continues to improve, especially for multimodal models like Qwen or Gemma, using one can be a bit like choosing a Swiss Army knife over a tool specially designed to accomplish a single specific task, to use Pete’s metaphor. It may get the job done, but it’s probably not the most effective way to do it. 36.10 Voice capabilities in the browser could be a game changer. Embedding a model directly in the browser—Chrome has a built-in ~4B parameter model that’s accessible from any website via JavaScript, for instance—makes it part of the operating system. “Once you are able to transcribe fast and accurately in the browser, it’s a way for people to easily start experimenting with this stuff,” Pete explains. Could this be an iPhone moment for LLMs? 39:58 The “gravitational pull” is toward on-prem. Unlike most recent technological advances that depend on the cloud to function, LLMs are well-suited to running locally, even with no internet connectivity. Enterprises are grappling with concerns about cost, privacy, capabilities changing with no notice, or even the models they depend on disappearing. Hosting your own model, whether on your laptop or in your corporate infrastructure, gives you the stability to plan for the long term. 44:21 GPUs are fantastic for training but “complete overkill for inference.” Pete likens it to “trying to use an oil tanker to go and do your shopping.” Memory bandwidth is the real limiting factor, and it’s a problem that companies like Apple, with its new chip designs and unified memory bandwidth, are working on solving. “Even if you’re running on the CPU, if you have something that’s got high-enough bandwidth to pull 27 billion weights in a fraction of a second, then the rest of it is fairly easy in terms of actually doing the processing,” Pete says. “I think we’re going to see a lot of really imaginative solutions now that people understand what the workload looks like.”

Real-time active load estimation on ESP32 using INT8 TFLite Micro with 121 µs latency and 772 B arena memory.

Non-invasive TinyML acoustic sensor on nRF54LM20 detecting pipe micro-fractures before major leaks occur.

Low-power Arm-based microcontrollers with integrated neural coprocessors now available as a ready-to-run low-cost development board.

Pete Warden has spent his career on the frontier of small, local AI, first as one of deep learning’s earliest engineers (he coined the term “TinyML”) and now as founder of Useful Sensors and Moonshine AI, where he builds voice models that run entirely on-device. Pete joined Ben to make the case that local AI no longer has to be a compromise. They get into what it actually takes to run a capable model on a laptop today; why the voice interface’s bad reputation is a consequence of rough, early implementations rather than a reflection of current capabilities; and where he stands in the ongoing debate between general “end-to-end” models and the compound AI approach of chaining specialized models together. Pete also explains why he thinks browser-based inference could be an “iPhone moment” for local AI and why more and more enterprises are considering self-hosted local models over commercial options. “The shape of [LLMs] is perfect for running locally,” Pete says, and local models could be a boon to enterprises worried about cost, privacy, and stability. About the Generative AI in the Real World podcast: In 2023, ChatGPT put AI on everyone’s agenda. In 2026, the challenge will be turning those agendas into reality. In Generative AI in the Real World, Ben Lorica interviews leaders who are building with AI. Learn from their experience to help put AI to work in your enterprise. Check out other episodes of this podcast on the O’Reilly learning platform or follow us on YouTube, Spotify, Apple, or wherever you get your podcasts. Takeaways 01.26 The usability gap is smaller than the marketing gap. The capabilities of local models are only a few months behind those from the big commercial companies, but because there’s no subscription revenue model behind local models, they often go unpromoted. “It’s very hard to make money off local models,” Pete explains, so the big companies aren’t focused on selling them. “Every company is going to go for the [product] that has an easy subscription revenue model. And that means you have a massive ton of marketing around all of these tools that are kind of like, ‘Oh, let’s have a little text box on a website.’ And so it means mostly that people have never heard of these local models.” 04.20 Local models are already good enough for most use cases. Pete compares the moment to the early web, when free alternatives like Apache eventually overtook expensive commercial servers. “All of these alternatives, once people actually had time to look around and they had a little bit of time to improve, they just wiped the floor with the commercial [offerings],” he points out. “I don’t know if we’re going to quite get there, but that’s the kind of pattern that I’m seeing.” 07.26 “The hardware barriers are a lot lower than people think.” Ben and Pete discuss what hardware you actually need to get up and running, from parameter counts, quantization (Q4, 8-bit), and VRAM requirements to the new Apple M5 Studio’s unified memory as a way to run very large models locally at usable speed. “The key thing is whether you can fit [your model] into your graphics card’s memory,” Pete says. “So with weight quantization, 9 billion [parameters] if it was 8 bits is like 9 GB. A lot of mid-end decent laptops that are shipping now have more than that.” 18.33 “It’s not that people don’t like voice interfaces. It’s that people don’t like bad voice interfaces.” We’ve solved most of the big problems, like dealing with background noise, phrasing, and speech in a range of accents—or at least have improved tools’ capabilities. However, “there’s no commercial incentive to kind of pull them all together,” Pete says. Most tools feel like they haven’t caught up to the LLM era, but “open source can be a really strong lever” to updating them, argues Pete. 28.26 We’re navigating the split between “LLM maximalist” end-to-end models (favored by big AI companies with the most capital) and the “compound AI” approach of chaining together specialized models from different sources. “If the future is end-to-end models, then only the people with the most money can actually build and train them,” Pete notes. Compound AI lets you “actually train all of the models independently” to accomplish your particular goals. While the performance of end-to-end models continues to improve, especially for multimodal models like Qwen or Gemma, using one can be a bit like choosing a Swiss Army knife over a tool specially designed to accomplish a single specific task, to use Pete’s metaphor. It may get the job done, but it’s probably not the most effective way to do it. 36.10 Voice capabilities in the browser could be a game changer. Embedding a model directly in the browser—Chrome has a built-in ~4B parameter model that’s accessible from any website via JavaScript, for instance—makes it part of the operating system. “Once you are able to transcribe fast and accurately in the browser, it’s a way for people to easily start experimenting with this stuff,” Pete explains. Could this be an iPhone moment for LLMs? 39:58 The “gravitational pull” is toward on-prem. Unlike most recent technological advances that depend on the cloud to function, LLMs are well-suited to running locally, even with no internet connectivity. Enterprises are grappling with concerns about cost, privacy, capabilities changing with no notice, or even the models they depend on disappearing. Hosting your own model, whether on your laptop or in your corporate infrastructure, gives you the stability to plan for the long term. 44:21 GPUs are fantastic for training but “complete overkill for inference.” Pete likens it to “trying to use an oil tanker to go and do your shopping.” Memory bandwidth is the real limiting factor, and it’s a problem that companies like Apple, with its new chip designs and unified memory bandwidth, are working on solving. “Even if you’re running on the CPU, if you have something that’s got high-enough bandwidth to pull 27 billion weights in a fraction of a second, then the rest of it is fairly easy in terms of actually doing the processing,” Pete says. “I think we’re going to see a lot of really imaginative solutions now that people understand what the workload looks like.”