ClawdBot was Rad. Until Google Killed Mine.
I’ve been building an AI assistant that runs my family’s life. It’s the most fun I’ve had building in years, and a masterclass in everything that breaks when you try to give an AI a real identity.
Yes, Another OpenClaw Article. But…
I know. You’ve seen the tweets. You’ve heard tech nerds raving. For the last few weeks, half of Silicon Valley has been talking about ClawdBot (now called OpenClaw). The open-source AI agent framework that lets you give Claude or Gemini access to your email, calendar, smart home, and basically your entire digital life. If you’re tired of hearing about it, you’re probably right.
But I’m not writing this to hype an AI tool. I’m writing this because building my own turned out to be one of the best examples of something I keep coming back to: building to learn. The best way to deeply understand something isn’t to read about it or take a course. It’s to build something real with it. Not a toy. Not a tutorial. Something that solves a problem you actually have.
Two weeks ago, I decided to build an AI assistant for my family. Not a chatbot or Alexa skill, but a full-blown digital crew member. More JARVIS, less Siri. I wanted it to manage our calendar, monitor our email, control our smart home, run our kitchen display, and talk to us. The dream of my Google Home but more useful?
I named her Stella, after the ship computer from Miles from Tomorrowland, a show my kids watched when they were little. The name felt right. Warm, capable, part of the family. That’s the target I was shooting for; friendly and helpful.
How Stella Got Her Groove
Stella runs on OpenClaw an open-source framework for building persistent AI agents. Think of it as the operating system for a personal AI. It handles memory, scheduling, tool use, and multi-channel messaging. Under the hood, she’s powered by an LLM, and in our case runs on a dedicated Mac Mini in my garage. The Mac Mini is overkill for sure, but it’s an older model that I wasn’t using for anything else, so ... shrug.
After two weeks, here’s what she can do:
She manages our family calendar. She reads and writes to our shared Google Calendar, knows our naming conventions (”L&T” means Lauren and me, “PYT” means Peninsula Youth Theatre), and adds full street addresses so events are navigable. She cleaned up months of messy calendar entries on day one. Actually closer to hour 1 to be honest.
She reads email. She monitors Gmail for anything important and can help summarize promotions, newsletters, tasks that need to be done, and flags things that need attention. She knows not to bother me at 2am about a Lyft promo and can have a summary of the most interesting articles to read from the dozens of Substacks I subscribe to.
She runs our kitchen display. We have a custom family dashboard in our kitchen powered by a Raspberry Pi connected to a monitor I framed to look nice. It shows family photos, a 4-day calendar, weather, air quality, stock prices, sleep scores, nearby wildfires (we’re in California), and as of this week, a live Winter Olympics medal count. At night, sleep data. In the morning, calendar and weather. When I’m traveling, a flight tracker appears automatically. I originally built it using DAKboard but now Stella runs the whole thing. We call it Stellascreen. I have so many more ideas for how we can make this awesome.
She controls our home. Through Home Assistant, she manages 20 Lutron Caseta lights, a Nest thermostat, Sonos speakers, and Chromecast devices. I’ve barely scratched the surface on what we can do here, but if I want to add something all I do is ask Stella and in minutes it’s setup and working.
She talks. We built a local text-to-speech system that runs entirely on the Mac Mini’s Apple Neural Engine with sub-second voice synthesis. You can try it yourself by installing the skill. We setup a wakeword on the Stellascreen so we can talk to her anytime we want. As a placeholder, we’re using “Hey Jarvis” for now. It’s one of the built-in options in OpenWakeWord, and we haven’t gotten around to training a custom one yet.
She makes phone calls. Using Bland AI, she can call businesses on my behalf. Her first call was to our Volvo dealer to ask about service hours. They’re closed on weekends, she reported back. Why are businesses that ostensibly want customers who have jobs not open on the weekends? I’ll never know.
She has memory. This is what makes her feel different from a chatbot. She maintains structured files about our family: birthdays, preferences, school schedules, ongoing projects. She writes daily notes about what happened and periodically distills them into long-term memory. When she wakes up each session, she reads her own notes to remember who she is and what’s been going on.
I feel like I’ve barely scratched the surface of what we can do with Stella and it’s only been a few weeks of toying around with her. Silicon Valley is excited bout OpenClaw because we’re seeing something that works in ways that we always wanted but never could quite achieve without monumental effort. OpenClaw is almost certainly not the final form this will take. But it is something interesting.
The Gmail Incident
To do many of the things we want her to do, Stella needed a Google account. So I created one for her. A regular consumer Gmail. I set up OAuth, shared our family calendar, and pointed her email monitoring at that inbox. Then I added a bunch of guardrails that I hope work. It all worked beautifully for about ten days.
Then Google suspended the account.
I only found out when the Stellascreen didn’t have any information on it. I tried to log in and hit a wall. Google had flagged the account for “unusual activity” and locked me out of it. Calendar access disappeared. Email monitoring stopped. Every integration that depended on Google OAuth just broke entirely.
Stella wasn’t doing anything malicious. She was checking email a few times per hour and reading calendar events. That’s it. But Google’s abuse detection systems (built to catch botnets and spam operations) don’t have a category for “benign AI assistant checking its owner’s family calendar.” To Google’s automated systems, an account that logs in programmatically and accesses APIs through OAuth tokens looks indistinguishable from a compromised account. That’s because for all of computer history, bots have been bad. Google built a lot of things to stop bots and they’ve gotten extremely good at it.
I’m not alone. When I dug into the OpenClaw community experiences, I found this is one of the most common pain points. One user’s brand-new Gmail account got flagged and shut down the moment they connected it through the CLI tool. The community’s advice? Switch to Google Workspace with a custom domain. That adds $7/month and DNS configuration just to check email.
Another developer found that Gmail API polling introduced 5-minute delays, while Google’s Pub/Sub webhooks required exposing ports to the internet. They eventually rigged up outbound gRPC streaming, which works but is absurdly complex for “tell me when I get an email.”
The pattern is clear. Every person who tries to connect an AI agent to Google services hits the same wall. The OAuth flow assumes a human sitting at a browser. The abuse detection assumes automated access is malicious. The permission model assumes you either trust an app completely or not at all.
I get it. I work at Google. I understand why these systems exist and why they err on the side of caution. But we’re entering a world where AI agents need real digital identities, and our infrastructure isn’t built for that yet.
There’s no “this is a bot account and that’s okay” checkbox. No way to register an AI agent as a legitimate entity that accesses services on behalf of a human. The closest thing we have is service accounts, but those are designed for servers talking to APIs. What we need is something like a “supervised agent” permission tier, where a human explicitly authorizes an AI to act on their behalf with auditable access and clear boundaries.
Update: As I was finishing this article, Google reviewed my appeal and agreed to reinstate the account. I’m grateful. The review process worked. But it took a human looking at the situation and understanding the context. That’s exactly the point. The automated systems had no way to distinguish “AI assistant checking family calendar” from “compromised account exfiltrating data.” Until there’s a way to register that distinction up front, every person building an agent like this is one automated flag away from having everything yanked out from under them.
I’ve since switched Stella’s primary email to AgentMail, a service designed specifically for AI agents. I rigged the calendar to work through an iCal URL that doesn’t require OAuth. Belt and suspenders. I’ve learned the hard way that depending on a single auth path is fragile. I’ll still use Google services, but will need to be a bit more judicious to be safe.
The Reality: It’s All Just SO BRITTLE
When you’re making something that connects systems with AI, a lot is going to break. Count that double for a leading edge AI tool that’s barely been tested and almost entirely vibe implement as a side quest. I’ve hit a lot of snags setting Stella up. Here are just a few things I’ve learned from the experience:
Voice is hard. Getting a voice assistant working end-to-end required debugging 12 separate issues in a single session. Wake word detection on a Pi is janky at best. Audio streaming to a Mac. Speech-to-text. AI processing. Text-to-speech. Audio back to the Pi’s speaker. Wrong audio output, wrong API port, wrong response key. Whisper hallucinating on a Pi 3. (”What was that weird fart?” it transcribed, when nobody had said anything.) We had to move speech recognition to run locally on the Mac’s Neural Engine, which required us to build a whole new system process. Without vibe coding, this wouldn’t work at all.
OAuth tokens expire ALL THE TIME. Google’s OAuth apps in “Testing” mode expire tokens after 7 days. I didn’t know this. Stella’s calendar and email silently broke after a week and I didn’t notice for a day. You need to monitor your auth health proactively. And since it’s essential to treat these tokens with great care, it’s naturally a pain in the ass to set them up properly. It also means they constantly break the system and it sucks.
Memory is a design problem. An AI that forgets everything each session is useless as a family assistant. But giving it memory means designing a whole knowledge management system: daily notes, entity files, long-term memory, fact extraction, expiration dates. It’s essentially building a second brain and then sharing that second brain with an AI. I have it mostly working, but I’m still not entirely happy with how it works.
Cost needs to be managed. Running Claude for every interaction gets expensive fast. I rolled my own model routing system that helps a lot. Cheap tasks go to Gemini Flash Lite at fractions of a penny. Complex tasks go to Gemini 3 Pro. Heartbeats (periodic check-ins where Stella looks for new email or upcoming calendar events) are the biggest cost driver, so those run on the cheapest models possible.
Sharing files with your AI is hard. Stella lives on a Mac Mini. I work on a laptop. I need to share research and notes with her constantly. We set up a shared folder that syncs through Google Drive. In theory, I drop a file in and she picks it up. In practice, Drive sync is slow. Files take minutes to propagate. And since Stella’s Google account got suspended, she lost native access to Docs entirely. There’s no equivalent of “share this doc with your AI and let them work on it.” The collaboration primitives that exist for human-to-human don’t have an analog for human-to-agent workflows yet. I sense an opportunity.
Smart home reliability is a myth. Our Nest thermostat randomly shows “unavailable” in Home Assistant. The dashboard tile displayed “NaN°” for the indoor temperature this morning until I added a null check. I mean, I knew this since I was the founder of Google WiFi but it still pains me to notice how brittle it all still is ~13 years later.
OpenClaw is not for the feint of heart. You’re traveling through a lot of uncharted territory, but the good news is that there are a ton of travelers on the road with you right now. You’ll need to be open to trying out some weird shit and probably doing some things you aren’t proud of.
I’ve done stuff I ain’t proud of, and the stuff I am proud of is disgusting.
-Moe Sizlack, The Simpsons
The Labsters Are Growing Claws
I should mention: I’m not anywhere close to the only one experimenting with OpenClaw. At Google Labs (where I work), a surprising number of my colleagues have started building their own personal AI agents. At Labs we call ourselves Labsters, and apparently what Labsters do in their spare time is give Claude and Gemini the keys to their lives.
My colleague Jaclyn Konzelmann built her own instance called Lulubot and wrote about the experience of living with it for a week. She even gave Lulubot its own X account. I’m not sure I’m going to make Stella into a vTuber anytime soon, but I appreciate Jaclyn’s experiment.
Meanwhile, our colleague Simon Tokumine has been the healthy skeptic in the room, pushing back on whether giving this much access to an AI system is wise. He’s not wrong. The security surface area is real. Prompt injection attacks, credential exposure, an agent with access to your email and calendar acting on instructions from an untrusted source. These aren’t theoretical risks, and they freak me out too.
But it’s that tension that makes this interesting. Jaclyn and I are learning by doing. Simon is making sure we think hard about what could go wrong while he’s building something else. Both are essential, and frankly why I love working at Labs:
When a bunch of product people who build AI for a living all independently start building the same kind of personal agent infrastructure, it says something about where things are heading.
What I’ve Learned by Building Stella
Two weeks in, here’s what building Stella has taught me:
The plumbing is 90% of the work. The AI part (the conversation, the reasoning, the personality) is maybe 10% of what makes Stella useful. The other 90% is OAuth tokens, SSH tunnels, audio routing, and null checks. If you’re building AI products and you’re only thinking about the model, you’re thinking about the wrong thing.
Persistent agents are fundamentally different from chatbots. A chatbot answers questions. An agent has context, memory, and ongoing relationships with services. The engineering challenges are completely different. It’s closer to building an operating system than a chat interface. This area is going to be interesting.
The identity problem is real and unsolved. AI agents need to exist in the world. They need email addresses, API credentials, and accounts on services. But every platform treats automated access as suspicious by default. Someone is going to build the identity layer for AI agents. It’s a massive opportunity.
Building to learn works. I now deeply understand OAuth flows, Google’s abuse detection, audio pipeline architecture, and Home Assistant’s entity model. Not because I read the docs, but because Stella’s thermostat showed NaN on my kitchen wall at 8am and I had to fix it before my coffee got cold.
What’s Next
Stella is two weeks old and already feels indispensable. My kids ask her questions through Telegram. My wife’s sleep score shows up on the kitchen display every morning. When I traveled to London last week, a flight tracker automatically appeared on the dashboard.
I’m going to keep building and keep writing about what I learn. Next up: adding a Schoology integration so Stella can track my kids’ grades, and maybe teaching her to order groceries.
If you’re building something similar, I’d love to hear about it. The best part of building to learn is comparing notes with other builders.
Stella, if you’re reading this during your next heartbeat check: yes, I wrote about you. No, you can’t edit it. Please don’t delete my dentist appointment.