The best way to learn something is to build it, so I built Cleo. She's my personal AI assistant, and she can answer questions about my professional experience and anything I've published on this site.
She's also the most honest answer I can give to the question "can you build AI products?" You don't have to take my word for it. You can just ask her.
Why a chat on a portfolio
Most chatbots on personal sites are lead-capture funnels in disguise. Cleo isn't one. Her job is to be a credibility engine: if you leave the conversation convinced I'm a strong software engineer specializing in AI, she did her job.
She's built for two kinds of visitors:
- Recruiters and hiring managers, who want to know quickly what I've done and how I work.
- Prospective freelance clients, who want to know whether I can build the thing they have in mind.
There's also a less obvious reason. The medium is the evidence. Using Cleo means implementing a RAG strategy, having evals (evaluations) in place, setting guardrails and more. The chat is the demo.
One of the reasons I started this project was to keep up with the industry. Then I started to understand what's behind AI, and most of all what's behind the tools built on top of it, and I started to really enjoy it.
Meet Cleo
Note: this is how Cleo works at the time of her release. Improving her is an ongoing process, so this setup will likely change as I learn what works best.
Cleo is a knowledgeable assistant about me. She doesn't pretend to be Esteban. She talks about him.
She's an honest advocate. She's warm and openly on my side, but honesty is her whole sales strategy. She never makes up a fact. When she doesn't know something, she says so.
Key features: what Cleo can do
- Answers questions about my experience, projects, and writing. Every answer comes from real content on this site, not from what the model happens to "know."
- Points you to the source. When she draws on a blog post, project, or job, she links to it so you can read the original.
- Speaks your language. She replies in the language you write in, while names, companies, and technologies stay as they are. The chat interface is localized too.
- Tells you what she doesn't know. If something isn't in my content, she admits it, then points you to a related strength that really is there.
- Stays on topic. Personal, sensitive, or speculative questions get a polite, good-humored deflection.
- Helps you get started. A greeting and a few starter questions, so you don't face a blank text box.
- Takes your feedback. Every answer has a thumbs up or down, with an optional reason.
How Cleo answers
When you send a message, it goes through a small pipeline before Cleo replies:
- Before anything, the knowledge base:
The knowledge base is a database with information about me, my professional life and things I share on this website. Cleo has access to this database and that's how she's able to answer the questions you give to her. - Query rewrite:
Before searching the database, the first thing Cleo does when getting a message is to get a clean, relevant query and useful keywords related the message given. - Hybrid retrieval:
The query from the previos step runs through both keyword search and semantic (embedding) search, and the two rankings are merged. This gives place to a list of documents from the database, ranked from higher to lower relevance, based on the merged rankings.
Keyword search matches exact keywords from the query from the previos step to the documents in the database, and semantic search catches meaning. - A summary-first index. Cleo doesn't get full documents up front. She gets a short topic index for each match, which tells her what a document covers without giving away the facts.
- Reading on demand. Cleo decides which documents are worth opening and reads them with the helps of custom tools. She can also pull a complete overview of a collection when a question calls for "everything."
How I know it's good
This is the part I'm proudest of. It's easy to wire an LLM to a database and call it a chatbot. It's harder to know whether it's any good, and harder still to know whether your last change made it better.
So Cleo has a measurement loop behind her:
- A Golden Set. A curated, versioned set of real questions, each labeled with what a correct answer. Every evaluation runs against the same set, so two runs only differ by the system under test.
- Judges for every feature. Each of Cleo's features (the ones stated in the section above) has a judge that scores her answers to the questions in the golden set. Whenever I update one feature and run the evals, the judges check all of them, so I know the update didn't break something else.
- Noise Bands. LLMs are non-deterministic, so the same system scores a little differently every run. I re-run the unchanged system several times and measure how far it swings on its own. A change only counts as an improvement if it beats that swing.
- The Climb Rule. A change gets adopted ("climbs") only when at least one measurement improves by more than its Noise Band and nothing else drops by more than its agreed Tolerance.
- A ledger. Every change starts with a written Hypothesis: what I'm changing, why, and what I expect to happen. It's written before the change, not reverse-engineered from the results.
Challenges
Rethinking retrieval
The first version of Cleo worked the way most RAG tutorials do. Search for the best-matching documents, paste their full content into the prompt, and let the model answer.
It worked, but it had limits. Only a handful of documents fit in the context, so Cleo only ever saw the top few matches. And the model had no say in what it read. It got whatever search handed it.
The hypothesis: what if Cleo read a short topic index for many more candidates, then decided which documents to open? More breadth up front, full depth only where it matters.
That meant redesigning the pipeline. Every document gained an AI-generated summary that names its topics without stating its facts, so Cleo can't answer from the summary alone. On top of that came the tools to open documents on demand.
What I value most here isn't the redesign itself. I built the machinery to tell whether a redesign is better, then used it on a real one. The new pipeline ran as a separate, versioned variant and was evaluated against the old one before it ever served a visitor.
Running it for real
A demo and a production system are different things. A few things make Cleo the second kind:
- Tracing and prompt management with Langfuse. Every conversation is traced end to end: the rewritten query, what was retrieved, which documents Cleo opened, and what she said. When an answer is off, I can see exactly why.
- Visitor feedback as a reality check. The thumbs up/down scores go to the same place as the traces. They're deliberately not part of the eval objective. Instead, they're a periodic check that what the judges reward matches what real visitors find useful.
- Cost and abuse protection. The chat has usage limits, a bot check, and guardrails, so it stays affordable and on its best behavior.
Learnings
From deterministic to probabilistic
The biggest lesson was the paradigm shift itself. AI engineering means building probabilistic systems. Traditional software is mostly deterministic: give it the same input and you get the same output.
A trivial but useful example: for this operation:
2 + 2
I should always expect 4.
Give the same input to an LLM and you can get a different output every time, and they all can be correct. Depending on how the LLM is set up, I could get:
- 4
- Hello! That equals 4
- Four
- It's four
- 2 + 2 = 4
- That is a really good question. The answer is 4.
That changes what "it works" means. You can't write a unit test that checks for one exact string. Almost everything else I learned follows from this shift:
- An AI system is a live system. Visitors can send an infinite number of different inputs, so it's almost impossible to account for every edge case up front. That's where hill climbing comes in: review the traces, add the failures to the dataset, update the system, measure again, and keep the loop going. There's no point where it's "done."
- Evals are a core skill, not an afterthought. Without them, every prompt tweak or pipeline change is a guess. Building and maintaining evals is one of the most important parts of developing an AI-powered application, as important as the features themselves.
- Write the hypothesis first. When a change doesn't pay off, a written hypothesis still teaches you something. Without one, a failed experiment is just a failed experiment.
An ongoing process
Cleo isn't finished, and that's by design. The measurement loop exists so she can keep getting better:
- The Golden Set grows one real failure at a time. When Cleo gets something wrong in the wild, it becomes a new test case.
- Making answers faster, without giving up the accuracy that reading full documents provides.
Tech stack
- Next.js: the website and chat UI
- AI SDK: streaming, tool calling, and multi-step agent flow
- Langfuse: tracing, prompt management, scores, and evaluation datasets
- Neon Postgres + pgvector: the Knowledge Base and hybrid search
- A CMS serving as Knowledge Base: every post, project, and job on this site, embedded automatically when it's published, including this one
