All the Hard Stuff Nobody Talks About when Building Products with LLMs (via) Phillip Carter shares lessons learned building LLM features for Honeycomb—hard won knowledge from building a query assistant for turning human questions into Honeycomb query filters.
This is very entertainingly written. “Use Embeddings and pray to the dot product gods that whatever distance function you use to pluck a relevant subset out of the embedding is actually relevant”.
Few-shot prompting with examples had the best results out of the approaches they tried.
The section on how they’re dealing with the threat of prompt injection—“The output of our LLM call is non-destructive and undoable, No human gets paged based on the output of our LLM call...” is particularly smart.
Recent articles
- Phoenix.new is Fly's entry into the prompt-driven app development space - 23rd June 2025
- Trying out the new Gemini 2.5 model family - 17th June 2025
- The lethal trifecta for AI agents: private data, untrusted content, and external communication - 16th June 2025