AI Engineer NYC (10/14/26)

A Roadmap for Evaluating Highly-Subjective Tasks

• Talk Slides

• Further reading:

  • Human evaluation

  • LLM-as-a-Judge

  • Rubrics

  • Statistics for LLM Evalation

  • How to Build Benchmarks

  • How to Evaluate Agents

• Work at Netflix:

  • LLM Judges for Synopsis

  • Small LLM Judges