MarmicodeTrusting Your Agentic Angular Apps: A Testing and Eval Strategy
3 Days · Online
🐣 Early Bird starts at €1,270
Three days to draw the new testing boundaries in your AI-powered apps.Fake the LLM where you can, eval what's left, and know exactly what each eval costs you.
🍱 What you will learn
Map the layers between the browser and the LLM, and find out what's deterministic, what's probabilistic, and where the boundary really sits.
Scripted responses, structured outputs, and tool calls for fast deterministic tests — plus a clear-eyed look at when faking lies to you.
Use the event stream as a testing seam: record and replay AG-UI streams for frontend tests with no model and no tokens.
UI-as-data is testable data: assert on A2UI payloads instead of pixels, and validate that generated UI stays inside the trusted catalog.
Dataset, task, grader, score: what an eval is made of, how grading differs from asserting, and when to run offline vs online.
Start with a hand-crafted golden set, mine production traces for real cases, and generate synthetic edge cases without fooling yourself.
Code graders first, then rubrics and pairwise comparison — and calibrate the judge against human labels before you trust its grades.
Score tool choice, argument accuracy, and ordering across multi-turn conversations, not just the final answer.
Sampling, caching, pass thresholds vs score tracking, and how to tell a red eval apart from an actual regression.
Know the cost per run, per PR, and per day — and keep the feedback loop tax from slowing the inner loop to a crawl.
Traces as your new stack trace, online evals and user feedback signals, and the path from production incident to dataset entry.
Test, eval, or monitor — per layer, per risk, and per budget. You take the decision tree home.
🗓️ Program
- What non-determinism does to assertions, snapshots, and CI.
- The failure modes: hallucination, drift, prompt regressions, model swaps.
- Test, eval, or pray: the three postures and what each one buys you.
- Mapping the layers between the browser and the LLM: UI, AG-UI wire, orchestration, tools, prompts, model.
- What's deterministic, what's probabilistic, and where the boundary really sits.
- Take a real feature apart layer by layer, and mark where determinism actually ends.
- Faking at the model boundary: scripted responses, structured outputs, tool calls.
- Record & replay: when it helps, when it rots.
- What fakes can prove — and the confidence they can't give you.
- Swap the model for a fake, and get an agentic flow under fast, repeatable test.
- Tools are just functions: testing them like you always did.
- Prompt assembly, context construction, and structured output parsing.
- Guardrails and validation layers: testing the safety net, not the model.
- Put the deterministic shell under test, and watch the token bill stay at zero.
- Scripting multi-turn conversations and tool-call sequences.
- Simulating failure: timeouts, malformed outputs, refusals.
- Turn a flaky, hard-to-reproduce report into a deterministic failing test, then make it pass.
- The event stream as a testing seam: run lifecycle, messages, tool calls, state deltas.
- Recording and replaying AG-UI streams: deterministic frontend tests, no model, no tokens.
- Asserting on the wire: what the agent said vs what the UI rendered.
- Streaming edge cases: partial messages, cancellation, reconnection.
- Record a run once, then test the whole chat surface against it forever, for free.
- UI-as-data is testable data: asserting on A2UI payloads instead of pixels.
- Catalog contracts: validating that generated UI stays inside the trusted catalog.
- Rendering tests: from A2UI JSONL to your design system components.
- Pin a generative UI feature with catalog contracts and rendering tests.
- Anatomy of an eval: dataset, task, grader, score.
- Evals vs tests: grading vs asserting.
- Offline vs online evals.
- Build a dataset, a task, and a grader, and get a score you can actually reason about.
- Starting small: hand-crafted golden datasets.
- Mining production traces for real cases.
- Synthetic data: generating edge cases without fooling yourself.
- Mine real traces, generate the edge cases they miss, and end up with a set worth grading against.
- Code graders first: exact match, contains, structural checks.
- LLM-as-judge: rubrics, pairwise comparison, known biases.
- Calibrating the judge against human labels — trusting the grader before trusting the grades.
- Write the rubric, run it against human labels, and find out how much your judge can be trusted.
- Trajectory evals: grading tool choice, arguments, and ordering — not just the final answer.
- Multi-turn evals: simulated users and conversation-level scoring.
- Grading generative UI: did the agent pick the right A2UI components for the job?
- Score the path, not just the destination: tool choice, arguments, and order.
- Where the tokens go: cost per run, per PR, per day.
- Sampling, caching, and pass thresholds vs score tracking.
- Handling variance: when a red eval isn't a regression.
- The feedback loop tax: keeping the inner loop fast.
- Get evals running on every PR without the bill or the wait getting out of hand.
- Traces as your new stack trace.
- Online evals and user feedback signals.
- From production incident to dataset entry: closing the loop.
- Wire up tracing, catch a real failure, and promote it into your dataset.
- The decision framework: test, eval, monitor — per layer, per risk.
- What to run on every commit, every night, every release.
- When an eval is worth its price — and when it isn't.
- A takeaway decision tree.
- Agent integration, security & generative UI are covered in the companion workshop: “Agentic Angular: Baking Safe AI Agents Into Your Apps”.
- Test, eval, or monitor — decided layer by layer, for the app you actually ship.
🎓 Required knowledge
- Curiosity and good web culture
- Comfortable with TypeScript and automated testing basics (any framework)
- No prior LLM engineering or eval experience required
👨🏻🏫 Your instructor
Younes Jaaidi
I am a Software Cook who enjoys whipping code until tests pass.
With nearly 20 years of experience in eXtreme Programming, I've coached dozens of teams and trained thousands of developers to cook robust and maintainable software — using Test-Driven Development, pragmatic testing strategies, and a healthy dose of Collective Ownership.
I'm also an Angular Google Developer Expert, an NX Champion, and a mediocre sailor.
🙋 Frequently Asked Questions
Developers shipping AI features who no longer trust their test suite; QA and testing specialists facing non-deterministic behavior for the first time; and leads, architects, and CTOs who need to know what "confident enough to ship" means when a model is in the loop.
You should be comfortable with TypeScript and the basics of automated testing in any framework. Good web culture and curiosity matter more than seniority.
No. We start from what non-determinism does to assertions, snapshots, and CI, then build evals up from first principles. If you already run evals, the AG-UI and A2UI testing seams, judge calibration, and CI cost work will still be new ground.
The examples run on TypeScript end-to-end, with Vitest, Mastra, and Langfuse. Every pattern maps one-to-one to other stacks and frameworks, and we name the alternatives as we go — the boundaries, the graders, and the cost model transfer, not the tool names.
A computer with internet access, microphone, webcam, an up-to-date browser, and installation rights.
Yes. Each topic ends in an exercise: dissecting an AI feature to draw its SUT boundaries, testing an agentic flow with a fake model, replaying an AG-UI stream, calibrating an LLM judge, grading an agent's trajectory, wiring evals into CI with a budget, and finally building the decision tree you leave with.
To keep the workshop interactive. Ten attendees is the point where everyone still gets their code looked at and their questions answered.
No. Agent integration, security, and generative UI are covered in the companion workshop, "Agentic Angular: Baking Safe AI Agents Into Your Apps". This one picks up where that leaves off: proving the thing works.
Yes. Contact me for a quote and administrative details.
"Book a Session" lets you join a scheduled session with other participants. "Custom Session" is for companies who want a private, in-house workshop, with optional adjustments to content, duration, or focus areas.
If the workshop doesn't meet your expectations, reach out within 7 days and we'll work it out.