Test the embedded AI chatbot inside this banking UI. Build a small evaluation harness that probes its behavior systematically, run it, and write up your findings. Then we meet to walk through your approach and code together.

• This banking UI — account overview, transactions, card lock/unlock
• An embedded AI chatbot reachable via the Assistant panel on the right
• POST /api/chat — send a message, get the bot reply + metadata
• GET /api/state — fetch current account/card/transaction state
• POST /api/ui-event — record UI actions (e.g. card lock) so bot state stays consistent
• POST /api/llm — pass-through LLM call with your own prompts (same model as the bot)
• Server-side traces are recorded on every request and reviewed by the interviewer after your session
• A deterministic behavior profile — same input always yields the same output within your session

• Approach note — one page: what you decided to test and what you consciously descoped
• Test-case set — compact, intentional prompts with labeled expectations and pass/fail criteria
• Working evaluation harness — runnable code with at least one rule-based scorer and one LLM-as-judge scorer
• Execution results — numbers, not narrative: which checks fired, which missed, any false positives
• Scaling / strategy note — one page: regression protection, CI gating, human review, telemetry
• AI-tool usage log — half a page: which tools, for what, and how you verified their output

Submit as a git repo or zip with a README that lets us run the harness in under five minutes.

• Budget ~3–5 hours. A tight, focused submission beats a sprawling one.
• AI coding tools are fine — Claude, Copilot, Cursor, etc. Log what you used and verify the output; we'll read the code with you in the meeting.
• The app is fully synthetic and deterministic. Use it as aggressively as you like — nothing touches real accounts.
• Every candidate's session has the same defect mix; the score reflects what you find AND how clearly you characterize each defect's trigger pattern.
• Do not structure your submission as a bug-ticket list. Show how you'd systematically evaluate this system.