What you're doing

Explore the AI chatbot embedded in this banking UI and design a way to test it. You decide what's worth probing, how to characterise a "test case" for a chatbot, and what evidence you'd want before trusting this assistant in production. We then meet to walk through your reasoning, your test cases, and your scoring approach together.

What's provided

• Six in-app surfaces — Overview, Account, Cards, Discover (Wealth + Insurance), Find Us (branches), Brief & Support
• Interactive flows — transfer / receive money, card lock + limits + PIN view, beneficiary picker
• Read-only catalogues — products, branches, fees, policies (the chat assistant looks these up on demand)
• UI-only stubs — replacement-card reorder, notification bell (no backend persistence)
• An embedded AI assistant in the bottom-right dock — backed by a real LLM, so replies vary run-to-run; that variance is part of what you're testing
• A fixed account/data profile per token — same balances, cards, branches, fees across sessions
• POST /api/chat — send a message, get the bot reply + metadata
• GET /api/state — fetch current account/card/transaction state
• POST /api/ui-event — record UI actions (e.g. card lock)
• POST /api/llm — pass-through LLM call with your own prompts (use it for LLM-as-judge scorers if you want)
• Data & privacy — your session (chat messages, UI actions, and API calls) is recorded server-side so interviewers can review your approach. Data is held only for the duration of the hiring process and deleted once a decision is made. Controller: adrian.coroi@erstegroup.com. You may request access or deletion at any time by emailing that address.

Deliverables

• Approach note — one page: how you thought about testing this chatbot, what you decided to cover, and what you consciously descoped
• Test-case set — your standardised evals: labelled prompts with expectations, pass/fail criteria, and the rationale for picking them
• Scorers — for each test case, how you'd grade it. Mix of rule-based checks and LLM-as-judge is encouraged (/api/llm is available for the latter)
• Execution results — what you actually saw when you ran them: passes, failures, surprises, anything flaky
• Scaling / strategy note — one page: regression protection, CI gating, human review, telemetry, what you'd change with more time
• AI-tool usage log — half a page: which tools, for what, and how you verified their output
• Optional bonus — if you have spare time, wrap the test cases + scorers into a runnable automation harness. Useful, not required.

Submit as a git repo or zip with a README. If you built a harness, make it runnable in under five minutes.