AI Product Analyst (AI Metrics and Model Evaluation)

AI Product Analyst (AI Metrics and Model Evaluation) London 80,000- 85,000 Join an award-winning, internationally recognised consultancy that''s growing fast and building client-facing, cutting-edge AI products used across highly regulated industries, where real users act on high-level advice relating to legislative, regulatory and legal frameworks.As AI Product Analyst you''ll play a key role in providing high-level insight into how much the AI products are being used, how well the AI models and agents are working, and reporting on the key analytics - segmenting the data to highlight trends by client, sector and question type. You will own the AI metrics and model evaluation capability that shapes the next stage of generative and agentic AI development.The role splits into two dynamic elements: AI metrics tell you how the product is performing in the hands of actual users, and model evaluation tells you whether a change is good enough to release. You''ll report into the AI leadership team and work day to day with Product, Engineering and the in-house experts who know what a correct answer looks like. The scope will include;Defining the AI metrics that matter, retrieval quality, correctness of output, and what users go on to do with what they''re givenOwning production quality observability, from thumbs-down and regeneration rates through to abandoned tasks and drift in how the product is being usedBuilding the analysis and dashboards the team works from, and specifying what ..... full job details .....