Senior AI Engineer (Observability)

Andersen is hiring a Senior AI Engineer (Observability) to build scalable AI infrastructure and observability solutions for a cloud platform, ensuring reliable performance and intelligent system monitoring.

The customer is an international company delivering professional and technology-enabled solutions that support effective collaboration, structured communication, and operational efficiency for organizations. It operates in a fast-growing environment, focusing on scalability, security, and continuous improvement while developing digital platforms used by diverse clients worldwide.

The project is focused on enhancing a secure, cloud-based board management platform with intuitive meeting tools, real-time collaboration, and advanced analytics. It also includes building and maintaining scalable AI infrastructure, orchestration patterns, and observability to ensure reliable and intelligent platform performance.

Responsibilities:

• Translating product requirements into AI workflows and quality criteria.
• Designing, implementing, and optimizing prompts using LangFuse.
• Building evaluation datasets from real customer scenarios.
• Establishing quality gates and measurement systems (precision, recall, accuracy).
• Setting up and maintaining LangFuse for observability and prompt experimentation.
• Tuning workflows based on production telemetry and evaluation results.
• Partnering with Product and QE on acceptance criteria and test strategy.

Must-have:

• Experience in software engineering for 5+ years, including delivering production systems.
• Bachelor's degree in computer science, Engineering, IT, or equivalent experience.
• Experience with LLM-powered features or AI pipelines (instrumentation, evaluation, or monitoring).
• Experience with LLMOps and observability tools (e.g., Langfuse, Arize, W&B, LangSmith).
• Understanding RAG, prompt engineering, vector search, and orchestration frameworks (e.g., Semantic Kernel, LangChain).
• Experience in designing LLM evaluations (datasets, scoring, LLM-as-judge, regression testing).
• Proficiency in Python and/or C#/.NET, including working with production codebases.
• Understanding observability fundamentals (tracing, logging, metrics, alerting) in cloud environments (e.g., OpenTelemetry, Azure Monitor).
• Experience with CI/CD and Git workflows, Azure DevOps preferred.
• Understanding LLM cost management (token tracking, budgeting, optimization).
• Experience working in regulated environments (e.g., SOC 2, ISO 27001).
• Experience with AI-assisted development tools (e.g., Claude Code, PlayerZero).
• Strong communication skills, including translating insights and educating teams on AI quality.
• Level of English – from Intermediate+ and above.

Nice to have:

• Experience with A/B testing or online experimentation frameworks applied to AI features.
• Familiarity with OpenTelemetry and its emerging application to LLM tracing (OpenLLMetry or similar).
• Experience monitoring guardrail and content safety systems — including tracking filter effectiveness, policy enforcement, and safety-related anomalies as a distinct observability concern.
• Experience with semantic similarity metrics, embedding models, and vector database platforms (e.g., Azure AI Search, Pinecone, Qdrant).
• Background in QA engineering, data engineering, or ML engineering that informs a rigorous approach to quality measurement.
• Experience extending observability or platform practices across multiple product lines or teams.

Reasons why this job would be interesting to you:

• Experience in teamwork with leaders in FinTech, Healthcare, Retail, Telecom, and others. Andersen cooperates with such businesses as Samsung, Siemens, Johnson & Johnson, BNP Paribas, Ryanair, Mercedes, TUI, Verivox, Allianz, T-Systems, etc.
• The opportunity to change the project and/or develop expertise in an interesting business domain.
• Job conditions – you can work both fully remotely and from the office or can choose a hybrid variant.
• Guarantee of professional, financial, and career growth! The company has introduced systems of mentoring and adaptation for each new employee.
• The opportunity to earn an additional up to 1,000 USD per month by participating in the company's activities.
• Access to the corporate training portal, where the entire knowledge base of the company is collected and which is constantly updated.
• Bright corporate life (parties / pizza days / PlayStation / fruits / coffee / snacks / movies).
• Certification compensation (AWS, PMP, etc).
• Referral program.
• English courses.
• Private health insurance and compensation for sports activities.

Join us!

Your personal data is protected in accordance with GDPR regulations.