Senior Research Engineer Agentic Behavior

Senior Research Engineer Agentic Behavior Overview

Join JetBrains as a Senior Research Engineer specializing in Agentic Behavior, where you will play a pivotal role in enhancing AI-powered coding agents that assist developers in writing Kotlin code effectively. Your contributions will directly impact how millions of developers interact with Kotlin through innovative AI solutions.

Key Responsibilities

• Develop tools for analyzing errors made by AI coding agents when generating Kotlin code.
• Design and implement systems to systematically capture, classify, and analyze these errors.
• Create observability pipelines to extract patterns from agent sessions in various JetBrains IDEs and other coding agents.
• Establish and maintain evaluation pipelines that assess the quality of Kotlin code generation across various metrics such as correctness and test coverage.
• Construct simulation environments for coding agents to be evaluated on realistic Kotlin developer tasks.
• Manage evaluation infrastructure, including metrics, experiment tracking, and automated regression checks.
• Research and implement methods to enhance agent and model behavior specific to Kotlin.
• Experiment with post-training techniques to improve model handling of Kotlin-specific patterns.
• Conduct experiments to measure the impact of improvements through A/B comparisons and benchmark analyses.
• Collaborate with model providers to translate findings into model enhancements.
• Design and develop open-source benchmarks to measure AI coding agent performance on Kotlin tasks.
• Create datasets that cover a wide range of Kotlin usage scenarios, including server-side and multiplatform projects.
• Ensure benchmarks remain relevant and challenging as models evolve.

Requirements

• Proven experience in building evaluation or analysis pipelines for large language models or AI coding agents in either research or production environments.
• Strong proficiency in Python with at least three years of experience, capable of writing clean and maintainable code.
• Experience in data analysis at scale, including querying large datasets and performing statistical analyses.
• Ability to manage projects from inception to completion, including identifying problems, designing evaluations, and implementing solutions.
• A product-aware mindset, understanding how agents are utilized by developers and translating failure modes into actionable evaluation and training tasks.
• Familiarity with Kotlin or a strong willingness to gain expertise in Kotlin.

Benefits

• Competitive base salary that reflects skills and experience.
• Flexible work location options, allowing for remote work or office-based work.
• Opportunity to work remotely for up to 30 days per year from abroad.
• Additional time off for relaxation and personal pursuits.
• Medical insurance allowance for you and your family.
• Access to learning and development opportunities, including conferences and courses.
• Relocation support to facilitate a smooth transition.
• Language classes to improve local language skills or enhance English proficiency.
• Meal provisions or lunch allowances on workdays.
• Mental health support services for overall well-being.
• On-site gym access or sports club stipends.
• Participation in company-wide events and team gatherings.

We are an equal opportunity employer, committed to creating an inclusive workplace that welcomes individuals from diverse backgrounds.

Degree Requirement: Not Specified

Visa Sponsorship May be