Member of Technical Staff (Answer Quality & Evals)
Could this role work from Kenya?
Kenya eligibility needs confirmation
- Stated location
- San Francisco
- Listing updated on our site
- 14 Nov 2026, 01:01 EAT
Last seen in the live employer feed: 14 Nov 2026, 01:01 EAT. This confirms a feed listing, not employer endorsement of an application.
- Mentions "eat"
These are automated wording checks, not confirmation from the employer. The update date records a site update, not a hiring check. Review location, working hours, experience, and current availability in the application portal. Employer identity remains hidden here.
Your application workflow
- 1. Check requirements
- 2. Tailor your CV below
- 3. Apply through the portal
- 4. Track your progress
AI Application Copilot
Prepare role-focused CV bullets and a cover letter to review
| Employer | Verified Direct Employer (Protected) |
|---|---|
| Location | San Francisco |
| Remote type | Remote |
| Posted | 7mo ago |
| Application Status | Pro Membership & Free Pass Access |
About this role (Direct Employer Details Protected)
[Employer] serves tens of millions of users daily with reliable, high-quality answers grounded in an LLM-first search engine and specialized data sources. The Answer Quality team ensures that our prompts, tools, search systems, datasets, and models work together to create the best possible experience for our users.
As our product and agent capabilities evolve, we need evaluation systems that are fast, reliable, production-faithful, and actionable. In this role, you will build and improve the technical foundations that support Answer Quality across [Employer]. This includes our shared evaluation infrastructure and the platform used to replay and analyze agent traces. You will work closely with data scientists, engineers, and product teams to identify quality problems, measure their impact, and turn evaluation findings into product improvements.
Responsibilities
- Build shared evaluation infrastructure that helps teams run reliable evals, analyze results, and make product and model decisions
- Develop the platform for replaying and analyzing agent traces to reproduce production behavior and diagnose failures
- Build and operate scalable systems for processing, storing, and monitoring interaction, trace, and evaluation data
- Partner with data scientists, engineers, and product teams to turn answer-quality problems into evaluations, analyses, and product improvements
- Operate in a small, high-impact team where your work directly shapes how [Employer] measures and improves Answer Quality
Qualifications
- 4+ years of software, data, or machine learning engineering experience shipping and operating production systems
- Strong proficiency in Python and SQL, with solid fundamentals in system design, data modeling, and distributed systems
- Experience building big-data systems, including distributed compute, large-scale storage, and high-volume pipelines
- Demonstrated ownership of ambiguous technical projects from initial design through production operation
- Ability to work effectively with data scientists, engineers, and product partners
Preferred Qualifications
- Experience building evaluation, experimentation, observability, or machine learning infrastructure
- Familiarity with LLM and agent systems, including tool use, execution traces, replay, and simulation
- Experience building on top of large-scale data processing platforms such as Databricks, Snowflake, or ClickHouse
Unlock 0-Second Direct Apply
This role was published 7mo ago. Global recruiters typically freeze applications after receiving the first 50 CVs. Pro members and Free Pass holders reveal the employer identity and get immediate 0-second direct apply access before positions fill.
Top of Recruiter Inbox
First 20 applicants receive 80% of recruiter interviews.
AI ATS Match Tailoring
Instant role-targeted bullets & cover letters.
Remote Contractor Toolkit
USD invoices, W-8BEN guide & CRM pipeline.
Instant M-Pesa STK push · 7-Day Money-Back Guarantee · Be in the first 20 applicants
Related Remote Jobs Open to Kenya
Explore similar international remote opportunities
Member of Technical Staff (Model Behavior)
Verified Direct Employer · Vetted
About the Role We're hiring software engineers for the Model Behavior team to help shape how [Employer]’s AI products behave: the style of their resp…
Member of Technical Staff (AI Infrastructure Engineer)
Verified Direct Employer · Vetted
We are looking for an AI Infra engineer to join our growing team. We work with Kubernetes, Slurm, Python, C++, PyTorch, and primarily on AWS. As an AI…