A founding Tech Lead role at an early-stage, research-led AI company that builds private evaluations, expert training data and RL environments for frontier AI labs. You'll own the technical foundation and lead the engineers who build on it. On-site in Yerevan with visa and relocation support, USD 3,000–5,000 per month plus early-stage equity.
We're recruiting on behalf of an early-stage, research-led AI company, on a confidential basis. Its customers are frontier AI labs, and its work starts where most training data ends: rigorous evaluation. The team designs private, uncontaminated benchmarks to find exactly where today's models fail, traces each failure to its root cause, and then builds the expert data and RL environments that close the gap. It works across coding, STEM, professional reasoning, computer use, agentic tool use and embodied AI.
The company was founded by operators who have run large post-training data programs for leading model builders, and its methods are grounded in peer-reviewed software-engineering research. Software engineering is its deepest domain.
The role
This is the first Tech Lead hire and the technical owner of everything the company ships. You'll turn research-grade ideas into reliable infrastructure: agentic evaluators that replay and perturb failing trajectories, pass@k difficulty calibration, programmatic verifiers, and a harness that runs thousands of tasks in parallel. You'll also set the engineering bar for a small core team and a network of expert contributors.
Read this first
Scope will not arrive pre-defined. One week it's a sandboxed terminal environment for repo-level coding tasks. The next, it's a mock web app with DOM-level state capture for a browser-use agent, or an LLM-as-judge pipeline that has to agree with expert graders. If you need a fixed roadmap and a large platform team underneath you, this is the wrong role. If you want to be the person who decides how it should be built, and then builds the first version yourself, read on.
What our client offers
Founding-level ownership. You'll define the architecture, tooling and engineering standards of a company working on one of the most valuable bottlenecks in AI.
Frontier exposure. Direct work with research teams at leading AI labs on problems they can't yet solve in-house.
Real equity. Early-stage equity on top of salary, so your upside moves with the company.
Relocation to Yerevan. Full visa and relocation support. The role is on-site, side by side with the founding team.
Compensation. USD 3,000–5,000 per month, depending on experience.
Responsibilities
- Own the technical architecture end-to-end: eval harnesses, RL environments, verifier suites, and the data pipelines that connect them.
- Design and build sandboxed tool-based and UI-based environments (containerized shells, mock desktop apps and websites) with reset, state capture and programmatic, assertion-based verification.
- Build and scale the evaluation stack: LLM-as-judge pipelines validated against expert labels, agentic evaluators that re-run and perturb failing trajectories, and pass@k difficulty calibration.
- Run model baselines and experiments: serve frontier and open-weight models, fine-tune judges or reward models where needed, and analyze results with statistical rigor.
- Build the orchestration layer that runs thousands of tasks in parallel across environments, models and operating systems (macOS, Windows, Ubuntu).
- Hire, lead and mentor a small engineering team. Set code-review, testing and documentation standards, and review expert-produced coding tasks and solutions for technical correctness.
- Act as the technical counterpart to client research teams: scope custom evals, translate capability gaps into task and environment specs, and ship deliverables their training infrastructure can consume directly.
- Own delivery quality and reliability: reproducible results across environment versions, QC tooling, and clear technical documentation.
Requirements
- 7+ years of professional software engineering, including 2+ years as a tech lead or owner of a production system end-to-end.
- 3+ years of hands-on production experience with ML and LLM systems. You have built, fine-tuned or evaluated models and shipped them, not only wrapped an API.
- Strong ML foundations: PyTorch and the Hugging Face stack (Transformers, Datasets, TRL, PEFT/LoRA), with real experience running training or fine-tuning jobs on GPUs.
- A solid grasp of the post-training stack (SFT, reward modeling, RLHF, DPO and GRPO-style methods) and of what makes training data genuinely useful to it.
- Deep LLM engineering: OpenAI, Anthropic and Gemini APIs; open-weight models (Llama, Qwen, Mistral, DeepSeek) served with vLLM or SGLang; tool calling, structured outputs, prompt and judge design, and managing rate limits and cost at scale.
- Proven LLM evaluation work: you've built eval pipelines and LLM-as-judge systems and validated them against human labels, using tools such as lm-evaluation-harness, Inspect AI, OpenAI Evals, DeepEval or promptfoo. You understand pass@k, inter-annotator agreement and benchmark contamination.
- Agent experience: frameworks and protocols such as LangGraph, LlamaIndex, DSPy, the OpenAI Agents SDK and MCP, plus working knowledge of agentic benchmarks like SWE-bench, Terminal-Bench, OSWorld, WebArena or GAIA.
- Expert Python (asyncio, typing, pytest) and strong infrastructure skills: Linux, Docker, Kubernetes, AWS or GCP, distributed job orchestration (Ray, Celery or similar), CI/CD with GitHub Actions, and experiment tracking with Weights & Biases or MLflow.
- Strong testing and code-review instincts. You can tell a real verifier from one that only pattern-matches, and you're at home in real-world repositories, builds and dependencies.
- Clear written and spoken English, and comfortable defending technical decisions in front of researchers at client labs.
Nice to Have
- Experience at a frontier AI lab or a post-training / human-data provider.
- You've trained models with RL on verifiable rewards (GRPO or PPO, with frameworks such as TRL, OpenRLHF or verl), or built RL environments with Gymnasium-style interfaces.
- Distributed training and inference optimization (DeepSpeed, FSDP, Megatron, multi-GPU serving).
- Browser and desktop automation for computer-use agents (Playwright, Selenium, accessibility trees, VMs).
- Publications in ML, NLP or software-engineering research.
- You've been an early engineer at a startup before.