Are you passionate about AI? 🤖
At Satori Analytics, we aim to change the world one algorithm at a time by bringing clarity to global brands through Data & AI. From cloud-based ecosystems for fintech to predictive models for airlines, our cutting-edge solutions cover the entire data lifecycle—from ingestion to AI applications.
As a fast-growing scale-up, our team of 100+ tech specialists—including Data Engineers, Data Scientists, and more—delivers innovative analytics solutions across industries like FMCG, retail, manufacturing and FSI. Join us as we lead the data revolution in South-Eastern Europe and beyond!
Together with a partnering company, we're looking for a strong Software Engineer to build the systems behind an AI agent evaluation platform — systems that test, grade, and stress AI agents at scale, and turn the results into actionable signal about how those agents behave.This is a software engineering role first — designing services, data models, and pipelines that have to be correct, tested, and maintainable — applied to an AI problem domain. You'll work with LLMs regularly, but the core of the job is engineering, not prompt-tuning.
What Your Day Might Look Like:
- Backend services & APIs: Build and maintain the services, data models, and APIs that power the platform — designed for correctness, testability, and scale.
- Simulation & orchestration: Work on the systems that coordinate complex, multi-step interactions between AI agents and external systems, improving their reliability and throughput.
- Evaluation & scoring: Design systems that grade agent outputs, combining deterministic checks with model-assisted judgment — and make scoring reliable, explainable, and reproducible.
- Data pipelines: Build pipelines that generate, transform, and quality-check large volumes of structured data and benchmark content.
- Quality & reliability: Add the tests, instrumentation, and safeguards needed to trust outputs from systems that are inherently non-deterministic.
Requirements
Your Superpowers 🚀
- 4+ years building and shipping production software, with strong proficiency in Python.
- Deep software engineering fundamentals: system and API design, data modeling, concurrency/async, testing strategy, debugging, and code review. You can own a non-trivial service end-to-end.
- Experience designing and operating distributed or service-oriented systems (queues, workers, APIs) — not just calling them.
- Comfort designing schemas and working with relational databases, plus the migrations and performance concerns that come with them.
- Working knowledge of LLM APIs — orchestration, structured outputs, and handling non-determinism. We expect you to use LLMs effectively, but this is not a prompt-engineering role.
- Ability to reason about correctness of probabilistic systems: how to test, measure, and trust outputs that aren't byte-for-byte deterministic.
- High quality bar: you write tests, types, and docs by default, and you keep changes small and reviewable.
Bonus points for:
- Experience building agentic or multi-agent systems, tool-use, or orchestration frameworks.
- Background in evaluation / benchmarking of ML or LLM systems (rubrics, golden datasets, model-as-judge, inter-rater reliability).
- Experience with distributed task queues and async workloads.
- Modern Python tooling and typed codebases (e.g. type checkers, linters, Pydantic, FastAPI).
- Retrieval / search experience and working with data ingest pipelines.
- Some comfort with the infra side (Docker, CI/CD) so you can ship what you build.
Benefits
Perks on Perks
- Competitive salary.
- Training budget to level up your skills from top tech partners like Microsoft, AWS, Salesforce, and Databricks – whether it’s certifications or courses, we’ve got you covered.
- Private insurance, top-tier tech gear, and the chance to work with a stellar crew.
Ready to create some data magic with us? Hit that apply button and let’s get started. ✨
Originally posted on