Note: The job is a remote job and is open to candidates in USA. Gene by Gene is dedicated to advancing the field of genomics through innovation and research. They are seeking a Senior Data Pipeline Engineer/Developer to design, build, and own data pipelines that facilitate genomic and operational data flow, ensuring data quality and compliance in a regulated environment.
Responsibilities
- Builds and operates production ETL and ELT for high-volume genomic data as well as operational and business data
- Integrates data across LIMS, lab instruments, internal applications, and third-party sources through robust, well-tested interfaces
- Develops across the stack in Python (data services, internal APIs, and supporting application code) and provides architectural and code-level guidance to engineering teams on data-layer integration
- Sets technical direction for batch and streaming data pipelines by evaluating frameworks, orchestration, storage, and processing patterns, making recommendations, and leading adoption
- Models and tunes data stores (Microsoft SQL Server and PostgreSQL, plus a cloud warehouse or lake) for performance and scale
- Defines and enforces engineering standards for testing, CI/CD, infrastructure as code, code review, and architecture decision records
- Owns data quality, lineage, and observability, including freshness, completeness, schema-drift detection, cost-per-job, and SLAs
- Builds pipelines for reproducibility and audit-readiness, incorporating versioned data and code, lineage, decision logging, access controls, and evidence collection
- Partners with security and compliance on data privacy, PII and PHI handling, and regulatory requirements across the data lifecycle
- Apply deep understanding of CAP/CLIA, HIPAA, GDPR, and GxP regulations specifically to data pipeline architecture
- Oversee protected health information (PHI) handling, data lineage, retention, and comprehensive audit logging
- Produce and maintain the critical operational evidence required to carry the data platform successfully through compliance audits
- Adhere to strict data-governance controls necessary for securely handling sensitive genomic data across global regions
- Enforce least-privilege access and ensure zero data egress to personal or unapproved infrastructure
- Utilize exclusively de-identified or synthetic data within development environments
- Maintain strict compliance with international data-residency and localization requirements
Skills
- Strong SQL proficiency on Microsoft SQL Server and PostgreSQL, including schema design, query tuning, and performance troubleshooting
- Strong Python proficiency across the stack (data pipelines, backend services, and APIs). This is a Python-first role
- Production experience with a workflow orchestrator (Argo Workflows, Step Functions, Prefect, Dagster, Airflow, or similar)
- Production experience with containers (Docker) and Kubernetes, which the lead orchestrator (Argo Workflows) runs on
- Strong testing discipline, including unit, integration, and data-quality or contract tests
- Excellent written and verbal communication, including the ability to explain data tradeoffs to technical and compliance stakeholders
- Genomics or NGS data formats and handling (FASTQ, BAM/CRAM, VCF)
- Bioinformatics workflow engines (Nextflow, WDL/Cromwell, or Snakemake)
- AWS data stack proficiency (S3, Glue, EMR, Batch, Lambda, Redshift) and/or AWS HealthOmics
- Distributed processing and streaming frameworks (Spark, Kafka, Kinesis)
- Data warehouse and transformation tooling (Snowflake, dbt)
- Infrastructure as code practices (Terraform, CDK, CloudFormation, or Pulumi)
- Familiarity with .NET (C#) and/or C++ as supporting languages for integrating with existing services
- Knowledge of LIMS integration and on-premises plus hybrid data architectures
- 8+ years of professional software or data engineering experience
- 4+ years building and operating production data pipelines at scale
- 3+ years of production cloud experience (AWS preferred)
- Demonstrable experience working in regulated environments. Compliance is a hard requirement for this role
- Experience supporting HIPAA or GxP audits
- Bachelor's degree in Computer Science, Bioinformatics, or a related field, or equivalent professional experience
- An advanced degree in a quantitative field is preferred
Company Overview
Company H1B Sponsorship