Note: The job is a remote job and is open to candidates in USA. CirrusLabs is undergoing a large-scale enterprise modernization initiative focused on Azure-native infrastructure and platform engineering. They are seeking a Senior Databricks Platform Engineer to support architecture, operational engineering, and modernization activities across the evolving Databricks-centered enterprise data platform.
Responsibilities
- Support architecture and operational engineering across enterprise Databricks environment
- Design and maintain scalable Lakehouse, Delta Lake, and Unity Catalog solution
- Support administration of Databricks Workspace
- Job
- Notebook
- Cluster
- Optimize Spark workloads, cluster configurations, and platform performance
- Support batch and streaming data processing workload
- Assist with multi-environment platform management across Dev, QA, and Production
- Support integration of Databricks platforms with Azure-native ecosystems and enterprise applications
- Work across integrations involving API
- Azure Storage Account
- Data Lake
- Reporting and analytics platform
- Collaborate with platform and DevOps teams to align Databricks activities with existing CI/CD and automation standards
- Support deployment workflows utilizing GitHub
- Azure DevOps
- Infrastructure as Code framework
- Help ensure modernization activities do not disrupt existing automated deployment pipelines and operational processes
- Support governance, access management, monitoring, logging, and operational security practices
- Assist with implementation of RBAC
- Lineage
- Encryption
- Platform governance controls
- Participate in troubleshooting, operational support, and root cause analysis activities
- Contribute to documentation, operational standards, and platform best practices
Skills
- 6+ years of experience within cloud, platform engineering, data engineering, or enterprise analytics ecosystem
- 3+ years of hands-on experience supporting enterprise Databricks environment
- Strong experience with Delta Lake
- Unity Catalog
- Spark workloads
- Cluster operation
- Databricks Workspace
- Experience with Azure cloud services and cloud-native platform concept
- Familiarity with Azure DevOps
- GitHub workflow
- CI/CD pipeline concept
- Deployment automation
- Experience supporting APIs, distributed application integrations, and enterprise data platform
- Strong troubleshooting and operational support capabilities
- Strong communication and stakeholder collaboration skills
- Experience with PySpark
- Python
- SQL
- Spark SQL
- Scala
- Familiarity with AKS/Kubernetes
- Terraform
- Bicep
- Infrastructure as Code practice
- Experience with Azure Data Lake
- Event Hub
- Streaming architecture
- Enterprise analytics environment
- Familiarity with monitoring and observability platforms such as Azure Monitor
- Datadog
- Exposure to enterprise modernization and cloud migration initiative
Company Overview
Company H1B Sponsorship