Platform Engineer (43439)

I am looking for a Platform Engineer with strong Databricks and streaming expertise. You will optimize platform performance, design real-time data pipelines, and ensure scalability and reliability. I expect deep knowledge of Spark, cloud environments, and streaming technologies.

🚀 Project
- act as SME for Databricks infrastructure including cluster configuration, performance tuning, autoscaling, and cost governance
- design, build, and optimize streaming data pipelines for real-time data acquisition
- diagnose and resolve performance bottlenecks across compute, storage, networking, and streaming workloads
- define best practices for workspace management, access control, and streaming architecture
- collaborate with solution architecture on infrastructure and streaming design decisions
- work with testing teams to validate data integrity and platform performance
- support migration activities to Databricks from a platform perspective
- document infrastructure setup, streaming pipelines, and operational procedures

🎯 Skills
- 6+ years of experience with Databricks platform engineering in production environments
- strong expertise in streaming/real-time data pipelines (Structured Streaming, Kafka, Event Hub, or similar)
- deep knowledge of Spark internals, cluster sizing, autoscaling, and optimization
- strong understanding of cloud platforms (Azure, AWS, or GCP)
- experience with performance monitoring and cost optimization
- ability to combine platform engineering and streaming pipeline design

💡 Nice to have
- experience with Infrastructure as Code (Terraform or similar)
- exposure to OT or industrial data systems