Loading profile…
Aditya Narayan Padhi
Open to opportunitiesLead Data Engineer
Lead Data Engineer building cloud-scale, observability-driven data platforms and GenAI-enabled pipelines.
IndiaLead Data Engineer at HTC Global Services
About
I am a Lead Data Engineer with 10+ years designing and delivering cloud-scale streaming and batch data platforms that enable analysts, data scientists, and ML/AI teams to ship experiments to production quickly and reliably.
I have led and mentored teams (recently a team of 5), built metadata-driven reusable platforms, migrated large systems to Azure with zero data loss, and implemented Medallion architectures and observability stacks that raised pipeline success rates to 98% while cutting pipeline build time by 80%.
Hands-on with Spark, Kafka, Airflow, Kubernetes, MLflow, vector stores and GenAI/RAG solutions, I focus on resilient, cost-efficient architectures, robust data quality, and product-aligned delivery that accelerates business outcomes.
Experience
Lead Data Engineer
Architected a metadata-driven, reusable data platform and led a team of 5 engineers to cut new-pipeline build time by 80%, raise pipeline success rate to 98%, and compress release cycles by ~70% while running 700+ production pipelines under SLAs.
PySparkKafkaAirflowKubernetesDelta LakeAzureMLFlowPrometheusGrafana
Senior Data Engineer
Architected and built large-scale real-time and batch data pipelines (700+) using PySpark, Kafka and Airflow to support low-latency power trading analytics and renewable energy optimization.
DatabricksPySparkKafkaAirflowKubernetes
Education
Liverpool John Moores University
Master of Science in Machine Learning & AI completed remotely with coursework and a thesis project focused on predictive trading and end-to-end data pipelines.
Chhattisgarh Swami Vivekanand Technical University
Bachelor of Telecommunication Engineering completed at an institution in Bhilai focusing on telecommunications and engineering fundamentals.