Sai Nagendra Pavan Kumar Pilli

Experience

1. Sr Software Engineer

   Designed and maintained scalable ETL/ELT pipelines using Azure Data Factory, Azure Databricks, PySpark, and Python for healthcare data. Built Delta Live Tables frameworks with CDC-based incremental loading and implemented Auto Loader and Delta Lake ingestion patterns.

   Azure Data FactoryAzure DatabricksPySparkPythonDelta Live TablesAuto LoaderDelta LakeUnity CatalogAzure Event HubsSpark Structured StreamingMicrosoft FabricOneLakeLakehouseData FactoryPower BIDirectLakeAzure DevOpsCI CDGitSQLT SQLData GovernancePerformance Optimization

2. Sr Software Engineer I

   Capgemini

   Developed and managed ETL pipelines using Azure Data Factory, Azure Databricks, and Python. Implemented Unity Catalog for data governance and created streaming and batch ingestion pipelines using Auto Loader and Delta Lake.

   Azure Data FactoryAzure DatabricksPythonUnity CatalogAuto LoaderDelta LakeAzure Stream AnalyticsAzure Event HubsAzure Logic AppsAzure SQL DatabaseData GovernanceRBACStreamingETL

3. Software Engineer

   Automated orchestration and monitoring workflows using ADF and Azure Logic Apps. Developed real-time streaming ingestion with Azure Event Hubs and Spark Structured Streaming, and supported live data feeds into Azure Synapse Analytics.

   Azure Data FactoryAzure Logic AppsAzure Event HubsSpark Structured StreamingAzure DatabricksADLS Gen2Delta LakeAzure SynapseData WarehouseStar SchemasDimensional ModelingData QualityMonitoring

4. Software Engineer

   Monitored and optimized data pipeline performance using Azure Data Factory and Databricks metrics. Collaborated with data scientists and analysts to deliver insights, and implemented data governance policies in streaming environments.

   Azure Data FactoryAzure DatabricksData Lake AnalyticsPower BIazure monitorLog AnalyticsDelta LakeSparkT SQLAzure DevOpsCI CDData GovernanceStreaming