Abhishek Gawali

Data Engineer

Create and optimize scalable ETL workflows using PySpark, SQL, and Databricks to ingest auto loan data into Delta Lake on Amazon S3. Develop real-time data ingestion pipelines using Kafka and Databricks Structured Streaming, and design a Databricks-based Medallion Architecture with robust data transformation and quality checks.

PySparkSQLDatabricksKafkaDelta LakeS3Spark Structured StreamingMedallion ArchitectureVector SearchRAGLLMNL To SQL

Software Developer

Enhanced data pipeline performance by optimizing MapReduce, Hive, and Apache Spark workflows, achieving a 20% increase in processing speed. Developed real-time analytics solutions on Azure, processing over 2 million records daily.

MapReduceHiveApache SparkAzureData Pipeline OptimizationReal Time Analytics

Associate

Equifax

Optimized the Talend ETL framework, slashing processing time for 500GB of commercial data by 40%. Automated 200+ repetitive database tasks with Bash scripts, delivering weekly time savings of 4 hours.

TalendETLApache SparkSQLBASHCassandraData Quality