Senior Data Engineer

SRM TechnologiesUnited States only

Apply now

Opens the original posting on Himalayas.

Apply now Source: Himalayas

Apply on the original site

Companies in this catalog do not charge applicants. Never send money to apply.

This is a remote position.

Summary:

Role: Senior Data Engineer
Experience: 8+ Years
Mandatory/Core: Python, PySpark, Snowflake, dbt, Apache Iceberg, AWS, SQL
Preferred: AWS Glue, S3, EMR, Lambda, Airflow, Snowpipe/Snowpark, CI/CD, Terraform, Data Modeling
Role Type: Senior hands-on Data Engineer
Focus: Cloud Data Engineering, Lakehouse, Data Transformation, Performance Optimization and Production Engineering Detailed information:

Senior Data Engineer:

Loading job…

Senior Data Engineer at SRM Technologies · ApplyPick

Data Pipeline Engineering
Design, develop, test, and maintain scalable ETL/ELT data pipelines .
Develop production-quality data-processing solutions using Python and PySpark .
Build reusable frameworks and components for ingestion, transformation, validation, and publishing of data.
Process large structured, semi-structured, and distributed datasets.
Implement incremental and batch-processing patterns where appropriate.
Snowflake Development
Design and develop scalable data solutions using Snowflake .
Develop complex SQL transformations, data models, views, and reusable data structures.
Optimize Snowflake workloads for performance, scalability, and cost.
Implement appropriate data-loading and transformation patterns between AWS data platforms and Snowflake.
Troubleshoot performance and data-quality issues across Snowflake workloads. dbt Development
Build and maintain transformation pipelines using dbt .
Develop modular, reusable, maintainable dbt models.
Implement dbt tests and documentation.
Follow appropriate development practices for source, staging, intermediate, and business-layer transformations.
Support automated deployment and CI/CD practices for dbt projects.
Apache Iceberg / Lakehouse
Design and implement data-lake and lakehouse solutions using Apache Iceberg .
Build scalable table structures for large analytical datasets.
Work with partitioning, schema evolution, incremental processing, and table-maintenance strategies.
Integrate Iceberg-based datasets with Spark and AWS-based data-processing services.
Ensure efficient storage and query patterns for high-volume datasets.
AWS Data Engineering
Design and implement cloud-native data solutions on AWS .
Build data-processing workloads leveraging services such as S3, Glue, EMR and Lambda where appropriate.
Implement secure access patterns using AWS IAM.
Monitor data workloads and troubleshoot operational issues.
Participate in designing scalable, reliable, secure, and cost-efficient cloud data architectures.
Performance & Scalability
Diagnose and optimize Spark/PySpark jobs , SQL queries, Snowflake workloads, and data pipelines.
Identify bottlenecks involving compute, storage, partitioning, data skew, transformations, and queries.
Design solutions capable of supporting increasing data volumes without unnecessary infrastructure cost.
Data Quality & Governance
Implement automated data-quality checks across ingestion and transformation layers.
Establish proper logging, monitoring, exception handling, and reconciliation mechanisms.
Follow organizational standards for data security, governance, lineage, and access controls.
Ensure production pipelines are reliable, auditable, and maintainable.
Engineering Best Practices
Write clean, modular, reusable, testable, and maintainable code.
Perform code reviews and enforce engineering standards.
Implement unit, integration, and data-validation testing.
Use Git-based version control and CI/CD practices.
Create and maintain appropriate technical documentation.
Senior-Level Responsibilities
Independently drive technically complex data-engineering requirements from design through production deployment.
Participate in solution design and architecture discussions.
Evaluate alternative implementation approaches and recommend appropriate solutions.
Troubleshoot complex production and performance issues.
Mentor junior and mid-level data engineers.
Collaborate with Architects, Product Owners, Business Analysts, Data Scientists, QA, DevOps, and application teams.
Translate business/data requirements into scalable technical solutions.
Identify technical risks and proactively recommend improvements.

Senior Data Engineer

Apply now

Summary:

Senior Data Engineer:

Senior Data Engineer

Apply now

Summary:

Senior Data Engineer:

Primary Skills:

Apache Iceberg

Data Lake / Lakehouse architecture

Secondary / Preferred Skills:

AWS services such as:

Apache Airflow or other workflow orchestration tools

Snowpipe / Snowpark

Git / GitHub / GitLab

Job Description:

Key Responsibilities:

Core Skills Expected

Apache Iceberg

Data Engineering

Data Architecture

Engineering Practices

Preferred Qualifications

Apply now