Virtusa
Senior Data Engineer
Job Description
Develop and Maintain ETL Pipelines: Design, develop, and implement scalable ETL workflows using PySpark, Python, and AWS Glue, Databricks.
Data Transformation and Integration: Extract, transform, and load data from various sources to AWS S3 and Redshift.
Performance Optimization: Identify and resolve performance bottlenecks in ETL processes, ensuring optimal performance across large datasets.
Debugging & Reverse Engineering: Should be able to debug PySpark programs/Job and reverse Engineer the code.
Automation and Monitoring: Implement automation scripts using AWS Lambda, Step Functions to schedule and monitor data pipelines.
Data Quality: Ensure data integrity and quality across all stages of the ETL pipeline.
Collaboration: Work closely with data architects, analysts, and stakeholders to understand requirements and provide clear communication throughout the project lifecycle.
Documentation: Create and maintain technical documentation, including data mapping, workflow designs, and ETL processes.
Knowledge of CI/CD pipelines and best practices in deployment automation.
Jobs at Bangalore
IBM
Product Manager for Cloud Monitoring
Professional
Bangalore, Karnataka
Last Date: Nov. 22, 2024
Infosys
Finacle Technology - Senior Develop…
5 - 8 Years Exp.
Bangalore, Karnataka
Last Date: Dec. 31, 2024
Virtusa
Senior Java Kafka Developer
Full Time
Bangalore, Karnataka
Last Date: Dec. 31, 2024
IBM
Applinx Senior Technical
Professional
Bangalore, Karnataka
Last Date: Nov. 22, 2024
IBM
Business Transformation Consultant:…
Professional
Bangalore, Karnataka
Last Date: Nov. 22, 2024
Amazon
IT Manager II, OPSTech IT -Service …
Freshers/Experienced
Bangalore, Karnataka
Last Date: Nov. 26, 2024
Amazon
Software Development Engineer - Ale…
Freshers/Experienced
Bangalore, Karnataka
Last Date: Nov. 19, 2024
Paypal
Lead Software Engineer (Full Stack)
Freshers/Experienced
Bangalore, Karnataka
Last Date: Nov. 24, 2024
More Jobs at Virtusa
Virtusa
Senior React JS Developer
Full Time
Chennai, Tamil Nadu
Last Date: Dec. 31, 2024
Virtusa
Product Support Engineer
Full Time
Chennai, Tamil Nadu
Last Date: Dec. 31, 2024
Virtusa
Senior Software Engineer
Full Time
Hyderabad, Telangana
Last Date: April 30, 2025
Virtusa
QA Automation - Banking/Payments
Full Time
Chennai, Tamil Nadu
Last Date: Dec. 31, 2024
Virtusa
ERP Lead Consultant Technical
Full Time
Hyderabad, Telangana
Last Date: Dec. 31, 2024
Actively Recruiting Companies at Bangalore, Karnataka