Virtusa

Data Engineer

Job Description

Roles and Responsibilities : Data Engineer Role
Data Pipeline Development:
  Design, build, and maintain scalable data pipelines in AWS Glue using Python code language to ingest, transform, and load data from sources (AWS S3 bucket) into data lakes (AWS S3 bucket).
Data Integration:
  Integrate data from multiple sources and systems (AWS S3 bucket) to enable unified and comprehensive views of clinical data.
  Implement ETL (Extract, Transform, Load) processes to prepare raw data for analysis.
Database Management:
  Manage and optimize databases (e.g., Starbust) for performance, scalability, and reliability.
  Implement database schemas and indexes to support efficient data querying and retrieval.
Data Governance and Security:
  Implement data governance policies and procedures to ensure data security, compliance, and privacy using AZ GxP process.
  Establish access controls, encryption, and auditing mechanisms to protect sensitive data.
Monitoring and Optimization:
  Monitor data pipelines and systems for performance issues, bottlenecks, and anomalies.
  Implement optimizations to improve data processing efficiency and reduce latency.
Collaboration and Documentation:
  Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and support analytics initiatives.
  Document data engineering processes, workflows, and architectures for knowledge sharing and future reference.
  Collaborate with testers while testing the developed pipeline and after which promoting to higher environment.
Deployment and Jira Test Execution:
  Deploying the code from Github repository from the master branch to higher environments like SIT, PPT and PROD.
  While deploying we capture evidences and execute the deployment scripts with different steps and corresponding evidences in Jira.
Automation:
  Automation to trigger the code automatically using AWS resources without manual intervention.
Skillset required for this project : Python, AWS, Starbust, Neo4j


Jobs at Chennai

Virtusa

AWS Cloud Engineer

Full Time

Chennai, Tamil Nadu

View Details

Last Date: Feb. 28, 2025

Virtusa

Snowflake DBT

Full Time

Chennai, Tamil Nadu

View Details

Last Date: Dec. 31, 2024

Virtusa

Camunda Developer

Full Time

Chennai, Tamil Nadu

View Details

Last Date: June 30, 2025

Virtusa

.Net Core FSD

Full Time

Chennai, Tamil Nadu

View Details

Last Date: Dec. 31, 2024

Virtusa

Business Analyst - Payments/Cash Ma…

Full Time

Chennai, Tamil Nadu

View Details

Last Date: Dec. 31, 2024

Virtusa

QA Automation Engineer

Full Time

Chennai, Tamil Nadu

View Details

Last Date: Dec. 31, 2024

Oracle

TAM - Oracle DBA

Professional

Chennai, Tamil Nadu

View Details

Last Date: May 7, 2025

Virtusa

Qliksense Power BI Migration

Full Time

Chennai, Tamil Nadu

View Details

Last Date: March 31, 2025

Virtusa

Senior UI Engineer

Full Time

Chennai, Tamil Nadu

View Details

Last Date: Dec. 31, 2024

Virtusa

UI Angular Lead

Full Time

Chennai, Tamil Nadu

View Details

Last Date: Feb. 28, 2025

Amazon

Senior Application Engineer, Specia…

Freshers/Experienced

Chennai, Tamil Nadu

View Details

Last Date: Dec. 24, 2024

Virtusa

Java Microservices Developer

Full Time

Chennai, Tamil Nadu

View Details

Last Date: Dec. 31, 2024




More Jobs at Virtusa

Virtusa

ACBS Business Analyst Tech

Full Time

Hyderabad, Telangana

View Details

Last Date: Dec. 31, 2024

Virtusa

Senior AIML Engineer

Full Time

Hyderabad, Telangana

View Details

Last Date: Dec. 31, 2024

Virtusa

Java Fullstack

Full Time

Chennai, Tamil Nadu

View Details

Last Date: Dec. 31, 2024

Virtusa

.NET Developer ( React)

Full Time

Hyderabad, Telangana

View Details

Last Date: Dec. 31, 2024

Virtusa

Pega KYC CLM

Full Time

Chennai, Tamil Nadu

View Details

Last Date: Dec. 31, 2024

Virtusa

QA Testing

Full Time

Bangalore, Karnataka

View Details

Last Date: Dec. 31, 2024

Virtusa

Angular/React Java Full Stack Devel…

Full Time

Hyderabad, Telangana

View Details

Last Date: March 31, 2025

Virtusa

Tech Lead

Full Time

Hyderabad, Telangana

View Details

Last Date: Dec. 31, 2024

Virtusa

Architect (ATC)

Full Time

Hyderabad, Telangana

View Details

Last Date: Dec. 31, 2024

Virtusa

CRM Associate Consultant

Full Time

Chennai, Tamil Nadu

View Details

Last Date: Dec. 31, 2024

Virtusa

Abinitio lead developer

Full Time

Hyderabad, Telangana

View Details

Last Date: Dec. 31, 2024

Virtusa

LN Functional SME

Full Time

Hyderabad, Telangana

View Details

Last Date: Dec. 31, 2024