Virtusa
Pyspark Dev & QA
Job Description
Pyspark Dev & QA
Mandatory Skills
BIG Data technology mentioned below Hadoop / Big Data (HDFS, PYTHON, SPARK-SQL, MapReduce) with PySpark
Build CI/CD pipelines is required Outstanding coding, debugging and analytical skills
Spark APIs to cleanse, explore, aggregate, transform, store & analyse available data
Knowledge of installing, configuring, debugging and troubleshooting Hadoop clusters
Secondary Skills
Knowledge of cloud-based infrastructure , AWS services EC2, EMR, S3, Lambda, EBS, IAM, Redshift, RDS
Knowledge of deploying and managing ETL pipelines, RDBMS technologies (PostgreSQL, MySQL, Oracle, etc.).
Knowledge of data frames, Pandas , data visualization tools & data mining
knowledge of JIRA, Bitbucket GitHub
JD
The position is seeking someone having atleast 4 years of experience with expertise in pyspark, databricks, SQL. An individual must have good experience in extracting and manipulating data from relational databases with advanced SQL, HIVE, Python/PySpark.
Responsibility
To develop and implement a comprehensive quality assurance strategy for data engineering projects, ensuring that data quality standards are met throughout the data pipeline.
To collaborate with cross-functional teams to define test plans and strategies for validating data pipelines, ETL processes, and transformations using PySpark, Databricks, and SQL.
To design and execute tests using SQL queries and data profiling techniques to validate the accuracy, completeness, and integrity of data stored in various data repositories, including data lakes and databases.
To adhere to formal QA processes, ensuring that the Systems Implementation (SI) team is using industry-accepted best Practices.
Jobs at Chennai
Virtusa
Business Analyst with Credit Risk
Full Time
Chennai, Tamil Nadu
Last Date: Dec. 31, 2024
Capgemini
Procurement -Process Lead I 5 to 7 …
Experienced
Chennai, Tamil Nadu
Last Date: Oct. 2, 2024
Amazon
Support Engineer III
Freshers/Experienced
Chennai, Tamil Nadu
Last Date: Oct. 8, 2024
Virtusa
Java Microservices Developer
Full Time
Chennai, Tamil Nadu
Last Date: Sept. 30, 2024
Virtusa
Data Engineer - Snowflake DBT
Full Time
Chennai, Tamil Nadu
Last Date: Dec. 31, 2024
Amazon
Support Engineer III, Global Store
Freshers/Experienced
Chennai, Tamil Nadu
Last Date: Oct. 8, 2024
More Jobs at Virtusa
Actively Recruiting Companies at Chennai, Tamil Nadu