QA ENGINEER – DATA

Descripción del puesto What you Will do.
Validates data and ETL pipelines to bring new data into a data warehouse.
Collaborate with cross-functional teams (Product/Data science/Data engineering) to develop, execute, and automate data testing processes, ensuring that our data assets meet the highest standards of accuracy, completeness, and consistency.
Identify and research issues reported by internal and external customers.
Manage defect resolution throughout the lifecycle and ensure issues are resolved prior to production.
Develop and execute comprehensive data quality tests to identify anomalies, inconsistencies, and data integrity issues for new product development initiatives, product changes, policy changes, database changes.
Data mining and detailed data analysis on data warehousing systems.
Create formal test plans to ensure the delivery of data related projects involving applications that use ETL components.
Provide input and support big data testing initiative.
Define and track quality assurance metrics such as defects, defect counts, test results, test status, test procedures.
Verify data accuracy, completeness, and consistency across various data sources and pipelines.
Create and maintain test data sets for regression testing.
Provide test support for any issues that require code changes or changes made directly to the ETL pipelines.
Implement and maintain automated testing framework for data validation.
Continuously improve and expand test coverage through automation.
Develop and maintain testing scripts and tools to streamline the testing process Collaborate with cross-teams to define data validation rules and criteria.
Validate data transformations, aggregations, and calculations to ensure accuracy and reliability.
Maintain comprehensive documentation of data quality issues and resolutions.
Evaluate and transform documentation into test scripts as needed.
Work closely with cross-functional teams, including engineers, project managers and other subject matter experts to understand data requirements and validation needs.
Communicate effectively with stakeholders to report on data quality findings and collaborate on improvements and identify gaps in test coverage.
Schedule or attend peer reviews of test logic to ensure it has been constructed correctly.
Communicate with subject matter experts to research source of issues and proposed resolutions, as well as, loading and examining data, business rules, and editorial policy to determine point of failure.
Perform data testing on new and changed customer output files by reviewing requirements, specifications, and technical design documents and participating in design review meetings.
Create and support data validation scripts for new and existing ETL pipeline changes.
Create visualization dashboards to analyze/monitor data for ETL pipeline changes and flag any defects or anomalies in data from regression data testing perspective.
Design, develop, automation tools to test ETL pipelines.
Write Python scripts in PySpark for data processing and manipulation Requisitos About you.
2+ years of proven experience in software engineering.
Bachelor’s degree in computer science, Engineering, or equivalent experience.
Proven experience as a QA Engineer with focus data/ETL pipeline testing, regression data testing.
Proven experience with one of BigData technologies such Pyspark, Pandas, Spark, Hadoop, Hive.
Experience with creation/maintenance of data validation tools and frameworks • Proficient in Python as well as AWS tools.
Knowledge of data modeling concepts and ETL processes.
Familiarity with data integration and data warehousing technologies such as Databricks/Snowflake.
Experience with system integration testing, end-to-end testing, databases, CI/CD pipelines Ability to document and troubleshoot errors.
Strong attention to detail and patience to track down difficult issues.
Possessing an analytical mind, critical-thinking skills, and problemsolving aptitude.
Strong organizational skills and ability to meticulously follow detailed steps.
Experience in creating robust test plans/strategies and test status for Big Data product deliverables.
Excellent verbal and written communication skills.
Willing and able to go above and beyond.
Ability to work collaboratively across different divisions.