| Business Analyst Test Lead Databricks Test Engineer at Remote at Remote, Remote, USA |
| Email: [email protected] |
|
From: vinay, vbeyond [email protected] Reply to: [email protected] Role: Business Analyst Test Lead / Databricks Test Engineer Location Required: Remote. Note: We are looking for Strong Insurance domain Business Analyst Life & Annuity Insurance experience, with good ETL Testing experience Job Description: We are seeking an experienced Databricks Test Engineer with AWS (Amazon Web Services) cloud experience to join our team. The successful candidate will be responsible for validating and testing data processing pipelines, ETL (Extract, Transform and Load) processes, and data transformations in the Databricks environment on AWS. This role involves collaborating with data engineers and stakeholders to ensure data accuracy, reliability, and quality across various data systems. Key Responsibilities: Test Planning and Execution: Develop and execute comprehensive test plans and test cases for data pipelines, ETL processes, and data transformations on Databricks. Automation of Data Quality Tests: Implement and maintain automated data quality tests using frameworks such as Great Expectations or custom scripts in Python/Py Spark. Data Validation and Verification: Ensure the integrity and accuracy of data by performing data profiling, anomaly detection, and reconciliation between source and target systems. Integration Testing: Collaborate with data engineers to test data integration workflows involving multiple data sources, including AWS services such as Redshift, S3, and RDS. Performance Testing: Conduct performance testing of Databricks jobs and clusters to ensure optimal data processing and scalability. Issue Resolution: Identify, troubleshoot, and resolve data quality issues and discrepancies in collaboration with the data engineering team. Documentation: Maintain detailed documentation of test plans, test cases, test results, and data validation processes. Continuous Improvement: Contribute to the continuous improvement of testing practices, tools, and frameworks to enhance data quality assurance processes. Qualifications: Education: Bachelor's or master's degree in computer science, Information Technology, or a related field. Experience: Overall 8+ years of Testing domain experience Good knowledge and experience in Life & Annuity insurance. Minimum of 3+ years of experience in data testing or quality assurance within a data engineering environment. At least 2 years of hands-on experience with Databricks and Apache Spark. Strong experience working with AWS cloud services, s3. Technical Skills: Proficient in SQL and Python/PySpark for data manipulation and testing. Strong understanding of ETL processes, data warehousing, and big data technologies. Experience with data quality frameworks such as Great Expectations or similar tools. Familiarity with AWS data services and experience testing in AWS environments. Knowledge of Databricks Lakehouse architecture and Delta Lake. Understanding of DevOps practices and tools relevant to data engineering and testing. Experience with CI/CD pipelines and automated testing frameworks. Problem-Solving: Excellent analytical and problem-solving skills, with the ability to troubleshoot complex data issues. Communication: Strong verbal and written communication skills, with the ability to convey complex technical concepts to non-technical stakeholders. Preferred Qualifications: Certification in Databricks, Apache Spark, or AWS. Experience with Delta Lake and other Databricks ecosystem components. Knowledge of data orchestration tools like Airflow and analytics tools. Best Regards, Vinay Pandey VBeyond Corporation I PARTNERING FOR GROWTH New Jersey, USA |Website : www.vbeyond.com [email protected] Keywords: continuous integration continuous deployment sthree Business Analyst Test Lead Databricks Test Engineer at Remote [email protected] |
| [email protected] View All |
| 04:50 AM 12-Mar-25 |