| Job opportunity _ Data Engineer _ REMOTE _ Long Term Contract at Remote, Remote, USA |
| Email: [email protected] |
|
Hi, Greetings from SourceMantra, Please have a look at the below opportunity and reply to me if you are interested. Job: Data Engineer Location: REMOTE Duration : Long Term Contract 10+ Years and passport Number is must for Submission. Job Description : Primary Skills Spark(PySpark), FiveTran, Airbyte, HVR, Apache flink, SQL, Iceberg on Azure Datalake (ADLS) Database and OS: Hive, Snowflake, DB2, Data Warehousing, Linux Secondary Skills Spark streaming, real time data processing from Azure Service Bus Job Responsibilities/KRAs: Design, develop and maintain ETL processes using Spark (PySpark) to integrate data as Iceberg on ADLS from multiple source systems Petabyte size database migration to ADLS on Iceberg tables Real-time data processing from Azure Service Bus using Spark streaming/airbyte/flink Optimize ETL workflows to ensure efficient data processing and loading. Develop scripts to automate data processing and loading tasks. Implement data quality checks and validation processes within ETL workflows. Understanding business requirements/scope of projects, create ETL code as per business logic/process; be able to provide estimation for the tasks as required with supporting data points. Ensure data governance policies are adhered to, including data lineage and metadata management. Provide support for data-related issues and troubleshoot ETL-related problems. Create and maintain technical documentation and reports for stakeholders. Requirements Spark (PySpark), FiveTran, Airbyte, HVR, Apache flink, SQL: 5-6 years of total technical experience on designing, developing, and implementing ETL solutions using Apache Spark. Experience in complete Software Development Life Cycle which includes Systems Analysis of various applications in Client/Server Environment. Working with various software applications with advanced knowledge (FiveTran, Airbyte, HVR, Apache flink, SQL) Experience in creating Streaming data processing using Spark streaming, scripting, deployment activities, performance tuning and error handling skills. Excellent hands-on experience in various data sources, transformations, Partition/De-partition, Databases, Datasets and JSON/XML/Parquet file formats Experience in data replication from RDBMS source using HVR Experience in working on airbyte/Apache flink is preferred Analytical problem solving and business interaction skills. Effective communication with entire offshore team, customer, and all concerned teams on day-to-day basis; provide daily status reporting to all stakeholders. Experience in the insurance (e.g., UW, Claim, Policy Issuance) or financial industry preferred. Thanks & Regards Sharath Reesu | Source Mantra, Inc. Senior Talent Acquisition 295 Durham Ave, Suite # 201, South Plainfield, NJ 07080 Phone : +1 908 373 8651 Ext: 243 | 732 377 8653 (Fax) [email protected] | www.sourcemantra.com -- Keywords: information technology Delaware New Jersey Job opportunity _ Data Engineer _ REMOTE _ Long Term Contract [email protected] |
| [email protected] View All |
| 07:41 PM 09-Aug-24 |