| Data Engineer (Spark, Airflow, GCP) at Bentonville, Arkansas, USA |
| Email: [email protected] |
|
From: Prudhvi Raju, Biztegy Analytics. Inc [email protected] Reply to: [email protected] Data Engineer (Spark, Airflow, GCP) Location: Bentonville, AR Job Description Designing and Building ETL pipeline using Sqoop, Hive, Map Reduce and Spark on on-prem and cloud environments. Functional Programming using Python and Scala for complex data transformations and in-memory computations. Using Erwin for Logical/Physical data modeling and Dimensional Data Modeling. Designing and developing UNIX/Linux scripts for handing complex File formats and structures Orchestration of workflows and jobs using Airflow and Automic, Creating Multiple Kafka producers and consumers for data transferring Performing Continous Integration and deployment (CI/CD) using tools like GIT, Jenkin to run test cases and build applications with code coverage using Scala test Analyzing data using SQL, Big Query monitoring the cluster performance, setting up alerts, documenting the designs, workflow. Providing production support, troubleshooting and fixing the issues by tracking the status of Running applications to perform System Administrator tasks. Keywords: continuous integration continuous deployment business analyst Arkansas Data Engineer (Spark, Airflow, GCP) [email protected] |
| [email protected] View All |
| 10:59 PM 29-Jan-25 |