| Urgently Hiring:: Lead Big Data Engineer::Remote at Remote, Remote, USA |
| Email: [email protected] |
|
From: Priyabrata pradhan, Vyze [email protected] Reply to: [email protected] Job Description - Position: Lead Big Data Engineer/Developer Location: Must be anywhere in an EST time zone. (NY OR DC preferred) Duration: Long term Contract Visa: USC/GC/GCEAD (Need LinkedIn) Job Description Lead Big Data Engineer/Developer -10+ years of experience -must have strong java or Scala (Scala preferred) -strong SQL -must have strong unit testing experience at the least and data testing is a big nice to have -Heavy AWS big data experience - EMR, EKS and Apache Spark -Heavy AWS server less - Lambda, Dynamo DB -JSON data formats -unit testing in Java Interview Process: -Glider: Glider Assessment - 1 hour screen -2 hour panel final interview Overview: We are searching for a Lead Big Data Developer to support our client, a large financial regulator, to be a part of a multi-year initiative to build a new system that assists with the collection and processing of up to 100 million records to support financial regulations. The Lead Big Data Developer will be responsible for designing, ingesting, storing, validating, and disseminating data in a consumable format for intelligence teams to gain insights. Responsibilities: Understand complex business requirements Design and develop ETL pipeline for collecting, validating and transforming data according to the specification Develop automated unit tests, functional tests and performance tests. Maintain optimal data pipeline architecture Design ETL jobs for optimal execution in AWS cloud environment Reduce processing time and cost of ETL workloads Lead peer reviews and design/code review meetings Provide support for production support operations team Implement data quality checks. Identify areas where machine learning can be used to identify data anomalies Requirements: BS degree in computer science or related field 7+ years of experience in programming language Java or Scala 7+ years of experience in ETL projects 5+ years of experience in Big Data projects 3+ years of experience with API development (REST API's) Strong experience in Java or Scala Strong experience in big data technologies like AWS EMR, AWS EKS, Apache Spark Strong experience with serverless technologies like AWS Dynamo DB, AWS Lambda Strong experience in processing with JSON and csv files Must be able to write complex SQL queries Experience in performance tuning and optimization Familiar with columnar storage formats (ORC, Parquet) and various compression techniques Experience in writing Unix scripts Unit testing using JUnit or ScalaTest Git/Maven/Gradle Code Reviews Experience with CI/CD pipelines BPM/ AWS Step Functions Python scripting Performance testing tools like Gatling or JMeter Thanks & Regards Priyabrata pradhan Email: [email protected] Keywords: continuous integration continuous deployment database green card New York Urgently Hiring:: Lead Big Data Engineer::Remote [email protected] |
| [email protected] View All |
| 10:02 AM 01-May-24 |