| Job opening for Infrastructure / Platform Engineer in TX at Remote, Remote, USA |
| Email: [email protected] |
|
Position Infrastructure / Platform Engineer Location San Antonio, TX Duration Long-Term Contract Job Description: Skills and Expertise: 1. AWS Managed Services: o Proficiency in AWS services such as Amazon MSK (Managed Streaming for Kafka), Amazon Kinesis, AWS Lambda, Amazon S3, Amazon EC2, Amazon RDS, Amazon VPC, and AWS IAM. o Ability to manage infrastructure as code with AWS CloudFormation or Terraform. 2. Apache Flink: o Understanding of Apache Flink for real-time stream processing and batch data processing. o Familiarity with Flinks integration with Kafka, or other messaging services. o Experience in managing Flink clusters on AWS (using EC2, EKS, or managed services). 3. Kafka Broker (Apache Kafka): o Deep knowledge of Kafka architecture, including brokers, topics, partitions, producers, consumers, and zookeeper. o Proficiency with Kafka management, monitoring, scaling, and optimization. o Hands-on experience with Amazon MSK (Managed Streaming for Kafka) or self-managed Kafka clusters on EC2. 4. DevOps & Automation: o Strong experience in automating deployments and infrastructure provisioning. o Familiarity with CI/CD pipelines using tools like Jenkins, GitLab, GitHub Actions, CircleCI, etc. o Experience with Docker and Kubernetes, especially for containerizing and orchestrating applications in cloud environments. 5. Programming & Scripting: o Strong scripting skills in Python, Bash, or Go for automation tasks. o Ability to write and maintain code for integrating data pipelines with Kafka, Flink, and other data sources. 6. Monitoring & Performance Tuning: o Knowledge of CloudWatch, Prometheus, Grafana, or similar monitoring tools to observe Kafka, Flink, and AWS service health. o Expertise in optimizing real-time data pipelines for scalability, fault tolerance, and performance. Responsibilities: 1. Infrastructure Design & Implementation: o Design and deploy scalable and fault-tolerant real-time data processing pipelines using Apache Flink and Kafka on AWS. o Build highly available, resilient infrastructure for data streaming, including Kafka brokers and Flink clusters. 2. Platform Management: o Manage and optimize the performance and scaling of Kafka clusters (using MSK or self-managed). o Configure, monitor, and troubleshoot Flink jobs on AWS infrastructure. o Oversee the deployment of data processing workloads, ensuring low-latency, high-throughput processing. 3. Automation & CI/CD: o Automate infrastructure provisioning, deployment, and monitoring using Terraform, CloudFormation, or other tools. o Integrate new applications and services into CI/CD pipelines for real-time processing. 4. Collaboration with Data Engineering Teams: o Work closely with Data Engineers, Data Scientists, and DevOps teams to ensure smooth integration of data systems and services. o Ensure the data platforms scalability and performance meet the needs of real-time applications. 5. Security and Compliance: o Implement proper security mechanisms for Kafka and Flink clusters (e.g., encryption, access control, VPC configurations). o Ensure compliance with organizational and regulatory standards, such as GDPR or HIPAA, where necessary. 6. Optimization & Troubleshooting: o Optimize Kafka and Flink deployments for performance, latency, and resource utilization. o Troubleshoot issues related to Kafka message delivery, Flink job failures, or AWS service outages. Tools & Technologies: Apache Kafka (Self-managed or MSK) AWS managed Apache Flink Amazon EC2, S3, RDS, and VPC Terraform/CloudFormation Docker, Kubernetes (EKS) Elk, CloudWatch Python, Bash (609) 623-5499 EXT - 107 Mail: [email protected] LinkedIn ID - linkedin.com/in/manish-kumar-aa179b24a -- Keywords: continuous integration continuous deployment sthree information technology golang Idaho Texas Job opening for Infrastructure / Platform Engineer in TX [email protected] |
| [email protected] View All |
| 08:30 PM 05-Mar-25 |