Home

Shiva ranajani - Data integration architect

[email protected]

Location: Richmond, Virginia, USA

Relocation: Yes

Visa: H1B

Resume file: Sivaranajani _ Lead DE_1781290013018.docx Please check the file(s) for viruses. Files are checked manually and then made available for download.

Sivaranjani M Sr. Data Engineer / Data Integration Architect PROFESSIONAL SUMMARY Over 10+ years of professional experience in Big Data, Hadoop, Python, PySpark, ETL, Cloud (AWS & Azure), Data Integration, and SQL & NoSQL development, delivering end-to-end solutions for Financial, Insurance, Retail, and other IT domains. Extensive hands-on experience in Hadoop ecosystem development, including Hive, HBase, Spark, Sqoop, Flume, MapReduce, and HDFS, implementing large-scale data processing pipelines. Expert in Apache Spark development, leveraging PySpark, Spark SQL, Spark Streaming, Spark DataFrames, and Scala for high-performance, distributed, and real-time data processing. Proficient in designing scalable and fault-tolerant data pipelines on AWS (S3, Redshift, RDS, Lambda, Glue, Kinesis, Step Functions, and EC2) and Azure (ADF, Databricks, Synapse Analytics, ADLS Gen2) for cloud-native ETL solutions. Extensive ETL development experience using Apache NiFi, StreamSets, SSIS, Airflow, Oozie, and scheduling tools like AutoSys, Control-M, Jenkins, delivering high-quality, automated pipelines. Strong programming and scripting skills in Python (Pandas, NumPy, SQLAlchemy), Java, SQL (T-SQL, PL/SQL), DataWeave, and UNIX Shell scripting, enabling efficient data extraction, transformation, and analytics workflows. Hands-on experience with RDBMS and NoSQL databases, including Oracle, SQL Server, PostgreSQL, MySQL, Teradata, Snowflake, MongoDB, AWS RDS, and Redshift, ensuring robust and optimized data storage and retrieval. Expertise in data warehousing, data lakes, and data lakehouse architectures, implementing ETL pipelines, data validation, reconciliation, and governance best practices. Proven experience in API-led connectivity and enterprise integration using MuleSoft Anypoint Platform, REST/SOAP services, RAML, OAuth 2.0, and CloudHub for seamless data exchange across systems. Skilled in version control and collaboration tools, including Git, GitHub, Azure DevOps, Jenkins, JIRA, and Confluence, facilitating seamless teamwork and project management. Strong experience in Agile/Scrum methodology, DevOps practices, and CI/CD pipelines, delivering projects efficiently in iterative and collaborative environments. Implemented real-time streaming solutions using Kafka, Spark Streaming, AWS Kinesis, and NiFi, enabling near real-time analytics and reporting for business-critical operations. Extensive experience in cloud-based data transformations on Databricks and Snowflake, utilizing Delta Lake, Parquet, ORC, and materialized views for versioned storage and optimized query performance. Skilled in migrating legacy ETL workflows to PySpark, Snowflake, and cloud platforms, maintaining transactional integrity and operational efficiency. Proven expertise in production support, monitoring, and incident resolution for large-scale ETL and big data pipelines, implementing alerts, automated recovery, and disaster recovery procedures. Strong business acumen and technical leadership, guiding teams, mentoring developers, reviewing code, and collaborating with stakeholders to align technical solutions with business objectives. CERTIFICATIONS Azure Data Fundamentals (DP-900) AWS Developer Associate TECHNICAL SKILLS Big Data & Analytics Hadoop, HDFS, Hive, HBase, Spark, PySpark, Spark SQL, Spark Streaming, MapReduce, Sqoop, Flume, Databricks, Delta Lake Cloud Platforms AWS (S3, Redshift, RDS, Lambda, Glue, Kinesis, EC2, Step Functions), Azure (ADF, Databricks, Synapse Analytics, ADLS Gen2) ETL & Data Integration Apache NiFi, StreamSets, SSIS, Airflow, Oozie, AutoSys, Control-M, Jenkins, MuleSoft Anypoint Platform Programming & Scripting Python (Pandas, NumPy, SQL Alchemy), Scala, Java, SQL (T-SQL, PL/SQL), Unix Shell Scripting, DataWeave Databases Oracle, SQL Server, PostgreSQL, MySQL, Teradata, MongoDB, Snowflake, AWS RDS, Redshift, DB2 Data Modeling & Warehousing Data Lake, Data Warehouse, Data Lakehouse, ETL Pipelines, Data Validation & Reconciliation, Data Governance API & Integration REST, SOAP, RAML, JSON, XML, API Manager, API-led Connectivity, OAuth 2.0, CloudHub Version Control & Collaboration Git, GitHub, Stash, Jenkins, JIRA, Rally, Azure DevOps BI & Reporting Tools Tableau, Matplotlib, Python-based Dashboards, Splunk, New Relic Methodologies Agile, Scrum, DevOps, CI/CD Pipelines, MUnit Testing, JUnit PROFESSIONAL EXPERIENCE Client: Genworth, Remote April 2025 Present Role: Sr. Lead Data Engineer / Data Integration Architect Responsibilities: Designed and implemented distributed data pipelines leveraging Apache Spark for efficient large-scale data processing, enabling high-performance handling of structured and unstructured data across multiple sources. Designed and implemented scalable ETL pipelines using AWS Glue and AWS Lambda, integrating S3, Redshift, and RDS to ensure efficient data ingestion from multiple sources. Designed and implemented metadata-driven ETL frameworks using configuration tables and control metadata to enable dynamic pipeline execution and reduce code duplication. Developed reusable ingestion frameworks leveraging metadata configurations for source-to-target mappings, validation rules, and transformation orchestration. Built parameterized Spark pipelines driven by metadata repositories, improving scalability and maintainability of enterprise data platforms. Designed and delivered Data Products to support business domains, enabling self-service analytics and data-driven decision making. Served as a technical lead for cross-functional initiatives involving Data Engineering, Security Operations, Compliance, and Infrastructure teams. Mentored engineers on security-focused development practices, operational excellence, and cloud platform governance. Led technical discussions and architecture reviews, balancing security requirements with scalability and performance objectives. Presented security platform processes, reporting metrics, and remediation status to business and technical stakeholders Collaborated with business stakeholders to define Data Product requirements, ownership, and lifecycle management. Implemented Data Governance frameworks, including data lineage, metadata management, data cataloguing, and data quality controls. Worked closely with Vulnerability Management teams to analyze, prioritize, and remediate security vulnerabilities across cloud and on-premises environments. Participated in vulnerability triage activities, validating findings and coordinating remediation efforts with application and infrastructure teams. Tracked vulnerability remediation SLAs and provided regular status reporting to stakeholders and security leadership. Supported patch management processes by identifying affected systems, validating remediation, and ensuring compliance with organizational security standards. Conducted root cause analysis for recurring security findings and collaborated with engineering teams to implement preventive controls. Assisted in evaluating and prioritizing vulnerabilities using CVSS scores, exploitability data, and business risk considerations. Contributed to incident response activities, including critical vulnerability and zero-day remediation efforts Utilized AWS Glue Data Catalog for metadata management and data discovery across enterprise datasets. Established data governance standards, ensuring compliance, consistency, and traceability across data platforms. Developed automated data validation and reconciliation scripts using Python and Pandas, and orchestrated pipelines via AWS Step Functions and Lambda triggers. Developed and optimized PySpark jobs within end-to-end data processing pipelines, ensuring data quality, scalability, and reliability across ingestion, transformation, and loading stages. Managed and monitored batch and streaming data pipelines using AWS Kinesis, Kafka, and CloudWatch, implementing alerts for data failures and performance issues. Built and maintained ETL workflows integrating multiple data sources with robust transformation logic to enhance data consistency and integrity across multiple data layers. Designed and implemented scalable Hadoop-based data pipelines using HDFS, MapReduce, and Hive, integrating Sqoop and Flume to ingest structured and unstructured data from multiple sources. Implemented complex data transformations using DataWeave, ensuring accurate and efficient data mapping and enrichment across systems and APIs. Developed and deployed integration modules using API-led connectivity with REST and SOAP services, enabling seamless loan and compliance data processing. Automated CI/CD pipelines using Azure DevOps YAML, incorporating unit and integration testing, automated deployments, and MUnit test suites. Designed serverless ETL workflows with AWS Lambda, S3, and DynamoDB, integrating Python scripts and SQL transformations for real-time data processing. Practiced Spec-Driven Development (SDD) for AI and data engineering initiatives by converting business requirements into detailed technical specifications, data contracts, workflow definitions, API schemas, and acceptance criteria before implementation. Leveraged AI-assisted engineering tools and LLM-based workflows to generate specifications, validate requirements, create implementation plans, automate documentation, and improve development consistency across data and AI platforms. Defined specification-first architectures for data pipelines, AI agents, and enterprise analytics solutions, ensuring traceability from requirements through design, development, testing, and deployment. Collaborated with product managers, analytics teams, and AI stakeholders to establish clear functional specifications, quality standards, testing criteria, and governance controls for scalable enterprise solutions. Implemented automated validation frameworks that compared delivered functionality against approved specifications, reducing defects and improving release quality for data and AI applications. Developed and optimized large-scale data pipelines using PySpark, integrating with Hive and HDFS to process structured and unstructured data efficiently. Optimized SSIS data flows using conditional splits, lookups, derived columns, and scheduled executions via SQL Server Agent, with strong error handling and email notifications. Developed and optimized complex SQL stored procedures, functions, views, and ad-hoc queries, supporting backend data processing and integrations. Collaborated with business stakeholders, compliance teams, QA, and platform support to ensure requirements were met and integrations delivered reliably. Environment: Apache Spark, PySpark, Hive, Python, Hadoop, AWS Glue, AWS Lambda, S3, Redshift, RDS, Kafka, AWS Kinesis, CloudWatch, Data Weave, Azure DevOps (YAML), Jenkins, MUnit, Git, MongoDB, SQL Server, SSIS, HDFS, MapReduce, Sqoop Client: Aflac, Columbus, GA Nov 2024 March 2025 Role: Big Data Developer Responsibilities: Developed PySpark jobs for data transformation and loading into final tables, ensuring adherence to updated schema and maintaining consistency across datasets. Developed data integration solutions using Azure Data Factory (ADF) and SSIS, connecting cloud and on-premise systems to streamline reporting and analytics. Designed and managed AWS EMR clusters for large-scale Spark and Hadoop workloads. Developed and optimized PySpark applications on AWS EMR for distributed data processing and ETL pipelines. Implemented cost optimization and performance tuning strategies for EMR-based data processing environments. Built and maintained data lakes on Hadoop ecosystem platforms using Hive, HBase, and Parquet, enabling efficient analytics and query performance. Designed and implemented scalable data pipelines for ingesting security telemetry, audit logs, and operational monitoring data. Built cloud-based data solutions supporting security analytics, risk reporting, and compliance initiatives. Processed high-volume log and event data using Spark, Kafka, and cloud-native services to enable near real-time security monitoring. Implemented data governance, lineage, and access control mechanisms to support security and regulatory requirements. Developed data quality and observability frameworks for security-related datasets and reporting platforms. Designed and implemented ETL pipelines in Azure Data Factory to ingest, transform, and load structured and unstructured data from multiple sources. Implemented data quality and validation frameworks using Azure Databricks, Python, and Pandas, ensuring accurate and reliable datasets for downstream applications. Built incremental ETL pipelines in NiFi integrating data from multiple SQL Server and MongoDB sources. Implemented data cleaning, transformation, and validation routines with Python, NumPy, and PySpark, improving data quality and consistency for downstream applications. Worked extensively on the Cloudera platform, utilizing Spark, Hive, and other Hadoop ecosystem tools to design, implement, and optimize large-scale data pipelines. Worked with vulnerability scanning solutions to review findings, validate remediation status, and support compliance reporting. Integrated vulnerability assessment data into enterprise reporting and analytics platforms for improved visibility and risk tracking. Automated collection and processing of security scan results using Python, SQL, and cloud-native services. Developed dashboards and reports to monitor vulnerability trends, remediation progress, and SLA compliance. Developed scalable data processing scripts with Python and Airflow, enabling automated scheduling, orchestration, and monitoring of batch and streaming workflows. Developed and maintained data warehouses on Azure Synapse Analytics, optimizing SQL queries and partitioning strategies to ensure high-performance reporting. Automated data ingestion and transformation processes using Apache NiFi and Kafka, streaming real-time data into HDFS and Hive tables. Built and managed data lakes in Azure Data Lake Storage (ADLS Gen2), leveraging PySpark and Databricks for large-scale data transformation and analytics. Built and maintained real-time streaming pipelines using PySpark Streaming and Kafka, enabling near real-time analytics for business reporting. Migrated legacy DTS packages to SSIS and built SSIS ETL packages to extract, transform, and integrate data, triggering workflows based on API events. Implemented data quality and validation pipelines in Spark and Hive, using Airflow for workflow orchestration and Oozie for Hadoop job scheduling. Provided production support for ETL and data replication processes, troubleshooting issues, resolving failures, and maintaining seamless pipeline operations. Environment: Spark, Hive, Python, PySpark, Cloudera, Azure Data Factory, SSIS, Azure Databricks, Airflow, Kafka, HBase, Parquet, ADLS Gen2, Azure Synapse Analytics, Oozie, SQL Server, MongoDB, NiFi, NumPy, Pandas Client: MAXPI Technologies, Chennai, India December 2017 Dec 2022 Role: Data Engineer / Integration Developer Responsibilities: Developed TPT scripts to extract and transfer data from Teradata to Amazon S3, ensuring efficient and secure data migration. Led cross-functional teams consisting of Data Engineers, QA, Business Analysts, and Product Owners to deliver end-to-end data solutions. Drove architecture discussions, technical design reviews, code reviews, and best practice adoption across data engineering teams. Managed complete project lifecycles from requirements gathering and solution design through deployment and production support. Designed and deployed REST APIs using AWS API Gateway integrated with AWS Lambda and backend data services. Implemented API security, throttling, monitoring, and authentication mechanisms using API Gateway. Delivered analytics and reporting solutions supporting consumer behavior analysis, sales performance, product segmentation, and marketing effectiveness. Developed serverless data ingestion frameworks leveraging API Gateway, Lambda, and S3. Developed and managed data pipelines using Microsoft Fabric Data Factory and Lakehouse architecture. Utilized Microsoft Fabric OneLake, Data Engineering, and Warehouse capabilities for enterprise analytics solutions Designed reusable Spark processing templates and automation frameworks to standardize ingestion, transformation, auditing, and reconciliation processes. Participated in data discovery workshops with business stakeholders to analyze source systems, profile datasets, and derive business requirements. Conducted exploratory data analysis and source-system assessments to define scalable data integration solutions. Developed common libraries and framework components to accelerate delivery of new data products. Built PySpark pipelines to load and transform data into Hive tables, leveraging Spark APIs over Cloudera Hadoop YARN for high-performance analytics. Implemented partitioning and caching strategies in PySpark, optimizing job execution times and integrating with Snowflake and Redshift for analytics. Created Python scripts to extract data from Teradata and other sources, and implemented REST APIs for seamless data transfer from on-premises systems to AWS S3. Designed and tuned Spark SQL queries for data transformation and aggregation, integrating PostgreSQL and Parquet files for high-performance storage. Tuned Snowflake performance using clustering, partitioning, and materialized views to accelerate reporting queries. Configured connectivity between AWS EC2 and RDS to provide secure access to Postgres databases, enabling smooth ETL and analytics operations. Built automated reporting dashboards using Python, Matplotlib, and Tableau APIs, transforming complex datasets into actionable business insights. Designed and deployed batch and streaming data solutions using Kafka, Spark Streaming, and Hive, enabling real-time analytics and reporting. Migrated legacy ETL workflows to PySpark on Databricks, leveraging Delta Lake for efficient versioned data storage and faster query performance. Created data pipelines integrating Python, Spark, and Hadoop, reducing processing time and supporting large-scale analytics initiatives. Architected and implemented enterprise-scale data pipelines processing high-volume structured and semi-structured datasets. Designed reusable ETL/ELT frameworks to support scalable and maintainable data integration solutions. Implemented monitoring, alerting, and automated recovery mechanisms to improve pipeline reliability and availability. Used AWS Aurora DB and MongoDB for profile data persistence and semi-structured data storage. Implemented Docker-based environments for integration components, reducing deployment time and improving environment consistency. Collaborated with Cloud Security and Infrastructure teams to implement secure data platform architectures on AWS and Azure. Supported security assessments of cloud resources and assisted in remediation of identified configuration risks. Applied security best practices for data access, encryption, identity management, and network controls across cloud environments. Participated in architecture reviews to ensure secure, scalable, and compliant data platform implementations. Used New Relic and custom logging frameworks to monitor request/response payloads and application health. Performed data reconciliation between source and target tables to ensure data accuracy and integrity. Guided business users to understand complex ETL processes and data flows, ensuring alignment between technical solutions and business expectations. Environment: PySpark, Spark SQL, Python, Hadoop, Hive, HDFS, Snowflake, Redshift, Databricks, Delta Lake, Kafka, Spark Streaming, AWS S3, EC2, RDS, Teradata, PostgreSQL, MongoDB, Docker, New Relic, Matplotlib, Tableau, Linux Client: Smarc Technologies, Chennai, India May 2016 November 2017 Role: Data Engineer Responsibilities: Developed and maintained ETL pipelines using Python, Pandas, and SQLAlchemy, optimizing data extraction, transformation, and loading processes across multiple relational databases. Implemented ETL workflows in PySpark leveraging Airflow for orchestration and S3 for cloud storage, improving data ingestion and pipeline reliability. Developed and optimized enterprise data pipelines leveraging Snowflake, Airflow, Python, SQL, and cloud-native services for scalable analytics solutions. Implemented Snowflake data modeling, performance tuning, clustering strategies, and workload optimization to support large-scale reporting environments. Provided post-production support for critical Spark pipelines, monitoring job executions, resolving failures, and ensuring SLA compliance. Performed root-cause analysis of data quality issues, job failures, and performance bottlenecks in production environments. Established operational dashboards and alerting mechanisms for proactive pipeline monitoring Developed and maintained dbt-based ELT frameworks for scalable data transformations, reusable models, source freshness validation, and automated documentation. Implemented dbt testing frameworks, including schema tests, data quality validations, and source integrity checks to improve trust in analytical datasets. Led refactoring initiatives for legacy Spark and ETL codebases, improving maintainability, reducing technical debt, and increasing processing efficiency. Modernized legacy data pipelines through reusable design patterns, modular code structures, and automated testing frameworks. Utilized dbt models to standardize transformation logic and improve maintainability across enterprise data platforms. Integrated dbt deployments into CI/CD pipelines using Git and automated build validation processes Developed modular ELT transformation pipelines using dbt, implementing reusable models, source freshness checks, schema tests, and automated documentation for enterprise analytics workloads. Leveraged dbt Core with Snowflake and AWS data platforms to build scalable transformation layers following software engineering best practices. Scheduled, monitored, and managed data jobs using AutoSys, automating pipeline executions and reducing manual intervention. Developed Scala applications to transform and aggregate raw data in Snowflake, enabling faster analytics for business stakeholders. Collaborated with Data Scientists and Marketing Analytics teams to support customer targeting models, lookalike audiences, and ad effectiveness analysis. Built attribution and conversion tracking pipelines to measure user journeys across multiple digital marketing channels. Supported Real-Time Bidding (RTB) analytics by processing bid requests, bid responses, win rates, and auction-level data streams. Implemented GDPR/CCPA-compliant data governance and privacy controls for advertising and customer behavioral datasets. Built automated data ingestion workflows from APIs and streaming sources using Python, Requests, and Kafka, ensuring real-time data availability for analytics. Designed and implemented Hadoop-based data pipelines using HDFS, MapReduce, and Hive, integrating data from multiple structured and unstructured sources. Developed and optimized ETL workflows in Spark and Pig, improving data processing speed while leveraging HDFS for distributed storage. Engineered Java-based microservices to automate ETL processes between relational databases and cloud-based data warehouses. Established CI/CD practices for data engineering solutions using Git, Jenkins, Azure DevOps, and automated deployment pipelines. Implemented automated testing, code quality checks, deployment validation, and rollback mechanisms for data platform releases. Wrote complex TSQL and PL/SQL scripts for data cleansing, validation, and enrichment across multiple relational databases. Implemented NoSQL (MongoDB) solutions to store semi-structured and unstructured data, reducing query latency for high-volume datasets. Used Jenkins for continuous integration and deployment, and Git/GitHub for version control and source management. Created instances (VMs) in AWS with required infrastructure stack for application teams. Participated in Agile/Scrum ceremonies including daily standups, sprint planning, and retrospectives. Environment: Python, PySpark, Spark, Hive, HDFS, Pig, Hadoop, Scala, Snowflake, Airflow, AutoSys, Kafka, AWS S3, MongoDB, SQL Server, Oracle, Java, Jenkins, Git, Linux Client: Zenser Technologies, Chennai, India June 2014 May 2016 Role: Data Engineer Responsibilities: Responsible for availability and reliability of data platforms, ensuring they met requirements of all business users. Worked with Hadoop, Pig, Hive, Spark, Oracle, and MS SQL Server for large-scale data processing and analytics. Implemented custom error handling in ETL jobs and worked on different methods of logging and monitoring. Experience with high-velocity, high-volume stream processing using Apache NiFi and StreamSets. Executed Jenkins pipelines for script deployments, table creation, and technical tests. Built data lakes and reporting layers to support advertising KPIs such as CTR, CPC, CPM, CPA, ROAS, Conversion Rate, and Customer Acquisition Cost. Processed billions of advertising events daily using Spark, Hadoop, Hive, and Snowflake, ensuring low-latency analytics and reporting. Developed audience segmentation pipelines leveraging behavioral, demographic, and campaign engagement data for targeted advertising initiatives. Implemented automated data quality validation frameworks, completeness checks, freshness monitoring, and SLA-based alerting mechanisms using Airflow, Python, and CloudWatch. Served as L3 Production Support Engineer for critical data platforms, performing incident triage, root-cause analysis, hotfix deployment, and post-incident reviews. Developed Spark batch and distributed processing applications using Scala, Spark SQL, DataFrames, and Dataset APIs. Built reusable Scala utility libraries for data validation, enrichment, auditing, and error handling. Optimized Spark jobs using Scala-based transformations, partitioning strategies, caching, and broadcast joins Developed code-first ETL solutions using Scala and Spark, avoiding low-code transformation tools and maintaining all business logic within version-controlled source code. Implemented reusable Spark libraries and Scala-based processing frameworks for enterprise-scale data transformation workloads. Enforced software engineering best practices including unit testing, code reviews, and CI/CD for Spark-based ETL development. Participated in 24x7 on-call support rotations, ensuring adherence to SLAs and minimizing downtime for business-critical data pipelines. Performed detailed analysis of pipeline failures, Spark job issues, orchestration failures, and data inconsistencies, implementing permanent corrective actions. Developed and maintained operational runbooks, troubleshooting guides, and knowledge base documentation for L1/L2 support teams. Coordinated with infrastructure, application, and business teams during production incidents and post-mortem reviews Designed and implemented Medallion Architecture (Bronze, Silver, Gold layers) to improve data quality, lineage, governance, and analytical performance across enterprise data platforms. Built curated Gold-layer datasets for business reporting and advanced analytics while maintaining traceability across data lifecycle stages. Participated in on-call support rotations and created operational runbooks for L2 support teams to improve incident response and reduce MTTR. Troubleshot distributed Spark, Kafka, Airflow, Snowflake, and AWS data processing environments during production outages. Established data observability dashboards for monitoring pipeline health, data drift, and operational metrics. Utilized AI-assisted development tools including ChatGPT, Claude, and Generative AI copilots to accelerate SQL optimization, Python development, ETL troubleshooting, documentation generation, and operational support activities. Leveraged LLM-based assistants for root cause analysis, code reviews, knowledge management, and production issue resolution. Ran regression tests end-to-end for checking application stability and data accuracy. Validated data loads by performing post-load checks and obtaining user confirmations to guarantee accuracy and completeness. Performed incident analysis to gather findings and identify follow-up actions that led to more reliable data products. Worked with adhoc requests including copying data between environments and managing data partitions. Served as the bridge between clients and technical teams for system issues, queries, and requirement gathering. Environment: Python, Hadoop, Pig, Spark, Hive, Impala, NiFi, StreamSets, Jenkins, Oracle, MS SQL Server, ETL, HDFS, Spark SQL, Linux EDUCATION Bachelor of Engineering and Technology -Sastra University (2010-2014) Keywords: continuous integration continuous deployment quality analyst artificial intelligence business intelligence sthree database active directory information technology microsoft mississippi procedural language Georgia