Home

Koti Reddy - data Engineer

[email protected]

Location: St Louis, Missouri, USA

Relocation:

Visa: H1B

Resume file: Koti Reddy Resume_1754936740011.doc Please check the file(s) for viruses. Files are checked manually and then made available for download.

Professional Summary: Dynamic IT professional with over 12+ years of experience specializing in Big Data Analytics, Cloud Data Engineering, Data Warehousing, Data Visualization, and Data Quality Solutions. Expertise in designing, developing, and deploying scalable ETL/ELT pipelines on Big Data and cloud platforms like Azure and Snowflake, delivering high-performance, data-driven applications. Utilized Medallion architecture data for efficient ingestion, cleaning, transformation, and aggregation, enabling accurate business intelligence and insights for performance monitoring and reporting. Skilled in creating optimized data models (e.g., Star Schema, Snowflake Schema) and implementing data marts and OLAP structures for analytics. Proficient in real-time data pipeline development for continuous integration and data processing, ensuring timely insights. Solid understanding of data governance principles, including compliance, security, and role-based access control (RBAC), using Azure Key Vault, Unity Catalog. Developed interactive dashboards and reports in Power BI, leveraging Power Query and DAX for efficient data transformation and analysis. Adept in business intelligence (BI) and data transformation using Power BI to support strategic decision-making. Expertise in managing and optimizing CI/CD workflows for data deployment using Azure DevOps and Git, ensuring efficient development processes. Demonstrated ability to lead and mentor teams in building and delivering collaborative data solutions within Agile frameworks like Scrum. Proven track record of automating and optimizing data workflows to improve operational efficiency, reduce processing times, and enhance performance. Skilled in data migration for large-scale projects, ensuring seamless transitions from on-premises to cloud environments. Extensive experience collaborating with cross-functional teams, including data scientists, business stakeholders, and executives, to deliver data-driven solutions aligned with business requirements. Strong problem-solving and analytical skills with a history of quickly identifying and resolving issues to improve data quality, performance, and business outcomes. Core Competencies: Data Engineering Tools: Databricks, Azure Data Factory, Snowflake, DBT, Power BI. Programming Languages: Python, SQL, Pyspark, DAX, Power Query Cloud Platforms: Azure, AWS Project Methodologies: Agile (Scrum), Waterfall Other Tools: GitLab, Jira, Azure DevOps. Certifications: Microsoft Certified: Azure Data Engineer Associate Microsoft Certified: Microsoft Power Platform Fundamentals AWS Certified AI Practitioner Professional Experience: Client: S&P Global March 2022 Till Date Senior Data Engineer Key Responsibilities: Data Integration, Oil & Gas Analysis Project Created and optimized Azure Data Factory (ADF) pipelines to ingest, transform, and move large volumes of structured and unstructured data from on-premises and cloud sources to Azure Data Lake, Azure SQL Database, and Azure Databricks. Developed Mapping Data Flows to perform complex data transformations such as aggregations, joins, lookups, conditional splits, and derived columns using ADF s built-in transformations and SQL-based logic for efficient data processing. Automate job scheduling and monitoring workflows using Azure Data Factory pipelines with triggers, ensuring consistent, reliable, and timely data processing for critical operations. Developed and managed Azure Data Lake Storage Gen2, structuring data for high availability and performance. Optimized storage using hierarchical namespace, folder partitioning, and compression techniques (Parquet, ORC) to improve data retrieval efficiency. Implemented the Medallion Architecture to design scalable and efficient data models. Designed Apache Spark-based ETL workflows in Azure Databricks, leveraging PySpark and SQL to process and transform large datasets efficiently for analytics and reporting. Optimize Databricks workloads for cost efficiency and high performance, leveraging Delta Live Pipelines and Databricks Runtime, improving query performance and reducing infrastructure costs. Implement data security and governance standards using Unity Catalog in Databricks, managing access controls and ensuring compliance with regulatory and organizational data policies for sensitive datasets. Implemented data partitioning, indexing, and caching strategies for high-performance data processing. Built interactive and data-driven Power BI reports by integrating data from Azure Data Lake, Databricks, Snowflake and SQL databases, ensuring real-time insights and business intelligence. Enhance system performance in Snowflake by implementing materialized views, partitioned tables, and optimized query execution plans, ensuring faster query processing and reduced compute costs. Optimized Snowflake queries and data models for faster performance by using clustering and partitioning. Applied best practices in Snowflake for schema design, normalization, and optimization to ensure long-term scalability and maintainability of the data warehouse. Implemented star and snowflake schema models, DAX (Data Analysis Expressions) optimizations, and aggregation techniques to enhance report performance, reduce load times, and improve query efficiency. Develop interactive Power BI dashboards for senior leadership, presenting visually compelling, data-driven insights that improve strategic decision-making across business units. Expert in designing intuitive and interactive Power BI reports, utilizing a wide range of visualizations (e.g., charts, maps, KPIs, tables) to effectively present business insights and key performance indicators, ensuring a user-friendly experience for stakeholders. Configured Power BI Dataflows, Incremental Refresh, and Scheduled Refresh to automate data updates. Conduct code reviews to provide mentorship and ensure compliance with engineering best practices, fostering a culture of excellence and improving overall team performance. Integrated unit tests and data validation scripts in the CI/CD pipeline to ensure code quality and compliance before deployment to Azure Data Factory, Snowflake, Databricks, and Power BI environments. Validating ETL processes to maintain data accuracy, consistency, and compliance with business rules and requirements. Designed and executed test cases for data validation and quality assurance, ensuring data integrity across multiple sources and pipelines. Incorporate feedback from code reviews to refine and improve code quality, consistently adhering to best practices and aligning with team development standards. Utilized problem-solving skills to identify and resolve data inconsistencies, ensuring high data quality, accuracy, and reliability across systems. Conducted Root Cause Analysis (RCA) and implemented Failure Mode and Effects Analysis (FMEA) to identify, resolve, and prevent defects, enhancing product quality, process reliability, and operational efficiency. Identify and manage potential risks by ensuring compliance with standards and implementing best practices to maintain data integrity and security. Communicate data insights and analysis findings to senior management and stakeholders through clear, concise reports and presentations, facilitating informed decision-making. Collaborate with cross-functional teams to identify and implement process automation opportunities, leading to significant cost savings, improved operational efficiency, and streamlined workflows. Led Agile practices, managing sprint planning, daily stand-ups, and retrospectives to ensure timely delivery of project milestones and alignment with reporting needs. Technologies: Azure Databricks, Azure Data Factory, Azure Data Lake, Snowflake, Snowpipes, Time travel, SnowSQL, Zero copy cloning, Failover, DBT, SQL, Azure Synapse Analytics, Power BI, Python, Pyspark, Unity Catalog, Cloud Functions, IAM, ETL/ELT, CI/CD, S3, Glue, Rest APIs, GitHub, Azure DevOps, Data Modelling, Jira, Agile Scrum. Client: IHSMarkit June 2020 Feb 2022 Data Engineer Key Responsibilities: Designed and developed scalable data warehouse solutions, optimizing data storage, retrieval, and performance for large-scale industry analysis, using SQL Server and snowflake Data Warehouse. Utilized Time Travel to retrieve historical data versions, ensuring data recovery, auditability, and consistency in case of accidental deletions or updates. Developed and implemented automation scripts and tools to optimize the migration process while minimizing downtime. Expertise in using SnowSQL for data loading, querying, and scripting. Streamlined daily data operations and automation using SnowSQL scripts and batch processing. Designed and implemented dimensional data models (star and snowflake schemas), optimizing data storage and query performance using SQL and Indexing Strategies for high-performance analytics. Implemented Snowpipe to automate continuous data ingestion, providing real-time streaming and loading of data from external stages into Snowflake with low latency. Optimized ETL performance by tuning complex SQL queries, improving execution time and reducing resource usage through indexing and partitioning techniques. Created and maintained DBT models, macros, and tests to ensure high-quality, reliable, and maintainable data transformations. Developed and optimized SQL-based data transformations using DBT to enable efficient ELT workflows in cloud data warehouses. Utilized DBT documentation and data lineage tracking to enhance data transparency and governance across teams. Optimized DBT macros and Jinja templating to automate repetitive transformations and improve code reusability. Developed and maintained clear and concise documentation for ETL processes, data flow, and transformations, ensuring transparency and reproducibility of data pipelines. Configured Power BI Dataflows, Incremental Refresh, and Scheduled Refresh to automate data updates. Deployed reports using Power BI Service, Power BI Embedded, and CI/CD pipelines via Azure DevOps for seamless version control and updates. Collaborated with cross-functional teams (data analysts, scientists, and business stakeholders) to understand requirements and deliver customized data solutions that supported various analytical needs. Technologies: SQL, Python, Data Store, Cloud Functions, Snowflake, Snowpipes, Time travel, SnowSQL, Zero copy cloning, Failover, DBT, Rest APIs, SQL server, Git, Data Modelling, Jira, Agile Scrum. Client: IHSMarkit Aug 2018 May 2020 Data Engineer Responsibilities: Analysed the business systems, gathered requirements from the users and documented business needs for decision support data. Utilized Kapow tools to automate data extraction from various state websites related to oil and gas fields, ensuring timely and accurate data collection. Extracted, transformed, and loaded (ETL) large datasets from state regulatory websites into IHS s databases, optimizing data flow and enhancing reporting capabilities for clients in the oil and gas industry. Developed and maintained automated web scraping solutions to collect data from government sites, including permits, production and well data, improving operational efficiency and reducing manual processing time. Applied data transformation techniques to clean, standardize, and structure raw data for analysis, ensuring it aligned with IHS s internal data models and reporting requirements. Conducted data validation and quality checks to ensure integrity and accuracy of data loaded into IHS s system, preventing discrepancies in reports used by stakeholders. Enhanced the performance of ETL processes, identifying and resolving bottlenecks in data extraction and transformation workflows, resulting in faster processing times and improved resource efficiency. Ensured data extraction processes adhered to regulatory and compliance standards specific to the oil and gas sector, including data privacy and security protocols. Technologies: Kapow, SQL, HTML, SQL server, MS Office, MS access. Client: IHS Inc April 2015 July 2018 Data Analyst Responsibilities: Collected and aggregated critical oil & gas and chemical industry data from various state and government websites to ensure accurate, up-to-date datasets for analysis. Applied business rules and logic to analyse complex data sets, ensuring consistency and alignment with industry standards and regulatory requirements. Conducted detailed data analysis using Excel and early versions of SQL Server, providing insights into industry trends, performance metrics, and data anomalies. Generated basic reports and charts using Microsoft Excel and early iterations of reporting tools like Crystal Reports, providing clear, visual representations of data trends for business stakeholders. Collaborated with cross-functional teams, including engineers and regulatory experts, to ensure accurate data integration and alignment with operational goals in the oil & gas and chemical sectors. Conducted research and analysis on market trends, regulatory changes, and industry reports to assist business units in strategic planning and forecasting. Assisted in the management of relational databases, including Microsoft Access and early versions of SQL Server, ensuring data could be queried efficiently for operational insights. Ensured that all collected and analyzed data adhered to relevant oil & gas and chemical industry regulations, including environmental and safety standards, utilizing documentation and regulatory systems in place at the time. Technologies: SQL Server, MS Office, ArcGis, MS Access. Client: tomtom May 2012 March 2015 Geospatial Analyst Responsibilities: Led efforts to collect, validate, and integrate geographic road data from multiple sources, ensuring accuracy and up-to-date information for navigation systems. Conducted rigorous quality checks on geospatial datasets, identifying and correcting discrepancies to maintain high-quality, reliable road navigation data. Managed and optimized large-scale road network databases, ensuring fast access and smooth updates to support navigation products. Collaborated with cross-functional teams to integrate road data with various navigation systems, enhancing real-time GPS-based applications for end-users. Performed in-depth analysis of road network structures and traffic flow, identifying critical areas for improvement to optimize routing algorithms and user experience. Served as a key point of contact for clients, providing technical support and troubleshooting to ensure seamless integration of Teleatlas data into their systems. Transformed raw geographic data into usable formats, ensuring compatibility with Teleatlas' navigation and map products for global deployment. Coordinated and managed multiple project timelines, ensuring successful and timely delivery of road data updates and navigation system enhancements. Worked closely with mapping and cartography experts to align road data with cartographic standards, ensuring consistency and visual accuracy in navigation maps. Technologies: TA Mapper, MS Office, ArcGis, MS Access. Education: Masters from Acharya Nagarjuna University, India. Keywords: continuous integration continuous deployment artificial intelligence business intelligence sthree information technology microsoft