| Azue Databricks Auburn Hills, MI at Auburn, Pennsylvania, USA |
| Email: [email protected] |
|
http://bit.ly/4ey8w48 https://jobs.nvoids.com/job_details.jsp?id=615719&uid= From: madhavi sharma, siriinfo [email protected] Reply to: [email protected] Develop deep understanding of the data sources, implement data standards, and maintain data quality and master data management. Expert in building Databricks notebooks in extracting the data from various source systems like DB2, Teradata and perform data cleansing, data wrangling, data ETL processing and loading to AZURE SQL DB. Expert in building Ephemeral Notebooks in Databricks like wrapper, driver and config for processing the data, back feeding the data to DB2 using multiprocessing thread pool. Expert in developing JSON Scripts for deploying the Pipeline in Azure Data Factory (ADF) that process the data. Expert in using Databricks with Azure Data Factory (ADF) to compute large volumes of data. Performed ETL operations in Azure Databricks by connecting to different relational database source systems using jdbc connectors. Developed Python scripts to do file validations in Databricks and automated the process using ADF. Analyzed the SQL scripts and designed it by using Pyspark SQL for faster performance. Worked on reading and writing multiple data formats like JSON, Parquet, and delta from various sources using Pyspark. Developed an automated process in Azure cloud which can ingest data daily from web service and load in to Keywords: database information technology http://bit.ly/4ey8w48 https://jobs.nvoids.com/job_details.jsp?id=615719&uid= |
| [email protected] View All |
| 03:13 AM 08-Sep-23 |