Naveen Paramasivam
Abu Dhabi
Summary
Desirous of an Engineer role in a challenging workplace that welcomes innovative ideas and offers growth opportunities and positive environment for accomplishing projects with 6+ years data engineering experience. Responsive expert experienced in monitoring database performance, troubleshooting issues and optimizing database environment. Possesses strong analytical skills, excellent problem-solving abilities, and deep understanding of database technologies and systems. Equally confident working independently and collaboratively as needed and utilizing excellent communication skills.
Overview
1822
1822
years of professional experience
1
1
Certification
Work History
Lead Data Engineer Consultant
ALDAR PROPERTIES (UAE)
12.203 - Current
- Managed data extraction from REST API and HTTP endpoints using Azure Data Factory, incorporating pagination for efficient data transfer to ADLS Gen2 and other blob stores
- Orchestrated Databricks Job Clusters for daily data loads, reducing infrastructure costs and improving performance and scalability based on load volume
- Completed a proof of concept on Microsoft Fabric and cluster performance tuning, initiating a new project
- Led seamless data migration projects, ensuring uninterrupted transitions and preserving data integrity
- Handled data extraction from over 20 API sources (e.g., JIRA, SAP, C4C, Datadog) using ADF for SCD Type 1 & Type 2 load scenarios
- Developed a Python framework in Databricks for web scraping and data extraction for data science analytics
- Used Azure DevOps for deploying ADB notebooks and ADF pipelines to higher environments
- Implemented data lifecycle management in Azure Blob Storage to delete outdated data, reducing storage and processing costs
- Integrated Git and other version control tools into the data engineering workflow for collaborative development
- Worked on all phases of the data modeling process, including research design, data extraction, cleaning, model building (Star & Snowflake Schema), and validation.
Lead Azure Data Engineer
TREDENCE ANALYTICS SOLUTION (IN)
10.2021 - 12.2023
- Developed Spark applications using Spark-SQL in Databricks for data extraction, transformation, and aggregation
- Migrated SQL databases, PostgreSQL, and on-premises databases to Azure Data Lake and Azure Databricks
- Implemented ACL using high concurrency clusters and Unity Catalog
- Created pipelines in Azure Data Factory to extract, transform, and load data from various sources
- Utilized Autoloader Spark Streaming for efficient data handling
- Implemented Spark Streaming to consume data from Kafka, Kinesis, and EventHub for real-time analytics
- Handled Dynamics CRM data, including mapping and native data types
- Developed Azure Functions to run custom Python codes
- Migrated code from on-premises Hadoop to Azure Databricks
- Played SME and lead roles in cloud migration projects
- Leveraged Redshift for high-performance analytics and optimized data processing workflows
- Implemented data quality checks and validation processes within Python scripts.
Azure Developer - Retail Domain
TATA CONSULTANCY SERVICES
01.2019 - 01.2021
- Develop Pipelines in Azure Data Factory for migration of Data across layers in Azure Storage Containers with various curation activities
- Developed Scala Scripts to Generate Delta Lake Structured Files across ADLS Gen 2 from the Source Formatted Input files
- Worked Cross Platform to load data from Data Bricks to Azure SQL DW
- Exposure in Creating Jar Files from Scala Source Code to trigger the codes using Azure Data Factory Pipelines
- Complex Pipeline build involving sequential and parallel execution using PowerShell scripts
- Develop Stored Procedure in SQL Server to Format the History Data from Hadoop Servers and move it to Data Warehouse Platform
- Analyze data in Azure Data bricks and generate reports to virtualize the data
- History data migration from on-prem Hadoop & hive servers to Azure SQL Data Ware Housing Platform.
Cloud Support Executive - Retail Domain
TATA CONSULTANCY SERVICES
01.2017 - 01.2018
- Developed Shell Scripts with Text Handling & File Handling commands to Validate the Backend queries for Productions Jobs and send the report as mail
- Worked on live monitoring of Production Jobs and immediate resolution activities on failure
- Following the project life cycle defined under the process framework, ensuring that customer deadlines were met within Planned Time
- Communicating with internal/external clients to determine specific requirements and expectations
- Managing client expectations as an indicator of quality
- Providing Various Automation Ideas for Manual misses encountered during the process and following up on the progress till implementation in it.
Education
Bachelor of Engineering Technology - Civil Engineering
Sri Sai Ram Institute of Technology
Chennai05.2017
Skills
- Data Pipeline Design
- Python Programming
- Spark Development
- ETL development
- Data Warehousing
- SQL Programming
- Data Migration
- Problem-Solving
- Teamwork and Collaboration
- Big Data Processing
- Relational databases
- Continuous Improvement
Certification
- DP-203 Certified Microsoft Azure Data Engineer Associate
- Databricks Certified Data Engineer Associate
- AZ-900 Azure Fundamentals
- DP-900 Azure Data Fundamentals
Accomplishments
- Developed a Game using Py game Module (Python 3) as a Learning Progress.
- Developed an Encode - Decoder Application Based Text encryption using Tkinter Python Module.
- Developed Unix shell scripts to pull large datasets from the backend, reducing UI effort and server load.
- Automated mail reminders for weekly team activities using Unix Scripts.
- Automated weekly file comparisons using shell and Python scripting with Pandas.
- Lead a Team of 3 Members for two consecutive projects, delivering work as planned.
- Initiated POCs with leadership teams to test and compare new services against implemented technologies: Microsoft Fabric, Cluster Performance Tuning, Delta Live Tables & Unity Catalog
Skillsexperience
6 years and 9 months, Azure – Data Factory , ADLS Gen 2, Synapse , Delta Lake , Data Bricks, AWS Redshift, Azure Functions, Logic Apps, Dynamics CRM, Python, Java, PL/SQL, Shell Scripting, Python Modules – Pygame ,Tkinter ,Pandas, Toad, Putty, PyCharm, Informatica, Microsoft SQL Developer, Anaconda, Eclipse, Agile, Waterfall, Retail, CPG, Infrastructure & Education Domain, Managed a Team of 3 members for Development Activities, Hive, Hadoop, Oracle, MS SQL Server, Postgre SQL, Delta Lake
Timeline
Lead Azure Data Engineer
TREDENCE ANALYTICS SOLUTION (IN)
10.2021 - 12.2023
Azure Developer - Retail Domain
TATA CONSULTANCY SERVICES
01.2019 - 01.2021
Cloud Support Executive - Retail Domain
TATA CONSULTANCY SERVICES
01.2017 - 01.2018
Bachelor of Engineering Technology - Civil Engineering
Sri Sai Ram Institute of Technology
Lead Data Engineer Consultant
ALDAR PROPERTIES (UAE)
12.203 - Current
Similar Profiles
Abdulla Ali AlmansoorAbdulla Ali Almansoor
Completed Military Service Program at UAECompleted Military Service Program at UAE
Laila ShehadeLaila Shehade
Executive Assistant at Aldar PropertiesExecutive Assistant at Aldar Properties
Kellina PillayKellina Pillay
MIDDLE SCHOOL MATH TEACHER at UAEMIDDLE SCHOOL MATH TEACHER at UAE