Shahid Ur Rehman is a results-driven Senior Software Engineer at S&P Global Market Intelligence, where he leverages over 4.5 years of extensive experience in data engineering to drive innovative solutions in the realm of big data. His expertise in harnessing technologies such as Apache Spark,...
Shahid Ur Rehman is a results-driven Senior Software Engineer at S&P Global Market Intelligence, where he leverages over 4.5 years of extensive experience in data engineering to drive innovative solutions in the realm of big data. His expertise in harnessing technologies such as Apache Spark, Scala, and Hive positions him as a key player in architecting and executing end-to-end data pipelines that are both efficient and scalable. Shahid has successfully orchestrated the deployment of Spark applications within an On-Prem Big Data environment, utilizing a robust stack that includes Apache NiFi, Yarn, Zeppelin, and Airflow on AWS EC2 instances.
One of his significant achievements includes the design and implementation of a sophisticated Big Data pipeline capable of processing over 6 TB of data weekly from diverse sources. This pipeline not only enhances data integration but also ensures comprehensive data analysis, enabling stakeholders to make informed decisions based on real-time insights. His proficiency in data warehousing solutions, combined with his hands-on experience with tools like Amazon S3 and PostgreSQL, allows him to create robust data architectures that support complex analytical workloads.
In addition to his technical skills, Shahid is well-versed in utilizing the Elastic Stack (ELK) for monitoring and logging, ensuring that data integrity and system performance are maintained at optimal levels. His ability to work with Apache Kafka for real-time data streaming further enhances his capability to deliver high-quality data solutions. As a dedicated professional, Shahid is committed to continuous learning and innovation, making him an invaluable asset to the S&P Global Market Intelligence team and the broader data engineering community.
