REZAWAHADJ

Microsoft Azure Research Project
Funded by the Microsoft Azure Research Center on June 30, 2016 Doha, Qatar

This project was awarded to me and the Microsoft Azure Research Center in June 2016 to develop a Real-Time IoT Monitoring & BIG DATA Harvesting Storage System for Energy Demand Response in the Middle East Region (GCC Countries).

Gulf Cooperation Council (GCC), the political and economic alliance of six Middle Eastern countries including Saudi Arabia, Kuwait, the United Arab Emirates, Qatar, Bahrain, and Oman, has recently faced oil prices falling to their lowest level since 2003. The effects of falling prices are being felt by economies around the world. However, oil-producing nations that rely on exports have been particularly hard hit, with many now experiencing social and, in some cases, political impacts.

The goal and scope of this project are to harvest, monitor, and forecast energy demand response data across the GCC countries to help each country in energy usage. This includes monitoring and storing historical Big Data in a real-time cloud-based architecture at the regional scale. The historical datasets and real-time monitoring will be utilized in IoT in conjunction with Microsoft Azure Cloud, making them accessible for researchers and governmental institutions from the Microsoft Azure Cloud for developing IoT applications.

At the beginning, we started without adhering to standard academic approaches that typically fall into deductive or inductive methodologies for each hypothesis. Instead, we decided to investigate the crucial importance of this project based on a practical and political understanding of the Gulf Region during the current energy crisis. This strong motivation drove our investigation. We proposed several hypothesis scenarios as solutions to educate, motivate, and help the community of the GCC understand the future of energy demand and take appropriate action before time runs out.

With this in mind, we initiated the harvesting of electricity datasets across all six countries, with a time interval of 5 seconds, storing them in Microsoft SQL Server on three different computers. Starting around June 2015, we now have 12 million electricity datasets, which can guide us in future modeling and forecasting. This is the first major step towards developing and integrating solutions.

The code has been written in Microsoft C# in cooperation with SQL Server over the HTTP Protocol. It runs every five seconds, harvesting electricity demand data from the Saudi Arabia Command Center in the Dammam region (close to Bahrain). The data is streamed into SQL Server, and simultaneously, raw data is also streamed into an IoT real-time dashboard in JSON format.

Since June 2015, we have been involved in this initiative by architecting and developing real-time IoT infrastructure for all six members of the GCC countries as backbone infrastructure. Currently, we have 120 million record sets in SQL Server.

However, since this project will address the real problem at the scale of the Gulf Region by covering all six countries simultaneously in real-time energy consumption demand, we are evaluating our current resources.

We are scaling out cloud-based processing not only for our storage but also for real-time demand response for each additional country.

Our architecture supports SOAP/ XML as well as RESTful / JSON format to address any application using such formats. Currently, we are using cloud-based accounts for our real-time streaming data with limited resources from several known IoT companies, such as ThinkSpeak, Xively, and PubNub, to close the technical loop without losing real-time datasets. This was a major step following our investigation.

The stream data flow (see the project summary description - below), as well as the entire architecture in detail with respect to IoT. It includes all resources such as Python Modelling, RESTful JSON, C# WebJobs, NodeJS Server, ETL SSIS, SSRS, SSAS, Raspberry Pi board, SQL Server, and the Forecasting Algorithm.

The dashboard is real and operational internally on our server back-end. It will be launched in the public domain soon, once we have tested and ensured that all parts of the model function properly.

This project is exactly suitable for Microsoft Azure Cloud or any other cloud platform, consisting of state-of-the-art real-time data actuation, on-the-fly analysis, monitoring through suitable dashboards, and cloud storage for future historic modeling and forecasting. We have all the necessary resources to achieve this goal and have been working on it since June 2015 from home.

This initiative will also encourage researchers and engineers from all six GCC countries to access energy demand response datasets synchronously, based on their local timestamps.

Meanwhile, the ultimate goal is to encourage GCC countries to migrate their datasets to Microsoft Azure Cloud by successfully executing this case study, which includes forecasting modeling, real-time monitoring of energy demand data, and data storage services. This project can easily be customized for any data science field, such as water data, meteorology data, healthcare data, etc., in the Middle East region.

Project Summary Description: [Click here to download PDF]