REZAWAHADJ

Microsoft Azure Research Project
Founded by Microsoft Azure Research Center on June 30, 2016 Doha, Qatar

This Project has been awarded between me and Microsoft Azure Research Center on June 2016 to develop a Real-Time IoT Monitoring & BIG DATA Harvesting Storage System for Energy Demand Response in the Middle East Region (GCC Countries).

Gulf Cooperation Council (GCC), the political and economic alliance of six Middle Eastern Countries including Saudi Arabia, Kuwait, the United Arab Emirates, Qatar, Bahrain, and Oman, recently they have been faced oil prices which have fallen to their lowest level since 2003. The effects of falling prices are being felt by economies around the world. But oil-producing nations that rely on exports have been particularly hard hit, with many now feeling the social and, in some cases, political impact.

The goal and scope of this project are harvesting, monitoring and forecasting energy demand response data at the scope of GCC countries to help each country in energy usage, monitoring & storing historical Big Data in real-time cloud-based architecture at the scale of the region. The historical datasets & real-time monitoring will be utilized in IoT in conjunction with Microsoft Azure Cloud, which will be accessible for researchers & governmental institute from the Microsoft Azure Cloud & developing IoT applications.

At the beginning, we have been started regardless of any standard academic approaches which normally fall into deductive or inductive approaches for each hypothesis, we have been decided to investigate how would be crucial this project based-on practical & political understanding of Gulf Region during current energy crisis, that was strong motivation towards this investigation. We proposed several hypothesis scenarios as solution to educate, motivate and help the community of GCC to realize & understand the future of energy demand by taking appropriate action before time runs out.

With this respect, we have initiated harvesting of electricity datasets in all six countries for the time interval of 5 seconds and store them in Microsoft SQL Server on three different computers. That was around June 2015, now we have 12 million electricity datasets which could lead us for future modeling & forecasting. This is the first major step towards modeling and integrating solutions.

The code has been written in Microsoft C# in cooperation with SQL Server over the HTTP Protocol, it runs every five seconds harvesting electricity demand from Saudi Arabia Command Center at Dammam region (close to Bahrain). The data has been streamed into SQL Server at the same time RAW data is also being streamed into IoT real-time dashboard in JSON format.

Since June 2015, we have been involved in this idea by architecting & developing real-time IoT for all six members of GCC countries as Backbone Infrastructure (right now we have 120 million record sets in SQL Server).

However, since this project will address the real problem at the scale of the Gulf Region by covering all six counties simultaneously in real-time Energy Consumption Demand, we are evaluating our current resources.

We are scaling out cloud-based processing not only for our storage but also for real-time demand response for each extra country.

Our architecture support SOAP/ XML as well as RESTful / JSON format to address any application that might be using such a format. Right now we are using cloud-based accounts for our real-time streaming data with limited resources from several known IoT companies such as ThinkSpeak, Xively, PubNub, etc. to close the technical loop without losing real-time datasets. This was a major step after our investigation was done.

The diagram shows the stream data flow, as well as entire architecture in detail with respect to IoT, all resources, Python Modelling, RESTful JSON, C# WebJobs, NodeJS Server, ETL SSIS, SSRS, SSAS, Raspberry PI board, SQL Server and Forecasting Algorithm.

The dashboard is real and working internally in our server back-end, it will be launched in public domain soon when we have tested & realized all parts of model works properly.

This project exactly suitable for Microsoft Azure Cloud or any other Cloud consist of state-of-art by actuating real-time data, analyzing them on the fly, monitoring them by suitable dashboard and storing them in cloud for future historic modeling and forecasting. We have all resources to achieve this goal. We already have been started since June 2015 in home.

This also will encourage all six GCC countries’ researchers & engineers to access synchronously to energy demand response datasets at the same time based on their local timestamps.

In the meanwhile, the ultimate goal is to encourage GCC countries to migrate their datasets to Microsoft Azure cloud by successfully executing this case study, forecasting modeling & real-time monitoring energy demand data as well as data storage services. This project can be easily customizable for any data science such as water data, metrology data, health care data and etc. in the Middle East region.