Datathon Sofia Air Solution – Air station measurement bias correction using Pearson correlation coefficient

Posted 5 CommentsPosted in Datathons Solutions

This article aims to improve the estimation of the measured PM10 pollutants. In Sofia, there are several air pollution measurement stations. They measure PM10 particles, which are particles found in the air with a diameter between 2.5 and 10 micrometers.

The measurement stations fall into two categories, official stations and citizen stations. The official stations provide reliable measurements, they are better monitored and documented. The down-side is that they are only 5 and they are all concentrated in a single region. The citizen stations represent devices mounted on people homes or properties which measure PM10 particles. There is a whole network of such devices. They are many in number and provide a good coverage of the city. The problem with those measurements is that they are biased because of many local factors. Therefore the measurements form the citizen stations are not as reliable as those from the official stations, but on the up-side they are many in numbers.

In this article we define a method to reduce the bias of the measurements from the citizen stations.

Air Sofia Pollution Case

Posted 7 CommentsPosted in Datathons Solutions

Script in R below:   library(stringr) #Step 1 ———————- rm(list=ls()) dd <- read.csv(“C:\\Users\\estoyanova\\OneDrive – VMware, Inc\\ES\\UNI\\master BA\\Boriana-Monthly challenge\\Air Tube\\data_bg_2017.csv”, header = TRUE, sep = “,”, na.strings = c(“”,” “, “NA”, “#NA”), stringsAsFactors = FALSE) topo <- read.csv(“C:\\Users\\estoyanova\\OneDrive – VMware, Inc\\ES\\UNI\\master BA\\Boriana-Monthly challenge\\TOPO-DATA\\sofia_topo.csv”, header = TRUE, sep = “,”, na.strings = c(“”,” “, “NA”, “#NA”), stringsAsFactors […]

Monthly Challenge – Sofia Air – Solution – [iseveryonehigh]

Posted 8 CommentsPosted in Prediction systems

I have just begun my machine learning course from Andrew Ng at Coursera so I thought that this challenge would be a good test of my learnings. I apologise for the delay for article writing as I was not sure if I should have taken this challenge or not since the dataset seemed difficult to […]

Monthly Challenge – Sofia Air – Solution – Kung Fu Panda

Posted 12 CommentsPosted in Prediction systems

1. Business Understanding The air quality in Sofia, Bulgaria, has been a problem for some time already. The population of the city is constantly increasing and this brings more traffic on the streets. The car ownership in Sofia is among the highest in Europe with around 600 cars per 1000 citizens. Another huge issue in […]

Monthly Challenge – Sofia Air – Solution – Jacob Avila

Posted 8 CommentsPosted in Prediction systems

Preliminary Analisys Due to the objective focused on predicting air quality forecast for the next 24 hours per station, first step should be data understanding for citizen science air quality measurements to group it by station and summarize them by day. To complete this task for inspection and pre-processing in order to find missing data, outliers and […]

October Data Science Monthly Challenge

Posted Leave a commentPosted in Prediction systems

Why you should join the Data Science Monthly Challenge and what you can expect?

The Data Science Monthly Challenge provides an exceptional opportunity for participants to be involved in finding a solution to a real data science problem [https://bit.ly/2CAg0V8] step by step. The proposed gradual approach towards advanced business problems will give participants a chance to familiarize themselves in depth with each of the important steps which should be considered during the development of an effective and high-quality data science projects.

And last but not least the monthly challenge is an excellent opportunity for data enthusiasts to prepare themselves for participation in the Global Datathon organized by the Data Science Society during which the time is constrained and there is a much higher level of competition. The acquired skills and deeper understanding during the monthly challenges will play a key role and serve as a competitive advantage of the teams in such large-scale events such as the Global Datathons. Nevertheless, the monthly challenge can also be inspiring for those with more competitive attitude because there will be voting for each article and peer-to-peer reviews and each week the best-voted articles in progress will be uploaded on the News section of the site. 

So, what are you waiting for? 🙂
Register now for the learning challenge before 15th of Oct at http://bit.ly/2QyNshI

Datathon Telenor Solution – WILDLINGS ANALYSIS ON TELENOR – GAME OF THRONES

Posted 3 CommentsPosted in Prediction systems

Cell phones have become a necessity for many people throughout the world. The ability to keep in touch with family, business associates, and access to email are only a few of the reasons for the increasing importance of cell phones. Today’s technically advanced cell phones are capable of not only receiving and placing phone calls, but storing data, taking pictures, and can even be used as walkie talkies, to name just a few of the available options.
Dataset, The Telenor Case – What do Game of Thrones and Telecoms Have in Common? contains the data of delays in networks (RAVENS). The delays of RAVENS are ranging from 26/07/2018 – 05/08/2018. Each RAVEN_NAME represents the Tower. There are 7847 unique RAVEN_NAMES for different networks like 2G/3G/4G. There are 5 unique families.
To provide optimum solution to business problems we are solving the problem in two steps (i) Data Analysis and coding in PYTHON and (ii) Time Series model building in R Studio.
In data analysis we have found the solutions for the problems and found the number of delays (failures) of RAVENS. We also found the Top_10 RAVENS with and without fails. We also detected the Family names and Member names with most and least fails in networks (failures).
The methods of prediction & forecasting of the problem is done by using Time Series model building. As the name suggests that it involves working on time (years, days, hours, minutes) based on data, to derive the hidden insights to make informed decision making. Time series models are very useful models when it is serially correlated data. Based on mobile data, to predict the four days we have divided the data into train and test .We have done Time series analysis by using Arima, Simple exponential analysis and Recurrent Neural networks (RNN).
Finally we conclude that by considering the Root mean square error for these algorithms, we got RNN (Recurrent Neural Networks) as the best algorithm to predict the future for days. Based on the RNN algorithm the prediction of delays for the next four days were analyzed. We have plotted the graphs based on the Time series model for all the algorithms.