1. Business Understanding¶

As the case intro states, the extreme weather conditions can negatively affect the reliability and capacity of the public transportation by potentially damaging the transport infrastructure. This leads to increased costs - not only for infrastructure and transport vehicles maintenance, but also by causing unexpected delays and decreasing the public transport reliability.

If we can predict such events, we would be able to take precautions - by planning costs in advance, by rescheduling in advance or by choosing (or advising for) alternative transportation methods.

As a side note, public transport disruptions do not affect only the government-provided transportation service - they could be used as a reference; as a signal for all drivers and passengers that there are problems on the road.

2. Data Understanding¶

Data collection

The data used in this project consists of the following:

• Dubai Weather data, provided in the case details
• Dubai public bus transportation data, downloaded from Dubai Pulse website. This includes all 640 files available between 01-01-2018 and 16-03-2020. This period is selected so that it fully covers the period of the weather data.

Data processing

The weather data needs to be cleaned of redundant columns. Visualizations need to be made, in order to look for dependencies, cyclic occurrences, relationships between features.

The bus transportation data needs to be downloaded, read and processed, most importantly - identifying the time difference between two consecutive rides of the same bus line. The aggregated information by date will be joined to the weather data and machine learning models will be applied in order to find relationships between weather conditions and transport duration disruptions. It is assumed that the weather conditions will mostly affect the bus ride duration, therefore the median time interval between bus rides will be evaluated. Median is chosen instead of mean value, because it is more reliable metric when dealing with non-normal data with outliers.

Other public transportation methods are out of scope.

Approach limitations

• Aggregated vs. granular data

One of the major limitations of the current project approach is that the public transportation data was aggregated at a daily (instead of an hourly) basis and therefore it lacks granularity.

• Additional data is needed

Looking for additional data could have been helpful, for example any historical infrastructure damages, government spend on road reconstruction by month, public transport rescheduling, etc. Also, weather forecasts for broader region, as well as traffic estimations, could improve the predictins and provide valuable information.

• Transportation type

Another limitation is that the current project is based on bus transportation data only, while other transport types could also be taken into account. It was assumed that the bus transportation is impacted at highest extent by the weather conditions and by other factors as well (e.g. other drivers on the road), therefore it was preferred over the other transportation types.

• Seasonal and hourly specifics have not been taken into account

Seasonality has not been implemented into the modeling part. Information about business holidays and hourly segments (day, night, peak, off-peak) could also improve the model results.

3. Data Preparation¶

3.1. Import libraries¶

3.2. Load raw data¶

3.2.1. Load weather data¶

There are 3 cases with hourly record having more than one weather condition recorded. Only the first one will be kept.

Creating timestamp:

There are missing values in both rain columns - they will be infilled with zeros.

3.2.2. Data Exploration¶

3.2.3.1 Data types¶

3.2.3.2. Data visualization¶

Several findings:

• The temperature is highest during Dust weather condition.
• While it's Clear, the average temperature is relatively high, but the range is also very high. However, highest temperatures (37+ degrees) were achieved only while it's clear, cloudy or dusty. On the other hand, the lowest temperatures were in foggy and misty weather.
• Lowest pressure is recorded in dusty weather.
• Low humidity while Clear, Cloudy, Dust and Smoke; highest during Fog, Mist and Haze.
• High wind spead during Dust, Thunderstorm and Smoke.
• No rain volume recorded during thunderstorms.

All temperature features are moving pretty much together, with highest temperatures recorded in July and August and lowest in January and February.

Generally, the atmospheric pressure varies within tight intervals on monthly basis, but shows a clear seasonal character, being highest in January and lowest in July. Later will be observed a correlation plot between the features that will likely highlight negative correlation between the atmospheric pressure and the temperature.

Although that the volatility of humidity varies greatly, here we can also observe some kind of cyclic behaviour, with highest humidity recorded in January and lowest in May-June.

It's slightly more difficult to look for patterns in the dynamics of cloudiness percentage, but it looks a bit lower in May and June, as well as in some parts of February, August and October.

Due to Dubai's climate specifics, there is no much evidence of raining. Data for "the last 3 hours" does not exist before the end of 2019 - has either not been observed or not been collected. There is some sort of seasonality, although not so well established:

• in 2018 the highest rain volumes have been recorded in March and October.
• in 2019 rains have been recorded in February, April and May, as well as high quantities in November and December.
• in 2020 the highest figures are in January.

From June to September typically there's no raining.

Several observations:

• Confirmed is the strong negative correlation between the temperature and the pressure that was mentioned above. Many years ago Gay-Lussac has claimed for the opposite, but it seems that his statement applies only in closed systems with constant gas volume (Source: https://en.wikipedia.org/wiki/Gay-Lussac%27s_law#Pressure-temperature_law)
• There is somewhat strong negative correlation between the temperature and humidity. However, this negative correlation goes somewhat weaker when calculated between the human perception of the temperature ('feels like temperature') and the humidity
• There is some evidence of a negative relationship between the wind speed and humidity, meaning that probably the wind carries away the little water drops in the air
• There is a positive correlatin between the wind speed and wind degrees - meaning that when the wind speed is higher for the higher degree value, and therefore - the winds from North and West more often have higher speed than the ones from South and East (Source: https://snowfence.umn.edu/Components/winddirectionanddegrees.htm)

In order to catch what specific changes happen during thunderstorm, we can observe the differences in correlation matrix during clear weather and during thunderstorms. Here are the main differences:

• In clear weather, there is a strong negative correlation between the temperature and atmospheric pressure. During thunderstorms such correlation almost does not exist.
• In clear weather, there is some, although weak, positive correlation between the temperature and the wind speed. During thunderstorms, there is no correlation. However, we can see that the human perception of the temperature does not correlate with the wind speed in clear weather, but correlates negatively with the wind speed during thunderstorms, and so does with the wind degrees - meaning that in low temperatures there wind directions are mainly North and West. This is also associated with higher humidity and higher cloudiness percentage.

There is some interesting evidence on the two charts above - that both atmospheric pressure and humidity are high during thunderstorms. They could be explored further, in order to find if a thunderstorm could be predicted by the dynamics of pressure and humidity, and probably wind speed/direction. However, this analysis will not be implemented in the current project.

3.2.3. Load public transportation data¶

The case description does not specify which particular type of public transportation to be analyzed, but due to the tight timelines the best approach would be to focus on the bus transportation - assuming that it should be the most affected type of transportation by weather conditions. Moreover, any bus transportation issues could be a signal for general road transportation ones - this is something that other public transportation types will not be able to hint about.

The Dubai Pulse website API seems to be inaccessible for non-UAE citizens, therefore the only way to acquire the data is by downloading separately each of the 640 CSV files for the bus transportation since 1st January 2018. To reduce the manual effort, Selenium webdriver is going to be set up to download the files one by one - putting a generous 10 seconds wait time between download requests to avoid server overloading. This action took about 5 hours and required a storage size of 60 GB.

The easiest way to find the position of each 'Download' button is by getting all such buttons by their xpath:

There are 1212 bus transportation files uploaded, but some of them are attributed to 2017. Below is the code that downloads the files - executed in several separate runs, using the i-th parameter.

3.2.3. Explore sample bus transportation data - proof of concept¶

The purpose of this section is to set up a proof of concept for a later use - explore one of the bus transportation files, observe the data and find potential ways to extract the information that could be useful for the task. The biggest challenge would be find the most efficient way to process 600+ files, having in mind that some of them contain more than a million records. This makes the looping approach undesirable, because iterating through all the files will probably exceed the Datathon article submission deadline!

Almost a minute to get the timestamp of the sample file is not great, but definitely better than iterating. The next task is to sort the data and calculate the time difference between consecutive transactions.

The initial approach was to find each transaction that is recorded 30 seconds or more after the previous (or before the following one) and assign a specific (increment) value to each. However, this process did not complete in more than half an hour for just one file and unfortunately the approach was discarded. A quicker one was to identify and get the first occurence of each group and put it into a new data frame.

From the initial 785k records in the file, about as much as 75% of the data was discarded and not used in the project. Now we can calculate the duration between the rides.

The last record has no duration, therefore needs manual calculation - its timestamp minus the timestamp of the last record from the initial (detailed) data.

For the purposes of the project I will only use aggregated data from the bus transportation files.

Expectedly, the data is non-normal and highly right-skewed. The mean value of duration is three times bigger than the median value, meaning that it is biased and affected by the outliers. The maximum duration (difference between two rides of a single bus line) is almost 24 hours - probably accounting to bus lines riding once per day. The standard deviation is also high - almost 3 hours. The most meaningful piece of the descriptive statistics is the median value, which is approx 17 minutes - this refers to the median time difference between different rides of a single bus line.

3.2.4. Read all bus transportation data¶

Now we have finished the proof of concept and we can try applying it on all the files. This is a huge effort, requiring more than 7 hours of processing time.

3.2.4. Combine weather and public transportation data¶

Unfortunately, bus transportation statistics is indeed not provided for these dates on the Dubai Pulse website. Moreover, the missing dates account for 4007 data records in the weather dataset, which is about 20% of the total. Dropping such a big part of the data is not recommended, therefore the better approach would be to infill the NULLs with the median values for the respective month, and thus, to keep the impact (if any) of the seasonal fluctuations.

As only the last 3 data records have no value for condition, and the several previous have the value 'Clear', we can fill 'Clear' to the last 3 as well. Several columns, including the description, will not be needed and can be dropped.

4. Modelling¶

4.1. Pre-processing¶

The 'condition' column is a categorical one, therefore needs to be one-hot encoded.

The distribution of the dependent variable is not normal, so we can improve the models performance if we transform it by log(1+y).

Now we can split the data into training and testing sets.

The metric that is going to be used to evaluate the models is the mean absolute error. There will also be a visual representation of the predictions.

4.2. Regression models¶

The following models will be put into action:

• Multiple Linear Regression
• Random Forest Regression
• Support Vector Regression
• Extra Gradient Boosting Regressor

4.2.1. Multiple Linear Regression¶

4.2.2. Random Forest Regression¶

Here we will also apply grid search to find the best parameters for the model

4.2.3. Support Vector Regression¶

The Support Vector models work best if the data is scaled first.

4.2.4. Extra Gradient Boosting (XGBoost) Regressor¶

5. Evaluation¶

5.1. Models summary¶

The lowest mean absolute error is returned by the SVR model. However, the task is not as trivial as usually - we need not only to find the best fitting model, but also the model that does the best job in predicting the outliers - because they are the ones that are causing the issue described in the business understanding section.

The chart above says much more than the mean absolute error - despite having the best score, the SVR model fails to predict any of the unexpected fluctuations in the test data. This means that none of SVR, Random Forest and Multi-Linear model is able go predict even a single outlier. This makes them inappropriate for the current task.

Several outliers have been well-predicted by the XGBoost regressor, but it is also unable to predict any of the major outliers - only some of the minor ones. Also, it doesn't do well in several cases, when returns a false alarm. Looking at the chart again, it seems that at times there is slight shifting of some predictions, but the shift is with a little step, so the model still has a chance of some 'almost acceptable' prediction for the day, if not for the exact hour.

It may be worth shifting the data with several positions and re-fitting the model, because the weather data is provided at hourly level. Therefore, it makes some sense events from several hours ago to affect the timings for several hours later (although there is a major limitation that the approach in this project is to use aggregated daily (instead of hourly) transportation data).

5.2. Data shift¶

One additional test will be done - shifting the data to check whether the weather condition by several hours ago could predict the current day's median bus ride intervals. Only the XGBoost regressor will be used for predictions.

After shifting the data with 5 hours:

• On the one hand, the model has decreased its ability to catch the little fluctuations, i.e. the usual/normal/standard bus ride interval values.
• On the other hand, it made several good predictions of outliers. Again, there are a few false alarms, but in this case they could be slightly preferred against no alarm when there should be one.

However, the overall performance of the model is still below par and the prediction expectations were not met.

6. Deployment¶

The business case that was worked on presented a good idea, which unfortunately was not fulfilled well enough by the current project. The final XGBoost Regressor model made a good job in predicting several unusual delays in the bus transportation in Dubai, caused by weather conditions, but they are only a small part of the total delays that have been registered.

There are several ideas that could be additionally applied and could potentially improve the model performance:

• Combine the current date with additional information, e.g. real cases of damaged infrastructure, flooding, thunderstorms, etc. Also, as suggested by the jury of this Datathon, researching for hourly data with official weather forecasts, as well as daily traffic estimates on an hourly basis, and the subsequent inclusion into the models could improve their prediction power or give much valuable insights.
• Explore the seasonality element in the data, that could potentially explain if the weather impact on the traffic varies according to the season. Distinction of working days and bank holidays could also be useful.
• Reprocess the transportation data and extract it on hourly, instead of daily basis. This could not only improve the predictability, but also to help us find out whether the atmospheric condition has a different impact on the traffic during day, night, peak, off-peak hours, etc.
• Make more tests with shifted the data, especially when transportation hourly data is available. Then it could probably be much easier to predict if the heavy rain from several hours ago is causing bus delays now.
• Put slightly more focus on whether the pressure, humidity and wind could predict heavy raining or thunderstorm events. As suggested by the jury, it is needed to detect causality in weather elements, which is something inappropriate to do with a single location data. Additional weather data for a broader region could help, as the weather in particular location cannot be observed as isolated from the rest of the world and it is highly dependent on the weather in the neighbouring regions.
• Explore additional machine learning models that could outperform the XGBoost regressor for this particular case. The task here is not to get as close to the usual values as possible, but rather to predict unusual events. Therefore, the task seems to be more appropriate for avant-garde models, rather than traditional ones like Random Forest, because the purpose is to find non-standard, outlying situations.