semifinal qualification progress:
- Approach:
datasets have been downloaded for public transport(RTA dubai pulse) for the dates which were common with the weather dataset.(01-Jan-2018 to 31-Mar-2018) in a file and have preprocessed the datasets and got the hourly traffic flow numbers on each of the days (between 01-Jan-2018 to 31-Mar-2018) these hourly traffic flows were appended to the weather dataset.
- Analysis aim:
To analyse how traffic would increase or decrease during various weather conditions by applying various machine learning and time series models.
python code:
import numpy as np
import pandas as pd
import statsmodels as sm
import sklearn as sk
import matplotlib.pyplot as plt
import seaborn as sns
import os
from datetime import datetime
sns.set()
Weather dataset preprocessing¶
keeping only the necessary features to predict how traffic is affected by various weather conditions. future aim: using machine learning models and time series analysis
main_data=pd.read_csv('C:\\Users\\Taha\\Desktop\\weather_data (2).csv')
data=main_data
data
data.describe()
list(data.columns.values)
data.notna().sum()
data=data.drop(['city_name','dt','lat','lon','main/temp_min','main/temp_max','weather/0/id','weather/1/id','weather/0/icon',
'weather/1/main','weather/1/description','timezone','weather/1/icon'],axis=1)
(data.columns.values)
data['rain/1h'].fillna(0, inplace=True)
data['rain/3h'].fillna(0, inplace=True)
column_names = ['dt_iso','main/temp', 'main/feels_like', 'main/pressure', 'main/humidity',
'wind/speed', 'wind/deg', 'clouds/all', 'rain/1h', 'rain/3h', 'weather/0/main',
'weather/0/description']
data = data.reindex(columns=column_names)
data=data.rename(columns={"main/temp":"temp",
"main/feels_like":"temp_feels_like",
"main/pressure":"main_pressure",
"main/humidity":"humidity",
"wind/speed":"wind_speed",
"wind/deg":"wind_deg",
"clouds/all":"clouds",
"rain/1h":"rain_1h",
"rain/3h":"rain_3h",
"weather/0/main":"weather_main",
"weather/0/description":"weather_description"})
data['dt_iso']=data['dt_iso'].str[0:19]
data
data["traffic"] = np.nan
data['dt_iso'] = data['dt_iso'].astype('str')
data.plot(kind='scatter',x='temp',y='weather_main',color='magenta')
plt.show()
data.plot(kind='scatter',x='temp',y='weather_description',color='orange')
plt.show()
data.plot(kind='scatter',x='temp',y='wind_speed',color='green')
plt.show()
data.plot(kind='scatter',x='temp',y='clouds',color='blue')
plt.show()
corrmatrix=data.corr()
sns.heatmap(corrmatrix, annot=True)
plt.show()
Traffic data Preprocessing¶
i have downloaded the datasets for public transport(RTA dubai pulse) for the dates which were common with the weather dataset.(01-Jan-2018 to 31-Mar-2018) in a file and have preprocessed the datasets and got the hourly traffic flow numbers on each of the days (between 01-Jan-2018 to 31-Mar-2018) these hourly traffic flows were appended to the weather dataset.
Future aim:To analyse how traffic would increase or decrease during various weather cconditions using machine learning models
with os.scandir("C:\\Users\\Taha\\Desktop\\traffic data\\") as files:
for file in files:
print("Accessing file"+':'+file.name)
date=str(file.name[22:24])
month=str(file.name[19:21])
date='2018-'+month+'-'+date+' '
#filter the data with only the dates of the filename and sort them by timestamp.
week_day = pd.read_csv("C:\\Users\\Taha\\Desktop\\traffic data\\"+file.name)
weekday=week_day[['txn_date','txn_time']]
weekday["datetime"] = weekday["txn_date"] +' ' +weekday["txn_time"]
weekday.drop(['txn_time','txn_date'],axis=1,inplace=True)
weekday.sort_values(by=['datetime'],inplace=True)
weekday=weekday.reset_index(drop=True)
weekday1 =weekday['datetime'].str.contains(date)
weekday1=weekday[weekday1]
weekday1= weekday1.reset_index(drop=True)
weekday1['datetime'] = pd.to_datetime(weekday1['datetime'])
weekday1= weekday1.reset_index(drop=True)
#finding the traffic flow during each hour on each of the common days between weather and transport datasets
#and appending it to the weather dataset for analysis
for i in range(0,24):
start_s=str(date)+str(i)+':00:00'
end_s=str(date)+str(i)+':59:59'
start= datetime.strptime(start_s, '%Y-%m-%d %H:%M:%S')
end=datetime.strptime(end_s,'%Y-%m-%d %H:%M:%S')
traffic=weekday1[(weekday1['datetime'] >=start) & (weekday1['datetime'] <=end )].count()
print(traffic[0])
data.loc[data['dt_iso'] == str(start_s), ['traffic']]=traffic[0]
6 thoughts on “Weather Disruption of Public Transport Analysis Using Python”
The same segments repeat across article.
The entire focus of the article is based on data analysis where we are missing all models which are aligning two datasets together and finds appropriate correlation and causality in data.
I know that time was short, so I would recommend teaming with someone else next time so that work can be split among team members.
Also, I would advise using some additional datasets which were not part of the initial dataset, like aggregated daily traffic estimates on an hourly basis provided by some navigation applications because that can additionally help with model precision. We all know that bus driers should be professionals but the majority of “normal” non-bus driers are not and they are heavily impacted in distracting sensor inputs (thunderstorm, rain, people cutting in, or even forgetting how to drive when weather condition changes). – I’m adding my last sentence about additional dataset to all teams focusing on this problem because no one did even consider it and that is something you can always do on any project – focus not on internal/provided data but find something to augment it 😉
my aim was to combine hourly analysis(which i got by processing the dubai traffic datatsets) of traffic to the weather data and later apply machine learning models and time series models to it.but time fell too short for me, since this was my first time ever.
anyways enjoyed the journey and yes…lesson learnt,always team up !.
the long rails of numbers u see in my article in the middle are the hourly dsitribution of traffic for each day of each month (eventhough the intersection dates were only for 3 months between the weather data and traffic data on the dubaipulse site). 😐
Hi taha-junaid3000, the approach was good and I could read through the long rails of numbers, but here when I talk about the same segments repeating, I talk about the page itself. If you do a search (find on page) for the chapter “Traffic data Preprocessing”, you will find it three times with completely and exactly the same text – or at least it is how I see it on my browser 🙁
Hi, taha-junaid3000 🙂
tomislavk is right… Splitting the work with someone would help you to achive better results and to learn much more while collaborating with others 🙂
Keeping in mind your work I would focus more on the analysis and conclusions regarding the data quality, variables for modelling, etc. This would be helpful to make next steps.