Using Machine Learning to Predict Flight Delays

Q Holidays Flight Delays

Q Holidays Flight Delays Determine whether or not a flight will be delayed. You work for a travel booking website that seeks to improve the customer experience for people whose flights were delayed. The company wants to let customers know, when they book their flights, if their flights will be delayed due to weather. You need to use ML to identify whether a flight will be delayed due to weather.

QA Holidays Flight Delay Problems What is the business problem? Customer experience is negatively effected by delayed flights. Furthermore, delayed flights may have knock on effects to their other holiday plans. This affects customer satisfaction and may lead to a loss of business in the future. What is the business goal? Being able to predict if a flight will be delayed will allow us to notify customers of this at the point of purchase and hopefully be able to manage customers expectations and improve their experience. What is the business metric? Improved Customer Satisfaction/Feedback Less complaints

Flight Delays Problems and Goals Why is the business problem appropriate for machine learning? Size, scale and complexity of the data lends itself to words machine learning. Furthermore, the fact we want to make predictions is an ideal use-case for ML What type of ML should be used? - Supervised we have a large amount of historical, labeled data that makes an ideal training dataset. This means we can make use of this valuable information. Worth nothing that we may be able to make use of some unsupervised techniques to aide our understanding of the data or reframe the problem. Reframe the business problem as a machine learning problem - Can we build a model to accurately predict if a future flight will be delayed or not.

Flight Delay Data The data provided to us to explore this problem has been collected by the Office of Airline Information, Bureau of Transportation Statistics (BTS) and covering domestic US flights between 2013 and 2018. The data they have collected data covering 110 features (columns) and having done so for five years results in the information of 1658130 flights being captured. Whilst there are 110 different features being captured for the flights, not all are relevant to our analysis and may just be noise to the prediction, or worst yet negatively impact the models accuracy. As such, the features we will be focussing on are: - Features associated to the time and date of the flight, - Origin and destination airports of flights, - Any delay associated features. Furthermore, at this time we will be focussing the analysis on the top airports and top airlines.

Data Preprocessing Many assumptions were made upon the data at the point of exploring if ML was an appropriate solution and why, such as: Large quantity of data Structured / tabular data Contain delay information / labels that we could train the model on. (Or if such data is missing, features we could use to create labels.) Contain other features which would be appropriate as inputs for training a model. Of the 1.6 million flights, it was identified that 22540 flightshad missing data. Whilst this may require further investigation in the future, we decided to remove these flights from the dataset as they only accounted for 1.3% of all flights, which meant there is still sufficient data for model training, testing and validation. The 110 columns, was a significant number of features as such many that were thought to be irrelevant or unneccesary were removed, leaving features related to flight times / dates, flight origins / destinations and delay information. This resulted in 20 features. These were then encoded (using one-hot encoding) for use in model training and outputted 94 columns including our prediction target.

Data Exploration A greater % of flights are delayed in Summer months , though there is also a slight rise in Winter also. Unknown if this could be due to increased flights operating. Delays are more likely later in the day, increasing from lunchtime, peaking at 20:00hrs. The delay of one flight is likely to have a knock on effect to other flights and so the issue compounds. Interesting delay spike at 02:00hrs Day 4 (Thursday?) has the highest % of delays, however it is relatively similar amongst all days. WN and OO airlines have a higher % of delays. Chicago (departure), San Francisco and LA airports (arrivals) have the highest % of delays. Unclear if flight distance impacts likelihood of delay. Requires further analysis/visualisation. Unclear from this visual if there is a particular issue at certain distances. Though flight distance is likely to be normally distributed.

Model Training & Selection Algorithm: Binary Classification Labelled Data Predict: Delay / No Delay (Binary) The now processed data was split for training, testing and validation: 80/10/10 Scikit-learn s train_test_split() was used to split the data, this function shuffles data by default. As identified earlier, later flights in the day may be delayed by other flights and shuffling may remove this factor.

Model Evaluation Given that we want to improve customer satisfaction by informing customers when they may be delayed, we want to: minimise flights predicted as on-time that are in fact delays (false negatives) As such, recall is the metric we want to optimise for. Results High accuracy terrible elsewhere Recall VERY BAD AUC 50% accurate = coin-flip We have a very weighted training set towards non-delayed flights. Hence the model can classify most things as non-delayed and be highly accurate. told no-delay incorrectly

Feature Engineering Adding real world experience and knowledge to the model may improve performance. Such as: airports are seemingly busier during the holidays, which may contribute to delays, add holiday information to the data weather can contribute to delays (and is in fact a detail of the brief), incorporate weather information to the data. Whilst future weather prediction is difficult, it is often seasonal so we may be able to associate historic data to dates/months. Incorporating this data resulted in: little impact on the accuracy (79%) Recall significantly improved, but still far from desirable (0.3% -> 5%)

Model Optimisation Due to little improvement in score, XGBoost was used instead of LinearLearner. We were then able to utilise Sagemaker s hyperparameter tuner on a number of hyperparameters to identify optimal model settings. The final model, using optimised hyperparameters resulted in: A model with much improved recall from previous attempts. Recall was more balanced with Precision (Also reflected in F1 score) AUC 77% which is much better than a coin

Model Evaluation Utilising the hyperparameter optimisation records, we are able to review what the importance of each feature is to the model. This identifies that: Departure Hour is most important logical based on previous statement about the knock-on effect of delays. Airwind and Temprature at both origin and destination airports are also highly important to the model. Again makes sense as this weather variables may impact flight durations, take-off/landing and routes. Distance is also of significant importance This is surprising as early visuals didn t indicate a

Evaluation The business success metric was defined as : Improved Customer Experience/Feedback // Less Complaints The current models performance shows: 82% accuracy in predicting delay or not. Recall is unfortunately limited to 29% Putting the confusion matrix into real terms: 76% no delay as expected correct prediction happy customer 6% delayed as expected correct prediction hopefully less annoyed customer 3% unexpectedly on time incorrect prediction but happy customer 15% were unexpectedly delayed incorrect prediction unhappy customer This means hopefully we can manage the expectations and improve the service and satisfaction of 85% of customers.

Evaluation Additional time may have allowed for a further improved model through: Further hyperparameter optimisation Improved feature engineering particularly after reviewing feature importance Potentially reduce dimensionality again from review of feature importance Explore and understand the dataset further Consider other factors from real-world experience or the knowledge of subject matter experts to further influence and inform the model From this I have learned: The importance of sufficient data exploration and subject matter expertise The capabilities of SageMaker and ways to utilise it both in training and optimisation Further understanding of interpreting results and the significance of this