Titanic: Machine Learning from Disaster¶
In [1]:
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
Train Data Wrangling¶
Overview¶
In [2]:
train = pd.read_csv("./dataset/train.csv")
train
Out[2]:
Suvived¶
In [3]:
train_survived = train[train['Survived'] == 0].shape[0]
train_not_survived = train[train['Survived'] == 1].shape[0]
plt.pie([train_survived, train_not_survived], labels=['Survived: ' + str(train_survived), 'Not Survived: ' + str(train_not_survived)])
plt.show()
Pclass¶
There are three different values of this field in train data.
In [4]:
pd.DataFrame({ 'PClass': train.Pclass.unique()})
Out[4]:
In [5]:
ax = plt.subplot()
ax.hist([train[train.Survived == 0].Pclass, train[train.Survived == 1].Pclass], label=['Not Survived', 'Survived'])
ax.legend()
ax.set_xlabel('Pclass')
plt.show()
It it pretty clear that people with Pclass = 1 have more possibility to survive. So we keep this feature Pclass.
Name¶
First, vectorize the names.
In [6]:
from sklearn.feature_extraction.text import CountVectorizer
name_vectorizer = CountVectorizer()
X = name_vectorizer.fit_transform(train.Name)
We got a sort of words.
In [7]:
pd.DataFrame({ 'name': name_vectorizer.get_feature_names() })
Out[7]:
And a matrix of features.
In [8]:
pd.DataFrame(X.toarray())
Out[8]:
Use k-means to cluster these features.
In [9]:
from sklearn.cluster import KMeans
n_clusters = 500
name_cls = KMeans(n_clusters = n_clusters)
name_cls.fit(X)
train['Name-Feature'] = name_cls.predict(X)
ax = plt.subplot()
ax.bar(range(0, 100), [train[(train['Name-Feature'] == i) & (train['Survived'] == 1)].shape[0] / (train[train['Name-Feature'] == i].shape[0]) for i in range(0, 100)])
ax.set_ylabel('Suvived Rate')
ax.set_xlabel('Name-Feature')
plt.show()
At least we have a distribution of survied rate under different clusters. So we keep this generated field Name-Feature.
Sex¶
There are two kinds of values of field Sex in train data.
In [10]:
pd.DataFrame({ 'Sex': train.Sex.unique() })
Out[10]:
Categorize the field Sex.
In [11]:
train.loc[train.Sex == 'male', 'Male'] = 1
train.loc[train.Sex == 'female', 'Male'] = 0
In [12]:
ax = plt.subplot()
ax.hist([train[train.Survived == 0].Male, train[train.Survived == 1].Male], label=['Not Survived', 'Survived'])
ax.legend()
ax.set_xlabel('Male')
plt.show()
It is pretty clear that females have more possibility to survive than males. So we keep this field Sex.
Age¶
The fields Age of some train instances are missed.
In [13]:
pd.DataFrame({ 'Age': train.Age.unique() })
Out[13]:
In [14]:
from sklearn.linear_model import LinearRegression
train_with_age = train[train.Age.notna()]
fig, axes = plt.subplots(2)
axes[0].hist([train_with_age[train_with_age.Survived == 0].Age, train_with_age[train_with_age.Survived == 1].Age], label=['Not Survived', 'Survived'])
axes[0].legend()
axes[0].set_xlabel('Age')
reg = LinearRegression()
reg.fit(np.array(train_with_age.Age).reshape(-1, 1), train_with_age.Survived)
X = np.arange(train_with_age.Age.min(), train_with_age.Age.max(), 0.1)
Y = reg.predict(X.reshape(-1, 1))
axes[1].plot(X, Y)
axes[1].set_xlabel('age')
axes[1].set_ylabel('survived')
plt.show()
reg.coef_
Out[14]:
In general, with increasing age, the survived rate shows a downward trend. So we keep this field Age.
And for the missing value of Age, we choose to use field Pclass to predict for a suitable value.
In [15]:
from sklearn.linear_model import LinearRegression
train_with_age = train[train.Age.notna()]
fig, axes = plt.subplots(2)
axes[0].hist([
train_with_age[train_with_age.Pclass == 1].Age,
train_with_age[train_with_age.Pclass == 2].Age,
train_with_age[train_with_age.Pclass == 3].Age], label=['Pclass=1', 'Pclass=2', 'Pclass=3'])
axes[0].legend()
axes[0].set_xlabel('Age')
reg_age = LinearRegression()
reg_age.fit(np.array(train_with_age.Pclass).reshape(-1, 1), train_with_age.Age)
X = np.arange(train_with_age.Pclass.min(), train_with_age.Pclass.max(), 0.1)
Y = reg_age.predict(X.reshape(-1, 1))
axes[1].plot(X, Y)
axes[1].set_xlabel('Age')
axes[1].set_ylabel('Pclass')
plt.show()
In [16]:
if train[train.Age.isna()].shape[0] > 0:
train.loc[train.Age.isna(), 'Age'] = reg_age.predict(np.array(train[train.Age.isna()].Pclass).reshape(-1, 1)).tolist()
SibSp¶
The field SibSp indicates the number of family members aboard the Titanic. Thre are 7 possible values of the field SibSp.
In [17]:
pd.DataFrame({ 'SibSp': train.SibSp.unique() })
Out[17]:
In [18]:
from sklearn.linear_model import LinearRegression
fig, axes = plt.subplots(2)
axes[0].hist([
train[train.Survived == 0].SibSp,
train[train.Survived == 1].SibSp], label=['Not Survived', 'Survived'])
axes[0].legend()
axes[0].set_xlabel('SibSp')
reg = LinearRegression()
reg.fit(np.array(train.SibSp).reshape(-1, 1), train.Survived)
X = np.arange(train.SibSp.min(), train.SibSp.max(), 0.1)
Y = reg.predict(X.reshape(-1, 1))
axes[1].plot(X, Y)
axes[1].set_xlabel('SibSp')
axes[1].set_ylabel('Survived')
plt.show()
It seems that with increasing number of siblings, the survied rate is reduced. So we keep this field SibSp.
Parch¶
The field Parch indicates the number of parents or children aboard the Titanic. There are also 7 possible values of this field.
In [19]:
pd.DataFrame({ 'Parch': train.Parch.unique() })
Out[19]:
In [20]:
from sklearn.linear_model import LinearRegression
fig, axes = plt.subplots(2)
axes[0].hist([
train[train.Survived == 0].Parch,
train[train.Survived == 1].Parch], label=['Not Survived', 'Survived'])
axes[0].legend()
axes[0].set_xlabel('Parch')
reg = LinearRegression()
reg.fit(np.array(train.Parch).reshape(-1, 1), train.Survived)
X = np.arange(train.Parch.min(), train.Parch.max(), 0.1)
Y = reg.predict(X.reshape(-1, 1))
axes[1].plot(X, Y)
axes[1].set_xlabel('Parch')
axes[1].set_ylabel('Survived Rate')
plt.show()
It seems that with increasing number of Parch, the survied rate is reduced. So we keep this field Parch.
Ticket¶
The field Ticket is the ticket number.
In [21]:
pd.DataFrame({ 'Ticket': train.Ticket.unique() })
Out[21]:
Vectorize this field and use k-means to generate a feature called Ticket-Feature.
In [22]:
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.cluster import KMeans
ticket_vectorizer = CountVectorizer()
X = ticket_vectorizer.fit_transform(train.Ticket)
n_clusters = 500
ticket_cls = KMeans(n_clusters = n_clusters)
ticket_cls.fit(X)
train['Ticket-Feature'] = ticket_cls.predict(X)
ax = plt.subplot()
ax.bar(range(0, 100), [train[(train['Ticket-Feature'] == i) & (train['Survived'] == 1)].shape[0] / (train[train['Ticket-Feature'] == i].shape[0]) for i in range(0, 100)])
ax.set_ylabel('Suvived Rate')
ax.set_xlabel('Ticket-Feature')
plt.show()
At least we have a distribution of survied rate under different clusters. So we keep this generated field Ticket-Feature.
Fare¶
In [23]:
fig, axes = plt.subplots(2)
axes[0].hist([
train[train.Survived == 0].Fare,
train[train.Survived == 1].Fare], label=['Not Survived', 'Survived'])
axes[0].set_xlabel('Fare')
reg = LinearRegression()
reg.fit(np.array(train.Fare).reshape(-1, 1), train.Survived)
X = np.arange(train.Fare.min(), train.Fare.max(), 0.1)
Y = reg.predict(X.reshape(-1, 1))
axes[1].plot(X, Y)
axes[1].set_xlabel('Parch')
axes[1].set_ylabel('Survived Rate')
plt.show()
Cabin¶
Convert the field Cabin to A-Z, - and categorize.
In [24]:
for i in range(ord('A'), ord('Z') + 1):
train.loc[train.Cabin.notna() & train.Cabin.str.contains(chr(i)), 'Cabin'] = chr(i)
for i in range(ord('A'), ord('Z') + 1):
train.loc[train.Cabin == chr(i), 'Cabin'] = i - ord('A')
if train[train.Cabin.isna()].shape[0] > 0:
train.loc[train.Cabin.isna(), 'Cabin'] = -1
In [25]:
train
Out[25]:
Embarked¶
The field Embarked indicates the place people aboard the Titanic. Some of them are missed.
In [26]:
pd.DataFrame({ 'Embarked': train.Embarked.unique() })
Out[26]:
In [27]:
t = train[train.notna()]
for label in train.Embarked.unique():
if not pd.isna(label):
plt.bar(label, t[(t.Survived == 0) & (t.Embarked == label)].shape[0], label = label + ' / Survived')
plt.bar(label, t[(t.Survived == 1) & (t.Embarked == label)].shape[0], label = label + ' / Not Survived')
plt.legend()
plt.show()
In [28]:
train.groupby('Embarked').count()
Out[28]:
Since the value S is the most common one, we choose S to fill in missied values.
In [29]:
train.loc[train.Embarked.isna(), 'Embarked'] = 'S'
And also, categorize this field.
In [30]:
train.loc[train.Embarked == 'C', 'Embarked'] = 0
train.loc[train.Embarked == 'Q', 'Embarked'] = 1
train.loc[train.Embarked == 'S', 'Embarked'] = 2
Wrangled Train Data¶
Now, select choosen fields.
In [90]:
train_x = train[['Pclass', 'Name-Feature', 'Male', 'Age', 'SibSp', 'Parch', 'Ticket-Feature', 'Fare', 'Cabin', 'Embarked']]
train_x = (train_x - train_x.min()) / (train_x.max() - train_x.min())
train_y = train.Survived
train_x
Out[90]:
Test Data Wrangling¶
Overview¶
In [32]:
test = pd.read_csv('./dataset/test.csv')
test
Out[32]:
Pclass¶
No missed values.
In [33]:
pd.DataFrame({ 'Pclass': test.Pclass.unique() })
Out[33]:
Name¶
Vectorize and cluster.
In [34]:
X = name_vectorizer.transform(test.Name)
test['Name-Feature'] = name_cls.predict(X)
Sex¶
No missed values.
In [35]:
pd.DataFrame({ 'Sex': test.Sex.unique() })
Out[35]:
Categorize.
In [36]:
test.loc[test.Sex == 'male', 'Male'] = 1
test.loc[test.Sex == 'female', 'Male'] = 0
Age¶
Use the field Pclass to predict missed values.
In [37]:
if test[test.Age.isna()].shape[0] > 0:
test.loc[test.Age.isna(), 'Age'] = reg_age.predict(np.array(test[test.Age.isna()].Pclass).reshape(-1, 1)).tolist()
SibSp¶
In [38]:
pd.DataFrame({ 'SibSp': test.SibSp.unique() })
Out[38]:
Parch¶
In [39]:
pd.DataFrame({ 'Parch': test.Parch.unique() })
Out[39]:
Ticket¶
Vectorize and cluster.
In [40]:
X = ticket_vectorizer.transform(test.Ticket)
test['Ticket-Feature'] = ticket_cls.predict(X)
Fare¶
Fill missed values with mean.
In [41]:
if test[test.Fare.isna()].shape[0] > 0:
test.loc[test.Fare.isna(), 'Fare'] = test.Fare.mean()
Cabin¶
Convert the field Cabin to A-Z, - and categorize.
In [42]:
for i in range(ord('A'), ord('Z') + 1):
test.loc[test.Cabin.notna() & test.Cabin.str.contains(chr(i)), 'Cabin'] = chr(i)
for i in range(ord('A'), ord('Z') + 1):
test.loc[test.Cabin == chr(i), 'Cabin'] = i - ord('A')
if test[test.Cabin.isna()].shape[0] > 0:
test.loc[test.Cabin.isna(), 'Cabin'] = -1
Embarked¶
No missed values.
In [43]:
test.groupby('Embarked').count()
Out[43]:
Categorize.
In [44]:
test.loc[test.Embarked == 'C', 'Embarked'] = 0
test.loc[test.Embarked == 'Q', 'Embarked'] = 1
test.loc[test.Embarked == 'S', 'Embarked'] = 2
Wrangled Test Data.¶
Select choosen fields only and normalize.
In [91]:
test_x = test[['Pclass', 'Name-Feature', 'Male', 'Age', 'SibSp', 'Parch', 'Ticket-Feature', 'Fare', 'Cabin', 'Embarked']]
test_x = (test_x - test_x.min()) / (test_x.max() - test_x.min())
test_x
Out[91]:
Performance Comparison¶
Now, we compare the performance of Random Forest, Support Vector Machine and Multi-Layer Perceptron.
In [99]:
from sklearn.neural_network import MLPClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.svm import SVC
from sklearn.model_selection import train_test_split
from sklearn.metrics import plot_roc_curve
x1, x2, y1, y2 = train_test_split(train_x, train_y)
rfc = RandomForestClassifier()
svc = SVC()
mlp = MLPClassifier(max_iter=1000)
rfc.fit(x1, y1)
svc.fit(x1, y1)
mlp.fit(x1, y1)
ax = plt.gca()
plot_roc_curve(rfc, x2, y2, ax = ax)
plot_roc_curve(svc, x2, y2, ax = ax)
plot_roc_curve(mlp, x2, y2, ax = ax)
plt.show()
Use Random Forest¶
In [107]:
clf = RandomForestClassifier()
clf.fit(train_x, train_y)
test_y = clf.predict(test_x)
prediction = pd.DataFrame({ 'PassengerId': test.PassengerId, 'Survived': test_y })
prediction
Out[107]:
Save Result¶
In [108]:
prediction.to_csv("./dataset/submission.csv", index = False)