Libraries Used¶

Authors:¶

Team 4:

Stephen Panev
Marin St
Dayana Hristova
Dimitar Lyubchev

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import matplotlib.pyplot as plt
from statsmodels.stats.outliers_influence import variance_inflation_factor
from pycaret.classification import *

Read the data and data prep¶

We are using the financial distress data of companies. This is part of a Kaggle dataset which can be viewed in the following link: https://www.kaggle.com/shebrahimi/financial-distress The below analysis is a remake of the analysis by the last notebook, only this time it is produced by the new Python library PyCaret.PyCaret is a new library that is presented as using less code. (https://pycaret.org/) Due to this reason we will not be as much descriptive as the last notebook. The results will be compared to the resorts produced by sklearn.

data = pd.read_csv('data/Financial Distress.csv')

data.shape

(3672, 86)

data.columns

Index(['Company', 'Time', 'Financial Distress', 'x1', 'x2', 'x3', 'x4', 'x5',
       'x6', 'x7', 'x8', 'x9', 'x10', 'x11', 'x12', 'x13', 'x14', 'x15', 'x16',
       'x17', 'x18', 'x19', 'x20', 'x21', 'x22', 'x23', 'x24', 'x25', 'x26',
       'x27', 'x28', 'x29', 'x30', 'x31', 'x32', 'x33', 'x34', 'x35', 'x36',
       'x37', 'x38', 'x39', 'x40', 'x41', 'x42', 'x43', 'x44', 'x45', 'x46',
       'x47', 'x48', 'x49', 'x50', 'x51', 'x52', 'x53', 'x54', 'x55', 'x56',
       'x57', 'x58', 'x59', 'x60', 'x61', 'x62', 'x63', 'x64', 'x65', 'x66',
       'x67', 'x68', 'x69', 'x70', 'x71', 'x72', 'x73', 'x74', 'x75', 'x76',
       'x77', 'x78', 'x79', 'x80', 'x81', 'x82', 'x83'],
      dtype='object')

data.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 3672 entries, 0 to 3671
Data columns (total 86 columns):
 #   Column              Non-Null Count  Dtype  
---  ------              --------------  -----  
 0   Company             3672 non-null   int64  
 1   Time                3672 non-null   int64  
 2   Financial Distress  3672 non-null   float64
 3   x1                  3672 non-null   float64
 4   x2                  3672 non-null   float64
 5   x3                  3672 non-null   float64
 6   x4                  3672 non-null   float64
 7   x5                  3672 non-null   float64
 8   x6                  3672 non-null   float64
 9   x7                  3672 non-null   float64
 10  x8                  3672 non-null   float64
 11  x9                  3672 non-null   float64
 12  x10                 3672 non-null   float64
 13  x11                 3672 non-null   float64
 14  x12                 3672 non-null   float64
 15  x13                 3672 non-null   float64
 16  x14                 3672 non-null   float64
 17  x15                 3672 non-null   float64
 18  x16                 3672 non-null   float64
 19  x17                 3672 non-null   float64
 20  x18                 3672 non-null   float64
 21  x19                 3672 non-null   float64
 22  x20                 3672 non-null   float64
 23  x21                 3672 non-null   float64
 24  x22                 3672 non-null   float64
 25  x23                 3672 non-null   float64
 26  x24                 3672 non-null   float64
 27  x25                 3672 non-null   float64
 28  x26                 3672 non-null   float64
 29  x27                 3672 non-null   float64
 30  x28                 3672 non-null   float64
 31  x29                 3672 non-null   float64
 32  x30                 3672 non-null   float64
 33  x31                 3672 non-null   float64
 34  x32                 3672 non-null   float64
 35  x33                 3672 non-null   float64
 36  x34                 3672 non-null   float64
 37  x35                 3672 non-null   float64
 38  x36                 3672 non-null   float64
 39  x37                 3672 non-null   float64
 40  x38                 3672 non-null   float64
 41  x39                 3672 non-null   float64
 42  x40                 3672 non-null   float64
 43  x41                 3672 non-null   float64
 44  x42                 3672 non-null   float64
 45  x43                 3672 non-null   float64
 46  x44                 3672 non-null   float64
 47  x45                 3672 non-null   float64
 48  x46                 3672 non-null   float64
 49  x47                 3672 non-null   float64
 50  x48                 3672 non-null   float64
 51  x49                 3672 non-null   float64
 52  x50                 3672 non-null   float64
 53  x51                 3672 non-null   float64
 54  x52                 3672 non-null   float64
 55  x53                 3672 non-null   float64
 56  x54                 3672 non-null   float64
 57  x55                 3672 non-null   float64
 58  x56                 3672 non-null   float64
 59  x57                 3672 non-null   float64
 60  x58                 3672 non-null   float64
 61  x59                 3672 non-null   float64
 62  x60                 3672 non-null   float64
 63  x61                 3672 non-null   float64
 64  x62                 3672 non-null   float64
 65  x63                 3672 non-null   float64
 66  x64                 3672 non-null   float64
 67  x65                 3672 non-null   float64
 68  x66                 3672 non-null   float64
 69  x67                 3672 non-null   float64
 70  x68                 3672 non-null   float64
 71  x69                 3672 non-null   float64
 72  x70                 3672 non-null   float64
 73  x71                 3672 non-null   float64
 74  x72                 3672 non-null   float64
 75  x73                 3672 non-null   float64
 76  x74                 3672 non-null   float64
 77  x75                 3672 non-null   float64
 78  x76                 3672 non-null   float64
 79  x77                 3672 non-null   float64
 80  x78                 3672 non-null   float64
 81  x79                 3672 non-null   float64
 82  x80                 3672 non-null   int64  
 83  x81                 3672 non-null   float64
 84  x82                 3672 non-null   int64  
 85  x83                 3672 non-null   int64  
dtypes: float64(81), int64(5)
memory usage: 2.4 MB

data['Company'].unique()

array([  1,   2,   3,   4,   5,   6,   7,   8,   9,  10,  11,  12,  13,
        14,  15,  16,  17,  18,  19,  20,  21,  22,  23,  24,  25,  26,
        27,  28,  29,  30,  31,  32,  33,  34,  35,  36,  37,  38,  39,
        40,  41,  42,  43,  44,  45,  46,  47,  48,  49,  50,  51,  52,
        53,  54,  55,  56,  57,  58,  59,  60,  61,  62,  63,  64,  65,
        66,  67,  68,  69,  70,  71,  72,  73,  74,  75,  76,  77,  78,
        79,  80,  81,  82,  83,  84,  85,  86,  87,  88,  89,  90,  91,
        92,  93,  94,  95,  96,  97,  98,  99, 100, 101, 102, 103, 104,
       105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117,
       118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130,
       131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143,
       144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156,
       157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169,
       170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182,
       183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195,
       196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208,
       209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221,
       222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234,
       235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247,
       248, 249, 250, 251, 252, 253, 254, 255, 256, 257, 258, 259, 260,
       261, 262, 263, 264, 265, 266, 267, 268, 269, 270, 271, 272, 273,
       274, 275, 276, 277, 278, 279, 280, 281, 282, 283, 284, 285, 286,
       287, 288, 289, 290, 291, 292, 293, 294, 295, 296, 297, 298, 299,
       300, 301, 302, 303, 304, 305, 306, 307, 308, 309, 310, 311, 312,
       313, 314, 315, 316, 317, 318, 319, 320, 321, 322, 323, 324, 325,
       326, 327, 328, 329, 330, 331, 332, 333, 334, 335, 336, 337, 338,
       339, 340, 341, 342, 343, 344, 345, 346, 347, 348, 349, 350, 351,
       352, 353, 354, 355, 356, 357, 358, 359, 360, 361, 362, 363, 364,
       365, 366, 367, 368, 369, 370, 371, 372, 373, 374, 375, 376, 377,
       378, 379, 380, 381, 382, 383, 384, 385, 386, 387, 388, 389, 390,
       391, 392, 393, 394, 395, 396, 397, 398, 399, 400, 401, 402, 403,
       404, 405, 406, 407, 408, 409, 410, 411, 412, 413, 414, 415, 416,
       417, 418, 419, 420, 421, 422])

data['Time'].unique()

array([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14])

plt.figure(figsize=(18,10))
sns.distplot(data['Financial Distress'],kde=False, rug=True, color = 'blue');
plt.title('Financial Distress Histogram')
plt.xlabel('Values')
plt.ylabel('Frequency')
plt.grid(True)
plt.show()

data['Financial Distress'].describe()

count    3672.000000
mean        1.040257
std         2.652227
min        -8.631700
25%         0.172275
50%         0.583805
75%         1.351750
max       128.400000
Name: Financial Distress, dtype: float64

plt.figure(figsize=(18,10))
sns.boxplot(x=data['Financial Distress'], color = 'green')
plt.title('Financial Distress Boxplot')
plt.xlabel('Values')
plt.grid(True)
plt.show()

max(data['Financial Distress'])

128.4

plt.figure(figsize=(17,8))
sns.distplot(data[data['Financial Distress'] < 100]['Financial Distress'], kde=False, rug=True, color = 'blue')
plt.title('Financial Distress Histogram without outlier')
plt.xlabel('Values')
plt.ylabel('Frequency')
plt.grid(True)
plt.show()

data_average = data.groupby("Company", as_index=False).mean() 
data_average = data_average.drop(['Time'], axis=1)
data_average.shape

(422, 85)

data_average.head()

data_one_period = data.drop_duplicates(subset=['Company'], keep='last')

time_to_merge = pd.DataFrame(data_one_period[['Company','Time']])
time_to_merge.head()

data_average_with_time = pd.merge(time_to_merge,data_average, on='Company')
data_average_with_time.head()

plt.figure(figsize=(17,8))
sns.distplot(data_average['Financial Distress'],kde=False, rug=True, color = 'blue')
plt.title('Financial Distress of the data average Histogram')
plt.xlabel('Values')
plt.ylabel('Frequency')
plt.grid(True)
plt.show()

plt.figure(figsize=(17,8))
sns.distplot(data_average[data_average['Financial Distress'] < 10]['Financial Distress'],kde=False, rug=True, color = 'blue')
plt.title('Financial Distress Histogram  of data average without outliers testing holdout')
plt.xlabel('Values')
plt.ylabel('Frequency')
plt.grid(True)
plt.show()

pd.qcut(data_average['Financial Distress'], 3).value_counts()

(0.929, 32.813]                141
(-5.6850000000000005, 0.25]    141
(0.25, 0.929]                  140
Name: Financial Distress, dtype: int64

data_average['status'] = np.where(data_average['Financial Distress'] < 0.25,0,np.where(data_average['Financial Distress'] < 0.929, 1,2))

data_average = data_average.drop(['Financial Distress', 'Company'], axis=1)

data_average.head()

Modeling¶

First we create a setup with the desired parameters. The report can be viewed below. The chosen methods are highlighted.

classification = setup(data_average, target = 'status', session_id = 42,normalize = True, 
                  transformation = True, 
                  ignore_low_variance = True,
                  remove_multicollinearity = True, multicollinearity_threshold = 0.75, train_size = 0.7,
                       silent = False)

 
Setup Succesfully Completed!

We compare some of the models based on some indicators. AUC for multiple classification is not present. The highest results are highlighted.

compare_models()

We take a deeper look at some of the models. There are 10 folds as a default. We can preview the mean and the standard deviation.

rf = create_model('rf')

tuned_rf = tune_model('rf')

The evaluate function lets us view some of the results of the model after it has been tuned in the above code. Not all of the results are active in this instance.

evaluate_model(tuned_rf)

gb = create_model('gbc')

tuned_gb = tune_model('gbc')

evaluate_model(tuned_gb)

dt = create_model('dt')

We will trying bagging. boosting and blending of decision trees. You can read more on the topic https://towardsdatascience.com/decision-tree-ensembles-bagging-and-boosting-266a8ba60fd9

bagged_dt = ensemble_model(dt)

print(bagged_dt)

OneVsRestClassifier(estimator=BaggingClassifier(base_estimator=DecisionTreeClassifier(ccp_alpha=0.0,
                                                                                      class_weight=None,
                                                                                      criterion='gini',
                                                                                      max_depth=None,
                                                                                      max_features=None,
                                                                                      max_leaf_nodes=None,
                                                                                      min_impurity_decrease=0.0,
                                                                                      min_impurity_split=None,
                                                                                      min_samples_leaf=1,
                                                                                      min_samples_split=2,
                                                                                      min_weight_fraction_leaf=0.0,
                                                                                      presort='deprecated',
                                                                                      random_state=42,
                                                                                      splitter='best'),
                                                bootstrap=True,
                                                bootstrap_features=False,
                                                max_features=1.0,
                                                max_samples=1.0,
                                                n_estimators=10, n_jobs=None,
                                                oob_score=False,
                                                random_state=42, verbose=0,
                                                warm_start=False),
                    n_jobs=None)

boosted_dt = ensemble_model(dt, method = 'Boosting', n_estimators=20)

tuned_bagged_dt = tune_model('dt', ensemble=True, method='Bagging')

blend_soft = blend_models(method = 'soft')

cb = create_model('catboost')

egb = create_model('xgboost')

lgb = create_model('lightgbm')

tuned_lgb = tune_model('lightgbm')

Finally we blend several models with 2 different methods of voting soft and hard version. You can read more information of different methods of ensemble methods. https://www.analyticsvidhya.com/blog/2018/06/comprehensive-guide-for-ensemble-models/.

blend_specific_soft = blend_models(estimator_list = [rf, egb, lgb, gb], method = 'soft')
# not catboost for the function

blend_specific_hard = blend_models(estimator_list = [rf,egb, lgb, gb], method = 'hard')

We can preview the agreement and disagreement of the models in a plot, this is not present for every model.

stack_soft = stack_models([rf,egb, lgb, gb, cb], plot=True)

stack_hard = stack_models([rf,egb, lgb, gb, cb],  method='hard', plot=True)

Test the models on a holdout sample¶

Finally we will test some of the models on the test sample. The predictions can be viewed below.

gb_holdout_pred = predict_model(gb)
gb_holdout_pred.head()

lgb_holdout_pred = predict_model(lgb)
lgb_holdout_pred.head()

tuned_gb_holdout_pred = predict_model(tuned_gb)
tuned_gb_holdout_pred.head()

rf_holdout_pred = predict_model(tuned_rf)
rf_holdout_pred.head()

Conclusion¶

The models produced by the PyCaret library have better results then similar models produces by sklearn. The differences between the training and testing scores are smaller.Some of the ensemble methods create better models other worsen the results. There instances that the test accuracy can be up to 15% higher using PyCaret. This discrepancy is suspicious and it needs to be investigated further. Due to the fact that sklearn is older and more widely used we will give it a higher faith for the results. For further research on the topic there needs to be a greater investigation of the methods and documentation applied by PyCaret to find the reasons behind the stark differences.

	Company	Financial Distress	x1	x2	x3	x4	x5	x6	x7	x8	...	x74	x75	x76	x77	x78	x79	x80	x81	x82	x83
0	1	-0.334323	1.179250	-0.011305	0.869128	0.940075	0.035843	0.126302	0.564090	-0.018738	...	92.050750	33.5625	32.486500	16.791750	15.750000	1.500000	22.0	-0.177584	31.5	50.5
1	2	1.966056	1.539892	0.204816	0.628511	0.931229	0.302304	0.251645	1.068073	0.218296	...	86.854643	92.1600	89.237286	17.770857	15.142857	-2.721429	29.0	1.895985	13.5	33.5
2	3	-1.659900	0.874400	-0.034676	0.793500	0.609520	-0.002632	-0.086847	0.506090	-0.056892	...	85.437000	27.0700	26.102000	16.000000	16.000000	0.200000	25.0	-0.303170	8.0	37.0
3	4	0.839656	1.553275	0.138410	0.462178	0.759583	0.185367	0.168315	0.472444	0.171318	...	86.854643	92.1600	89.237286	17.770857	15.142857	-2.721429	12.0	0.748936	34.5	50.5
4	5	1.969673	1.127500	0.107643	0.743549	0.449420	0.108686	0.089244	0.664047	0.249389	...	86.854643	92.1600	89.237286	17.770857	15.142857	-2.721429	23.0	1.921633	18.5	43.5

	Company	Time	Financial Distress	x1	x2	x3	x4	x5	x6	x7	...	x74	x75	x76	x77	x78	x79	x80	x81	x82	x83
0	1	4	-0.334323	1.179250	-0.011305	0.869128	0.940075	0.035843	0.126302	0.564090	...	92.050750	33.5625	32.486500	16.791750	15.750000	1.500000	22.0	-0.177584	31.5	50.5
1	2	14	1.966056	1.539892	0.204816	0.628511	0.931229	0.302304	0.251645	1.068073	...	86.854643	92.1600	89.237286	17.770857	15.142857	-2.721429	29.0	1.895985	13.5	33.5
2	3	1	-1.659900	0.874400	-0.034676	0.793500	0.609520	-0.002632	-0.086847	0.506090	...	85.437000	27.0700	26.102000	16.000000	16.000000	0.200000	25.0	-0.303170	8.0	37.0
3	4	14	0.839656	1.553275	0.138410	0.462178	0.759583	0.185367	0.168315	0.472444	...	86.854643	92.1600	89.237286	17.770857	15.142857	-2.721429	12.0	0.748936	34.5	50.5
4	5	14	1.969673	1.127500	0.107643	0.743549	0.449420	0.108686	0.089244	0.664047	...	86.854643	92.1600	89.237286	17.770857	15.142857	-2.721429	23.0	1.921633	18.5	43.5

	x1	x2	x3	x4	x5	x6	x7	x8	x9	x10	...	x75	x76	x77	x78	x79	x80	x81	x82	x83	status
0	1.179250	-0.011305	0.869128	0.940075	0.035843	0.126302	0.564090	-0.018738	-0.163632	-0.027849	...	33.5625	32.486500	16.791750	15.750000	1.500000	22.0	-0.177584	31.5	50.5	0
1	1.539892	0.204816	0.628511	0.931229	0.302304	0.251645	1.068073	0.218296	0.583685	0.238587	...	92.1600	89.237286	17.770857	15.142857	-2.721429	29.0	1.895985	13.5	33.5	2
2	0.874400	-0.034676	0.793500	0.609520	-0.002632	-0.086847	0.506090	-0.056892	-0.167930	-0.034521	...	27.0700	26.102000	16.000000	16.000000	0.200000	25.0	-0.303170	8.0	37.0	0
3	1.553275	0.138410	0.462178	0.759583	0.185367	0.168315	0.472444	0.171318	0.245349	0.195789	...	92.1600	89.237286	17.770857	15.142857	-2.721429	12.0	0.748936	34.5	50.5	1
4	1.127500	0.107643	0.743549	0.449420	0.108686	0.089244	0.664047	0.249389	0.443456	0.123553	...	92.1600	89.237286	17.770857	15.142857	-2.721429	23.0	1.921633	18.5	43.5	2

	Model	Accuracy	Recall	Prec.	F1	Kappa
0	Random Forest Classifier	0.847400	0.846300	0.858700	0.845800	0.771000
1	Extreme Gradient Boosting	0.837100	0.836700	0.858600	0.838100	0.755800
2	Gradient Boosting Classifier	0.837000	0.836300	0.852100	0.838800	0.755600
3	Light Gradient Boosting Machine	0.836900	0.837000	0.856100	0.838400	0.755700
4	CatBoost Classifier	0.833700	0.833700	0.851200	0.834500	0.750600
5	Ada Boost Classifier	0.817200	0.817800	0.841500	0.817700	0.726100
6	Extra Trees Classifier	0.810100	0.809300	0.831200	0.809600	0.715000
7	Logistic Regression	0.793200	0.793000	0.810300	0.786700	0.689900
8	SVM - Linear Kernel	0.773000	0.772600	0.782100	0.770700	0.659600
9	Linear Discriminant Analysis	0.732400	0.732600	0.748300	0.729400	0.598900
10	Decision Tree Classifier	0.728700	0.728100	0.762900	0.724200	0.592900
11	Ridge Classifier	0.715200	0.715900	0.719300	0.707600	0.573200
12	Quadratic Discriminant Analysis	0.678700	0.678100	0.701900	0.680400	0.518000
13	K Neighbors Classifier	0.647600	0.647000	0.699800	0.652800	0.471400
14	Naive Bayes	0.610900	0.612600	0.713100	0.602500	0.417400

	Accuracy	Recall	Prec.	F1	Kappa
0	0.8667	0.8667	0.8727	0.8682	0.8000
1	0.8000	0.8000	0.8000	0.7949	0.7000
2	0.9000	0.9000	0.9231	0.9019	0.8500
3	0.8333	0.8333	0.8357	0.8331	0.7500
4	0.8667	0.8667	0.8727	0.8682	0.8000
5	0.8966	0.8963	0.8959	0.8947	0.8446
6	0.8966	0.8926	0.8985	0.8960	0.8444
7	0.8276	0.8333	0.8387	0.8198	0.7425
8	0.7241	0.7222	0.7649	0.7329	0.5872
9	0.8621	0.8519	0.8851	0.8486	0.7914
Mean	0.8474	0.8463	0.8587	0.8458	0.7710
SD	0.0516	0.0512	0.0462	0.0502	0.0770

Data Science Society

Exploring Financial Distress part 2

Leave a Reply Cancel reply

	Description	Value
0	session_id	42
1	Target Type	Multiclass
2	Label Encoded	None
3	Original Data	(422, 84)
4	Missing Values	False
5	Numeric Features	83
6	Categorical Features	0
7	Ordinal Features	False
8	High Cardinality Features	False
9	High Cardinality Method	None
10	Sampled Data	(422, 84)
11	Transformed Train Set	(295, 46)
12	Transformed Test Set	(127, 46)
13	Numeric Imputer	mean
14	Categorical Imputer	constant
15	Normalize	True
16	Normalize Method	zscore
17	Transformation	True
18	Transformation Method	yeo-johnson
19	PCA	False
20	PCA Method	None
21	PCA Components	None
22	Ignore Low Variance	True
23	Combine Rare Levels	False
24	Rare Level Threshold	None
25	Numeric Binning	False
26	Remove Outliers	False
27	Outliers Threshold	None
28	Remove Multicollinearity	True
29	Multicollinearity Threshold	0.750000
30	Clustering	False
31	Clustering Iteration	None
32	Polynomial Features	False
33	Polynomial Degree	None
34	Trignometry Features	False
35	Polynomial Threshold	None
36	Group Features	False
37	Feature Selection	False
38	Features Selection Threshold	None
39	Feature Interaction	False
40	Feature Ratio	False
41	Interaction Threshold	None

	Accuracy	Recall	Prec.	F1	Kappa
0	0.8000	0.8000	0.8308	0.8020	0.7000
1	0.8000	0.8000	0.8141	0.7989	0.7000
2	0.9000	0.9000	0.9231	0.9019	0.8500
3	0.8667	0.8667	0.8788	0.8677	0.8000
4	0.8333	0.8333	0.8320	0.8313	0.7500
5	0.8276	0.8259	0.8305	0.8231	0.7411
6	0.8621	0.8593	0.8750	0.8644	0.7925
7	0.7931	0.8000	0.8759	0.7893	0.6926
8	0.7931	0.7926	0.8096	0.7958	0.6904
9	0.8621	0.8556	0.8637	0.8566	0.7921
Mean	0.8338	0.8333	0.8533	0.8331	0.7509
SD	0.0356	0.0343	0.0340	0.0361	0.0529

	Accuracy	Recall	Prec.	F1	Kappa
0	0.7333	0.7333	0.7718	0.7397	0.6000
1	0.7333	0.7333	0.7235	0.7249	0.6000
2	0.9333	0.9333	0.9444	0.9346	0.9000
3	0.8333	0.8333	0.8500	0.8357	0.7500
4	0.8667	0.8667	0.8667	0.8667	0.8000
5	0.7931	0.7963	0.8292	0.7890	0.6904
6	0.9310	0.9296	0.9310	0.9310	0.8964
7	0.8621	0.8667	0.9045	0.8646	0.7943
8	0.7586	0.7593	0.7893	0.7655	0.6394
9	0.7931	0.7852	0.7994	0.7846	0.6876
Mean	0.8238	0.8237	0.8410	0.8236	0.7358
SD	0.0703	0.0705	0.0684	0.0710	0.1054

	Accuracy	Recall	Prec.	F1	Kappa
0	0.6333	0.6333	0.7208	0.6212	0.4500
1	0.7000	0.7000	0.7175	0.6985	0.5500
2	0.7667	0.7667	0.8380	0.7593	0.6500
3	0.7667	0.7667	0.7685	0.7640	0.6500
4	0.8000	0.8000	0.8214	0.7963	0.7000
5	0.7586	0.7630	0.7931	0.7618	0.6394
6	0.7241	0.7185	0.7685	0.7214	0.5827
7	0.7931	0.7926	0.8022	0.7846	0.6893
8	0.7241	0.7222	0.7440	0.7307	0.5872
9	0.6207	0.6185	0.6552	0.6046	0.4304
Mean	0.7287	0.7281	0.7629	0.7242	0.5929
SD	0.0589	0.0595	0.0523	0.0622	0.0885

	Accuracy	Recall	Prec.	F1	Kappa
0	0.6333	0.6333	0.6899	0.6120	0.4500
1	0.7333	0.7333	0.7478	0.7302	0.6000
2	0.7667	0.7667	0.8380	0.7593	0.6500
3	0.8000	0.8000	0.8214	0.7963	0.7000
4	0.9000	0.9000	0.8993	0.8982	0.8500
5	0.7241	0.7296	0.7385	0.7224	0.5879
6	0.8276	0.8222	0.8632	0.8260	0.7397
7	0.6207	0.6185	0.6148	0.6166	0.4304
8	0.6897	0.6889	0.7124	0.6958	0.5356
9	0.6897	0.6889	0.7123	0.6814	0.5348
Mean	0.7385	0.7381	0.7638	0.7338	0.6078
SD	0.0828	0.0826	0.0842	0.0854	0.1240

	Accuracy	Recall	Prec.	F1	Kappa
0	0.7667	0.7667	0.7980	0.7709	0.6500
1	0.8333	0.8333	0.8426	0.8341	0.7500
2	0.9000	0.9000	0.9231	0.9019	0.8500
3	0.8667	0.8667	0.8788	0.8677	0.8000
4	0.8667	0.8667	0.8667	0.8667	0.8000
5	0.7931	0.7963	0.8292	0.7890	0.6904
6	0.8621	0.8593	0.8750	0.8644	0.7925
7	0.8621	0.8667	0.8793	0.8609	0.7940
8	0.7586	0.7593	0.7893	0.7655	0.6394
9	0.8276	0.8222	0.8303	0.8239	0.7401
Mean	0.8337	0.8337	0.8512	0.8345	0.7506
SD	0.0447	0.0446	0.0390	0.0439	0.0668

	Accuracy	Recall	Prec.	F1	Kappa
0	0.8000	0.8000	0.8439	0.8052	0.7000
1	0.8333	0.8333	0.8426	0.8341	0.7500
2	0.8667	0.8667	0.9048	0.8704	0.8000
3	0.8667	0.8667	0.8788	0.8677	0.8000
4	0.8667	0.8667	0.8667	0.8667	0.8000
5	0.7931	0.7926	0.8011	0.7924	0.6898
6	0.8621	0.8593	0.8750	0.8644	0.7925
7	0.8621	0.8667	0.9045	0.8574	0.7943
8	0.7586	0.7593	0.7893	0.7655	0.6394
9	0.8621	0.8556	0.8798	0.8573	0.7917
Mean	0.8371	0.8367	0.8586	0.8381	0.7558
SD	0.0374	0.0371	0.0374	0.0355	0.0557

	Accuracy	Recall	Prec.	F1	Kappa
0	0.8000	0.8000	0.8393	0.8056	0.7000
1	0.8667	0.8667	0.8727	0.8682	0.8000
2	0.8667	0.8667	0.9048	0.8704	0.8000
3	0.9000	0.9000	0.9061	0.9015	0.8500
4	0.8667	0.8667	0.8667	0.8667	0.8000
5	0.8621	0.8667	0.8879	0.8603	0.7940
6	0.8276	0.8259	0.8370	0.8308	0.7411
7	0.7931	0.7963	0.8128	0.7902	0.6909
8	0.7241	0.7259	0.7706	0.7339	0.5887
9	0.8621	0.8556	0.8637	0.8566	0.7921
Mean	0.8369	0.8370	0.8561	0.8384	0.7557
SD	0.0490	0.0483	0.0402	0.0468	0.0728

	Accuracy	Recall	Prec.	F1	Kappa
0	0.8333	0.8333	0.8604	0.8379	0.7500
1	0.8667	0.8667	0.8727	0.8682	0.8000
2	0.8667	0.8667	0.9048	0.8704	0.8000
3	0.8667	0.8667	0.8788	0.8677	0.8000
4	0.8667	0.8667	0.8667	0.8667	0.8000
5	0.8276	0.8296	0.8527	0.8278	0.7420
6	0.8621	0.8593	0.8750	0.8644	0.7925
7	0.8621	0.8667	0.8793	0.8609	0.7940
8	0.7241	0.7259	0.7706	0.7339	0.5887
9	0.8621	0.8556	0.8637	0.8566	0.7921
Mean	0.8438	0.8437	0.8625	0.8454	0.7659
SD	0.0422	0.0415	0.0334	0.0395	0.0625

	Accuracy	Recall	Prec.	F1	Kappa
0	0.7333	0.7333	0.7607	0.7387	0.6000
1	0.8333	0.8333	0.8350	0.8329	0.7500
2	0.8667	0.8667	0.8674	0.8624	0.8000
3	0.8000	0.8000	0.8148	0.8038	0.7000
4	0.8333	0.8333	0.8485	0.8360	0.7500
5	0.7931	0.7963	0.8292	0.7890	0.6904
6	0.8621	0.8556	0.8695	0.8597	0.7921
7	0.6897	0.6926	0.6931	0.6749	0.5364
8	0.7586	0.7556	0.7686	0.7623	0.6381
9	0.7241	0.7222	0.7303	0.7260	0.5865
Mean	0.7894	0.7889	0.8017	0.7886	0.6844
SD	0.0579	0.0570	0.0573	0.0592	0.0864

	x4	x9	x11	x12	x15	x17	x18	x19	x22	x23	...	x72	x73	x74	x78	x80	x82	x83	status	Label	Score
0	-0.235880	-0.962146	0.303608	-0.186713	-0.273263	-0.207795	-1.845254	-0.005883	-0.449907	-1.406979	...	0.863079	1.862990	0.875683	0.662868	0.789812	0.713907	-0.025607	0	0	0.9896
1	0.294453	0.456432	0.500978	-0.186581	-0.270637	-0.205210	-0.285810	-0.050554	-0.068685	0.264763	...	0.113475	-0.111385	-0.150287	-0.180539	1.225575	-0.183361	-1.277243	2	1	0.9908
2	0.234019	1.956273	-1.621492	-0.186284	4.162752	-0.200972	-0.261811	-0.077996	-0.926736	0.901133	...	-0.256419	-0.544432	-0.065215	-0.246617	-0.672648	-0.376299	-1.666447	2	2	0.9869
3	0.232117	-1.710759	0.742225	-0.186687	-0.271420	-0.216866	-1.163790	-0.064286	-0.304550	-0.478708	...	1.421555	1.862951	0.552170	0.662770	-0.794697	0.663312	-0.062555	0	0	0.9968
4	-0.477883	-0.102368	-0.513372	-0.186612	-0.272248	-0.209478	-0.182208	-0.037135	-0.205190	0.026181	...	0.113475	-0.111385	-0.150287	-0.180539	1.225575	1.018293	0.804925	1	1	0.9850

	x4	x9	x11	x12	x15	x17	x18	x19	x22	x23	...	x72	x73	x74	x78	x80	x82	x83	status	Label	Score
0	-0.235880	-0.962146	0.303608	-0.186713	-0.273263	-0.207795	-1.845254	-0.005883	-0.449907	-1.406979	...	0.863079	1.862990	0.875683	0.662868	0.789812	0.713907	-0.025607	0	0	0.9993
1	0.294453	0.456432	0.500978	-0.186581	-0.270637	-0.205210	-0.285810	-0.050554	-0.068685	0.264763	...	0.113475	-0.111385	-0.150287	-0.180539	1.225575	-0.183361	-1.277243	2	1	0.9900
2	0.234019	1.956273	-1.621492	-0.186284	4.162752	-0.200972	-0.261811	-0.077996	-0.926736	0.901133	...	-0.256419	-0.544432	-0.065215	-0.246617	-0.672648	-0.376299	-1.666447	2	2	0.9991
3	0.232117	-1.710759	0.742225	-0.186687	-0.271420	-0.216866	-1.163790	-0.064286	-0.304550	-0.478708	...	1.421555	1.862951	0.552170	0.662770	-0.794697	0.663312	-0.062555	0	0	0.9995
4	-0.477883	-0.102368	-0.513372	-0.186612	-0.272248	-0.209478	-0.182208	-0.037135	-0.205190	0.026181	...	0.113475	-0.111385	-0.150287	-0.180539	1.225575	1.018293	0.804925	1	1	0.9852

	x4	x9	x11	x12	x15	x17	x18	x19	x22	x23	...	x72	x73	x74	x78	x80	x82	x83	status	Label	Score
0	-0.235880	-0.962146	0.303608	-0.186713	-0.273263	-0.207795	-1.845254	-0.005883	-0.449907	-1.406979	...	0.863079	1.862990	0.875683	0.662868	0.789812	0.713907	-0.025607	0	0	0.9974
1	0.294453	0.456432	0.500978	-0.186581	-0.270637	-0.205210	-0.285810	-0.050554	-0.068685	0.264763	...	0.113475	-0.111385	-0.150287	-0.180539	1.225575	-0.183361	-1.277243	2	1	1.0000
2	0.234019	1.956273	-1.621492	-0.186284	4.162752	-0.200972	-0.261811	-0.077996	-0.926736	0.901133	...	-0.256419	-0.544432	-0.065215	-0.246617	-0.672648	-0.376299	-1.666447	2	2	0.9940
3	0.232117	-1.710759	0.742225	-0.186687	-0.271420	-0.216866	-1.163790	-0.064286	-0.304550	-0.478708	...	1.421555	1.862951	0.552170	0.662770	-0.794697	0.663312	-0.062555	0	0	1.0000
4	-0.477883	-0.102368	-0.513372	-0.186612	-0.272248	-0.209478	-0.182208	-0.037135	-0.205190	0.026181	...	0.113475	-0.111385	-0.150287	-0.180539	1.225575	1.018293	0.804925	1	1	1.0000

	x4	x9	x11	x12	x15	x17	x18	x19	x22	x23	...	x72	x73	x74	x78	x80	x82	x83	status	Label	Score
0	-0.235880	-0.962146	0.303608	-0.186713	-0.273263	-0.207795	-1.845254	-0.005883	-0.449907	-1.406979	...	0.863079	1.862990	0.875683	0.662868	0.789812	0.713907	-0.025607	0	0	0.9315
1	0.294453	0.456432	0.500978	-0.186581	-0.270637	-0.205210	-0.285810	-0.050554	-0.068685	0.264763	...	0.113475	-0.111385	-0.150287	-0.180539	1.225575	-0.183361	-1.277243	2	1	0.7414
2	0.234019	1.956273	-1.621492	-0.186284	4.162752	-0.200972	-0.261811	-0.077996	-0.926736	0.901133	...	-0.256419	-0.544432	-0.065215	-0.246617	-0.672648	-0.376299	-1.666447	2	2	0.8973
3	0.232117	-1.710759	0.742225	-0.186687	-0.271420	-0.216866	-1.163790	-0.064286	-0.304550	-0.478708	...	1.421555	1.862951	0.552170	0.662770	-0.794697	0.663312	-0.062555	0	0	0.9752
4	-0.477883	-0.102368	-0.513372	-0.186612	-0.272248	-0.209478	-0.182208	-0.037135	-0.205190	0.026181	...	0.113475	-0.111385	-0.150287	-0.180539	1.225575	1.018293	0.804925	1	1	0.8137