Skip to content

matbench_v0.1: MODNet (v0.1.12)

Algorithm description:

MODNet, the Materials Optimal Descriptor Network (v0.1.12). A feed-forward neural network, using all compatible matminer features and a relevance-redundancy based feature selection algorithm. Hyperparameter optimisation is performed with a nested grid search for the 9 smaller tasks, and with a genetic algorithm for the 4 larger tasks (matbench_perovskites, matbench_mp_gap, matbench_mp_is_metal, matbench_mp_eform. Benchmark results were loaded from https://github.com/ml-evs/modnet-matbench/releases/tag/v0.4.0, archived at 10.5281/zenodo.5109941. This latest benchmark uses an improved GA-based hyperparameter optimization.

Notes:

None

Raw data download and example notebook available on the matbench repo.

References (in bibtex format):

('@article{De_Breuck_2021, doi = {10.1088/1361-648x/ac1280}, url = '
 '{https://doi.org/10.1088/1361-648x/ac1280}, year = 2021, month = {jul}, '
 'publisher = {{IOP} Publishing}, volume = {33}, number = {40}, pages = '
 '{404002}, author = {Pierre-Paul De Breuck and Matthew L Evans and Gian-Marco '
 'Rignanese}, title = {Robust model benchmarking and bias-imbalance in '
 'data-driven materials science: a case study on {MODNet}}, journal = {Journal '
 'of Physics: Condensed Matter}, abstract = {As the number of novel '
 'data-driven approaches to material science continues to grow, it is crucial '
 'to perform consistent quality, reliability and applicability assessments of '
 'model performance. In this paper, we benchmark the Materials Optimal '
 'Descriptor Network (MODNet) method and architecture against the recently '
 'released MatBench v0.1, a curated test suite of materials datasets. MODNet '
 'is shown to outperform current leaders on 6 of the 13 tasks, while closely '
 'matching the current leaders on a further 2 tasks; MODNet performs '
 'particularly well when the number of samples is below 10\xa0000. Attention '
 'is paid to two topics of concern when benchmarking models. First, we '
 'encourage the reporting of a more diverse set of metrics as it leads to a '
 'more comprehensive and holistic comparison of model performance. Second, an '
 'equally important task is the uncertainty assessment of a model towards a '
 'target domain. Significant variations in validation errors can be observed, '
 'depending on the imbalance and bias in the training set (i.e., similarity '
 'between training and application space). By using an ensemble MODNet model, '
 'confidence intervals can be built and the uncertainty on individual '
 'predictions can be quantified. Imbalance and bias issues are often '
 'overlooked, and yet are important for successful real-world applications of '
 'machine learning in materials science and condensed matter.}}, '
 '@article{DeBreuck2021, doi = {10.1038/s41524-021-00552-2}, url = '
 '{https://doi.org/10.1038/s41524-021-00552-2}, year = {2021}, month = jun, '
 'publisher = {Springer Science and Business Media {LLC}}, volume = {7}, '
 'number = {1}, author = {Pierre-Paul De Breuck and Geoffroy Hautier and '
 'Gian-Marco Rignanese}, title = {Materials property prediction for limited '
 'datasets enabled by feature selection and joint learning with {MODNet}}, '
 'journal = {npj Computational Materials}}')

User metadata:

{}

Metadata:

tasks recorded 13/13
complete?
composition complete?
structure complete?
regression complete?
classification complete?

Software Requirements

{'python': ['modnet==0.1.12', 'matbench==0.2.0']}

Task data:

matbench_dielectric

Fold scores
fold mae rmse mape* max_error
fold_0 0.1691 0.6273 0.0541 14.3880
fold_1 0.2410 1.0270 0.0786 18.1817
fold_2 0.3899 2.9174 0.0759 59.1179
fold_3 0.2775 2.2353 0.0535 52.1521
fold_4 0.2781 1.6090 0.0762 28.0821
Fold score stats
metric mean max min std
mae 0.2711 0.3899 0.1691 0.0714
rmse 1.6832 2.9174 0.6273 0.8221
mape* 0.0677 0.0786 0.0535 0.0113
max_error 34.3844 59.1179 14.3880 18.0529
Fold parameters
fold params dict
fold_0 {}
fold_1 {}
fold_2 {}
fold_3 {}
fold_4 {}

matbench_expt_gap

Fold scores
fold mae rmse mape* max_error
fold_0 0.3121 0.7028 0.3201 5.7101
fold_1 0.3388 0.7483 0.3066 6.8526
fold_2 0.3763 0.8917 0.4073 9.8955
fold_3 0.3121 0.7313 0.3327 6.0927
fold_4 0.3241 0.7684 0.3777 6.9757
Fold score stats
metric mean max min std
mae 0.3327 0.3763 0.3121 0.0239
rmse 0.7685 0.8917 0.7028 0.0653
mape* 0.3489 0.4073 0.3066 0.0377
max_error 7.1053 9.8955 5.7101 1.4723
Fold parameters
fold params dict
fold_0 {}
fold_1 {}
fold_2 {}
fold_3 {}
fold_4 {}

matbench_expt_is_metal

Fold scores
fold accuracy balanced_accuracy f1 rocauc
fold_0 0.9269 0.9269 0.9255 0.9269
fold_1 0.9136 0.9136 0.9121 0.9136
fold_2 0.9177 0.9177 0.9173 0.9177
fold_3 0.9177 0.9177 0.9169 0.9177
fold_4 0.9045 0.9045 0.9049 0.9045
Fold score stats
metric mean max min std
accuracy 0.9161 0.9269 0.9045 0.0073
balanced_accuracy 0.9161 0.9269 0.9045 0.0072
f1 0.9153 0.9255 0.9049 0.0068
rocauc 0.9161 0.9269 0.9045 0.0072
Fold parameters
fold params dict
fold_0 {}
fold_1 {}
fold_2 {}
fold_3 {}
fold_4 {}

matbench_glass

Fold scores
fold accuracy balanced_accuracy f1 rocauc
fold_0 0.9789 0.9743 0.9851 0.9743
fold_1 0.9701 0.9618 0.9790 0.9618
fold_2 0.9639 0.9539 0.9747 0.9539
fold_3 0.9621 0.9545 0.9733 0.9545
fold_4 0.9710 0.9570 0.9798 0.9570
Fold score stats
metric mean max min std
accuracy 0.9692 0.9789 0.9621 0.0059
balanced_accuracy 0.9603 0.9743 0.9539 0.0075
f1 0.9784 0.9851 0.9733 0.0042
rocauc 0.9603 0.9743 0.9539 0.0075
Fold parameters
fold params dict
fold_0 {}
fold_1 {}
fold_2 {}
fold_3 {}
fold_4 {}

matbench_jdft2d

Fold scores
fold mae rmse mape* max_error
fold_0 25.5515 61.1616 18.6554 487.7418
fold_1 30.4506 74.1756 0.1995 366.3580
fold_2 45.1925 134.5285 0.5392 871.3962
fold_3 26.9801 58.2340 0.2126 318.7579
fold_4 37.7845 155.5663 0.4803 1564.8245
Fold score stats
metric mean max min std
mae 33.1918 45.1925 25.5515 7.3428
rmse 96.7332 155.5663 58.2340 40.3638
mape* 4.0174 18.6554 0.1995 7.3203
max_error 721.8157 1564.8245 318.7579 464.0333
Fold parameters
fold params dict
fold_0 {}
fold_1 {}
fold_2 {}
fold_3 {}
fold_4 {}

matbench_log_gvrh

Fold scores
fold mae rmse mape* max_error
fold_0 0.0731 0.1089 0.0576 0.9014
fold_1 0.0738 0.1111 0.0579 1.1745
fold_2 0.0731 0.1101 0.0587 0.9076
fold_3 0.0738 0.1115 0.0567 0.9225
fold_4 0.0718 0.1101 0.0560 0.8007
Fold score stats
metric mean max min std
mae 0.0731 0.0738 0.0718 0.0007
rmse 0.1103 0.1115 0.1089 0.0009
mape* 0.0574 0.0587 0.0560 0.0009
max_error 0.9413 1.1745 0.8007 0.1243
Fold parameters
fold params dict
fold_0 {}
fold_1 {}
fold_2 {}
fold_3 {}
fold_4 {}

matbench_log_kvrh

Fold scores
fold mae rmse mape* max_error
fold_0 0.0536 0.1013 0.0356 1.5366
fold_1 0.0559 0.1079 0.0366 1.2998
fold_2 0.0510 0.0949 0.0340 1.1808
fold_3 0.0585 0.1126 0.0418 1.1355
fold_4 0.0549 0.1046 0.0370 1.3202
Fold score stats
metric mean max min std
mae 0.0548 0.0585 0.0510 0.0025
rmse 0.1043 0.1126 0.0949 0.0060
mape* 0.0370 0.0418 0.0340 0.0026
max_error 1.2946 1.5366 1.1355 0.1397
Fold parameters
fold params dict
fold_0 {}
fold_1 {}
fold_2 {}
fold_3 {}
fold_4 {}

matbench_mp_e_form

Fold scores
fold mae rmse mape* max_error
fold_0 0.0402 0.0817 0.3786 4.0438
fold_1 0.0497 0.1018 0.3121 4.8803
fold_2 0.0475 0.0905 0.2562 1.6230
fold_3 0.0464 0.0889 0.3515 1.5189
fold_4 0.0400 0.0812 0.2882 3.3787
Fold score stats
metric mean max min std
mae 0.0448 0.0497 0.0400 0.0039
rmse 0.0888 0.1018 0.0812 0.0075
mape* 0.3173 0.3786 0.2562 0.0436
max_error 3.0889 4.8803 1.5189 1.3281
Fold parameters
fold params dict
fold_0 {}
fold_1 {}
fold_2 {}
fold_3 {}
fold_4 {}

matbench_mp_gap

Fold scores
fold mae rmse mape* max_error
fold_0 0.2147 0.4441 2.8966 5.0558
fold_1 0.2161 0.4484 2.6899 6.2874
fold_2 0.2165 0.4433 4.1912 7.5685
fold_3 0.2309 0.4705 4.6749 6.9325
fold_4 0.2211 0.4564 4.9590 4.9406
Fold score stats
metric mean max min std
mae 0.2199 0.2309 0.2147 0.0059
rmse 0.4525 0.4705 0.4433 0.0101
mape* 3.8823 4.9590 2.6899 0.9248
max_error 6.1570 7.5685 4.9406 1.0299
Fold parameters
fold params dict
fold_0 {}
fold_1 {}
fold_2 {}
fold_3 {}
fold_4 {}

matbench_mp_is_metal

Fold scores
fold accuracy balanced_accuracy f1 rocauc
fold_0 0.9169 0.9135 0.9028 0.9135
fold_1 0.9030 0.8995 0.8867 0.8995
fold_2 0.9131 0.9096 0.8984 0.9096
fold_3 0.8874 0.8849 0.8699 0.8849
fold_4 0.9140 0.9116 0.9003 0.9116
Fold score stats
metric mean max min std
accuracy 0.9069 0.9169 0.8874 0.0108
balanced_accuracy 0.9038 0.9135 0.8849 0.0106
f1 0.8916 0.9028 0.8699 0.0122
rocauc 0.9038 0.9135 0.8849 0.0106
Fold parameters
fold params dict
fold_0 {}
fold_1 {}
fold_2 {}
fold_3 {}
fold_4 {}

matbench_perovskites

Fold scores
fold mae rmse mape* max_error
fold_0 0.0932 0.1304 0.0970 0.8705
fold_1 0.0939 0.1283 0.1058 1.0063
fold_2 0.0861 0.1216 0.0939 0.9432
fold_3 0.0892 0.1274 0.0995 0.8501
fold_4 0.0914 0.1310 0.0894 1.1780
Fold score stats
metric mean max min std
mae 0.0908 0.0939 0.0861 0.0028
rmse 0.1277 0.1310 0.1216 0.0033
mape* 0.0971 0.1058 0.0894 0.0055
max_error 0.9696 1.1780 0.8501 0.1180
Fold parameters
fold params dict
fold_0 {}
fold_1 {}
fold_2 {}
fold_3 {}
fold_4 {}

matbench_phonons

Fold scores
fold mae rmse mape* max_error
fold_0 34.7662 87.4531 0.0580 1079.1280
fold_1 36.3582 75.3959 0.0631 640.3050
fold_2 36.5373 71.7215 0.0636 575.7557
fold_3 32.1725 61.8200 0.0658 456.9764
fold_4 31.5413 53.9441 0.0592 396.9667
Fold score stats
metric mean max min std
mae 34.2751 36.5373 31.5413 2.0781
rmse 70.0669 87.4531 53.9441 11.5011
mape* 0.0619 0.0658 0.0580 0.0029
max_error 629.8264 1079.1280 396.9667 240.4189
Fold parameters
fold params dict
fold_0 {}
fold_1 {}
fold_2 {}
fold_3 {}
fold_4 {}

matbench_steels

Fold scores
fold mae rmse mape* max_error
fold_0 103.5846 185.4940 0.0659 1121.0504
fold_1 72.3160 96.1504 0.0534 394.0216
fold_2 80.5134 117.5962 0.0562 452.9860
fold_3 81.6766 133.7020 0.0587 711.4582
fold_4 100.7231 190.9186 0.0777 932.3040
Fold score stats
metric mean max min std
mae 87.7627 103.5846 72.3160 12.2188
rmse 144.7722 190.9186 96.1504 37.4511
mape* 0.0624 0.0777 0.0534 0.0087
max_error 722.3641 1121.0504 394.0216 276.9541
Fold parameters
fold params dict
fold_0 {}
fold_1 {}
fold_2 {}
fold_3 {}
fold_4 {}