The descriptor "se_atten_v2" performs worse than "se_a."

[jpqiu]

I trained four potential functions using the same dataset with descriptors "se_a" and "se_atten_v2," with the only difference being in the descriptor settings. When I re-evaluated them on the same dataset, I found that "se_atten_v2" performed worse than "se_a." The curve for "se_atten_v2" was not as smooth. Why did this happen?

The following is my "param.json" file. Since I only extracted a subset of the dataset for testing, this subset actually only contains four elements: C, Cu, H, and N.

{
"type_map": [
"C",
"Cu",
"H",
"N",
"O",
"S"
],
"mass_map": [
12.011,
63.546,
1.0080,
14.007,
15.999,
32.06
],
"init_data_prefix": "/data/jpqiu/0_project/delta_test/0_dpa/00.init_data",
"init_data_sys": [
"06.dpa_DMDA",
"07.dpa_DMDA",
"dpa2_DMDA",
"dpa2_manual_DMDA",
"dp_opt_DMDA",
"manual_DMDA",
"multi_DMDA",
"opt_DMDA"
],
"init_batch_size": [
1, 1, 1, 1, 1, 1, 1, 1
],
"sys_configs": [
],
"sys_batch_size": [
],
"charges": [
],
"uks": [
],
"_comment": " that's all ",
"numb_models": 4,
"train_param": "input.json",
"default_training_param": {
"model": {
"descriptor": {
"type": "se_atten_v2",
"rcut_smth": 0.50,
"rcut": 6.00,
"sel": "auto",
"neuron": [25, 50, 100],
"axis_neuron": 16,
"resnet_dt": false,
"attn": 128,
"attn_layer": 2,
"attn_mask": false,
"attn_dotr": true,
"precision": "float64",
"seed": 1
},
"fitting_net": {
"neuron": [
240,
240,
240
],
"resnet_dt": true,
"precision": "float64",
"seed": 1
}
},
"learning_rate": {
"type": "exp",
"start_lr": 0.001,
"decay_steps": 2500,
"_comment": "nope",
"decay_rate": 0.95
},
"loss": {
"start_pref_e": 0.02,
"limit_pref_e": 1,
"start_pref_f": 1000,
"limit_pref_f": 1,
"start_pref_v": 0,
"limit_pref_v": 0
},
"training": {
"systems": [],
"set_prefix": "set",
"stop_batch": 500000,
"batch_size": 1,
"seed": 1,
"disp_file": "lcurve.out",
"disp_freq": 100,
"numb_test": 10,
"save_freq": 1000,
"save_ckpt": "model.ckpt",
"disp_training": true,
"time_training": true,
"profiling": false,
"profiling_file": "timeline.json"
}
},

[njzjz]

We fixed a bug for se_atten_v2 in #2905. Do you use the latest version?

[jpqiu]

I would like to ask you another question. If I haven't set up a test set, for example, if I have a total of 2000 datasets, is there any difference in training between putting all 2000 datasets in one folder and separating them into 10 folders? Does it affect the training?

[wanghan-iapcm]

The ML model has no ability for out-of-distribution generalization.
it is quite normal to see a large generalization error if you try to generalize on a system with complete new additive molecule that has never been seen in the training dataset. (May not always true if you have a large enough training dataset, but it is not your case.)

[jpqiu]

Thank you very much for your response. I would like to ask you another question.
If I haven't set up a test set, for example, if I have a total of 2000 datasets, is there any difference in training between putting all 2000 datasets in one folder and separating them into 10 folders? Does it affect the training?
If I want to prevent overfitting, should I consider removing structures with similar energies?

[jpqiu]

The figure above is an energy prediction chart where I have combined all the datasets into a single init_data_sys for training.

The following figure shows that I have divided the dataset into 8 init_data_sys, with a total of 1390 data points. At this point, I am using datasets from the same system.

[wanghan-iapcm]

how may sets in a data system should not have significant affect on the model accuracy. It is recommended dividing the data into training data and validation data, and supervising the gap between them to check the over-fitting.