How to ensure labels accuracy in DPA-2 distillation

[scott-5]

In the usage paradigm of DPA2, pre-trained models are more suitable for the finetune+distillation mode, but in the distillation step, how can we ensure that the student model is provided with sufficient accuracy labels by finetuned pre-trained model?

The plan I envisioned is that:
Perhaps we can calculate manually model_devi these labels using finetuned models with different initial seed. Assuming that some frames in these labels have insufficient accuracy, then put them in the dataset and finetune again. This loop may go through many times. (Yes, I have already done so, but strangely, some predicted-well previously frames has become inaccurate)

Overall, the benefit of finetune+distillation may be that it saves time in DFT calculations. But I feel that the time required for a complete finetuned model may be comparable with the time required for DFT calculations, or even more.
So, I have another question is that the application scenario of DPA-2 does not include obtaining a high-precision and efficient dp model for production?

[wanghan-iapcm]

1. The fine-tuning is conducted by a concurrent learning (CL) workflow. The fine-tuned model, at the convergence of the CL, is assumed to be of uniform accuracy in the application. At this stage the configurations are labeled by DFT.
2. The distilled model is supposed to learn the fine-tuned model by a CL workflow. At this stage the configurations are labeled by the fine-tuned model.
3. the DPA-2 model has much higher generalizability than the DPA-1 model thus the data requirement in 1. is much less than 2. thus computational cost for labeling is saved.

[scott-5]

Thanks for your reply. My confusion focus on 1..

Though DPA-2 has powerful generalizability and requires few DFT labdels, but very little phase space is covered, which makes people worried about the accuracy of the data.

Of course, we can use CL to complete labels the fine-tuned model needed. However fine-tuning models 1)has low inference efficiency in LAMMPS, resulting in a long time consumption for exploring new representative phase spaces on a large scale. 2)has a slower training speed, about dozens of times.

So if we don't use CL, it's difficult to ensure the accuracy of the labels, but if do, then whether the consumption of iterative 1) and 2) is much less than the saved DFT consumption still needs to be evaluated.

It is undeniable that if our goal is to generate a rough model, DPA-2 can almost save the majority of DFT consumption. I speculate whether DPA-2 is more suitable for like this scenarios. Also it is similar to ChatGPT, where it always answers people with a rough answer, and then humans accept the answer based on their own thinking.

[wanghan-iapcm]

typically the majority of the computational cost is spend on the labeling stage. by using DPA-2 the save in the labeling stage is 1-3 orders of magnitudes. the extra requirement in the training and exploration stage is usually acceptable.

In general i cannot answer how much would you save by using dpa-2 in a CL, that depends on the cost of training, exploration and labeling stages. you may test and report in the discussion.