Different results during training and testing for an approximate GP model

[pengzhi1998]

Hi, I'm having an issue of different evaluation results during training and testing.

Here is the defined GPModel:

class GPModel(gpytorch.models.ApproximateGP):
    def __init__(self, inducing_points, use_ard=True):
        variational_distribution = gpytorch.variational.NaturalVariationalDistribution(inducing_points.size(0))
        # variational_strategy = gpytorch.variational.CiqVariationalStrategy(
        #     self, inducing_points, variational_distribution, learn_inducing_locations=True
        # )
        variational_strategy = gpytorch.variational.VariationalStrategy(
            self, inducing_points, variational_distribution, learn_inducing_locations=True
        )
        super(GPModel, self).__init__(variational_strategy)
        ard_num_dims = None
        if use_ard:
            ard_num_dims = inducing_points.shape[-1]
        self.mean_module = gpytorch.means.ZeroMean()
        self.covar_module = gpytorch.kernels.ScaleKernel(gpytorch.kernels.RBFKernel(ard_num_dims=ard_num_dims))
    def forward(self, x):
        mean_x = self.mean_module(x)
        covar_x = self.covar_module(x)
        return gpytorch.distributions.MultivariateNormal(mean_x, covar_x)

I used: model.load_state_dict(checkpoint['model_state_dict']) to load the model and used preds = model(x_batch) for prediction, and found all the model's parameters are the same in training and testing (which means the model has been successfully saved and loaded). And I checked the input x in forward function and the value of mean_x and covar_x.evaluate() but they are all the same between training and testing scenarios. But strangely, the outputs of forward are different.

What might be the problem which leads to different results? Would gpytorch.distributions.MultivariateNormal(mean_x, covar_x) lead to similar issues?

But during training, I found this warning: NumericalWarning: Negative variance values detected. This is likely due to numerical instabilities. Rounding negative variances up to 1e-10., would this be the reason to the problem?
(Update: I checked this, when there is no warning of negative variances during training, the distribution's values are still different during tests between training and testing. This is so wired problem.) Or, is it a normal phenomenon that the performance during testing would be worse than evaluations during training for the variational model (when using the same test data and both are in evaluation mode)?

Look forward to your reply and thank you for your time and help!