Compile troubles when trying latest FMS 2024.01.02

[kshedstrom]

I thought I would try compiling on chinook again, with the latest FMS/FMScoupler. I've also merged in a recent CEFI MOM6. Something isn't playing nice:

//import/c1/AKWATERS/kate/ESMG/ESMG-configs/src/MOM6/config_src/external/GFDL_ocean_BGC/generic_tracer_utils.F90:10:7:

   10 |   use MOM_diag_mediator, only : g_diag_ctrl=>diag_ctrl
      |       1
Fatal Error: Mismatch in components of derived type ‘__vtype_C26A70’ from ‘fms_diag_time_reduction_mod’ at (1): expecting ‘do_time_averaging’, but got ‘copy’
compilation terminated.
make[1]: *** [Makefile:465: generic_tracer_utils.o] Error 1
make[1]: *** Waiting for unfinished jobs....
//import/c1/AKWATERS/kate/ESMG/ESMG-configs/src/MOM6/src/parameterizations/stochastic/MOM_stochastics.F90:17:5:

   17 | use mpp_domains_mod,         only : domain2d, mpp_get_layout, mpp_get_global_domain
      |     1
Fatal Error: Mismatch in components of derived type ‘domain1d’ from ‘mpp_domains_mod’ at (1): expecting ‘data’, but got ‘domain_data’
compilation terminated.
make[1]: *** [Makefile:319: MOM_stochastics.o] Error 1
//import/c1/AKWATERS/kate/ESMG/ESMG-configs/src/MOM6/src/user/Neverworld_initialization.F90:17:5:

   17 | use random_numbers_mod, only: initializeRandomNumberStream, getRandomNumbers, randomNumberStream
      |     1
Fatal Error: Mismatch in components of derived type ‘domain1d’ from ‘mpp_domains_mod’ at (1): expecting ‘data’, but got ‘domain_data’
compilation terminated.

[kshedstrom]

Arg, Marshall had it right, old FMS module files. Now I'm getting

FATAL from PE     0: NetCDF: Name contains illegal characters: netcdf_add_variable: file:./ocean_geometry.nc variable:lath

[marshallward]

Anything odd in ocean_geometry.nc? Maybe we are passing null characters or something bad in the metadata? A backtrace to this error might also help.

[kshedstrom]

Full stack trace:

/import/c1/AKWATERS/kate/ESMG/MOM6-examples/ocean_only/resting/z/ocean.stats.gnu: FATAL from PE     0: NetCDF: Unknown file format: netcdf_file_open:./ocean_geometry.nc
/import/c1/AKWATERS/kate/ESMG/MOM6-examples/ocean_only/resting/z/ocean.stats.gnu: 
/import/c1/AKWATERS/kate/ESMG/MOM6-examples/ocean_only/resting/z/ocean.stats.gnu: #0  0x2c3abbe in __mpp_mod_MOD_mpp_error_basic
/import/c1/AKWATERS/kate/ESMG/MOM6-examples/ocean_only/resting/z/ocean.stats.gnu: 	at //import/c1/AKWATERS/kate/ESMG/ESMG-configs/src/FMS2/mpp/include/mpp_util_mpi.inc:74
/import/c1/AKWATERS/kate/ESMG/MOM6-examples/ocean_only/resting/z/ocean.stats.gnu: #1  0x2015b46 in __fms_io_utils_mod_MOD_error
/import/c1/AKWATERS/kate/ESMG/MOM6-examples/ocean_only/resting/z/ocean.stats.gnu: 	at //import/c1/AKWATERS/kate/ESMG/ESMG-configs/src/FMS2/fms2_io/fms_io_utils.F90:190
/import/c1/AKWATERS/kate/ESMG/MOM6-examples/ocean_only/resting/z/ocean.stats.gnu: #2  0x1ed1d2e in __netcdf_io_mod_MOD_check_netcdf_code
/import/c1/AKWATERS/kate/ESMG/MOM6-examples/ocean_only/resting/z/ocean.stats.gnu: 	at //import/c1/AKWATERS/kate/ESMG/ESMG-configs/src/FMS2/fms2_io/netcdf_io.F90:386
/import/c1/AKWATERS/kate/ESMG/MOM6-examples/ocean_only/resting/z/ocean.stats.gnu: #3  0x1ecfcb9 in __netcdf_io_mod_MOD_netcdf_file_open
/import/c1/AKWATERS/kate/ESMG/MOM6-examples/ocean_only/resting/z/ocean.stats.gnu: 	at //import/c1/AKWATERS/kate/ESMG/ESMG-configs/src/FMS2/fms2_io/netcdf_io.F90:665
/import/c1/AKWATERS/kate/ESMG/MOM6-examples/ocean_only/resting/z/ocean.stats.gnu: #4  0x2a2f975 in __fms_netcdf_domain_io_mod_MOD_open_domain_file
/import/c1/AKWATERS/kate/ESMG/MOM6-examples/ocean_only/resting/z/ocean.stats.gnu: 	at //import/c1/AKWATERS/kate/ESMG/ESMG-configs/src/FMS2/fms2_io/fms_netcdf_domain_io.F90:389
/import/c1/AKWATERS/kate/ESMG/MOM6-examples/ocean_only/resting/z/ocean.stats.gnu: #5  0x100c057 in __mom_io_infra_MOD_mom_file_exists
/import/c1/AKWATERS/kate/ESMG/MOM6-examples/ocean_only/resting/z/ocean.stats.gnu: 	at //import/c1/AKWATERS/kate/ESMG/ESMG-configs/src/MOM6/config_src/infra/FMS2/MOM_io_infra.F90:152
/import/c1/AKWATERS/kate/ESMG/MOM6-examples/ocean_only/resting/z/ocean.stats.gnu: #6  0x100a539 in __mom_io_infra_MOD_open_file
/import/c1/AKWATERS/kate/ESMG/MOM6-examples/ocean_only/resting/z/ocean.stats.gnu: 	at //import/c1/AKWATERS/kate/ESMG/ESMG-configs/src/MOM6/config_src/infra/FMS2/MOM_io_infra.F90:336
/import/c1/AKWATERS/kate/ESMG/MOM6-examples/ocean_only/resting/z/ocean.stats.gnu: #7  0x578878 in __mom_io_file_MOD_open_file_infra
/import/c1/AKWATERS/kate/ESMG/MOM6-examples/ocean_only/resting/z/ocean.stats.gnu: 	at //import/c1/AKWATERS/kate/ESMG/ESMG-configs/src/MOM6/src/framework/MOM_io_file.F90:941
/import/c1/AKWATERS/kate/ESMG/MOM6-examples/ocean_only/resting/z/ocean.stats.gnu: #8  0xe9a65c in __mom_io_MOD_create_mom_file
/import/c1/AKWATERS/kate/ESMG/MOM6-examples/ocean_only/resting/z/ocean.stats.gnu: 	at //import/c1/AKWATERS/kate/ESMG/ESMG-configs/src/MOM6/src/framework/MOM_io.F90:344
/import/c1/AKWATERS/kate/ESMG/MOM6-examples/ocean_only/resting/z/ocean.stats.gnu: #9  0x44f553 in __mom_shared_initialization_MOD_write_ocean_geometry_file
/import/c1/AKWATERS/kate/ESMG/MOM6-examples/ocean_only/resting/z/ocean.stats.gnu: 	at //import/c1/AKWATERS/kate/ESMG/ESMG-configs/src/MOM6/src/initialization/MOM_shared_initialization.F90:1420
/import/c1/AKWATERS/kate/ESMG/MOM6-examples/ocean_only/resting/z/ocean.stats.gnu: #10  0x199fdf6 in __mom_MOD_initialize_mom
/import/c1/AKWATERS/kate/ESMG/MOM6-examples/ocean_only/resting/z/ocean.stats.gnu: 	at //import/c1/AKWATERS/kate/ESMG/ESMG-configs/src/MOM6/src/core/MOM.F90:2887
/import/c1/AKWATERS/kate/ESMG/MOM6-examples/ocean_only/resting/z/ocean.stats.gnu: #11  0x1346b95 in mom6
/import/c1/AKWATERS/kate/ESMG/MOM6-examples/ocean_only/resting/z/ocean.stats.gnu: 	at //import/c1/AKWATERS/kate/ESMG/ESMG-configs/src/MOM6/config_src/drivers/solo_driver/MOM_driver.F90:280
/import/c1/AKWATERS/kate/ESMG/MOM6-examples/ocean_only/resting/z/ocean.stats.gnu: #12  0x1349059 in main
/import/c1/AKWATERS/kate/ESMG/MOM6-examples/ocean_only/resting/z/ocean.stats.gnu: 	at //import/c1/AKWATERS/kate/ESMG/ESMG-configs/src/MOM6/config_src/drivers/solo_driver/MOM_driver.F90:27

[kshedstrom]

Sorry, that was the stack trace the second time around, when it tries to check the existing ocean_geometry.nc and finds that it's not a complete netCDF file. If you start without the file, it gives:

FATAL: NetCDF: Name contains illegal characters: netcdf_add_variable: file:./ocean_geometry.nc variable:lath

#0  0x2c3abbe in __mpp_mod_MOD_mpp_error_basic
	at //import/c1/AKWATERS/kate/ESMG/ESMG-configs/src/FMS2/mpp/include/mpp_util_mpi.inc:74
#1  0x2015b46 in __fms_io_utils_mod_MOD_error
	at //import/c1/AKWATERS/kate/ESMG/ESMG-configs/src/FMS2/fms2_io/fms_io_utils.F90:190
#2  0x1ed1d2e in __netcdf_io_mod_MOD_check_netcdf_code
	at //import/c1/AKWATERS/kate/ESMG/ESMG-configs/src/FMS2/fms2_io/netcdf_io.F90:386
#3  0x1ec9676 in __netcdf_io_mod_MOD_netcdf_add_variable
	at //import/c1/AKWATERS/kate/ESMG/ESMG-configs/src/FMS2/fms2_io/netcdf_io.F90:1002
#4  0x2a2d761 in __fms_netcdf_domain_io_mod_MOD_register_domain_variable
	at //import/c1/AKWATERS/kate/ESMG/ESMG-configs/src/FMS2/fms2_io/fms_netcdf_domain_io.F90:548
#5  0xfe975c in __mom_io_infra_MOD_write_metadata_axis
	at //import/c1/AKWATERS/kate/ESMG/ESMG-configs/src/MOM6/config_src/infra/FMS2/MOM_io_infra.F90:1935
#6  0x578500 in __mom_io_file_MOD_register_axis_infra
	at //import/c1/AKWATERS/kate/ESMG/ESMG-configs/src/MOM6/src/framework/MOM_io_file.F90:1005
#7  0xe9d031 in __mom_io_MOD_create_mom_file
	at //import/c1/AKWATERS/kate/ESMG/ESMG-configs/src/MOM6/src/framework/MOM_io.F90:432
#8  0x44f553 in __mom_shared_initialization_MOD_write_ocean_geometry_file
	at //import/c1/AKWATERS/kate/ESMG/ESMG-configs/src/MOM6/src/initialization/MOM_shared_initialization.F90:1420
#9  0x199fdf6 in __mom_MOD_initialize_mom
	at //import/c1/AKWATERS/kate/ESMG/ESMG-configs/src/MOM6/src/core/MOM.F90:2887
#10  0x1346b95 in mom6
	at //import/c1/AKWATERS/kate/ESMG/ESMG-configs/src/MOM6/config_src/drivers/solo_driver/MOM_driver.F90:280
#11  0x1349059 in main
	at //import/c1/AKWATERS/kate/ESMG/ESMG-configs/src/MOM6/config_src/drivers/solo_driver/MOM_driver.F90:27

[kshedstrom]

Line 336 in the original stack trace is:
if ((file_mode == APPEND_FILE) .and. file_exists(filename_tmp, MOM_Domain)) then
and it is checking file_exists even if the mode is overwrite.

[kshedstrom]

All is well with 2024.01, broken with 2024.01.02.