-
Notifications
You must be signed in to change notification settings - Fork 62
Parallel Execution Tutorial
This demo is based on a clone of ngen at commit 70545d1
More general information on building ngen can be found in Building.
This tutorial assumes that the ngen dependencies are installed and in sane enough location that standard cmake3 can find them.
Additional dependencies required by this tutorial are python, SQLite3, NetCDF, and OpenMPI (refer to Building for more details on optional and required dependencies.)
git clone https://github.com/noaa-owp/ngen
cd ngengit submodule update --init --recursive -- test/googletest
git submodule update --init --recursive -- extern/pybind11
git submodule update --init --recursive -- extern/cfe
git submodule update --init --recursive -- extern/noah-owp-modular/noah-owp-modular
git submodule update --init --recursive -- extern/bmi-cxx
git submodule update --init --recursive -- extern/sloth
git submodule update --init --recursive -- extern/evapotranspiration/cmake -B extern/cfe/cmake_build -S extern/cfe/cfe/ -DNGEN=ON
make -C extern/cfe/cmake_build
cmake -B extern/noah-owp-modular/cmake_build -S extern/noah-owp-modular -DNGEN_IS_MAIN_PROJECT=ON
make -C extern/noah-owp-modular/cmake_build
cmake -B extern/sloth/cmake_build -S extern/sloth
make -C extern/sloth/cmake_build
cmake -B extern/evapotranspiration/cmake_build -S extern/evapotranspiration/evapotranspiration/
make -C extern/evapotranspiration/cmake_buildpython3 -m venv venv
source venv/bin/activateNote this demo used the following python version
(venv) nels ngen % python --version
Python 3.11.8For t-route, need to downgrade pip for now... (see t-route issue #621)
pip install -U pip==23.0pip install -r extern/test_bmi_py/requirements.txtcd ..
git clone https://github.com/noaa-owp/t-route
cd t-route
pip install -r requirements.txtOn my system, I need to point $NETCDF to the location of the netcdf.mod file for netcdf fortran. I also have to provide a custom $LIBRARY_PATH which informs the extension builds of where to find libgfortran and libnetcdf and libnetcdff libraries to link.
You may need to edit these ENV variables for your environment.
The no-e flag installs t-route into the venv directly and doesn't use editable installs.
# verified to work with netcdf-fortran 4.6.1 and gcc 13.2.0
NETCDF=$(brew --prefix netcdf-fortran)/include \
LIBRARY_PATH=$(brew --prefix gcc)/lib/gcc/current/:$(brew --prefix)/lib:$LIBRARY_PATH \
./compiler.sh no-ecd ../ngen
cmake -B cmake_build -S . \
-DNGEN_WITH_PYTHON:BOOL=ON \
-DNGEN_WITH_BMI_C:=ON \
-DNGEN_WITH_BMI_FORTRAN:BOOL=ON \
-DUDUNITS_QUIET:BOOL=ON \
-DNGEN_WITH_NETCDF:=ON \
-DNGEN_WITH_SQLITE:=ON \
-DNGEN_WITH_ROUTING:=ON \
-DNGEN_UPDATE_GIT_SUBMODULES:=OFF \
-DNGEN_WITH_EXTERN_TOPMODEL:=OFF \
-DNGEN_WITH_MPI:=ON
make -j 8 -C cmake_buildcd cmake_build; ctest; cd ..Remove the artifacts from the t-route test and run these tests again. May need to do this twice to get both routing tests passing in isolation.
rm test/data/routing/*.parquet
cd cmake_build; ctest --rerun-failed; cd ..Symlink the following binaries from the previous build. In this case, I have created a demo directory to run simulations in
mkdir demo
cd demo
ln -s ../cmake_build/ngen .
ln -s ../cmake_build/partitionGenerator .In the demo folder I have a vpu hydrofabric of huc 01, nextgen_01.gpkg.
Note
This can be replaced by any hydrofabric geopackage which has 8 or more catchments in it.
Just replace nextgen_01 with your hydrofabric file name in the following steps.
./partitionGenerator nextgen_01.gpkg nextgen_01.gpkg huc_1_partitions.json 8 "" ""Reading 19194 features from layer divides using ID column `divide_id`
Partitioning 19194 catchments into 8 partitions.This creates the parallel partition file huc_1_partitions.json.
For more information on the partition generator, see partition‐generator.
Note that stream_output_dir is the output directory in the routing_config.yaml and if it doesn't exist, the routing step will throw an error. Similarly, the output_dir below is the output_root defined in the realization.json input to ngen.
mkdir output_dir
mkdir stream_output_dirCreate a simple forcing file for demonstration purposes
cp ../data/gauge_01073000/forcing/cat-11410.csv ./forcing.csv The supplementary section contains the various definitions of the input files used for this demo. To run the following simulations, your demo directory should look like:
demo
├── bmi_configs
│ ├── cfe.ini
│ ├── noah.namelist
│ ├── parameters
│ │ ├── GENPARM.TBL
│ │ ├── MPTABLE.TBL
│ │ └── SOILPARM.TBL
│ └── pet.ini
├── forcing.csv
├── huc_1_partitions.json
├── nextgen_01.gpkg
├── ngen -> ../cmake_build/ngen
├── output_dir
├── partitionGenerator -> ../cmake_build/partitionGenerator
├── realization.json
├── routing_config.yaml
└── stream_output_dir
Run the configured simulation using 8 processors
mpirun -n 8 ./ngen nextgen_01.gpkg all nextgen_01.gpkg all realization.json huc_1_partitions.jsonThis takes ~ 9 minutes on a 16 core M2 MacBook.
The bmi configuration files are the same for ALL catchments. A more realistic realization would configure the various init_config keys in the realization.json using a pattern, e.g.
"init_config": "./bmi_configs/pet/pet_{{id}}.ini"Similarly, the forcings applied to each catchment in this example come from a single file, forcings.csv and are applied to each catchment in the domain. A more realistic realization could use a similar pattern substitution as seen above, or it could use the NetCDF and provide a single NetCDF file with each data for each catchment by id. An example NetCDF file can be found here.
"forcing": {
"path": "./forcing.nc",
"provider": "NetCDF"
}Named realization.json in the top level demo directory. This realization configures
-
slothto provide static inputs to cfe (from a C++ BMI libaray) -
noah-owp-modularto provide land surface outputs (from a Fortran BMI library) -
petto provide evapotranspiration from (from a C BMI library) -
cfeto provide rainfall runoff hydrologoy (from a C BMI library)
as the formulation used by all catchments in the hydrofabric. It runs a 720 hour (30 day) hourly simulation, forcing all catchments with the data from forcing.csv and puts the formulation outputs in output_dir. Finally, it configures a routing simulation (via the routing Python module) to run at the end of the rainfall runoff simulations and passes the routing module the routing_config.yaml.
More details on ngen realization configurations can be found in Realization Config and details on BMI formulations can be reviewd in Formulations-and-BMI
realization.json
{
"global": {
"formulations": [
{
"name": "bmi_multi",
"params": {
"model_type_name": "bmi_multi_noahowp_cfe",
"forcing_file": "",
"init_config": "",
"allow_exceed_end_time": true,
"main_output_variable": "Q_OUT",
"modules": [
{
"name": "bmi_c++",
"params": {
"model_type_name": "bmi_c++_sloth",
"library_file": "../extern/sloth/cmake_build/libslothmodel",
"init_config": "/dev/null",
"allow_exceed_end_time": true,
"main_output_variable": "z",
"uses_forcing_file": false,
"model_params": {
"sloth_ice_fraction_schaake(1,double,m,node)": 0.0,
"sloth_ice_fraction_xinanjiang(1,double,1,node)": 0.0,
"sloth_smp(1,double,1,node)": 0.0
}
}
},
{
"name": "bmi_fortran",
"params": {
"model_type_name": "bmi_fortran_noahowp",
"library_file": "../extern/noah-owp-modular/cmake_build/libsurfacebmi",
"forcing_file": "",
"init_config": "./bmi_configs/noah.namelist",
"allow_exceed_end_time": true,
"main_output_variable": "QINSUR",
"variables_names_map": {
"PRCPNONC": "atmosphere_water__liquid_equivalent_precipitation_rate",
"Q2": "atmosphere_air_water~vapor__relative_saturation",
"SFCTMP": "land_surface_air__temperature",
"UU": "land_surface_wind__x_component_of_velocity",
"VV": "land_surface_wind__y_component_of_velocity",
"LWDN": "land_surface_radiation~incoming~longwave__energy_flux",
"SOLDN": "land_surface_radiation~incoming~shortwave__energy_flux",
"SFCPRS": "land_surface_air__pressure"
},
"uses_forcing_file": false
}
},
{
"name": "bmi_c",
"params": {
"model_type_name": "bmi_c_pet",
"library_file": "../extern/evapotranspiration/cmake_build/libpetbmi",
"forcing_file": "",
"init_config": "./bmi_configs/pet.ini",
"allow_exceed_end_time": true,
"main_output_variable": "water_potential_evaporation_flux",
"registration_function": "register_bmi_pet",
"variables_names_map": {
"water_potential_evaporation_flux": "potential_evapotranspiration"
},
"uses_forcing_file": false
}
},
{
"name": "bmi_c",
"params": {
"model_type_name": "bmi_c_cfe",
"library_file": "../extern/cfe/cmake_build/libcfebmi",
"forcing_file": "",
"init_config": "./bmi_configs/cfe.ini",
"allow_exceed_end_time": true,
"main_output_variable": "Q_OUT",
"registration_function": "register_bmi_cfe",
"variables_names_map": {
"water_potential_evaporation_flux": "potential_evapotranspiration",
"atmosphere_water__liquid_equivalent_precipitation_rate": "QINSUR",
"ice_fraction_schaake" : "sloth_ice_fraction_schaake",
"ice_fraction_xinanjiang" : "sloth_ice_fraction_xinanjiang",
"soil_moisture_profile" : "sloth_smp"
},
"uses_forcing_file": false
}
}
],
"uses_forcing_file": false
}
}
],
"forcing": {
"path": "./forcing.csv",
"provider": "CsvPerFeature"
}
},
"time": {
"start_time": "2015-12-01 00:00:00",
"end_time": "2015-12-30 23:00:00",
"output_interval": 3600
},
"output_root": "./output_dir/",
"routing": {
"t_route_config_file_with_path": "./routing_config.yaml"
}
}Named routing_config.yaml in the top level demo directory
More details on current ngen and t-route integration can be found in Routing
routing_config.yaml
#--------------------------------------------------------------------------------
log_parameters:
#----------
showtiming: True
log_level : DEBUG
#--------------------------------------------------------------------------------
network_topology_parameters:
#----------
supernetwork_parameters:
#----------
network_type: HYFeaturesNetwork
geo_file_path: ./nextgen_01.gpkg
columns:
key: 'id'
downstream: 'toid'
dx : 'length_m'
n : 'n'
ncc : 'nCC'
s0 : 'So'
bw : 'BtmWdth'
waterbody : 'rl_NHDWaterbodyComID'
gages : 'rl_gages'
tw : 'TopWdth'
twcc : 'TopWdthCC'
musk : 'MusK'
musx : 'MusX'
cs : 'ChSlp'
alt: 'alt'
mainstem: 'mainstem'
#duplicate_wb_segments: None
waterbody_parameters:
#----------
break_network_at_waterbodies: False
#--------------------------------------------------------------------------------
compute_parameters:
#----------
parallel_compute_method: by-subnetwork-jit-clustered #serial
compute_kernel : V02-structured
assume_short_ts : True
subnetwork_target_size : 10000
cpu_pool : 1
restart_parameters:
#----------
start_datetime: "2015-12-01 00:00:00"
forcing_parameters:
#----------
qts_subdivisions : 12
dt : 300 # [sec]
qlat_input_folder : ./output_dir/
qlat_file_pattern_filter : "nex-*"
binary_nexus_file_folder : #./ #NOTE: If memory issues arise while preprocessing forcing data, use this to create hourly binary files for forcing data.
nts : 288 #288 for 1day
max_loop_size : 24 # [hr]
data_assimilation_parameters:
#----------
usgs_timeslices_folder : ./usgs_TimeSlice/
usace_timeslices_folder : #usace_TimeSlice/
timeslice_lookback_hours : #48
qc_threshold : #1
streamflow_da:
#----------
streamflow_nudging: False
reservoir_da:
#----------
reservoir_persistence_da:
#----------
reservoir_persistence_usgs : False
reservoir_persistence_usace: False
reservoir_rfc_da:
#----------
reservoir_rfc_forecasts: False
#--------------------------------------------------------------------------------
output_parameters:
#----------
stream_output :
stream_output_directory: ./stream_output_dir/
stream_output_time: 1 #[hr]
stream_output_type: '.nc' #please select only between netcdf '.nc' or '.csv' or '.pkl'
stream_output_internal_frequency: 60 #[min] it should be order of 5 minutes. For instance if you want to output every hour put 60 In this demo, these should go in a directory named bmi_configs alongside the
realization and routing configs above. Note that these configuartions are valid, but are not realistic. They were copied from the ngen test data configs.
cfe.ini
forcing_file=BMI
surface_partitioning_scheme=Schaake
soil_params.depth=2.0[m]
soil_params.b=5.0[]
soil_params.satdk=4.413e-06[m s-1]
soil_params.satpsi=0.112305152[m]
soil_params.slop=0.021551854[m/m]
soil_params.smcmax=0.436131508[m/m]
soil_params.wltsmc=0.042880427[m/m]
soil_params.expon=1.0[]
soil_params.expon_secondary=1.0[]
refkdt=0.555369953
max_gw_storage=0.012863189697[m]
Cgw=1.8e-05[m h-1]
expon=7.0[]
gw_storage=0.35[m/m]
alpha_fc=0.33
soil_storage=0.35[m/m]
K_nash=0.03[]
K_lf=0.01[]
nash_storage=0.0,0.0
num_timesteps=1
verbosity=1
DEBUG=0
giuh_ordinates=0.64,0.36noah.namelist
&timing ! and input/output paths
dt = 3600.0 ! timestep [seconds]
startdate = "200710010000" ! UTC time start of simulation (YYYYMMDDhhmm)
enddate = "201912310000" ! UTC time end of simulation (YYYYMMDDhhmm)
forcing_filename = "./forcing/cat-12.csv" ! file containing forcing data
output_filename = "out_cat-12.csv"
/
¶meters
parameter_dir = "./bmi_configs/parameters//" ! location of input parameter files
general_table = "GENPARM.TBL" ! general param tables and misc params
soil_table = "SOILPARM.TBL" ! soil param table
noahowp_table = "MPTABLE.TBL" ! model param tables (includes veg)
soil_class_name = "STAS" ! soil class data source - "STAS" or "STAS-RUC"
veg_class_name = "USGS" ! vegetation class data source - "USGS" or "USGS"
/
&location ! for point runs
lat = 43.162699841798855 ! latitude [degrees] (-90 to 90)
lon = -71.01223053521309 ! longitude [degrees] (-180 to 180)
terrain_slope = 0.0 ! terrain slope [degrees]
azimuth = 0.0 ! terrain azimuth or aspect [degrees clockwise from north]
/
&forcing
ZREF = 10.0 ! measurement height for wind speed (m)
rain_snow_thresh = 1.0 ! rain-snow temperature threshold (degrees Celcius)
/
&model_options ! see OptionsType.f90 for details
precip_phase_option = 1
snow_albedo_option = 1
dynamic_veg_option = 4
runoff_option = 3
drainage_option = 8
frozen_soil_option = 1
dynamic_vic_option = 1
radiative_transfer_option = 3
sfc_drag_coeff_option = 1
canopy_stom_resist_option = 1
crop_model_option = 0
snowsoil_temp_time_option = 3
soil_temp_boundary_option = 2
supercooled_water_option = 1
stomatal_resistance_option = 1
evap_srfc_resistance_option = 4
subsurface_option = 2
/
&structure
isltyp = 3 ! soil texture class
nsoil = 4 ! number of soil levels
nsnow = 3 ! number of snow levels
nveg = 27 ! number of vegetation types
vegtyp = 15 ! vegetation type
croptype = 0 ! crop type (0 = no crops; this option is currently inactive)
sfctyp = 1 ! land surface type, 1:soil, 2:lake
soilcolor = 4 ! soil color code
/
&initial_values
dzsnso = 0.0, 0.0, 0.0, 0.1, 0.3, 0.6, 1.0 ! level thickness [m]
sice = 0.0, 0.0, 0.0, 0.0 ! initial soil ice profile [m3/m3]
sh2o = 0.3, 0.3, 0.3, 0.3 ! initial soil liquid profile [m3/m3]
zwt = -2.0 ! initial water table depth below surface [m]
/
pet.ini
verbose=0
pet_method=3
forcing_file=BMI
run_unit_tests=0
yes_aorc=1
yes_wrf=0
wind_speed_measurement_height_m=10.0
humidity_measurement_height_m=2.0
vegetation_height_m=16.0
zero_plane_displacement_height_m=0.0003
momentum_transfer_roughness_length=0.0
heat_transfer_roughness_length_m=0.0
surface_longwave_emissivity=1.0
surface_shortwave_albedo=0.17
cloud_base_height_known=FALSE
latitude_degrees=43.162699841798855
longitude_degrees=-71.01223053521309
site_elevation_m=64
time_step_size_s=3600
num_timesteps=720
shortwave_radiation_provided=1In addtion to these configs, the noah.namelist assumes a parameters subdirectory in the bmi_configs directory which includes the noah-mp parameter tables. These can be copied from the ngen test data
or from the demo folder, they can be copied from the ngen example data
cp -r ../data/gauge_01073000/NOAH/parameters bmi_configs/Tutorial
Getting Started
Configuration
Technical References