Scripts

Post-Processing

CS_calculation_genmodel.py

        CS Dynamic Workflow

        A unified interface for computing and analyzing dynamic model coefficients.

        This tool provides three independent but interoperable sub-commands.
          - 'compute' — compute dynamic coefficient fields
             for selected SGS models and store them as NetCDF output files. (default)

          - 'vprof' — compute vertical profiles by reading the coefficient
             datasets produced by 'compute' and writing the results back to disk.

          - 'plot' — generate basic diagnostic plots from the outputs of 'compute';
             plots can be shown interactively or saved to disk.

        Can run the sub-command individually or together, provided the necessary
        input or intermediate files exist on disk. If no sub-command is given
        will run just 'compute'
        

cs_dynamic [-h] [-V {0,1,2,3}] [--h_resolution H_RESOLUTION] [--t_range T_RANGE T_RANGE]
           [--z_range Z_RANGE Z_RANGE] [-v] [--fname_suffix FNAME_SUFFIX]
           [--z_chunk_size Z_CHUNK_SIZE] [--t_chunk_size T_CHUNK_SIZE]
           [--h_chunk_size H_CHUNK_SIZE] [--mem_limit_MB MEM_LIMIT_MB]
           [--sgs_model {all,vel_all,theta_all,Smag_vel,Smag_vel_diag,Carati,Kosovic,Smag_theta,Smag_theta_diag} [{all,vel_all,theta_all,Smag_vel,Smag_vel_diag,Carati,Kosovic,Smag_theta,Smag_theta_diag} ...]]
           [--filter_type {box,gaussian}] [--filter_scales FILTER_SCALES [FILTER_SCALES ...]]
           [--regularize_filter_type {box,gaussian}]
           [--regularize_filter_scales REGULARIZE_FILTER_SCALES [REGULARIZE_FILTER_SCALES ...]]
           [--plot_show] [--plot_path PLOT_PATH]
           [{compute,vprof,plot} ...] input_files {um,monc,sgs} output_path

positional arguments

• commands - Workflow steps to execute (default: compute)

options

• -h, --help - show this help message and exit

• --sgs_model SGS_MODEL - Choice of models for which to compute dynamic coefficients. (default: all)

Package version

• -V GIT_VERSION, --git_version GIT_VERSION - show package version (0=installed, 1=+last commit, 2=+modified files, 3=+ full file difference) and exit (default: 0)

I/O datasets on disk

• input_files -

  Location of NetCDF diagnostic file(s). Recognizes glob patterns and walks directory trees, e.g. './my_file_p[br]*nc' All files in a glob pattern should belong to the same simulation.

• input_format - Type of 'input_files'. Only support different NetCDF flavours from various production codes. 'sgs' refers to files produced by sgs_tools.

• --h_resolution H_RESOLUTION -

  horizontal resolution in meters. ONLY used for UM ideal simulations (will use to overwrite horizontal coordinates).

  (default: 0)

• --t_range T_RANGE - time interval to consider, in code coordinates, negative value are interpreted as min/max respectively (default: [-1, -1])

• --z_range Z_RANGE - vertical interval to consider, in code coordinates, negative values are interpreted as take the min/max respectively (default: [-1, -1])

Output datasets on disk

• -v, --verbosity - verbosity level. Increase by -v , -vv, etc. (default: 0)

• output_path -

  Output directory, where to write netcdf output files. Will create/overwrite existing file and create any missing intermediate directories

• --fname_suffix FNAME_SUFFIX -

  Optional suffix appended to output filenames. Final pattern: <model_name>_<fname_suffix>_pp.nc where <model_name> is detemined by the sgs_model

  (default: )

Dask parameters

• --z_chunk_size Z_CHUNK_SIZE -

  Size of dask array chunks in the vertical direction. Should divide the total number of levels. Smaller size leads to smaller memory footprint, but may penalize walltime.

• --t_chunk_size T_CHUNK_SIZE -

  Size of dask array chunks in the time direction. Should divide the total number time snapshots. Smaller size leads to smaller memory footprint, but may penalize walltime.

• --h_chunk_size H_CHUNK_SIZE -

  Size of dask array chunks in the "horizontal" directions. Should divide into the domain size. Smaller size leads to smaller memory footprint, but can penalize walltime heavily (building DAGs). Default -1 means don’t chunk, keep dimesions together – useful for horizontal filtering. If it doesn’t fit in memory try 1/2 to 1/4 of domain size.

  (default: -1)

• --mem_limit_MB MEM_LIMIT_MB -

  Target memory limit in MB used to control chunk size of dask array chunks. If on a dask cluster it is May not be fully enforced by this CLI. Smaller size leads to smaller memory footprint, but may penalize walltime.

Compute dynamic coefficients fields

Compute dynamic coefficient fields for the selected SGS models, filtering and regularization kernels and scale. Save each model to separate NetCDF file.

• --filter_type FILTER_TYPE - Shape of filter kernel to use for scale separation. (default: box)

• --filter_scales FILTER_SCALES - Scales to perform filter at, in number of cells. If a single value is given, it will be usedfor all regularize_filter_scales. Otherwise, must provide as many values as for regularize_filter_scales (default: (2,))

• --regularize_filter_type REGULARIZE_FILTER_TYPE - Shape of filter kernel used for coefficient regularization. (default: box)

• --regularize_filter_scales REGULARIZE_FILTER_SCALES - Scales to perform regularization at, in number of cells. If a single value is given, it will be used for all filter_scale. Otherwise, must provide as many values as for filter_scale (default: (2,))

Plotting parameters

Generate basic diagnostic plots from the compute output. Plots can be shown interactively or saved to disk.

• --plot_show - flag to display generated plots

• --plot_path PLOT_PATH - output directory, for storing generated plots

post_process.py

Post process a simulation and save result as NetCDF files

post_process [-h] [-V {0,1,2,3}] [--h_resolution H_RESOLUTION] [--t_range T_RANGE T_RANGE]
             [--z_range Z_RANGE Z_RANGE] [-v] [--overwrite_existing]
             [--fname_suffix FNAME_SUFFIX] [--hdims HDIMS [HDIMS ...]] [--vertical_profiles]
             [--vprofile_fields VPROFILE_FIELDS [VPROFILE_FIELDS ...]]
             [--vprofile_stats VPROFILE_STATS [VPROFILE_STATS ...]]
             [--vprofile_fname_out VPROFILE_FNAME_OUT] [--horizontal_spectra]
             [--power_spectra_fields POWER_SPECTRA_FIELDS [POWER_SPECTRA_FIELDS ...]]
             [--cross_spectra_fields CROSS_SPECTRA_FIELDS [CROSS_SPECTRA_FIELDS ...]]
             [--radial_smooth_factor RADIAL_SMOOTH_FACTOR] [--radial_truncation]
             [--hspectra_fname_out HSPECTRA_FNAME_OUT] [--anisotropy]
             [--box_domain_scales BOX_DOMAIN_SCALES [BOX_DOMAIN_SCALES ...]]
             [--box_meter_scales BOX_METER_SCALES [BOX_METER_SCALES ...]]
             [--box_delta_scales BOX_DELTA_SCALES [BOX_DELTA_SCALES ...]]
             [--gauss_scales GAUSS_SCALES [GAUSS_SCALES ...]]
             [--filter_shapes FILTER_SHAPES [FILTER_SHAPES ...]]
             [--aniso_fname_out ANISO_FNAME_OUT] [--z_chunk_size Z_CHUNK_SIZE]
             [--t_chunk_size T_CHUNK_SIZE] [--h_chunk_size H_CHUNK_SIZE]
             [--mem_limit_MB MEM_LIMIT_MB]
             input_files {um,monc,sgs} output_path

options

• -h, --help - show this help message and exit

• --hdims HDIMS - List of horizontal dimension names. (default: ['x', 'y'])

Package version

• -V GIT_VERSION, --git_version GIT_VERSION - show package version (0=installed, 1=+last commit, 2=+modified files, 3=+ full file difference) and exit (default: 0)

I/O datasets on disk

• input_files -

  Location of NetCDF diagnostic file(s). Recognizes glob patterns and walks directory trees, e.g. './my_file_p[br]*nc' All files in a glob pattern should belong to the same simulation.

• input_format - Type of 'input_files'. Only support different NetCDF flavours from various production codes. 'sgs' refers to files produced by sgs_tools.

• --h_resolution H_RESOLUTION -

  horizontal resolution in meters. ONLY used for UM ideal simulations (will use to overwrite horizontal coordinates).

  (default: 0)

• --t_range T_RANGE - time interval to consider, in code coordinates, negative value are interpreted as min/max respectively (default: [-1, -1])

• --z_range Z_RANGE - vertical interval to consider, in code coordinates, negative values are interpreted as take the min/max respectively (default: [-1, -1])

Output datasets on disk

• -v, --verbosity - verbosity level. Increase by -v , -vv, etc. (default: 0)

• output_path -

  output directory, where to store post-processed results will create any missing intermediate directories

• --overwrite_existing - behaviour if diagnostic file already exists. default: skip

• --fname_suffix FNAME_SUFFIX - Optional suffix appended to output filenames. Final pattern: <base_name>_<fname_suffix>_<statistics_tag>.nc, where <base_name> and <statistics_tag> are detemined in the respective statistics group: Vertical profiles | Horizontal spectra | Anisotropy diagnostics (default: )

Vertical Profiles

• --vertical_profiles - Swtitch for computing vertical profiles.

• --vprofile_fields VPROFILE_FIELDS - List of fields whose vertical profile to compute. (default: ['vel', 'vel_horiz', 'theta', 's', 'vert_heat_flux', 'vert_mom_flux', 'fluct_ke', 'tke', 'smag_visc_m', 'smag_visc_h', 'smag_visc_m_vert', 'smag_visc_h_vert', 'csDelta', 'cs_theta', 'cs_diag', 'cs_theta_diag'])

• --vprofile_stats VPROFILE_STATS - Statistics to generate vertical profiles with. (default: ['mean', 'std', 'median'])

• --vprofile_fname_out VPROFILE_FNAME_OUT -

  Core filename where to save the generated vertical profile.

  relative to output_path. Will add an '.nc' extension (so don’t give one). The final pattern will be <vprofile_fname_out>_<fname_suffix>_vprof.nc

  (default: post_proc)

Horizontal spectra

• --horizontal_spectra - Horizontal power spectra and cross spectra.

• --power_spectra_fields POWER_SPECTRA_FIELDS - Fields whose power spectra to compute. (default: ['u', 'v', 'w', 'theta'])

• --cross_spectra_fields CROSS_SPECTRA_FIELDS -

  Fields whose cross spectra to compute. Use spaces to between cross spectra and commas between fields in each cross-spectrum, e.g. u,v v,w

  (default: [('u', 'w'), ('v', 'w'), ('u', 'v'), ('theta', 'w')])

• --radial_smooth_factor RADIAL_SMOOTH_FACTOR -

  Radial binning of radial horizontal spectrum

  in units of delta_kx spacings

  (default: 1)

• --radial_truncation -

  Truncation of radial horizontal spectrum. If True will disregard wavenumbers above the maximum linear wavenumber.

• --hspectra_fname_out HSPECTRA_FNAME_OUT -

  Core filename where to save the generated horisontal spectra.

  relative to output_path. Will add an '.nc' extension (so don’t give one). The final pattern will be <hspectra_fname_out>_<fname_suffix>_spectra.nc

  (default: post_proc)

Anisotropy diagnostics

• --anisotropy - Anisotropy of velocity strain and stress

• --box_domain_scales BOX_DOMAIN_SCALES -

  Anisotropy box filter and coarse-graining scales in fraction of the horizontal domain size. Will round to nearest integer number of horizontal grid cells. Will combine all box scales and ignore entries which are less than 2 delta apart.

  (default: [])

• --box_meter_scales BOX_METER_SCALES -

  Anisotropy box filter and coarse-graining scales in meters. Will round to nearest integer number of horizontal grid cells. Will combine all box scales and ignore entries which are less than 2 delta apart.

  (default: [])

• --box_delta_scales BOX_DELTA_SCALES -

  Anisotropy box filter and coarse-graining scales in units of horizontal grid spacing delta. Will combine all box scales and ignore entries which are less than 2 delta apart.

  (default: [])

• --gauss_scales GAUSS_SCALES -

  Anisotropy Gaussian filter scales in units of horizontal grid spacing.

  Support 2 and 4

  (default: [])

• --filter_shapes FILTER_SHAPES - Anisotropy filter shapes. Support any of ['gauss', 'box', 'coarse'] (default: ['gauss', 'box', 'coarse'])

• --aniso_fname_out ANISO_FNAME_OUT -

  Core filename where to save the generated anisotropy eigen values. relative to output_path. Will append the filter label. Will add an '.nc' extension (so don’t give one). The final pattern will be <aniso_fname_out>_<fname_suffix>_anisotropy.nc

  (default: post_proc)

Dask parameters

• --z_chunk_size Z_CHUNK_SIZE -

  Size of dask array chunks in the vertical direction. Should divide the total number of levels. Smaller size leads to smaller memory footprint, but may penalize walltime.

• --t_chunk_size T_CHUNK_SIZE -

  Size of dask array chunks in the time direction. Should divide the total number time snapshots. Smaller size leads to smaller memory footprint, but may penalize walltime.

• --h_chunk_size H_CHUNK_SIZE -

  Size of dask array chunks in the "horizontal" directions. Should divide into the domain size. Smaller size leads to smaller memory footprint, but can penalize walltime heavily (building DAGs). Default -1 means don’t chunk, keep dimesions together – useful for horizontal filtering. If it doesn’t fit in memory try 1/2 to 1/4 of domain size.

  (default: -1)

• --mem_limit_MB MEM_LIMIT_MB -

  Target memory limit in MB used to control chunk size of dask array chunks. If on a dask cluster it is May not be fully enforced by this CLI. Smaller size leads to smaller memory footprint, but may penalize walltime.

BasicComparisonSimAnalysis.py

Create (and optionally save) standard diagnostic plots for a dry atmospheric boundary layer UM simulation Best-suited to one-parameter suite of simulations, but can handle several varying parameters through plot_style_file

sim_comparison [-h] [-V {0,1,2,3}] [--h_resolution H_RESOLUTION [H_RESOLUTION ...]]
               [--times [TIMES ...]] [--z_range Z_RANGE Z_RANGE] [--plot_show]
               [--plot_path PLOT_PATH] [--plot_styles PLOT_STYLES]
               [--hor_slice_levels [HOR_SLICE_LEVELS ...]] [--skip_vert_profiles]
               [--skip_clouds] [--verbose] [--z_chunk_size Z_CHUNK_SIZE]
               [--t_chunk_size T_CHUNK_SIZE] [--h_chunk_size H_CHUNK_SIZE]
               [--mem_limit_MB MEM_LIMIT_MB]
               input_files [input_files ...] {um,monc,sgs}

options

• -h, --help - show this help message and exit

• --verbose - More granular command line output.

Package version

• -V GIT_VERSION, --git_version GIT_VERSION - show package version (0=installed, 1=+last commit, 2=+modified files, 3=+ full file difference) and exit (default: 0)

I/O datasets on disk

• input_files -

  Location of a set NetCDF simulation file(s). Recognizes glob patterns and walks directory trees, e.g. './my_file_p[br]*nc' Can have multiple files per simulation, but only one glob pattern per simulation. All files in a glob pattern should belong to the same simulation.

• input_format - Type of 'input_files'. Only support different NetCDF flavours from various production codes. 'sgs' refers to files produced by sgs_tools. All simulations must have the same format

• --h_resolution H_RESOLUTION -

  horizontal resolution in meters. ONLY used for UM ideal simulations (will use to overwrite horizontal coordinates). If a single resolution is given, assume it applies to all input files. Else, must give as many resolutions as inpu_file glob patterns.

  (default: [0])

• --times TIMES -

  times at which to perform the analysis; in code coordinates; will find nearest available match. default (which is empty) means the full data range at 1h intervals.

  (default: [])

• --z_range Z_RANGE - vertical interval to consider, in code coordinates, negative values are interpreted as take the min/max respectively (default: [-1, -1])

Plotting parameters

• --plot_show - flag to display generated plots

• --plot_path PLOT_PATH - output directory, for storing generated plots

• --plot_styles PLOT_STYLES -

  JSON configuration describing a list of plot styles and decorations matched sequentially to each input simulation. Can pass as a json-compatible string, but better a path to a JSON file. See plot_config_template.json for a template. If absent, will use default_plotting_style and cycle through different colors.

• --hor_slice_levels HOR_SLICE_LEVELS -

  Vertical height at which to plot horizontal slices. If not given will omit these plots.

  (default: [])

• --skip_vert_profiles - skip vertical profiles from plotting

• --skip_clouds - skip cloud plot

Dask parameters

• --z_chunk_size Z_CHUNK_SIZE -

  Size of dask array chunks in the vertical direction. Should divide the total number of levels. Smaller size leads to smaller memory footprint, but may penalize walltime.

• --t_chunk_size T_CHUNK_SIZE -

  Size of dask array chunks in the time direction. Should divide the total number time snapshots. Smaller size leads to smaller memory footprint, but may penalize walltime.

• --h_chunk_size H_CHUNK_SIZE -

  Size of dask array chunks in the "horizontal" directions. Should divide into the domain size. Smaller size leads to smaller memory footprint, but can penalize walltime heavily (building DAGs). Default -1 means don’t chunk, keep dimesions together – useful for horizontal filtering. If it doesn’t fit in memory try 1/2 to 1/4 of domain size.

  (default: -1)

• --mem_limit_MB MEM_LIMIT_MB -

  Target memory limit in MB used to control chunk size of dask array chunks. If on a dask cluster it is May not be fully enforced by this CLI. Smaller size leads to smaller memory footprint, but may penalize walltime.

ReferenceComparisonSimAnalysis.py

Create (and optionally save) standard diagnostic plots for a dry atmospheric boundary layer UM simulation Best-suited to one-parameter suite of simulations, but can handle several varying parameters through plot_style_file

ref_comparison [-h] [-V {0,1,2,3}] [--h_resolution H_RESOLUTION [H_RESOLUTION ...]]
               [--times [TIMES ...]] [--z_range Z_RANGE Z_RANGE] [--plot_show]
               [--plot_path PLOT_PATH] [--only_diff] [--plot_styles PLOT_STYLES]
               [--hor_slice_levels [HOR_SLICE_LEVELS ...]] [--skip_vert_profiles]
               [--skip_clouds] [--verbose] [--z_chunk_size Z_CHUNK_SIZE]
               [--t_chunk_size T_CHUNK_SIZE] [--h_chunk_size H_CHUNK_SIZE]
               [--mem_limit_MB MEM_LIMIT_MB]
               target reference {um,monc,sgs}

options

• -h, --help - show this help message and exit

• --verbose - More granular command line output.

Package version

• -V GIT_VERSION, --git_version GIT_VERSION - show package version (0=installed, 1=+last commit, 2=+modified files, 3=+ full file difference) and exit (default: 0)

I/O datasets on disk

• target -

  Location of target simulation outputs – UM NetCDF diagnostic files. Recognizes glob patterns and walks directory trees, e.g. './my_file_p[br]*nc' Can have multiple files, but only one glob pattern. (All files in a glob pattern should belong to the simulation).

• reference -

  Location of reference simulation outputs – UM NetCDF diagnostic files. Recognizes glob patterns and walks directory trees, e.g. './my_file_p[br]*nc'. Can have multiple files, but only one glob pattern. (All files in a glob pattern should belong to the simulation).

• input_format - Type of 'input_files'. Only support different NetCDF flavours from various production codes. 'sgs' refers to files produced by sgs_tools. All simulations must have the same format

• --h_resolution H_RESOLUTION -

  horizontal resolution in meters. ONLY used for UM ideal simulations (will use to overwrite horizontal coordinates). If a single resolution is given, assume it applies to all input files. Else, must give as many resolutions as inpu_file glob patterns.

  (default: [0])

• --times TIMES -

  times at which to perform the analysis; in code coordinates; will find nearest available match. default (which is empty) means the full data range at 1h intervals.

  (default: [])

• --z_range Z_RANGE - vertical interval to consider, in code coordinates, negative values are interpreted as take the min/max respectively (default: [-1, -1])

Plotting parameters

• --plot_show - flag to display generated plots

• --plot_path PLOT_PATH - output directory, for storing generated plots

• --only_diff -

  Make only difference and relative difference figures. Skip overplotting comparison figures.

• --plot_styles PLOT_STYLES -

  JSON configuration describing a list of plot styles and decorations matched sequentially to (1) target and (2) reference. Can pass as a json-compatible string, but better a path to a JSON file. See plot_config_template.json for a template. If absent, will use default_plotting_style.

• --hor_slice_levels HOR_SLICE_LEVELS -

  Vertical height at which to plot horizontal slices. If not given will omit these plots.

  (default: [])

• --skip_vert_profiles - skip vertical profiles from plotting

• --skip_clouds - skip cloud plot

Dask parameters

• --z_chunk_size Z_CHUNK_SIZE -

  Size of dask array chunks in the vertical direction. Should divide the total number of levels. Smaller size leads to smaller memory footprint, but may penalize walltime.

• --t_chunk_size T_CHUNK_SIZE -

  Size of dask array chunks in the time direction. Should divide the total number time snapshots. Smaller size leads to smaller memory footprint, but may penalize walltime.

• --h_chunk_size H_CHUNK_SIZE -

  Size of dask array chunks in the "horizontal" directions. Should divide into the domain size. Smaller size leads to smaller memory footprint, but can penalize walltime heavily (building DAGs). Default -1 means don’t chunk, keep dimesions together – useful for horizontal filtering. If it doesn’t fit in memory try 1/2 to 1/4 of domain size.

  (default: -1)

• --mem_limit_MB MEM_LIMIT_MB -

  Target memory limit in MB used to control chunk size of dask array chunks. If on a dask cluster it is May not be fully enforced by this CLI. Smaller size leads to smaller memory footprint, but may penalize walltime.

Pre-processing

make_UM_level_sets.py

Create a constant level set namelist for the UM from given model top height and number of levels

um_levels [-h] [-V {0,1,2,3}] output_file z_top_of_model n_z_rho

positional arguments

• output_file - output path, will create/overwrite existing file and create any missing intermediate directories

• z_top_of_model - top of model a.k.a. last theta level in meters

• n_z_rho - number of rho-levels, will have one more theta level

options

• -h, --help - show this help message and exit

Package version

• -V GIT_VERSION, --git_version GIT_VERSION - show package version (0=installed, 1=+last commit, 2=+modified files, 3=+ full file difference) and exit (default: 0)

Miscellaneous

version.py

Show package version

sgs_tools_version [-h] [-v] [--raw]

options

• -h, --help - show this help message and exit

• --raw -

  Output directory, where to write netcdf output files. Will create/overwrite existing file and create any missing intermediate directories

Output

• -v, --verbosity - verbosity level. Increase by -v , -vv, etc. (default: 0)

Utilities:

class VersionAction(option_strings, dest, nargs=None, const=None, default=None, type=None, choices=None, required=False, help=None, metavar=None)

print_header(title, version_verbosity=1)

Print a consistent, centered CLI header.

Return type:

None

configure_matplotlib_backend(argv=None, strict=False)

Configure matplotlib backend based on presence of --plot_show in argv or sys.argv

Must be called before any import of matplotlib.pyplot.

Parameters:

• argv (Iterable | None) – argument list. If None will look at sys.argv instead

• strict (bool) – if True will raise a RuntimeError if it detects pyplot in sys.modules. Otherwise just prints a Warning.