Pipeline

class prism.Pipeline(modellink_obj, *, root_dir=None, working_dir=None, prefix=None, prism_par=None, emul_type=None, comm=None, **kwargs)[source]

Defines the Pipeline class of the PRISM package.

The Pipeline class is the main user class of the PRISM package and provides a user-friendly environment that gives access to all operations within the pipeline.

_Projection__analyze_proj_hcube(hcube)

Analyzes an emulator projection hypercube hcube.

Parameters:hcube (1D array_like of int of length {2, 3}) – Array containing the internal integer identifiers of the main model parameters that require a projection hypercube.
Returns:
  • impl_min_hcube (1D ndarray object) – List containing the lowest implausibility value that can be reached in every single grid point on the given hypercube.
  • impl_los_hcube (1D ndarray object) – List containing the fraction of the total amount of evaluated samples in every single grid point on the given hypercube, that still satisfied the implausibility cut-off criterion.
_Projection__draw_2D_proj_fig(hcube)

Draws the 2D projection figure for the provided hcube.

Parameters:hcube (1D array_like of int of length 2) – Array containing the internal integer identifiers of the main model parameters that require a projection figure.
_Projection__draw_3D_proj_fig(hcube)

Draws the 3D projection figure for the provided hcube.

Parameters:hcube (1D array_like of int of length 3) – Array containing the internal integer identifiers of the main model parameters that require a projection figure.
_Projection__get_default_input_arguments()

Generates a dict containing default values for all input arguments.

Returns:kwargs_dict (dict) – Dict containing all default input argument values.
_Projection__get_default_parameters()

Generates a dict containing default values for all projection parameters.

Returns:par_dict (dict) – Dict containing all default projection parameter values.
_Projection__get_fig_path(hcube)

Determines the absolute path of a projection figure corresponding to a provided projection hypercube hcube and returns it.

Parameters:hcube (1D array_like of int of length {2, 3} or str) – Array containing the internal integer identifiers of the main model parameters that require a projection hypercube. If str, the name of hcube instead (_Projection__get_hcube_name()).
Returns:
  • fig_path (str) – The absolute path to the requested projection figure.
  • fig_path_s (str) – The absolute path to the smoothed version.
_Projection__get_grid_idx(hcube)

Returns the indices of all parameters that are in the grid points of the given projection hypercube hcube.

Parameters:hcube (1D array_like of int of length {2, 3}) – Array containing the internal integer identifiers of the main model parameters that require a projection hypercube.
Returns:grid_idx (list of int) – Indices of all grid point parameters.
_Projection__get_hcube_name(hcube)

Determines the name of a provided projection hypercube hcube and returns it.

Parameters:hcube (1D array_like of int of length {2, 3}) – Array containing the internal integer identifiers of the main model parameters that require a projection hypercube.
Returns:hcube_name (str) – The name of this projection hypercube.
_Projection__get_proj_data(hcube)

Returns the projection data belonging to the provided hypercube hcube.

Parameters:hcube (1D array_like of int of length {2, 3}) – Array containing the internal integer identifiers of the main model parameters that require a projection hypercube.
Returns:
  • impl_min_hcube (1D ndarray object) – List containing the lowest implausibility value that can be reached in every single grid point on the given hypercube.
  • impl_los_hcube (1D ndarray object) – List containing the fraction of the total amount of evaluated samples in every single grid point on the given hypercube, that still satisfied the implausibility cut-off criterion.
  • proj_res (int) – Number of emulator evaluations used to generate the grid for the given hypercube.
  • proj_depth (int) – Number of emulator evaluations used to generate the samples in every grid point for the given hypercube.
  • proj_space (2D ndarray object) – The boundaries of the hypercube that encloses the parameter space in which the specified projection is defined.
_Projection__get_proj_hcube(hcube)

Generates a projection hypercube hcube containing emulator evaluation samples The output of this function depends on the requested projection type.

Parameters:hcube (1D array_like of int of length {2, 3}) – Array containing the internal integer identifiers of the main model parameters that require a projection hypercube.
Returns:proj_hcube (3D ndarray object) – 3D projection hypercube of emulator evaluation samples. For 3D projections, the grid uses matrix indexing (second parameter varies the fastest).
_Projection__get_proj_space(emul_i)

Returns the boundaries of the hypercube that encloses the parameter space in which the projection space of the provided emulator iteration emul_i is defined.

Parameters:emul_i (int) – Number indicating the requested emulator iteration.
Returns:proj_space (2D ndarray object) – The requested hypercube boundaries.
_Projection__get_req_hcubes(emul_i, proj_par)

Determines which projection hypercubes have been requested by the user. Also checks if these projection hypercubes have been calculated before, and depending on the value of force, either skips them or recreates them.

Parameters:
  • emul_i (int) – Number indicating the requested emulator iteration.
  • proj_par (1D array_like of {int; str} or None) – For which model parameters to construct the projection figures. If 1D array_like, construct projection figures for all combinations of provided model parameters that are active, with a string referring to the name of the model parameter and an integer referring to the position in which the model parameter is shown in the details() method. If None, projection figures are made for all active model parameters.

Generates

hcubes : list of lists
List containing the parameter indices of the requested projection hypercubes.
create_hcubes : list of lists
List containing the parameter indices of the requested projection hypercubes that need to be created first.
_Projection__prepare_projections(emul_i, proj_par, **kwargs)

Prepares the pipeline for the creation of the requested projections.

Parameters:
  • emul_i (int or None) – Number indicating the requested emulator iteration.
  • proj_par (1D array_like of {int; str} or None) – For which model parameters to construct the projection figures. If 1D array_like, construct projection figures for all combinations of provided model parameters that are active, with a string referring to the name of the model parameter and an integer referring to the position in which the model parameter is shown in the details() method. If None, projection figures are made for all active model parameters.
  • kwargs (keyword arguments) – Keyword arguments that were provided to project().
_Projection__process_input_arguments(emul_i, **kwargs)

Processes the input arguments given to the project() method.

Parameters:
  • emul_i (int or None) – Number indicating the requested emulator iteration.
  • kwargs (keyword arguments) – Keyword arguments that were provided to project().

Generates

All default and provided arguments are saved to their respective properties.

_Projection__read_proj_space(hcube_group)

Reads in and transforms the projection parameter space hypercube that is stored in the provided hcube_group.

Parameters:hcube_group (Group object) – The HDF5-group from which the projection parameter space hypercube needs to be read in.
Returns:proj_space (2D ndarray object) – The read-in hypercube boundaries.
_Projection__save_data(emul_i, data_dict)

Saves a given data dict {keyword: data} at the emulator iteration this class was initialized for, to the HDF5-file.

Parameters:
  • emul_i (int) – Number indicating the requested emulator iteration.
  • data_dict (dict) – Dict containing the data that needs to be saved to the HDF5-file.
Keyword Arguments:
 
  • keyword ({‘nD_proj_hcube’}) – String specifying the type of data that needs to be saved.
  • data ({int; float; str; array_like} or dict) – The actual data that needs to be saved at data keyword keyword. If dict, save every item individually.

Generates

The specified data is saved to the HDF5-file.

_Projection__set_parameters()

Sets the Projection parameters from the prism_dict property and saves them in the current Projection instance.

__call__(emul_i=None, *, force=False)[source]

Calls the construct() method to start the construction of the given iteration of the emulator and creates the projection figures right afterward if this construction was successful.

Other Parameters:
 
  • emul_i (int or None. Default: None) – If int, number indicating the requested emulator iteration. If None, the next iteration of the emulator will be constructed.
  • force (bool. Default: False) – Controls what to do if the specified emulator iteration emul_i already (partly) exists. If False, finish construction of the specified iteration or skip it if already finished. If True, reconstruct the specified iteration entirely.
__init__(modellink_obj, *, root_dir=None, working_dir=None, prefix=None, prism_par=None, emul_type=None, comm=None, **kwargs)[source]

Initialize an instance of the Pipeline class.

Parameters:

modellink_obj (ModelLink object) – Instance of the ModelLink subclass that links the emulated model to this Pipeline instance.

Other Parameters:
 
  • root_dir (str or None. Default: None) – String containing the absolute path of the root directory where all working directories are stored. If None, root directory will be set to the directory this class was initialized in.
  • working_dir (str, bool or None. Default: None) – String containing the name of the working directory of the emulator in root_dir. If True, a new working directory will be created in root_dir. If None or False, working directory is set to the last one that was created in root_dir that starts with the given prefix. Note that providing an integer instead of a bool will not work here. If no directories are found, one will be created.
  • prefix (str or None. Default: None) – String containing a prefix that is used for naming new working directories or scan for existing ones. If None, all directories in root_dir are considered working directories and ‘prism_’ will be used as a prefix for new ones.
  • prism_par (array_like, dict, str or None. Default: None) – A dict containing the values for the PRISM parameters that need to be changed from their default values. If array_like, dict(prism_par) must generate a dict with the correct lay-out. If str, the string is the absolute or relative path to the file that contains the keys in the first column and the dict values in the second column. If a relative path is given, its path must be relative to root_dir or the current directory. If None, no changes will be made to the default parameters.
  • emul_type (Emulator subclass or None. Default: None) – The type of Emulator to use in this Pipeline instance. If None, use the default emulator instead.
  • comm (Intracomm object or None. Default: None) – The MPI intra-communicator to use in this Pipeline instance or MPI.COMM_WORLD if comm is None.
_call_model(emul_i, par_set, data_idx)[source]

Obtain the output corresponding to the provided data_idx that is generated by the model for a given model parameter sample par_set. The current emulator iteration emul_i is also provided in case it is required by the ModelLink subclass.

Parameters:
  • emul_i (int) – Number indicating the requested emulator iteration.
  • par_set (1D array_like) – Model parameter sample to calculate the model output for.
  • data_idx (list of tuples) – The list of data identifiers for which the model is requested to return the corresponding data values.
Returns:

mod_out (1D ndarray object) – Model output corresponding to given par_set.

_compile_code_snippets()[source]

Compiles all pre-defined built-in code snippets to code objects and saves them to code_objects. These code objects are used for performing standard operations in the _evaluate_sam_set() method.

_do_impl_check(emul_i, uni_impl_val)[source]

Performs an implausibility cut-off check on the provided implausibility values uni_impl_val at emulator iteration emul_i.

Parameters:
  • emul_i (int) – Number indicating the requested emulator iteration.
  • uni_impl_val (1D array_like) – Array containing all univariate implausibility values corresponding to a certain parameter set for all data points.
Returns:

  • result (bool) – 1 if check was successful, 0 if it was not.
  • impl_cut_val (float) – Implausibility value at the first real implausibility cut-off.

_evaluate_model(emul_i, sam_set, data_idx)[source]

Evaluates the model for provided evaluation sample set sam_set at given data points data_idx.

This method automatically distributes the samples according to the various flags set in the ModelLink subclass.

Parameters:
  • emul_i (int) – Number indicating the requested emulator iteration.
  • sam_set (1D or 2D array_like) – Parameter/sample set to evaluate in the model.
  • data_idx (list of tuples) – The list of data identifiers for which the model is requested to return the corresponding data values.
Returns:

  • sam_set (2D ndarray object of shape (n_sam, n_par)) – Array containing the sample set used to evaluate the model.
  • mod_set (2D ndarray object of shape (n_sam, n_data)) – Array containing the data values corresponding to the requested data points.

_evaluate_sam_set(emul_i, sam_set, exec_code)[source]

Evaluates a provided set of emulator evaluation samples sam_set at a given emulator iteration emul_i. The provided tuple of code snippets exec_code are executed using Python’s exec() function at specific points during the analysis.

Parameters:
  • emul_i (int) – Number indicating the requested emulator iteration.
  • sam_set (2D ndarray object) – Array containing model parameter value sets to be evaluated in all emulator systems in emulator iteration emul_i.
  • exec_code ({‘analyze’, ‘evaluate’, ‘hybrid’, ‘project’} or tuple) – Tuple of five code snippets (pre_code, eval_code, anal_code, post_code, exit_code) to be executed at specific points during the analysis. If string, use one of the built-in tuples in code_objects instead.
Other Parameters:
 
  • pre_code (str or code object) – Code snippet to be executed before the evaluation of sam_set starts.
  • eval_code (str or code object) – Code snippet to be executed after the evaluation of each sample in sam_set.
  • anal_code (str or code object) – Code snippet to be executed after the analysis of each sample in sam_set. This code snippet is only executed by the controller.
  • post_code (str or code object) – Code snippet to be executed after the evaluation of sam_set ends.
  • exit_code (str or code object) – Code snippet to be executed before returning the results of the evaluation of sam_set. This code snippet is only executed by the controller.
Returns:

results (object) – The object that is assigned to a local variable called results, which is defaulted to None if no code snippet sets it. Preferably, the execution of post_code and/or exit_code sets this variable. All MPI ranks return it.

Notes

If any of the code snippets is provided as a string, it will be compiled into a code object before starting the evaluation.

_get_default_parameters()[source]

Generates a dict containing default values for all pipeline parameters.

Returns:par_dict (dict) – Dict containing all default pipeline parameter values.
_get_eval_sam_set(emul_i)[source]

Generates an emulator evaluation sample set to be used for analyzing an emulator iteration. Currently uses the lhd() function.

Parameters:emul_i (int) – Number indicating the requested emulator iteration.
Returns:eval_sam_set (2D ndarray object) – Array containing the evaluation samples.
_get_ext_real_set(emul_i, ext_real_set)[source]

Processes an externally provided model realization set ext_real_set, containing the used sample set and the corresponding data value set, to be used for the provided emul_i.

Parameters:
  • emul_i (int) – Number indicating the requested emulator iteration.
  • ext_real_set (str, list, dict or None) – String indicating the suffix of the backup file that needs to be used; list of dicts containing an externally calculated set of model evaluation samples and its data values; a dict with keys ['sam_set', 'mod_set'] containing these dicts; or None if no external set needs to be used.
Returns:

  • ext_sam_set (1D or 2D ndarray object) – Array containing the externally provided model evaluation samples.
  • ext_mod_set (1D or 2D ndarray object) – Array containing the model outputs of all specified externally provided model evaluation samples.

_get_f_impl(emul_i)[source]

Returns the fraction of parameter space that passed the implausibility checks during the analysis of the provided emulator iteration emul_i.

Parameters:emul_i (int) – Number indicating the requested emulator iteration.
Returns:f_impl (float) – The fraction of parameter space that is still plausible.
_get_impl_space(emul_i)[source]

Returns the boundaries of the hypercube that encloses the parameter space in which the plausible space of the provided emulator iteration emul_i is defined.

Parameters:emul_i (int) – Number indicating the requested emulator iteration.
Returns:impl_space (2D ndarray object) – The requested hypercube boundaries.

Note

The parameter space over which plausible space is defined is always equal to the emulator space of the next iteration. This means that reanalyzing an iteration can change the result of this function.

_get_iteration_data(emul_i, sam_set, ext_sam_set, ext_mod_set)[source]

Obtains the model realization data for given emulator iteration emul_i by evaluating the provided sam_set in the model and distributing model outputs to the correct emulator systems.

Parameters:
  • emul_i (int) – Number indicating the requested emulator iteration.
  • sam_set (2D ndarray object) – Array containing the model evaluation samples.
  • ext_sam_set (1D or 2D ndarray object) – Array containing the externally provided model evaluation samples.
  • ext_mod_set (1D or 2D ndarray object) – Array containing the model outputs of all specified externally provided model evaluation samples.

Generates

sam_set : 2D ndarray object
Array containing the model evaluation samples for emulator iteration emul_i.
mod_set : 2D ndarray object
Array containing the model outputs of all specified model evaluation samples for emulator iteration emul_i.
_get_md_var(emul_i, par_set)[source]

Retrieves the model discrepancy variances, which includes all variances that are created by the model provided by the ModelLink instance. This method tries to call the get_md_var() method, and assumes a default model discrepancy variance of 1/6th the data value if it cannot be called. If the data value space is not linear, then this default value is calculated such to reflect that.

Parameters:
  • emul_i (int) – Number indicating the requested emulator iteration.
  • par_set (1D ndarray object) – Model parameter value set to calculate the model discrepancy variances for.
Returns:

var_md (2D ndarray object) – Variance of the model discrepancy.

_get_mock_data(mock_par)[source]

Generates mock data and loads it into the ModelLink object that was provided during class initialization. This function overwrites the ModelLink properties holding the parameter estimates, data values and data errors.

Parameters:mock_par (1D array_like or None) – If 1D array_like, use the provided parameter estimates to create the mock data. If None, a random parameter set will be generated as parameter estimates.

Generates

Overwrites the corresponding ModelLink class properties with the generated values.

_get_n_eval_sam(emul_i)[source]

This function calculates the total number of emulator evaluation samples at a given emulator iteration emul_i from base_eval_sam.

Parameters:emul_i (int) – Number indicating the requested emulator iteration.
Returns:n_eval_sam (int) – Total number of emulator evaluation samples.
_get_paths(root_dir, working_dir, prefix)[source]

Obtains the path for the root directory, working directory and parameters file for PRISM.

Parameters:
  • root_dir (str or None) – String containing the absolute path of the root directory where all working directories are stored. If None, root directory will be set to the directory this class was initialized in.
  • working_dir (str, bool or None) – String containing the name of the working directory of the emulator in root_dir. If True, a new working directory will be created in root_dir. If None or False, working directory is set to the last one that was created in root_dir that starts with the given prefix. Note that providing an integer instead of a bool will not work here. If no directories are found, one will be created.
  • prefix (str or None) – String containing a prefix that is used for naming new working directories or scan for existing ones. If None, all directories in root_dir are considered working directories and ‘prism_’ will be used as a prefix for new ones.

Generates

The absolute paths to the root directory, working directory, emulator master HDF5-file and PRISM parameters file.

_get_uni_impl(emul_i, par_set, adj_exp_val, adj_var_val)[source]

Calculates the univariate implausibility values at a given emulator iteration emul_i for specified expectation and variance values adj_exp_val and adj_var_val, corresponding to given par_set.

Parameters:
  • emul_i (int) – Number indicating the requested emulator iteration.
  • par_set (1D ndarray object) – Model parameter value set to calculate the univariate implausibility values for. Only used to pass to the get_md_var() method.
  • adj_exp_val, adj_var_val (1D array_like) – The adjusted expectation and variance values to calculate the univariate implausibility for.
Returns:

uni_impl_val (1D ndarray object) – Univariate implausibility value for all requested emulator systems.

_load_data()[source]

Loads in all the important pipeline data into memory for the controller rank. If it is detected that the last emulator iteration has not been analyzed yet, the implausibility analysis parameters are taken from the PRISM parameters dict and temporarily stored in memory.

Generates

All relevant pipeline data up to the last emulator iteration is loaded into memory.

_make_call(exec_fn, *args, **kwargs)[source]

Sends the provided exec_fn to all worker ranks, if they are listening for calls, and tells them to execute it using the provided args and kwargs. All ranks that call this function will execute exec_fn as well.

If used within the WorkerMode context manager, this function should only be called by the controller. If not, it should be called by all valid ranks that must execute exec_fn.

Parameters:
  • exec_fn (str or callable) – If string, a callable attribute of this Pipeline instance or a callable object that the workers should execute if not.
  • args (positional arguments) – Positional arguments that need to be provided to exec_fn.
  • kwargs (keyword arguments) – Keyword arguments that need to be provided to exec_fn.
Returns:

out (object) – The object returned by executing exec_fn. Note that only ranks that directly call this function return, as workers in worker mode cannot do so.

Note

Changed in version 1.2.0: If any entry in args or kwargs is a string written as ‘pipe.XXX’, it is assumed that ‘XXX’ refers to a Pipeline attribute of the MPI rank receiving the call. It will be replaced with the corresponding attribute before exec_fn is called.

_make_call_workers(exec_fn, *args, **kwargs)[source]

Sends the provided exec_fn to all worker ranks, if they are listening for calls, and tells them to execute it using the provided args and kwargs.

If used within the WorkerMode context manager, this function should only be called by the controller. If not, it should be called by all valid ranks that must execute exec_fn.

Parameters:
  • exec_fn (str or callable) – If string, a callable attribute of this Pipeline instance or a callable object that the workers should execute if not.
  • args (positional arguments) – Positional arguments that need to be provided to exec_fn.
  • kwargs (keyword arguments) – Keyword arguments that need to be provided to exec_fn.
Returns:

out (object) – The object returned by executing exec_fn. Note that only ranks that directly call this function return, as workers in worker mode cannot do so.

Note

Changed in version 1.2.0: If any entry in args or kwargs is a string written as ‘pipe.XXX’, it is assumed that ‘XXX’ refers to a Pipeline attribute of the MPI rank receiving the call. It will be replaced with the corresponding attribute before exec_fn is called.

_multi_call_model(emul_i, sam_set, data_idx)[source]

Obtain the output set corresponding to the provided data_idx that is generated by the model for a given model parameter sample set sam_set. The current emulator iteration emul_i is also provided in case it is required by the ModelLink subclass.

This is a multi-version of _call_model().

Parameters:
  • emul_i (int) – Number indicating the requested emulator iteration.
  • sam_set (2D array_like) – Model parameter sample set to calculate the model output for.
  • data_idx (list of tuples) – The list of data identifiers for which the model is requested to return the corresponding data values.
Returns:

mod_set (2D ndarray object) – Model output set corresponding to given sam_set.

_read_parameters(prism_par)[source]

Reads in all parameters in the provided prism_par and saves them as a dict in the current Pipeline instance.

_save_data(data_dict)[source]

Saves a given data dict {keyword: data} at the last emulator iteration to the HDF5-file and as an data attribute to the current Pipeline instance.

Parameters:

data_dict (dict) – Dict containing the data that needs to be saved to the HDF5-file.

Keyword Arguments:
 
  • keyword ({‘impl_par’; ‘impl_sam’; ‘n_eval_sam’}) – String specifying the type of data that needs to be saved.
  • data ({int; float; str; array_like} or dict) – The actual data that needs to be saved at data keyword keyword. If dict, save every item individually.

Generates

The specified data is saved to the HDF5-file.

_save_statistics(emul_i, stat_dict)[source]

Saves a given statistics dict {keyword: [value, unit]} at emulator iteration emul_i to the HDF5-file. The provided values are always saved as strings.

Parameters:

emul_i (int) – Number indicating the requested emulator iteration.

Keyword Arguments:
 
  • keyword (str) – String containing the name/keyword of the statistic that is being saved.
  • value (int, float or str) – The value of the statistic.
  • unit (str) – The unit of the statistic.
_set_impl_par(impl_cut)[source]

Sets the impl_cut and cut_idx properties for implausibility evaluations using prism_dict and the provided impl_cut.

Parameters:impl_cut (list of float or None) – Incomplete, shortened impl_cut-offs list to be used during the analysis of this emulator iteration. If None, use prism_dict instead.

Generates

impl_cut : 1D ndarray object
Full list containing the impl_cut-offs for all data points provided to the emulator.
cut_idx : int
Index of the first impl_cut-off in the impl_cut list that is not 0.
_set_parameters()[source]

Sets the Pipeline parameters from the prism_dict property and saves them in the current Pipeline instance.

analyze(*, impl_cut=None)[source]

Analyzes the emulator at the last emulator iteration for a large number of emulator evaluation samples. All samples that survive the implausibility checks set by the provided impl_cut, are used in the construction of the next emulator iteration.

Other Parameters:
 impl_cut (list of float or None. Default: None) – Incomplete, shortened implausibility cut-offs list to be used during the analysis of this emulator iteration. If None, the currently set implausibility cut-off values (impl_cut) will be used.

Generates

impl_sam : 2D ndarray object
Array containing all emulator evaluation samples that survived the implausibility checks.
construct(emul_i=None, *, analyze=True, ext_real_set=None, force=False)[source]

Constructs the emulator at the specified emulator iteration emul_i, and performs an implausibility analysis on the emulator iteration right afterward if requested (analyze()).

Other Parameters:
 
  • emul_i (int or None. Default: None) – If int, number indicating the requested emulator iteration. If None, the next iteration of the emulator will be constructed.
  • analyze (bool. Default: True) – Bool indicating whether or not to perform an analysis after the specified emulator iteration has been successfully constructed, which is required for making projections (project()) and constructing the next iteration.
  • ext_real_set (str, list, dict or None. Default: None) – String indicating the suffix of the backup file that needs to be used; list of dicts containing an externally calculated set of model evaluation samples and its data values; a dict with keys ['sam_set', 'mod_set'] containing these dicts; or None if no external set needs to be used.
  • force (bool. Default: False) – Controls what to do if the specified emulator iteration emul_i already (partly) exists. If False, finish construction of the specified iteration or skip it if already finished. If True, reconstruct the specified iteration entirely.

Generates

A new HDF5-group with the emulator iteration as its name, in the loaded emulator master file, containing emulator data required for this emulator iteration.

Notes

Using an emulator iteration that has been (partly) constructed before, will finish construction or skip it if already finished when force is False; or it will delete that and all following iterations, and reconstruct the specified iteration when force is True. Using emul_i = 1 and force is True is equivalent to reconstructing the entire emulator.

If no implausibility analysis is requested, then the implausibility parameters are taken from the PRISM parameters dict and temporarily stored in memory in order to enable the usage of the evaluate() method.

crystal()

Creates an instance of QApplication or retrieves it if one already exists, and starts Crystal, PRISM’s Projection GUI.

Crystal provides an interactive way of creating projection figures, as opposed to the static and linear method provided by project(). It is made to make it easier to create; view; compare; and analyze large numbers of projection figures. All options available in the project() method can also be accessed through Crystal.

As with all Pipeline user methods, this function must be called by all MPI ranks.

New in version 1.2.0.

details(emul_i=None)[source]

Prints the details/properties of the currently loaded Pipeline instance at given emulator iteration emul_i. See Props for detailed descriptions of all printed properties.

Other Parameters:
 emul_i (int or None. Default: None) – If int, number indicating the requested emulator iteration. If None, the last iteration of the emulator will be used.

Props

Working directory
The relative path to the working directory of the emulator starting at the current working directory.
Emulator type
The type of this emulator, corresponding to the emul_type of the provided emul_type during Pipeline initialization. If no emulator type was provided during initialization, this is ‘default’.
ModelLink subclass
Name of the ModelLink subclass used to construct this emulator.
Emulation method
Indicates the combination of regression and Gaussian emulation methods that have been used for this emulator.
Mock data used?
Whether or not mock data has been used to construct this emulator. If so, the printed estimates for all model parameters are the parameter values used to create the mock data.
Emulator iteration
The iteration of the emulator this details overview is about. By default, this is the last (partly) constructed iteration.
Construction completed?
Whether or not the construction of this emulator iteration is completed. If not, the missing components for each emulator system are listed and the remaining information of this iteration is not printed.
Plausible regions?
Whether or not plausible regions have been found during the analysis of this emulator iteration. If no analysis has been done yet, “N/A” will be printed.
Projections available?
Whether or not projections have been created for this emulator iteration. If projections are available and analysis has been done, but with different implausibility cut-offs, a “desynced” note is added. Also prints number of available projections versus maximum number of projections in parentheses.
# of model evaluation samples
The total number of model evaluation samples used to construct all emulator iterations up to this iteration, with the number for every individual iteration in parentheses.
# of plausible/analyzed samples
The number of emulator evaluation samples that passed the implausibility check out of the total number of analyzed samples in this emulator iteration. This is the number of model evaluation samples that was/will be used for the construction of the next emulator iteration. If no analysis has been done, the numbers show up as “-“.
% of parameter space remaining
The percentage of the total number of analyzed samples that passed the implausibility check in this emulator iteration. If no analysis has been done, the number shows up as “-“.
# of active/total parameters
The number of model parameters that was considered active during the construction of this emulator iteration, compared to the total number of model parameters defined in the used ModelLink subclass.
# of emulated data points
The number of data points that have been emulated in this emulator iteration.
# of emulator systems
The total number of emulator systems that are required in this emulator. The number of active emulator systems is equal to the number of data points.
Parameter space
Lists the name, lower and upper value boundaries and estimate (if provided) of all model parameters defined in the used ModelLink subclass. An asterisk is printed in front of the parameter name if this model parameter was considered active during the construction of this emulator iteration. A question mark is used instead if the construction of this emulator iteration is not finished.
evaluate(sam_set, emul_i=None)[source]

Evaluates the given model parameter sample set sam_set up to given emulator iteration emul_i. The output of this function depends on the number of dimensions in sam_set. The output is always provided on the controller rank.

Parameters:sam_set (1D or 2D array_like or dict) – Array containing model parameter value sets to be evaluated in the emulator up to emulator iteration emul_i.
Other Parameters:
 emul_i (int or None. Default: None) – If int, number indicating the requested emulator iteration. If None, the last iteration of the emulator will be used.
Returns:
  • impl_check (list of bool) – List of bool indicating whether or not the given samples passed the implausibility check at the given emulator iteration emul_i.
  • emul_i_stop (list of int) – List containing the last emulator iterations at which the given samples are still within the plausible region of the emulator.
  • adj_exp_val (2D ndarray object) – Array containing the adjusted expectation values for all given samples.
  • adj_var_val (2D ndarray object) – Array containing the adjusted variance values for all given samples.
  • uni_impl_val (2D ndarray object) – Array containing the univariate implausibility values for all given samples.

Prints (if 1D sam_set)

emul_i_stop : int
Last emulator iteration at which the given sample is still within the plausible region of the emulator.
adj_exp_val : 1D ndarray object
The adjusted expectation values for the given sample.
adj_var_val : 1D ndarray object
The adjusted variance values for the given sample.
sigma_val : 1D ndarray object
The corresponding sigma value for the given sample.
uni_impl_val : 1D ndarray object
The univariate implausibility values for the given sample.

Notes

If given emulator iteration emul_i has been analyzed before, the implausibility parameters of the last analysis are used. If not, then the values are used that were read in when the emulator was loaded or that have been set by the user.

project(emul_i=None, proj_par=None, **kwargs)

Analyzes the emulator iteration emul_i and constructs a series of projection figures detailing the behavior of the model parameters corresponding to the given proj_par. The input and output depend on the number of model parameters n_par.

All optional keyword arguments (except force) control various aspects of drawing the projection figures and do not affect the projection data that is saved to HDF5. This is instead influenced by the proj_res and proj_depth properties.

Parameters:
  • emul_i (int or None. Default: None) – If int, number indicating the requested emulator iteration. If None, the last iteration of the emulator will be used.
  • proj_par (1D array_like of {int; str} or None. Default: None) – For which model parameters to construct the projection figures. If 1D array_like, construct projection figures for all combinations of provided model parameters that are active, with a string referring to the name of the model parameter and an integer referring to the position in which the model parameter is shown in the details() method. If None, projection figures are made for all active model parameters.
Keyword Arguments:
 
  • proj_type ({‘2D’; ‘3D’; ‘2D+3D’}. Default: ‘2D’ (2D), ‘2D+3D’ (nD)) – String indicating which projection type to create for all supplied active parameters. If n_par == 2, this is always ‘2D’ (and cannot be modified).
  • figure (bool. Default: True) – Whether or not to create the projection figures. If True, the figures are calculated, drawn and saved. If False, the figures are calculated and their data is returned in a dict.
  • align ({‘row’/’horizontal’; ‘col’/’column’/’vertical’}. Default: ‘col’) – If figure is True, string indicating how to position the two subplots. If ‘row’/’horizontal’, the subplots are positioned on a single row. If ‘col’/’column’/’vertical’, the subplots are positioned on a single column.
  • show_cuts (bool. Default: False) – If figure is True and proj_type is not ‘3D’, whether to show all implausibility cut-offs in the 2D projections (True) or only the first cut-off (False).
  • smooth (bool. Default: False) – Controls what to do if a grid point contains no plausible samples, but does contain a minimum implausibility value below the first non-wildcard cut-off. If False, these values are kept, which can show up as artifact-like features in the projection figure. If True, these values are set to the first cut-off, removing them from the projection figure. Doing this may also remove interesting features. This does not affect the projection data saved to HDF5. Smoothed figures have an ‘_s’ string appended to their filenames.
  • use_par_space (bool. Default: False) – Controls whether to use the model parameter space (True) or the parameter space in which emulator iteration emul_i is defined (False) as the axes limits in the projection figure.
  • force (bool. Default: False) – Controls what to do if a projection hypercube has been calculated at the emulator iteration emul_i before. If False, it will use the previously acquired projection data to create the projection figure. If True, it will recalculate all the data required to create the projection figure. Note that this will also delete all associated projection figures.
  • fig_kwargs (dict. Default: {‘figsize’: (6.4, 4.8), ‘dpi’: 100}) – Dict of keyword arguments to be used when creating the subplots figure. It takes all arguments that can be provided to the figure() function.
  • impl_kwargs_2D (dict. Default: {}) – Dict of keyword arguments to be used for making the minimum implausibility (top/left) plot in the 2D projection figures. It takes all arguments that can be provided to the plot() function.
  • impl_kwargs_3D (dict. Default: {‘cmap’: ‘cmr.rainforest_r’}) – Dict of keyword arguments to be used for making the minimum implausibility (top/left) plot in the 3D projection figures. It takes all arguments that can be provided to the hexbin() function.
  • los_kwargs_2D (dict. Default: {}) – Dict of keyword arguments to be used for making the line-of-sight (bottom/right) plot in the 2D projection figures. It takes all arguments that can be provided to the plot() function.
  • los_kwargs_3D (dict. Default: {‘cmap’: ‘cmr.freeze’}) – Dict of keyword arguments to be used for making the line-of-sight (bottom/right) plot in the 3D projection figures. It takes all arguments that can be provided to the hexbin() function.
  • line_kwargs_est (dict. Default: {‘linestyle’: ‘–’, ‘color’: ‘grey’}) – Dict of keyword arguments to be used for drawing the parameter estimate lines in both plots. It takes all arguments that can be provided to the plot() function.
  • line_kwargs_cut (dict. Default: {‘color’: ‘r’}) – Dict of keyword arguments to be used for drawing the implausibility cut-off line(s) in the top/left plot in the 2D projection figures. It takes all arguments that can be provided to the plot() function.
Returns:

fig_data (dict of dicts) – Dict containing the data for every requested projection figure, split up into the ‘impl_min’ and ‘impl_los’ dicts. For 2D projections, every dict contains a list with the x and y values. For 3D projections, it contains the x, y and z values. Note that due to the figures being interpolations, the y/z values can be below zero or the line-of-sight values being above unity.

Generates (if figure is True)

A series of projection figures detailing the behavior of the model. The lay-out and output of the projection figures depend on the type of figure:

2D projection figure: The output will feature a figure with two subplots for every active model parameter (n_par). Every figure gives details about the behavior of the corresponding model parameter, by showing the minimum implausibility value (top/left) and the line-of-sight depth (bottom/right) obtained at the specified parameter value, independent of the values of the other parameters.

3D projection figure (only if n_par > 2): The output will feature a figure with two subplots for every combination of two active model parameters that can be made (n_par*(n_par-1)/2). Every figure gives details about the behavior of the corresponding model parameters, as well as their dependency on each other. This is done by showing the minimum implausibility (top/left) and the line-of-sight depth (bottom/right) obtained at the specified parameter values, independent of the values of the remaining model parameters.

Notes

If given emulator iteration emul_i has been analyzed before, the implausibility parameters of the last analysis are used. If not, then the values are used that were read in when the emulator was loaded or that have been set by the user.

All colormaps defined in the e13tools package are loaded automatically when PRISM is imported and can be used.

run(emul_i=None, *, force=False)[source]

Calls the construct() method to start the construction of the given iteration of the emulator and creates the projection figures right afterward if this construction was successful.

Other Parameters:
 
  • emul_i (int or None. Default: None) – If int, number indicating the requested emulator iteration. If None, the next iteration of the emulator will be constructed.
  • force (bool. Default: False) – Controls what to do if the specified emulator iteration emul_i already (partly) exists. If False, finish construction of the specified iteration or skip it if already finished. If True, reconstruct the specified iteration entirely.
start_gui()

Creates an instance of QApplication or retrieves it if one already exists, and starts Crystal, PRISM’s Projection GUI.

Crystal provides an interactive way of creating projection figures, as opposed to the static and linear method provided by project(). It is made to make it easier to create; view; compare; and analyze large numbers of projection figures. All options available in the project() method can also be accessed through Crystal.

As with all Pipeline user methods, this function must be called by all MPI ranks.

New in version 1.2.0.

File

Custom File class that has added logging and automatically uses hdf5_file as the HDF5-file to open.

Type:File
base_eval_sam

Base number of emulator evaluations used to analyze the emulator systems. This number is scaled up by the number of model parameters to generate the true number of emulator evaluations (n_eval_sam).

Type:int
code_objects

Collection of pre-compiled built-in code snippets that are used in the _evaluate_sam_set() method.

Type:dict of code objects
comm

The MPI intra-communicator that is used in this Pipeline instance. By default, this is MPI.COMM_WORLD.

Type:Intracomm
criterion

Value indicating which criterion to use in the lhd() function.

Type:str, float or None
cut_idx

The index of the first non-wildcard in a list of implausibility values. This is equivalent to the number of wildcards leading the cut-off values in impl_cut.

Type:int
do_active_anal

Whether or not to do an active parameters analysis during the construction of the emulator systems.

Type:bool
do_logging

Whether or not to save all logging messages. If False, all logging messages of level INFO and below are ignored. It also enables/disables the progress bar when making projections.

Type:bool
emulator

The Emulator instance created during Pipeline initialization.

Type:Emulator
freeze_active_par

Whether or not previously active parameters always stay active if possible.

Type:bool
hdf5_file

Absolute path to the loaded master HDF5-file.

Type:str
impl_cut

The non-wildcard univariate implausibility cut-off values for an emulator iteration. Setting it with the reduced implausibility cut-off list will change the values of cut_idx and this property at the last emulator iteration.

Type:list of float
impl_sam

The model evaluation samples that will be added to the next emulator iteration.

Type:ndarray
is_controller

Whether or not this MPI process is a controller rank. If no MPI is used, this is always True.

Type:bool
is_worker

Whether or not this MPI process is a worker rank. If no MPI is used, this is always False.

Type:bool

The ModelLink instance provided during Pipeline initialization.

Type:ModelLink
n_eval_sam

The number of evaluation samples used to analyze an emulator iteration of the emulator systems. The number of plausible evaluation samples is stored in n_impl_sam. It is zero if the specified iteration has not been analyzed yet.

Type:int
n_impl_sam

Number of model evaluation samples that passed the implausibility checks during the analysis of an emulator iteration. It is zero if the specified iteration has not been analyzed yet or has no plausible samples.

Type:int
n_sam_init

Number of evaluation samples that will be used to construct the first iteration of the emulator systems.

Type:int
pot_active_par

The potentially active parameters. Only parameters from this list can become active during the active parameters analysis. If do_active_anal is False, all parameters in this list will be active.

Type:list of str
prism_dict

Dictionary containing all PRISM parameters that were provided during Pipeline initialization.

Type:dict
proj_depth

Number of emulator evaluations that will be used to generate the samples in every grid point for the projection figures. Note that when making 2D projections of nD models, the used depth will be this number multiplied by proj_res.

Type:int
proj_res

Number of emulator evaluations that will be used to generate the grid for the projection figures.

Type:int
rank

The rank of this MPI process in comm. If no MPI is used, this is always 0.

Type:int
root_dir

Absolute path to the root directory.

Type:str
size

The number of MPI processes in comm. If no MPI is used, this is always 1.

Type:int
worker_mode

Special context manager within which all code is executed in worker mode. In worker mode, all worker ranks are continuously listening for calls from the controller rank made with _make_call() or _make_call_workers().

Note that all code within the context manager is executed by all ranks, with the worker ranks executing it after the controller rank exits. It is therefore advised to use an if-statement inside to make sure only the controller rank executes the code.

Using this context manager allows for easier use of PRISM in combination with serial/OpenMP codes (like MCMC methods). It also makes it easier to write long complex code that is mostly executed on the controller rank (but the worker ranks sometimes need to execute something).

All worker modes are independent of each other and can be created in a nested fashion.

Type:WorkerMode
working_dir

Absolute path to the working directory.

Type:str
class prism._pipeline.WorkerMode(pipeline_obj)[source]
__enter__()[source]

The provided Pipeline object enters worker mode, making all worker ranks start listening for calls from the controller rank until this context manager exits.

__exit__(etype, value, tb)[source]

The provided Pipeline objects exits worker mode, making all worker ranks stop listening for calls from the controller rank and resume normal code execution.

__init__(pipeline_obj)[source]

Initialize the WorkerMode class using the MPI ranks defined in the provided pipeline_obj. This class should solely be initialized and finalized through the Pipeline class.

New in version 1.2.0.

Parameters:pipeline_obj (Pipeline object) – The instance of the Pipeline class that is enabling this worker mode.
static _process_call(pipeline_obj, exec_fn, args, kwargs)[source]

Processes a call that was made with the make_call() or make_call_workers() method.

This function should solely be called through either of these methods and never directly.

Parameters:
  • pipeline_obj (Pipeline object) – The instance of the Pipeline class that is making this call.
  • exec_fn (str or callable) – If string, a callable attribute of this Pipeline instance or a callable object that should be executed if not.
  • args (tuple) – Positional arguments that need to be provided to exec_fn.
  • kwargs (dict) – Keyword arguments that need to be provided to exec_fn.
Returns:

out (object) – The object returned by executing exec_fn.

Note

If any entry in args or kwargs is a string written as ‘pipe.XXX’, it is assumed that ‘XXX’ refers to a Pipeline attribute of the MPI rank receiving the call. It will be replaced with the corresponding attribute before exec_fn is called.

static _process_call_str(pipeline_obj, string)[source]

Processes a provided string that was provided as an argument value to _process_call().

Parameters:
  • pipeline_obj (Pipeline object) – The instance of the Pipeline class that is making this call.
  • string (str) – String value that must be processed.
Returns:

out (str or object) – If string starts with ‘pipe.’, the corresponding Pipeline attribute will be returned. Else, string is returned.

listen_for_calls()[source]

All worker ranks in the comm communicator start listening for calls from the corresponding controller rank and will attempt to execute the received message. Listening for calls continues until this context manager exits (__exit__() is called).

This method is automatically initialized and finalized when using the worker_mode context manager.

static make_call(pipeline_obj, exec_fn, *args, **kwargs)[source]

Sends the provided exec_fn to all worker ranks, if they are listening for calls, and tells them to execute it using the provided args and kwargs. All ranks that call this function will execute exec_fn as well.

If used within the WorkerMode context manager, this function should only be called by the controller. If not, it should be called by all valid ranks that must execute exec_fn.

Parameters:
  • pipeline_obj (Pipeline object) – The instance of the Pipeline class that is making this call.
  • exec_fn (str or callable) – If string, a callable attribute of this Pipeline instance or a callable object that the workers should execute if not.
  • args (positional arguments) – Positional arguments that need to be provided to exec_fn.
  • kwargs (keyword arguments) – Keyword arguments that need to be provided to exec_fn.
Returns:

out (object) – The object returned by executing exec_fn. Note that only ranks that directly call this function return, as workers in worker mode cannot do so.

Note

Changed in version 1.2.0: If any entry in args or kwargs is a string written as ‘pipe.XXX’, it is assumed that ‘XXX’ refers to a Pipeline attribute of the MPI rank receiving the call. It will be replaced with the corresponding attribute before exec_fn is called.

static make_call_workers(pipeline_obj, exec_fn, *args, **kwargs)[source]

Sends the provided exec_fn to all worker ranks, if they are listening for calls, and tells them to execute it using the provided args and kwargs.

If used within the WorkerMode context manager, this function should only be called by the controller. If not, it should be called by all valid ranks that must execute exec_fn.

Parameters:
  • pipeline_obj (Pipeline object) – The instance of the Pipeline class that is making this call.
  • exec_fn (str or callable) – If string, a callable attribute of this Pipeline instance or a callable object that the workers should execute if not.
  • args (positional arguments) – Positional arguments that need to be provided to exec_fn.
  • kwargs (keyword arguments) – Keyword arguments that need to be provided to exec_fn.
Returns:

out (object) – The object returned by executing exec_fn. Note that only ranks that directly call this function return, as workers in worker mode cannot do so.

Note

Changed in version 1.2.0: If any entry in args or kwargs is a string written as ‘pipe.XXX’, it is assumed that ‘XXX’ refers to a Pipeline attribute of the MPI rank receiving the call. It will be replaced with the corresponding attribute before exec_fn is called.

__weakref__

list of weak references to the object (if defined)