Pipeline¶
-
class
prism.
Pipeline
(modellink_obj, *, root_dir=None, working_dir=None, prefix=None, prism_par=None, emul_type=None, comm=None)[source]¶ Defines the
Pipeline
class of the PRISM package.The
Pipeline
class is the main user class of the PRISM package and provides a user-friendly environment that gives access to all operations within the pipeline.-
_Projection__analyze_proj_hcube
(hcube)¶ Analyzes an emulator projection hypercube hcube.
Parameters: hcube (1D array_like of int of length {2, 3}) – Array containing the internal integer identifiers of the main model parameters that require a projection hypercube. Returns: - impl_min_hcube (1D
ndarray
object) – List containing the lowest implausibility value that can be reached in every single grid point on the given hypercube. - impl_los_hcube (1D
ndarray
object) – List containing the fraction of the total amount of evaluated samples in every single grid point on the given hypercube, that still satisfied the implausibility cut-off criterion.
- impl_min_hcube (1D
-
_Projection__draw_2D_proj_fig
(hcube)¶ Draws the 2D projection figure for the provided hcube.
Parameters: hcube (1D array_like of int of length 2) – Array containing the internal integer identifiers of the main model parameters that require a projection figure.
-
_Projection__draw_3D_proj_fig
(hcube)¶ Draws the 3D projection figure for the provided hcube.
Parameters: hcube (1D array_like of int of length 3) – Array containing the internal integer identifiers of the main model parameters that require a projection figure.
-
_Projection__draw_estimate_arrowline
(par_est1, par_est2, ax)¶ Draws parameter estimate arrow and lines in the given ax, for the provided estimates par_est1 and par_est2. Whether an arrow and/or a line is drawn, and what their layout and position is, depends on the given estimate values.
Parameters: - par_est1, par_est2 (float or None) – The parameter estimates for which an arrow or line must be drawn. If the estimate is within the axes limits, a line is drawn. If the estimate is outside of the axes limits, an arrow is drawn in the direction of (par_est1, par_est2). If None, a default value is used for the respective argument.
- ax (
Axes
object) – The Axes object in which the arrow and lines must be drawn.
Note
Because the arrow is drawn using relative DPI-coordinates, the axes limits and positions of ax must be final before calling this function.
-
_Projection__get_default_input_arguments
()¶ Generates a dict containing default values for all input arguments.
Returns: kwargs_dict (dict) – Dict containing all default input argument values.
-
_Projection__get_default_parameters
()¶ Generates a dict containing default values for all projection parameters.
Returns: par_dict (dict) – Dict containing all default projection parameter values.
-
_Projection__get_fig_path
(hcube)¶ Determines the absolute path of a projection figure corresponding to a provided projection hypercube hcube and returns it.
Parameters: hcube (1D array_like of int of length {2, 3} or str) – Array containing the internal integer identifiers of the main model parameters that require a projection hypercube. If str, the name of hcube instead ( _Projection__get_hcube_name()
).Returns: - fig_path (str) – The absolute path to the requested projection figure.
- fig_path_s (str) – The absolute path to the smoothed version.
-
_Projection__get_grid_idx
(hcube)¶ Returns the indices of all parameters that are in the grid points of the given projection hypercube hcube.
Parameters: hcube (1D array_like of int of length {2, 3}) – Array containing the internal integer identifiers of the main model parameters that require a projection hypercube. Returns: grid_idx (list of int) – Indices of all grid point parameters.
-
_Projection__get_hcube_name
(hcube)¶ Determines the name of a provided projection hypercube hcube and returns it.
Parameters: hcube (1D array_like of int of length {2, 3}) – Array containing the internal integer identifiers of the main model parameters that require a projection hypercube. Returns: hcube_name (str) – The name of this projection hypercube.
-
_Projection__get_proj_data
(hcube)¶ Returns the projection data belonging to the provided hypercube hcube.
Parameters: hcube (1D array_like of int of length {2, 3}) – Array containing the internal integer identifiers of the main model parameters that require a projection hypercube. Returns: proj_data (dict) – Dict containing all the data associated with the specified hcube.
-
_Projection__get_proj_hcube
(hcube)¶ Generates a projection hypercube hcube containing emulator evaluation samples The output of this function depends on the requested projection type.
Parameters: hcube (1D array_like of int of length {2, 3}) – Array containing the internal integer identifiers of the main model parameters that require a projection hypercube. Returns: proj_hcube (3D ndarray
object) – 3D projection hypercube of emulator evaluation samples. For 3D projections, the grid uses matrix indexing (second parameter varies the fastest).
-
_Projection__get_proj_space
(emul_i)¶ Returns the boundaries of the hypercube that encloses the parameter space in which the projection space of the provided emulator iteration emul_i is defined.
Parameters: emul_i (int) – Number indicating the requested emulator iteration. Returns: proj_space (2D ndarray
object) – The requested hypercube boundaries.
-
_Projection__get_req_hcubes
(emul_i, proj_par)¶ Determines which projection hypercubes have been requested by the user. Also checks if these projection hypercubes have been calculated before, and depending on the value of
force
, either skips them or recreates them.Parameters: - emul_i (int) – Number indicating the requested emulator iteration.
- proj_par (1D array_like of {int; str} or None) – For which model parameters to construct the projection figures.
If 1D array_like, construct projection figures for all combinations
of provided model parameters that are active, with a string
referring to the name of the model parameter and an integer
referring to the position in which the model parameter is shown in
the
details()
method. If None, projection figures are made for all active model parameters.
Generates
- hcubes : list of lists
- List containing the parameter indices of the requested projection hypercubes.
- create_hcubes : list of lists
- List containing the parameter indices of the requested projection hypercubes that need to be created first.
-
_Projection__prepare_projections
(emul_i, proj_par, **kwargs)¶ Prepares the pipeline for the creation of the requested projections.
Parameters: - emul_i (int or None) – Number indicating the requested emulator iteration.
- proj_par (1D array_like of {int; str} or None) – For which model parameters to construct the projection figures.
If 1D array_like, construct projection figures for all combinations
of provided model parameters that are active, with a string
referring to the name of the model parameter and an integer
referring to the position in which the model parameter is shown in
the
details()
method. If None, projection figures are made for all active model parameters. - kwargs (keyword arguments) – Keyword arguments that were provided to
project()
.
-
_Projection__process_input_arguments
(emul_i, **kwargs)¶ Processes the input arguments given to the
project()
method.Parameters: - emul_i (int or None) – Number indicating the requested emulator iteration.
- kwargs (keyword arguments) – Keyword arguments that were provided to
project()
.
Generates
All default and provided arguments are saved to their respective properties.
-
_Projection__read_proj_space
(hcube_group)¶ Reads in and transforms the projection parameter space hypercube that is stored in the provided hcube_group.
Parameters: hcube_group ( Group
object) – The HDF5-group from which the projection parameter space hypercube needs to be read in.Returns: proj_space (2D ndarray
object) – The read-in hypercube boundaries.
-
_Projection__save_data
(emul_i, data_dict)¶ Saves a given data dict
{keyword: data}
at the emulator iteration this class was initialized for, to the HDF5-file.Parameters: - emul_i (int) – Number indicating the requested emulator iteration.
- data_dict (dict) – Dict containing the data that needs to be saved to the HDF5-file.
Keyword Arguments: - keyword ({‘nD_proj_hcube’}) – String specifying the type of data that needs to be saved.
- data ({int; float; str; array_like} or dict) – The actual data that needs to be saved at data keyword keyword. If dict, save every item individually.
Generates
The specified data is saved to the HDF5-file.
-
_Projection__set_parameters
()¶ Sets the Projection parameters from the
prism_dict
property and saves them in the currentProjection
instance.
-
__call__
(emul_i=None, *, force=False)[source]¶ Calls the
construct()
method to start the construction of the given iteration of the emulator and creates the projection figures right afterward if this construction was successful.Other Parameters: - emul_i (int or None. Default: None) – If int, number indicating the requested emulator iteration. If None, the next iteration of the emulator will be constructed.
- force (bool. Default: False) – Controls what to do if the specified emulator iteration emul_i already (partly) exists. If False, finish construction of the specified iteration or skip it if already finished. If True, reconstruct the specified iteration entirely.
-
__init__
(modellink_obj, *, root_dir=None, working_dir=None, prefix=None, prism_par=None, emul_type=None, comm=None)[source]¶ Initialize an instance of the
Pipeline
class.Parameters: modellink_obj (
ModelLink
object) – Instance of theModelLink
subclass that links the emulated model to thisPipeline
instance.Other Parameters: - root_dir (str or None. Default: None) – String containing the absolute path of the root directory where all working directories are stored. If None, root directory will be set to the directory this class was initialized in.
- working_dir (str, bool or None. Default: None) – String containing the name of the working directory of the emulator in root_dir. If True, a new working directory will be created in root_dir. If None or False, working directory is set to the last one that was created in root_dir that starts with the given prefix. Note that providing an integer instead of a bool will not work here. If no directories are found, one will be created.
- prefix (str or None. Default: None) – String containing a prefix that is used for naming new working directories or scan for existing ones. If None, all directories in root_dir are considered working directories and ‘prism_’ will be used as a prefix for new ones.
- prism_par (array_like, dict, str or None. Default: None) – A dict containing the values for the PRISM parameters that need to be changed from their default values. If array_like, dict(prism_par) must generate a dict with the correct lay-out. If str, the string is the absolute or relative path to the file that contains the keys in the first column and the dict values in the second column. If a relative path is given, its path must be relative to root_dir or the current directory. If None, no changes will be made to the default parameters.
- emul_type (
Emulator
subclass or None. Default: None) – The type ofEmulator
to use in thisPipeline
instance. If None, use the default emulator instead. - comm (
Intracomm
object or None. Default: None) – The MPI intra-communicator to use in thisPipeline
instance orMPI.COMM_WORLD
if comm is None.
-
_call_model
(emul_i, par_set, data_idx)[source]¶ Obtain the output corresponding to the provided data_idx that is generated by the model for a given model parameter sample par_set. The current emulator iteration emul_i is also provided in case it is required by the
ModelLink
subclass.Parameters: - emul_i (int) – Number indicating the requested emulator iteration.
- par_set (1D array_like) – Model parameter sample to calculate the model output for.
- data_idx (list of tuples) – The list of data identifiers for which the model is requested to return the corresponding data values.
Returns: mod_out (1D
ndarray
object) – Model output corresponding to given par_set.
-
_compile_code_snippets
()[source]¶ Compiles all pre-defined built-in code snippets to code objects and saves them to
code_objects
. These code objects are used for performing standard operations in the_evaluate_sam_set()
method.
-
_do_impl_check
(emul_i, uni_impl_vals)[source]¶ Performs an implausibility cut-off check on the provided implausibility values uni_impl_vals at emulator iteration emul_i.
Parameters: - emul_i (int) – Number indicating the requested emulator iteration.
- uni_impl_vals (2D array_like) – Array containing all univariate implausibility values corresponding to all parameter sets for all data points.
Returns:
-
_evaluate_model
(emul_i, sam_set, data_idx)[source]¶ Evaluates the model for provided evaluation sample set sam_set at given data points data_idx.
This method automatically distributes the samples according to the various flags set in the
ModelLink
subclass.Parameters: - emul_i (int) – Number indicating the requested emulator iteration.
- sam_set (1D or 2D array_like) – Parameter/sample set to evaluate in the model.
- data_idx (list of tuples) – The list of data identifiers for which the model is requested to return the corresponding data values.
Returns:
-
_evaluate_sam_set
(emul_i, sam_set, exec_code)[source]¶ Evaluates a provided set of emulator evaluation samples sam_set at a given emulator iteration emul_i. The provided tuple of code snippets exec_code are executed using Python’s
exec()
function at specific points during the analysis.Parameters: - emul_i (int) – Number indicating the requested emulator iteration.
- sam_set (2D
ndarray
object) – Array containing model parameter value sets to be evaluated in all emulator systems in emulator iteration emul_i. - exec_code ({‘analyze’, ‘evaluate’, ‘hybrid’, ‘project’} or tuple) – Tuple of five code snippets
(pre_code, eval_code, anal_code, post_code, exit_code)
to be executed at specific points during the analysis. If string, use one of the built-in tuples incode_objects
instead.
Other Parameters: - pre_code (str or code object) – Code snippet to be executed before the evaluation of sam_set starts.
- eval_code (str or code object) – Code snippet to be executed after the evaluation of each sample in sam_set.
- anal_code (str or code object) – Code snippet to be executed after the analysis of each sample in sam_set. This code snippet is only executed by the controller.
- post_code (str or code object) – Code snippet to be executed after the evaluation of sam_set ends.
- exit_code (str or code object) – Code snippet to be executed before returning the results of the evaluation of sam_set. This code snippet is only executed by the controller.
Returns: results (object) – The object that is assigned to a local variable called results, which is defaulted to None if no code snippet sets it. Preferably, the execution of post_code and/or exit_code sets this variable. All MPI ranks return it.
Notes
If any of the code snippets is provided as a string, it will be compiled into a code object before starting the evaluation.
-
_get_default_parameters
()[source]¶ Generates a dict containing default values for all pipeline parameters.
Returns: par_dict (dict) – Dict containing all default pipeline parameter values.
-
_get_eval_sam_set
(emul_i)[source]¶ Generates an emulator evaluation sample set to be used for analyzing an emulator iteration. Currently uses the
lhd()
function.Parameters: emul_i (int) – Number indicating the requested emulator iteration. Returns: eval_sam_set (2D ndarray
object) – Array containing the evaluation samples.
-
_get_ext_real_set
(emul_i, ext_real_set)[source]¶ Processes an externally provided model realization set ext_real_set, containing the used sample set and the corresponding data value set, to be used for the provided emul_i.
Parameters: - emul_i (int) – Number indicating the requested emulator iteration.
- ext_real_set (str, list, dict or None) – String indicating the suffix of the backup file that needs to be
used; list of dicts containing an externally calculated set of
model evaluation samples and its data values; a dict with keys
['sam_set', 'mod_set']
containing these dicts; or None if no external set needs to be used.
Returns:
-
_get_f_impl
(emul_i)[source]¶ Returns the fraction of parameter space that passed the implausibility checks during the analysis of the provided emulator iteration emul_i.
Parameters: emul_i (int) – Number indicating the requested emulator iteration. Returns: f_impl (float) – The fraction of parameter space that is still plausible.
-
_get_impl_space
(emul_i)[source]¶ Returns the boundaries of the hypercube that encloses the parameter space in which the plausible space of the provided emulator iteration emul_i is defined.
Parameters: emul_i (int) – Number indicating the requested emulator iteration. Returns: impl_space (2D ndarray
object) – The requested hypercube boundaries.Note
The parameter space over which plausible space is defined is always equal to the emulator space of the next iteration. This means that reanalyzing an iteration can change the result of this function.
-
_get_iteration_data
(emul_i, sam_set, ext_sam_set, ext_mod_set)[source]¶ Obtains the model realization data for given emulator iteration emul_i by evaluating the provided sam_set in the model and distributing model outputs to the correct emulator systems.
Parameters: - emul_i (int) – Number indicating the requested emulator iteration.
- sam_set (2D
ndarray
object) – Array containing the model evaluation samples. - ext_sam_set (1D or 2D
ndarray
object) – Array containing the externally provided model evaluation samples. - ext_mod_set (1D or 2D
ndarray
object) – Array containing the model outputs of all specified externally provided model evaluation samples.
Generates
-
_get_md_var
(emul_i, par_set)[source]¶ Retrieves the model discrepancy variances, which includes all variances that are created by the model provided by the
ModelLink
instance. This method tries to call theget_md_var()
method, and assumes a default model discrepancy variance of1/6th
the data value if it cannot be called. If the data value space is not linear, then this default value is calculated such to reflect that.Parameters: - emul_i (int) – Number indicating the requested emulator iteration.
- par_set (1D
ndarray
object) – Model parameter value set to calculate the model discrepancy variances for.
Returns: var_md (2D
ndarray
object) – Variance of the model discrepancy.
-
_get_mock_data
(mock_par)[source]¶ Generates mock data and loads it into the
ModelLink
object that was provided during class initialization. This function overwrites theModelLink
properties holding the parameter estimates, data values and data errors.Parameters: mock_par (1D array_like or None) – If 1D array_like, use the provided parameter estimates to create the mock data. If None, a random parameter set will be generated as parameter estimates. Generates
Overwrites the corresponding
ModelLink
class properties with the generated values.
-
_get_n_eval_sam
(emul_i)[source]¶ This function calculates the total number of emulator evaluation samples at a given emulator iteration emul_i from
base_eval_sam
.Parameters: emul_i (int) – Number indicating the requested emulator iteration. Returns: n_eval_sam (int) – Total number of emulator evaluation samples.
-
_get_paths
(root_dir, working_dir, prefix)[source]¶ Obtains the path for the root directory, working directory and parameters file for PRISM.
Parameters: - root_dir (str or None) – String containing the absolute path of the root directory where all working directories are stored. If None, root directory will be set to the directory this class was initialized in.
- working_dir (str, bool or None) – String containing the name of the working directory of the emulator in root_dir. If True, a new working directory will be created in root_dir. If None or False, working directory is set to the last one that was created in root_dir that starts with the given prefix. Note that providing an integer instead of a bool will not work here. If no directories are found, one will be created.
- prefix (str or None) – String containing a prefix that is used for naming new working directories or scan for existing ones. If None, all directories in root_dir are considered working directories and ‘prism_’ will be used as a prefix for new ones.
Generates
The absolute paths to the root directory, working directory, emulator master HDF5-file and PRISM parameters file.
-
_get_uni_impl
(emul_i, par_set, adj_exp_val, adj_var_val)[source]¶ Calculates the univariate implausibility values at a given emulator iteration emul_i for specified expectation and variance values adj_exp_val and adj_var_val, corresponding to given par_set.
Parameters: - emul_i (int) – Number indicating the requested emulator iteration.
- par_set (1D
ndarray
object) – Model parameter value set to calculate the univariate implausibility values for. Only used to pass to theget_md_var()
method. - adj_exp_val, adj_var_val (1D array_like) – The adjusted expectation and variance values to calculate the univariate implausibility for.
Returns: uni_impl_val (1D
ndarray
object) – Univariate implausibility value for all requested emulator systems.
-
_load_data
()[source]¶ Loads in all the important pipeline data into memory for the controller rank. If it is detected that the last emulator iteration has not been analyzed yet, the implausibility analysis parameters are taken from the PRISM parameters dict and temporarily stored in memory.
Generates
All relevant pipeline data up to the last emulator iteration is loaded into memory.
-
_make_call
(exec_fn, *args, **kwargs)[source]¶ Sends the provided exec_fn to all worker ranks, if they are listening for calls, and tells them to execute it using the provided args and kwargs. All ranks that call this function will execute exec_fn as well.
If used within the
WorkerMode
context manager, this function should only be called by the controller. If not, it should be called by all valid ranks that must execute exec_fn.Parameters: - exec_fn (str or callable) – If string, a callable attribute of this
Pipeline
instance or a callable object that the workers should execute if not. - args (positional arguments) – Positional arguments that need to be provided to exec_fn.
- kwargs (keyword arguments) – Keyword arguments that need to be provided to exec_fn.
Returns: out (object) – The object returned by executing exec_fn. Note that only ranks that directly call this function return, as workers in worker mode cannot do so.
Note
Changed in version 1.2.0: If any entry in args or kwargs is a string written as ‘pipe.XXX’, it is assumed that ‘XXX’ refers to a
Pipeline
attribute of the MPI rank receiving the call. It will be replaced with the corresponding attribute before exec_fn is called.- exec_fn (str or callable) – If string, a callable attribute of this
-
_make_call_workers
(exec_fn, *args, **kwargs)[source]¶ Sends the provided exec_fn to all worker ranks, if they are listening for calls, and tells them to execute it using the provided args and kwargs.
If used within the
WorkerMode
context manager, this function should only be called by the controller. If not, it should be called by all valid ranks that must execute exec_fn.Parameters: - exec_fn (str or callable) – If string, a callable attribute of this
Pipeline
instance or a callable object that the workers should execute if not. - args (positional arguments) – Positional arguments that need to be provided to exec_fn.
- kwargs (keyword arguments) – Keyword arguments that need to be provided to exec_fn.
Returns: out (object) – The object returned by executing exec_fn. Note that only ranks that directly call this function return, as workers in worker mode cannot do so.
Note
Changed in version 1.2.0: If any entry in args or kwargs is a string written as ‘pipe.XXX’, it is assumed that ‘XXX’ refers to a
Pipeline
attribute of the MPI rank receiving the call. It will be replaced with the corresponding attribute before exec_fn is called.- exec_fn (str or callable) – If string, a callable attribute of this
-
_multi_call_model
(emul_i, sam_set, data_idx)[source]¶ Obtain the output set corresponding to the provided data_idx that is generated by the model for a given model parameter sample set sam_set. The current emulator iteration emul_i is also provided in case it is required by the
ModelLink
subclass.This is a multi-version of
_call_model()
.Parameters: - emul_i (int) – Number indicating the requested emulator iteration.
- sam_set (2D array_like) – Model parameter sample set to calculate the model output for.
- data_idx (list of tuples) – The list of data identifiers for which the model is requested to return the corresponding data values.
Returns: mod_set (2D
ndarray
object) – Model output set corresponding to given sam_set.
-
_read_parameters
(prism_par)[source]¶ Reads in all parameters in the provided prism_par and saves them as a dict in the current
Pipeline
instance.
-
_save_data
(data_dict)[source]¶ Saves a given data dict
{keyword: data}
at the last emulator iteration to the HDF5-file and as an data attribute to the currentPipeline
instance.Parameters: data_dict (dict) – Dict containing the data that needs to be saved to the HDF5-file.
Keyword Arguments: - keyword ({‘impl_par’; ‘impl_sam’; ‘n_eval_sam’}) – String specifying the type of data that needs to be saved.
- data ({int; float; str; array_like} or dict) – The actual data that needs to be saved at data keyword keyword. If dict, save every item individually.
Generates
The specified data is saved to the HDF5-file.
-
_save_statistics
(emul_i, stat_dict)[source]¶ Saves a given statistics dict
{keyword: [value, unit]}
at emulator iteration emul_i to the HDF5-file. The provided values are always saved as strings.Parameters: emul_i (int) – Number indicating the requested emulator iteration.
Keyword Arguments: - keyword (str) – String containing the name/keyword of the statistic that is being saved.
- value (int, float or str) – The value of the statistic.
- unit (str) – The unit of the statistic.
-
_set_impl_par
(impl_cut)[source]¶ Sets the
impl_cut
andcut_idx
properties for implausibility evaluations usingprism_dict
and the provided impl_cut.Parameters: impl_cut (list of float or None) – Incomplete, shortened impl_cut-offs list to be used during the analysis of this emulator iteration. If None, use prism_dict
instead.Generates
- impl_cut : 1D
ndarray
object - Full list containing the impl_cut-offs for all data points provided to the emulator.
- cut_idx : int
- Index of the first impl_cut-off in the impl_cut list that is not 0.
- impl_cut : 1D
-
_set_parameters
()[source]¶ Sets the Pipeline parameters from the
prism_dict
property and saves them in the currentPipeline
instance.
-
analyze
(*, impl_cut=None)[source]¶ Analyzes the emulator at the last emulator iteration for a large number of emulator evaluation samples. All samples that survive the implausibility checks set by the provided impl_cut, are used in the construction of the next emulator iteration.
Other Parameters: impl_cut (list of float or None. Default: None) – Incomplete, shortened implausibility cut-offs list to be used during the analysis of this emulator iteration. If None, the currently set implausibility cut-off values ( impl_cut
) will be used.Generates
- impl_sam : 2D
ndarray
object - Array containing all emulator evaluation samples that survived the implausibility checks.
- impl_sam : 2D
-
construct
(emul_i=None, *, analyze=True, ext_real_set=None, force=False)[source]¶ Constructs the emulator at the specified emulator iteration emul_i, and performs an implausibility analysis on the emulator iteration right afterward if requested (
analyze()
).Other Parameters: - emul_i (int or None. Default: None) – If int, number indicating the requested emulator iteration. If None, the next iteration of the emulator will be constructed.
- analyze (bool. Default: True) – Bool indicating whether or not to perform an analysis after the
specified emulator iteration has been successfully constructed,
which is required for making projections (
project()
) and constructing the next iteration. - ext_real_set (str, list, dict or None. Default: None) – String indicating the suffix of the backup file that needs to be
used; list of dicts containing an externally calculated set of
model evaluation samples and its data values; a dict with keys
['sam_set', 'mod_set']
containing these dicts; or None if no external set needs to be used. - force (bool. Default: False) – Controls what to do if the specified emulator iteration emul_i already (partly) exists. If False, finish construction of the specified iteration or skip it if already finished. If True, reconstruct the specified iteration entirely.
Generates
A new HDF5-group with the emulator iteration as its name, in the loaded emulator master file, containing emulator data required for this emulator iteration.
Notes
Using an emulator iteration that has been (partly) constructed before, will finish construction or skip it if already finished when force is False; or it will delete that and all following iterations, and reconstruct the specified iteration when force is True. Using emul_i = 1 and force is True is equivalent to reconstructing the entire emulator.
If no implausibility analysis is requested, then the implausibility parameters are taken from the PRISM parameters dict and temporarily stored in memory in order to enable the usage of the
evaluate()
method.
-
crystal
()¶ Creates an instance of
QApplication
or retrieves it if one already exists, and starts Crystal, PRISM’s Projection GUI.Crystal provides an interactive way of creating projection figures, as opposed to the static and linear method provided by
project()
. It is made to make it easier to create; view; compare; and analyze large numbers of projection figures. All options available in theproject()
method can also be accessed through Crystal.As with all
Pipeline
user methods, this function must be called by all MPI ranks.New in version 1.2.0.
-
details
(emul_i=None)[source]¶ Prints the details/properties of the currently loaded
Pipeline
instance at given emulator iteration emul_i. SeeProps
for detailed descriptions of all printed properties.Other Parameters: emul_i (int or None. Default: None) – If int, number indicating the requested emulator iteration. If None, the last iteration of the emulator will be used. Props
- Working directory
- The relative path to the working directory of the emulator starting at the current working directory.
- Emulator type
- The type of this emulator, corresponding to the
emul_type
of the provided emul_type duringPipeline
initialization. If no emulator type was provided during initialization, this is ‘default’. - ModelLink subclass
- Name of the
ModelLink
subclass used to construct this emulator. - Emulation method
- Indicates the combination of regression and Gaussian emulation methods that have been used for this emulator.
- Mock data used?
- Whether or not mock data has been used to construct this emulator. If so, the printed estimates for all model parameters are the parameter values used to create the mock data.
- Emulator iteration
- The iteration of the emulator this details overview is about. By default, this is the last (partly) constructed iteration.
- Construction completed?
- Whether or not the construction of this emulator iteration is completed. If not, the missing components for each emulator system are listed and the remaining information of this iteration is not printed.
- Plausible regions?
- Whether or not plausible regions have been found during the analysis of this emulator iteration. If no analysis has been done yet, “N/A” will be printed.
- Projections available?
- Whether or not projections have been created for this emulator iteration. If projections are available and analysis has been done, but with different implausibility cut-offs, a “desynced” note is added. Also prints number of available projections versus maximum number of projections in parentheses.
- # of model evaluation samples
- The total number of model evaluation samples used to construct all emulator iterations up to this iteration, with the number for every individual iteration in parentheses.
- # of plausible/analyzed samples
- The number of emulator evaluation samples that passed the implausibility check out of the total number of analyzed samples in this emulator iteration. This is the number of model evaluation samples that was/will be used for the construction of the next emulator iteration. If no analysis has been done, the numbers show up as “-“.
- % of parameter space remaining
- The percentage of the total number of analyzed samples that passed the implausibility check in this emulator iteration. If no analysis has been done, the number shows up as “-“.
- # of active/total parameters
- The number of model parameters that was considered active during
the construction of this emulator iteration, compared to the total
number of model parameters defined in the used
ModelLink
subclass. - # of emulated data points
- The number of data points that have been emulated in this emulator iteration.
- # of emulator systems
- The total number of emulator systems that are required in this emulator. The number of active emulator systems is equal to the number of data points.
- Parameter space
- Lists the name, lower and upper value boundaries and estimate (if
provided) of all model parameters defined in the used
ModelLink
subclass. An asterisk is printed in front of the parameter name if this model parameter was considered active during the construction of this emulator iteration. A question mark is used instead if the construction of this emulator iteration is not finished.
-
evaluate
(sam_set, emul_i=None)[source]¶ Evaluates the given model parameter sample set sam_set up to given emulator iteration emul_i. The output of this function depends on the number of dimensions in sam_set. The output is always provided on the controller rank.
Parameters: sam_set (1D or 2D array_like or dict) – Array containing model parameter value sets to be evaluated in the emulator up to emulator iteration emul_i. Other Parameters: emul_i (int or None. Default: None) – If int, number indicating the requested emulator iteration. If None, the last iteration of the emulator will be used. Returns: - impl_check (list of bool) – List of bool indicating whether or not the given samples passed the implausibility check at the given emulator iteration emul_i.
- emul_i_stop (list of int) – List containing the last emulator iterations at which the given samples are still within the plausible region of the emulator.
- adj_exp_val (2D
ndarray
object) – Array containing the adjusted expectation values for all given samples. - adj_var_val (2D
ndarray
object) – Array containing the adjusted variance values for all given samples. - uni_impl_val (2D
ndarray
object) – Array containing the univariate implausibility values for all given samples.
Prints (if 1D sam_set)
- emul_i_stop : int
- Last emulator iteration at which the given sample is still within the plausible region of the emulator.
- adj_exp_val : 1D
ndarray
object - The adjusted expectation values for the given sample.
- adj_var_val : 1D
ndarray
object - The adjusted variance values for the given sample.
- sigma_val : 1D
ndarray
object - The corresponding sigma value for the given sample.
- uni_impl_val : 1D
ndarray
object - The univariate implausibility values for the given sample.
Notes
If given emulator iteration emul_i has been analyzed before, the implausibility parameters of the last analysis are used. If not, then the values are used that were read in when the emulator was loaded or that have been set by the user.
-
project
(emul_i=None, proj_par=None, **kwargs)¶ Analyzes the emulator iteration emul_i and constructs a series of projection figures detailing the behavior of the model parameters corresponding to the given proj_par. The input and output depend on the number of model parameters
n_par
.All optional keyword arguments (except force) control various aspects of drawing the projection figures and do not affect the projection data that is saved to HDF5. This is instead influenced by the
proj_res
andproj_depth
properties.Parameters: - emul_i (int or None. Default: None) – If int, number indicating the requested emulator iteration. If None, the last iteration of the emulator will be used.
- proj_par (1D array_like of {int; str} or None. Default: None) – For which model parameters to construct the projection figures.
If 1D array_like, construct projection figures for all combinations
of provided model parameters that are active, with a string
referring to the name of the model parameter and an integer
referring to the position in which the model parameter is shown in
the
details()
method. If None, projection figures are made for all active model parameters.
Keyword Arguments: proj_type ({‘2D’; ‘3D’; ‘2D+3D’}. Default: ‘2D’ (2D), ‘2D+3D’ (nD)) – String indicating which projection type to create for all supplied active parameters. If
n_par
== 2, this is always ‘2D’ (and cannot be modified).figure (bool. Default: True) – Whether or not to create the projection figures. If True, the figures are calculated, drawn and saved. If False, the figures are calculated and their data is returned in a dict.
align ({‘row’/’horizontal’; ‘col’/’column’/’vertical’}. Default: ‘col’) – If figure is True, string indicating how to position the two subplots. If ‘row’/’horizontal’, the subplots are positioned on a single row. If ‘col’/’column’/’vertical’, the subplots are positioned on a single column.
show_cuts (bool. Default: False) – If figure is True and proj_type is not ‘3D’, whether to show all implausibility cut-offs in the 2D projections (True) or only the first cut-off (False).
smooth (bool. Default: False) – Controls what to do if a grid point contains no plausible samples, but does contain a minimum implausibility value below the first non-wildcard cut-off. If False, these values are kept, which can show up as artifact-like features in the projection figure. If True, these values are set to the first cut-off, removing them from the projection figure. Doing this may also remove interesting features. This does not affect the projection data saved to HDF5. Smoothed figures have an ‘_s’ string appended to their filenames.
use_par_space (bool. Default: False) – Controls whether to use the model parameter space (True) or the parameter space in which emulator iteration emul_i is defined (False) as the axes limits in the projection figure. If this is True and a parameter estimate is outside of the axes limit, a parameter estimate arrow is used instead of a line.
full_impl_rng (bool. Default: False) – Controls whether to use zero (True) or the lowest plotted value (False) as the lower bound for the minimum implausibility plot. Set this to True for the same plotting behavior as in versions earlier than v1.2.4.
New in version 1.2.4.
force (bool. Default: False) – Controls what to do if a projection hypercube has been calculated at the emulator iteration emul_i before. If False, it will use the previously acquired projection data to create the projection figure. If True, it will recalculate all the data required to create the projection figure. Note that this will also delete all associated projection figures.
fig_kwargs (dict. Default: {‘dpi’: 100}) – Dict of keyword arguments to be used when creating the subplots figure. It takes all optional arguments that can be provided to the
figure()
function.impl_kwargs_2D (dict. Default: {}) – Dict of keyword arguments to be used for making the minimum implausibility (top/left) plot in the 2D projection figures. It takes all optional arguments that can be provided to the
plot()
function.impl_kwargs_3D (dict. Default: {‘cmap’: ‘cmr.rainforest_r’}) – Dict of keyword arguments to be used for making the minimum implausibility (top/left) plot in the 3D projection figures. It takes all optional arguments that can be provided to the
hexbin()
function.los_kwargs_2D (dict. Default: {}) – Dict of keyword arguments to be used for making the line-of-sight (bottom/right) plot in the 2D projection figures. It takes all optional arguments that can be provided to the
plot()
function.los_kwargs_3D (dict. Default: {‘cmap’: ‘cmr.freeze’}) – Dict of keyword arguments to be used for making the line-of-sight (bottom/right) plot in the 3D projection figures. It takes all optional arguments that can be provided to the
hexbin()
function.line_kwargs_est (dict. Default: {‘linestyle’: ‘–’, ‘color’: ‘grey’}) – Dict of keyword arguments to be used for drawing the parameter estimate lines in both plots. It takes all optional arguments that can be provided to the
plot()
function.arrow_kwargs_est (dict. Default: {‘color’: ‘grey’, ‘fh_arrowlength’: 0.015, ‘fh_arrowwidth’: 0.025, ‘ft_arrowlength’: 1.5, ‘ft_arrowwidth’: 0.003, ‘rel_xpos’: 0.5, ‘rel_ypos’: 0.5}) – Dict of keyword arguments to be used for drawing the parameter estimate arrows in both plots. It takes a special set of arguments that are described in the `Notes`_.
line_kwargs_cut (dict. Default: {‘color’: ‘r’}) – Dict of keyword arguments to be used for drawing the implausibility cut-off line(s) in the top/left plot in the 2D projection figures. It takes all optional arguments that can be provided to the
plot()
function.
Returns: fig_data (dict of dicts) – Dict containing the data for every requested projection figure, split up into the ‘impl_min’ and ‘impl_los’ dicts. For 2D projections, every dict contains a list with the x and y values. For 3D projections, it contains the x, y and z values. Note that due to the figures being interpolations, the y/z values can be below zero or the line-of-sight values being above unity.
Generates (if figure is True)
A series of projection figures detailing the behavior of the model. The lay-out and output of the projection figures depend on the type of figure:
2D projection figure: The output will feature a figure with two subplots for every active model parameter (
n_par
). Every figure gives details about the behavior of the corresponding model parameter, by showing the minimum implausibility value (top/left) and the line-of-sight depth (bottom/right) obtained at the specified parameter value, independent of the values of the other parameters.3D projection figure (only if
n_par
> 2): The output will feature a figure with two subplots for every combination of two active model parameters that can be made (n_par*(n_par-1)/2
). Every figure gives details about the behavior of the corresponding model parameters, as well as their dependency on each other. This is done by showing the minimum implausibility (top/left) and the line-of-sight depth (bottom/right) obtained at the specified parameter values, independent of the values of the remaining model parameters.Notes
All colormaps defined in the
cmasher
package are loaded automatically when PRISM is imported and can be used.Some kwargs dicts have a subset of arguments that are reserved for this function (like x/y for impl_kwargs_2D and co.). These arguments are ignored when provided.
The arrow_kwargs_est argument takes all optional arguments that
arrow()
takes, plus a few special arguments that are described below:- fh_arrowlength : float. Default: 0.015
- The fraction of the appropriate axis limit used as the length of the arrow head.
- fh_arrowwidth : float. Default: 0.03
- The fraction of the appropriate axis limit used as the width of the arrow head.
- ft_arrowlength : float. Default: 1.5
- The relative length of the arrow tail compared to its head.
- ft_arrowwidth : float. Default: 0.002
- The fraction of the appropriate axis limit used as the width of the arrow tail.
- rel_xpos : float. Default: 0.5
- The relative x-position of the arrow. Only used if the arrow is drawn vertically and has no anchor in 3D projections.
- rel_ypos : float. Default: 0.5
- The relative y-position of the arrow. Only used if the arrow is drawn horizontally and has no anchor in 3D projections.
-
run
(emul_i=None, *, force=False)[source]¶ Calls the
construct()
method to start the construction of the given iteration of the emulator and creates the projection figures right afterward if this construction was successful.Other Parameters: - emul_i (int or None. Default: None) – If int, number indicating the requested emulator iteration. If None, the next iteration of the emulator will be constructed.
- force (bool. Default: False) – Controls what to do if the specified emulator iteration emul_i already (partly) exists. If False, finish construction of the specified iteration or skip it if already finished. If True, reconstruct the specified iteration entirely.
-
start_gui
()¶ Creates an instance of
QApplication
or retrieves it if one already exists, and starts Crystal, PRISM’s Projection GUI.Crystal provides an interactive way of creating projection figures, as opposed to the static and linear method provided by
project()
. It is made to make it easier to create; view; compare; and analyze large numbers of projection figures. All options available in theproject()
method can also be accessed through Crystal.As with all
Pipeline
user methods, this function must be called by all MPI ranks.New in version 1.2.0.
-
File
¶ Custom
File
class that has added logging and automatically useshdf5_file
as the HDF5-file to open.Type: File
-
base_eval_sam
¶ Base number of emulator evaluations used to analyze the emulator systems. This number is scaled up by the number of model parameters to generate the true number of emulator evaluations (
n_eval_sam
).Type: int
-
code_objects
¶ Collection of pre-compiled built-in code snippets that are used in the
_evaluate_sam_set()
method.Type: dict of code object
-
comm
¶ The MPI intra-communicator that is used in this
Pipeline
instance. By default, this isMPI.COMM_WORLD
.Type: Intracomm
-
cut_idx
¶ The index of the first non-wildcard in a list of implausibility values. This is equivalent to the number of wildcards leading the cut-off values in
impl_cut
.Type: int
-
do_active_anal
¶ Whether or not to do an active parameters analysis during the construction of the emulator systems.
Type: bool
-
do_logging
¶ Whether or not to save all logging messages. If False, all logging messages of level
INFO
and below are ignored. It also enables/disables the progress bar when making projections.Type: bool
-
freeze_active_par
¶ Whether or not previously active parameters always stay active if possible.
Type: bool
-
impl_cut
¶ The non-wildcard univariate implausibility cut-off values for an emulator iteration. Setting it with the reduced implausibility cut-off list will change the values of
cut_idx
and this property at the last emulator iteration.Type: list of float
-
impl_sam
¶ The model evaluation samples that will be added to the next emulator iteration.
Type: list of dict
-
is_controller
¶ Whether or not this MPI process is a controller rank. If no MPI is used, this is always True.
Type: bool
-
is_worker
¶ Whether or not this MPI process is a worker rank. If no MPI is used, this is always False.
Type: bool
-
n_eval_sam
¶ The number of evaluation samples used to analyze an emulator iteration of the emulator systems. The number of plausible evaluation samples is stored in
n_impl_sam
. It is zero if the specified iteration has not been analyzed yet.Type: int
-
n_impl_sam
¶ Number of model evaluation samples that passed the implausibility checks during the analysis of an emulator iteration. It is zero if the specified iteration has not been analyzed yet or has no plausible samples.
Type: int
-
n_sam_init
¶ Number of evaluation samples that will be used to construct the first iteration of the emulator systems.
Type: int
-
pot_active_par
¶ The potentially active parameters. Only parameters from this list can become active during the active parameters analysis. If
do_active_anal
is False, all parameters in this list will be active.Type: list of str
-
prism_dict
¶ Dictionary containing all PRISM parameters that were provided during
Pipeline
initialization.Type: dict
-
proj_depth
¶ Number of emulator evaluations that will be used to generate the samples in every grid point for the projection figures. Note that when making 2D projections of nD models, the used depth will be this number multiplied by
proj_res
.Type: int
-
proj_res
¶ Number of emulator evaluations that will be used to generate the grid for the projection figures.
Type: int
-
worker_mode
¶ Special context manager within which all code is executed in worker mode. In worker mode, all worker ranks are continuously listening for calls from the controller rank made with
_make_call()
or_make_call_workers()
.Note that all code within the context manager is executed by all ranks, with the worker ranks executing it after the controller rank exits. It is therefore advised to use an if-statement inside to make sure only the controller rank executes the code.
Using this context manager allows for easier use of PRISM in combination with serial/OpenMP codes (like MCMC methods). It also makes it easier to write long complex code that is mostly executed on the controller rank (but the worker ranks sometimes need to execute something).
All worker modes are independent of each other and can be created in a nested fashion.
Type: WorkerMode
-
-
class
prism._pipeline.
WorkerMode
(pipeline_obj)[source]¶ -
__enter__
()[source]¶ The provided
Pipeline
object enters worker mode, making all worker ranks start listening for calls from the controller rank until this context manager exits.
-
__exit__
(etype, value, tb)[source]¶ The provided
Pipeline
object exits worker mode, making all worker ranks stop listening for calls from the controller rank and resume normal code execution.
-
__init__
(pipeline_obj)[source]¶ Initialize the
WorkerMode
class using the MPI ranks defined in the provided pipeline_obj. This class should solely be initialized and finalized through thePipeline
class.New in version 1.2.0.
Parameters: pipeline_obj ( Pipeline
object) – The instance of thePipeline
class that is enabling this worker mode.
-
static
_process_call
(pipeline_obj, exec_fn, args, kwargs)[source]¶ Processes a call that was made with the
make_call()
ormake_call_workers()
method.This function should solely be called through either of these methods and never directly.
Parameters: - pipeline_obj (
Pipeline
object) – The instance of thePipeline
class that is making this call. - exec_fn (str or callable) – If string, a callable attribute of this
Pipeline
instance or a callable object that should be executed if not. - args (tuple) – Positional arguments that need to be provided to exec_fn.
- kwargs (dict) – Keyword arguments that need to be provided to exec_fn.
Returns: out (object) – The object returned by executing exec_fn.
Note
If any entry in args or kwargs is a string written as ‘pipe.XXX’, it is assumed that ‘XXX’ refers to a
Pipeline
attribute of the MPI rank receiving the call. It will be replaced with the corresponding attribute before exec_fn is called.- pipeline_obj (
-
static
_process_call_str
(pipeline_obj, string)[source]¶ Processes a provided string that was provided as an argument value to
_process_call()
.Parameters: Returns: out (str or object) – If string starts with ‘pipe.’, the corresponding
Pipeline
attribute will be returned. Else, string is returned.
-
listen_for_calls
()[source]¶ All worker ranks in the
comm
communicator start listening for calls from the corresponding controller rank and will attempt to execute the received message. Listening for calls continues until this context manager exits (__exit__()
is called).This method is automatically initialized and finalized when using the
worker_mode
context manager.
-
static
make_call
(pipeline_obj, exec_fn, *args, **kwargs)[source]¶ Sends the provided exec_fn to all worker ranks, if they are listening for calls, and tells them to execute it using the provided args and kwargs. All ranks that call this function will execute exec_fn as well.
If used within the
WorkerMode
context manager, this function should only be called by the controller. If not, it should be called by all valid ranks that must execute exec_fn.Parameters: - pipeline_obj (
Pipeline
object) – The instance of thePipeline
class that is making this call. - exec_fn (str or callable) – If string, a callable attribute of this
Pipeline
instance or a callable object that the workers should execute if not. - args (positional arguments) – Positional arguments that need to be provided to exec_fn.
- kwargs (keyword arguments) – Keyword arguments that need to be provided to exec_fn.
Returns: out (object) – The object returned by executing exec_fn. Note that only ranks that directly call this function return, as workers in worker mode cannot do so.
Note
Changed in version 1.2.0: If any entry in args or kwargs is a string written as ‘pipe.XXX’, it is assumed that ‘XXX’ refers to a
Pipeline
attribute of the MPI rank receiving the call. It will be replaced with the corresponding attribute before exec_fn is called.- pipeline_obj (
-
static
make_call_workers
(pipeline_obj, exec_fn, *args, **kwargs)[source]¶ Sends the provided exec_fn to all worker ranks, if they are listening for calls, and tells them to execute it using the provided args and kwargs.
If used within the
WorkerMode
context manager, this function should only be called by the controller. If not, it should be called by all valid ranks that must execute exec_fn.Parameters: - pipeline_obj (
Pipeline
object) – The instance of thePipeline
class that is making this call. - exec_fn (str or callable) – If string, a callable attribute of this
Pipeline
instance or a callable object that the workers should execute if not. - args (positional arguments) – Positional arguments that need to be provided to exec_fn.
- kwargs (keyword arguments) – Keyword arguments that need to be provided to exec_fn.
Returns: out (object) – The object returned by executing exec_fn. Note that only ranks that directly call this function return, as workers in worker mode cannot do so.
Note
Changed in version 1.2.0: If any entry in args or kwargs is a string written as ‘pipe.XXX’, it is assumed that ‘XXX’ refers to a
Pipeline
attribute of the MPI rank receiving the call. It will be replaced with the corresponding attribute before exec_fn is called.- pipeline_obj (
-
__weakref__
¶ list of weak references to the object (if defined)
-