dials.algorithms.refinement

Contains classes used to manage the reflections used during refinement, principally ReflectionManager.

class dials.algorithms.refinement.reflection_manager.BlockCalculator(experiments, reflections)

Bases: object

Utility class to calculate and set columns in the provided reflection table, which will be used during scan-varying refinement. The columns are a ‘block’ number and an associated ‘block_centre’, giving the image number in the centre of the block

__init__(experiments, reflections)

per_image()

Set one block per image for all experiments

per_width(width, deg=True)

Set blocks for all experiments according to a constant width

class dials.algorithms.refinement.reflection_manager.ReflectionManager(reflections, experiments, nref_per_degree=None, max_sample_size=None, min_sample_size=0, close_to_spindle_cutoff=0.02, scan_margin=0.0, outlier_detector=None, weighting_strategy_override=None)

Bases: object

A class to maintain information about observed and predicted reflections for refinement.

This new version keeps the reflections as a reflection table. Initialisation is not complete until the ReflectionManager is paired with a target function. Then, prediction can be done, followed by outlier rejection and any random sampling to form the working subset.

__init__(reflections, experiments, nref_per_degree=None, max_sample_size=None, min_sample_size=0, close_to_spindle_cutoff=0.02, scan_margin=0.0, outlier_detector=None, weighting_strategy_override=None)

experiment_type = 'scans'

filter_obs(sel)

Perform a flex array selection on the managed observations, so that external classes can filter according to criteria not available here

finalise(analysis=None)

Complete initialisation by performing outlier rejection and any requested subsetting. If a list of results from a CentroidAnalysis object is provided, these may be used to determine outlier rejection block widths

get_accepted_refs_size()

Return the number of observations that pass inclusion criteria and can potentially be used for refinement

get_centroid_analyser(debug=False)

Create a CentroidAnalysis object for the current reflections

get_free_reflections()

Return all reflections that were accepted for refinement but not chosen in the working set

get_indexed()

Return the reflections passed in as input

get_matches()

For every observation used in refinement return (a copy of) all data

get_obs()

Get the list of managed observations

get_sample_size()

Return the number of observations in the working set to be used for refinement

print_stats_on_matches()

Print some basic statistics on the matches

reset_accepted_reflections(reflections=None)

Reset use flags for all observations in preparation for a new set of predictions

class dials.algorithms.refinement.reflection_manager.ReflectionManagerFactory

Bases: object

static from_parameters_reflections_experiments(params, reflections, experiments, do_stills=False)

Given a set of parameters and models, build a reflection manager

Params:

params The input parameters

Returns

The reflection manager instance

class dials.algorithms.refinement.reflection_manager.StillsReflectionManager(reflections, experiments, nref_per_degree=None, max_sample_size=None, min_sample_size=0, close_to_spindle_cutoff=0.02, scan_margin=0.0, outlier_detector=None, weighting_strategy_override=None)

Bases: dials.algorithms.refinement.reflection_manager.ReflectionManager

Overloads for a Reflection Manager that does not exclude reflections too close to the spindle, and reports only information about X, Y, DelPsi residuals

experiment_type = 'stills'

print_stats_on_matches()

Print some basic statistics on the matches

Contains classes used to provide weighting schemes as strategies for ReflectionManagers.

class dials.algorithms.refinement.weighting_strategies.ConstantWeightingStrategy(wx, wy, wz, stills=False)

Bases: object

__init__(wx, wy, wz, stills=False)

calculate_weights(reflections)

Set weights to constant terms. If stills, the z weights are the ‘delpsical.weights’ attribute of the reflection table. Otherwise, use the usual ‘xyzobs.mm.weights’

class dials.algorithms.refinement.weighting_strategies.ExternalDelPsiWeightingStrategy

Bases: dials.algorithms.refinement.weighting_strategies.StatisticalWeightingStrategy

Defines a single method that provides a ReflectionManager with a strategy for calculating weights for stills refinement. This version uses statistical weights for X and Y and assume that the Delta Psi part is already provided in the reflection table

calculate_weights(reflections)

Statistical weights for X, Y. Weights for DeltaPsi must be already provided in the reflection table

class dials.algorithms.refinement.weighting_strategies.StatisticalWeightingStrategy

Bases: object

Defines a single method that provides a ReflectionManager with a strategy for calculating weights for refinement

static calculate_weights(reflections)

set ‘statistical weights’, that is w(x) = 1/var(x)

class dials.algorithms.refinement.weighting_strategies.StillsWeightingStrategy(delpsi_constant=1000000)

Bases: dials.algorithms.refinement.weighting_strategies.StatisticalWeightingStrategy

Defines a single method that provides a ReflectionManager with a strategy for calculating weights for refinement. This version uses statistical weights for X and Y and a fixed constant for the delta Psi part, defaulting to 1000000

__init__(delpsi_constant=1000000)

calculate_weights(reflections)

Include weights for DeltaPsi

Refiner is the refinement module public interface. RefinerFactory is what should usually be used to construct a Refiner.

class dials.algorithms.refinement.refiner.Refiner(experiments, pred_param, param_reporter, refman, target, refinery)

Bases: object

Public interface for performing DIALS refinement.

Public methods:

run rmsds get_experiments get_matches get_param_reporter parameter_correlation_plot selection_used_for_refinement predict_for_reflection_table predict_for_indexed

Notes

• The return value of run is a recorded history of the refinement

• The experiments accessor provides a copy of the experiments used by refinement

• get_matches exposes the function of the same name from the privately stored reflection manager

• The return value of selection_used_for_refinement is a flex.bool

__init__(experiments, pred_param, param_reporter, refman, target, refinery)

Mandatory arguments:

experiments - a dxtbx ExperimentList object pred_param - An object derived from the PredictionParameterisation class param_reporter -A ParameterReporter object refman - A ReflectionManager object target - An object derived from the Target class refinery - An object derived from the Refinery class

get_experiments()

Return a copy of the current refiner experiments

get_free_reflections()

Delegated to the reflection manager

get_matches()

Delegated to the reflection manager

get_param_reporter()

Get the ParameterReport object linked to this Refiner

get_parameter_correlation_matrix(step, col_select=None)

Return the correlation matrix between columns of the Jacobian at the specified refinement step. The parameter col_select can be used to select subsets of the full number of columns. The column labels are also returned as a list of strings

property history

Get the refinement engine’s step history

predict_for_indexed()

perform prediction for all the indexed reflections passed into refinement and additionally set the used_in_refinement flag. Do not compose the derivatives of states of the model as this is expensive and they are not needed outside of a refinement run

predict_for_reflection_table(reflections, skip_derivatives=False)

perform prediction for all reflections in the supplied table

print_exp_rmsd_table()

print useful output about refinement steps in the form of a simple table

print_out_of_sample_rmsd_table()

print out-of-sample RSMDs per step, if these were tracked

print_panel_rmsd_table()

print useful output about refinement steps in the form of a simple table

print_step_table()

print useful output about refinement steps in the form of a simple table

rmsds()

Return rmsds of the current model

rmsds_for_reflection_table(reflections)

Calculate unweighted RMSDs for the specified reflections

run()

Run refinement

selection_used_for_refinement()

Return a selection as a flex.bool in terms of the input reflection data of those reflections that were used in the final step of refinement.

class dials.algorithms.refinement.refiner.RefinerFactory

Bases: object

Factory class to create refiners

static config_refinery(params, target, pred_param, constraints_manager)

Given a set of parameters, a target class, a prediction parameterisation class and a constraints_manager (which could be None), build a refinery

Params:

params The input parameters

Returns

The refinery instance

static config_restraints(params, pred_param)

Given a set of user parameters plus a model parameterisation, create restraints plus a parameterisation of these restraints

Params:

params: The input PHIL parameters pred_param: A PredictionParameters object

Returns

A restraints parameterisation or None

static config_sparse(params, experiments)

Configure whether to use sparse datatypes

static config_target(params, experiments, reflection_manager, predictor, pred_param, restraints_param, do_stills, do_sparse)

classmethod from_parameters_data_experiments(params, reflections, experiments)

class dials.algorithms.refinement.refiner.ScanVaryingRefiner(experiments, pred_param, param_reporter, refman, target, refinery)

Bases: dials.algorithms.refinement.refiner.Refiner

Includes functionality to update the models with their states at scan-points after scan-varying refinement

Contains classes for refinement engines. Refinery is the shared interface, LevenbergMarquardtIterations, GaussNewtonIterations, SimpleLBFGS and LBFGScurvs are the current concrete implementations

class dials.algorithms.refinement.engine.AdaptLbfgs(*args, **kwargs)

Bases: dials.algorithms.refinement.engine.Refinery

Adapt Refinery for L-BFGS minimiser

__init__(*args, **kwargs)

callback_after_step(minimizer)

Do journalling, evaluate rmsds and return True if the target is reached to terminate the refinement.

compute_functional_and_gradients()

compute_functional_gradients_and_curvatures()

run_lbfgs(curvatures=False)

Run the minimiser, keeping track of its log.

class dials.algorithms.refinement.engine.AdaptLstbx(target, prediction_parameterisation, constraints_manager=None, log=None, tracking=None, max_iterations=None)

Bases: dials.algorithms.refinement.engine.Refinery, scitbx_lstbx_normal_equations_ext.non_linear_ls, scitbx.lstbx.normal_eqns.non_linear_ls_mixin

Adapt Refinery for lstbx

__init__((object)arg1, (object)n_parameters) → None :

C++ signature :

void __init__(_object*,int)

__init__( (object)arg1, (object)n_equations, (object)objective, (double)opposite_of_grad_objective, (object)normal_matrix) -> None :

C++ signature :

void __init__(_object*,unsigned long,double,scitbx::af::shared<double>,scitbx::af::versa<double, scitbx::af::packed_u_accessor>)

build_up(objective_only=False)

calculate_esds()

Calculate ESDs of parameters

parameter_vector_norm()

restart()

set_cholesky_factor()

Set the Cholesky factor required for ESD calculation. This method is valid only for the LSTBX dense matrix interface

step_backward()

step_forward()

class dials.algorithms.refinement.engine.DisableMPmixin

Bases: object

A mixin class that disables setting of nproc for multiprocessing

set_nproc(nproc)

class dials.algorithms.refinement.engine.GaussNewtonIterations(target, prediction_parameterisation, constraints_manager=None, log=None, tracking=None, max_iterations=20, **kwds)

Bases: dials.algorithms.refinement.engine.AdaptLstbx, scitbx.lstbx.normal_eqns_solving.iterations

Refinery implementation, using lstbx Gauss Newton iterations

__init__((object)arg1, (object)n_parameters) → None :

C++ signature :

void __init__(_object*,int)

__init__( (object)arg1, (object)n_equations, (object)objective, (double)opposite_of_grad_objective, (object)normal_matrix) -> None :

C++ signature :

void __init__(_object*,unsigned long,double,scitbx::af::shared<double>,scitbx::af::versa<double, scitbx::af::packed_u_accessor>)

convergence_as_shift_over_esd = 1e-05

damping_value = 0.0007

gradient_threshold = 1e-10

max_shift_over_esd = 15

run()

To be implemented by derived class. It is expected that each step of refinement be preceded by a call to prepare_for_step and followed by calls to update_journal and test_for_termination (in that order).

step_threshold = None

class dials.algorithms.refinement.engine.Journal

Bases: dict

Container in which to store information about refinement history.

This is simply a dict but provides some extra methods for access that maintain values as columns in a table. Refinery classes will use these methods while entering data to ensure the table remains consistent. Methods inherited from dict are not hidden for ease of use of this object when returned to the user.

add_column(key)

Add a new column named by key

add_row()

Add an element to the end of each of the columns. Fail if any columns are the wrong length

del_last_row()

Delete the last element from the each of the columns. Fail if any columns are the wrong length

classmethod from_json_file(filename)

get_nrows()

reason_for_termination = None

set_last_cell(key, value)

Set last cell in column given by key to value. Fail if the column is the wrong length

to_json_file(filename)

class dials.algorithms.refinement.engine.LBFGScurvs(*args, **kwargs)

Bases: dials.algorithms.refinement.engine.AdaptLbfgs

Refinery implementation using cctbx LBFGS with curvatures

compute_functional_gradients_diag()

run()

To be implemented by derived class. It is expected that each step of refinement be preceded by a call to prepare_for_step and followed by calls to update_journal and test_for_termination (in that order).

class dials.algorithms.refinement.engine.LevenbergMarquardtIterations(target, prediction_parameterisation, constraints_manager=None, log=None, tracking=None, max_iterations=20, **kwds)

Bases: dials.algorithms.refinement.engine.GaussNewtonIterations

Refinery implementation, employing lstbx Levenberg Marquadt iterations

add_constant_to_diagonal(mu)

Add the constant value mu to the diagonal of the normal matrix

property mu

report_progress(objective)

Callback within the refinement main loop that can be overridden to report the value of the objective function (and possibly) other details for long-running methods

run()

To be implemented by derived class. It is expected that each step of refinement be preceded by a call to prepare_for_step and followed by calls to update_journal and test_for_termination (in that order).

setup_mu()

Setup initial value for mu

tau = 0.001

class dials.algorithms.refinement.engine.Refinery(target, prediction_parameterisation, constraints_manager=None, log=None, tracking=None, max_iterations=None)

Bases: object

Interface for Refinery objects. This should be subclassed and the run method implemented.

__init__(target, prediction_parameterisation, constraints_manager=None, log=None, tracking=None, max_iterations=None)

get_correlation_matrix_for_step(step)

For each type of residual (e.g. X, Y, Phi), decompress and return the full 2D correlation matrix between columns of the Jacobian that was stored in the journal at the given step number. If not available, return None

get_num_steps()

jacobian_condition_number()

Calculate the condition number of the Jacobian, for tracking in the refinement journal, if requested. The condition number of a matrix A is defined as cond(A) = ||A|| ||inv(A)||. For a rectangular matrix the inverse operation refers to the Moore-Penrose pseudoinverse. Various matrix norms can be used, resulting in numerically different condition numbers, however the 2-norm is commonly used. In that case, the definition is equivalent to the ratio of the largest to smallest singular values of the matrix: cond(A) = sig_(A) / sig_min(A). That is the calculation that is performed here.

The condition number is a measure of how accurate the solution x to the equation Ax = b will be. Essentially it measures how errors are amplified through the linear equation. The condition number is large in the case that the columns of A are nearly linearly-dependent (and infinite for a singular matrix). We use it here then to detect situations where the correlation between effects of different parameter shifts becomes large and therefore refinement is problematic.

Note, the Jacobian used here does not include any additional rows due to restraints terms that might be applied, or any parameter reduction due to constraints. Therefore this condition number relates to the pure linearised (Gauss-Newton) step, which might not actually be what the refinement engine uses. It can be indicative of issues in the fundamental set up of the least squares problem, even if these issues are avoided in practice (e.g. by use of an algorithm like Levenberg-Marquardt, inclusion of restraints or parameter reduction).

prepare_for_step()

Update the parameterisation and prepare the target function

run()

To be implemented by derived class. It is expected that each step of refinement be preceded by a call to prepare_for_step and followed by calls to update_journal and test_for_termination (in that order).

set_nproc(nproc)

Set number of processors for multiprocessing. Override in derived classes if a policy dictates that this must not be user-controlled

split_jacobian_into_blocks()

Split the Jacobian into blocks each corresponding to a separate residual

test_for_termination()

Return True if refinement should be terminated

test_objective_increasing_but_not_nref()

Test for an increase in the objective value between steps. This could be caused simply by the number of matches between observations and predictions increasing. However, if the number of matches stayed the same or reduced then this is a bad sign.

test_rmsd_convergence()

Test for convergence of RMSDs

update_journal()

Append latest step information to the journal attributes

class dials.algorithms.refinement.engine.SimpleLBFGS(*args, **kwargs)

Bases: dials.algorithms.refinement.engine.AdaptLbfgs

Refinery implementation, using cctbx LBFGS with basic settings

run()

To be implemented by derived class. It is expected that each step of refinement be preceded by a call to prepare_for_step and followed by calls to update_journal and test_for_termination (in that order).

Contains classes used to construct a target function for refinement, principally Target and ReflectionManager.

class dials.algorithms.refinement.target.LeastSquaresPositionalResidualWithRmsdCutoff(experiments, predictor, reflection_manager, prediction_parameterisation, restraints_parameterisation, frac_binsize_cutoff=0.33333, absolute_cutoffs=None, gradient_calculation_blocksize=None)

Bases: dials.algorithms.refinement.target.Target

An implementation of the target class providing a least squares residual in terms of detector impact position X, Y and phi, terminating on achieved rmsd (or on intrisic convergence of the chosen minimiser)

__init__(experiments, predictor, reflection_manager, prediction_parameterisation, restraints_parameterisation, frac_binsize_cutoff=0.33333, absolute_cutoffs=None, gradient_calculation_blocksize=None)

achieved()

RMSD criterion for target achieved

class dials.algorithms.refinement.target.LeastSquaresPositionalResidualWithRmsdCutoffSparse(experiments, predictor, reflection_manager, prediction_parameterisation, restraints_parameterisation, frac_binsize_cutoff=0.33333, absolute_cutoffs=None, gradient_calculation_blocksize=None)

Bases: dials.algorithms.refinement.target.SparseGradientsMixin, dials.algorithms.refinement.target.LeastSquaresPositionalResidualWithRmsdCutoff

A version of the LeastSquaresPositionalResidualWithRmsdCutoff Target that uses a sparse matrix data structure for memory efficiency when there are a large number of Experiments

class dials.algorithms.refinement.target.SparseGradientsMixin

Bases: object

Mixin class to build a sparse Jacobian from gradients of the prediction formula stored as sparse vectors, and allow concatenation of gradient vectors that employed sparse storage.

class dials.algorithms.refinement.target.Target(experiments, predictor, reflection_manager, prediction_parameterisation, restraints_parameterisation=None, gradient_calculation_blocksize=None)

Bases: object

Abstract interface for a target function class

A Target object will be used by a Refinery. It will refer to a Reflection Manager to get a list of observations. It will perform reflection prediction on those observations and update the reflection manager with those predictions. It will then query the reflection manager to get a list of accepted reflections with observed and calculated positions. These are the reflections for use in refinement. It obtains the gradients of reflection positions from a relevant prediction parameterisation object. With all of this information in place, it calculates the value of the target function, the gradients of the target function and auxiliary information (e.g. RMSDs).

Concrete instances of this class implement the actual target function calculation. The base class should not determine what type of target function is used (e.g. least squares target), or limit whether the object is used for a detector space & phi residual, or a reciprocal space residual. This should all be set by a derived class.

__init__(experiments, predictor, reflection_manager, prediction_parameterisation, restraints_parameterisation=None, gradient_calculation_blocksize=None)

achieved()

return True to terminate the refinement.

calculate_gradients(reflections=None, callback=None)

delegate to the prediction_parameterisation object to calculate gradients for all the matched reflections, or just for those specified

compute_functional_gradients_and_curvatures(block=None)

calculate the value of the target function and its gradients. Set approximate curvatures as a side-effect

compute_residuals()

return the vector of residuals plus their weights

compute_residuals_and_gradients(block=None)

return the vector of residuals plus their gradients and weights for non-linear least squares methods

compute_restraints_functional_gradients_and_curvatures()

use the restraints_parameterisation object, if present, to calculate the least squares restraints objective plus gradients and approximate curvatures

compute_restraints_residuals_and_gradients()

delegate to the restraints_parameterisation object, if present, to calculate the vector of restraints residuals plus their gradients and weights for non-linear least squares methods

property dim

Get the number of dimensions of the target function

get_num_matches()

return the number of reflections currently used in the calculation

get_num_matches_for_experiment(iexp=0)

return the number of reflections currently used in the calculation

get_num_matches_for_panel(ipanel=0)

return the number of reflections currently used in the calculation

predict()

perform reflection prediction for the working reflections and update the reflection manager

predict_for_free_reflections()

perform prediction for the reflections not used for refinement

predict_for_reflection_table(reflections, skip_derivatives=False)

perform prediction for all reflections in the supplied table

rmsd_names = ['RMSD_X', 'RMSD_Y', 'RMSD_Phi']

rmsd_units = ['mm', 'mm', 'rad']

rmsds()

calculate unweighted RMSDs for the matches

rmsds_for_experiment(iexp=0)

calculate unweighted RMSDs for the selected experiment.

rmsds_for_panel(ipanel=0)

calculate unweighted RMSDs for the selected panel.

rmsds_for_reflection_table(reflections)

calculate unweighted RMSDs for the specified reflections. Caution: this assumes that the table reflections has the keys expected by _rmsds_core

split_matches_into_blocks(nproc=1)

Return a list of the matches, split into blocks according to the gradient_calculation_blocksize parameter and the number of processes (if relevant). The number of blocks will be set such that the total number of reflections being processed by concurrent processes does not exceed gradient_calculation_blocksize

update_matches(force=False)

ensure the observations matched to predictions are up to date

class dials.algorithms.refinement.target.TargetFactory

Bases: object

static from_parameters_and_experiments(params, experiments, reflection_manager, predictor, pred_param, restraints_param, do_stills=False, do_sparse=False)

Given a set of parameters, configure a factory to build a target function

Params:

params The input parameters

Returns

The target factory instance

class dials.algorithms.refinement.target_stills.LeastSquaresStillsResidualWithRmsdCutoff(experiments, predictor, reflection_manager, prediction_parameterisation, restraints_parameterisation, frac_binsize_cutoff=0.33333, absolute_cutoffs=None, gradient_calculation_blocksize=None)

Bases: dials.algorithms.refinement.target.Target

An implementation of the target class providing a least squares residual in terms of detector impact position X, Y and minimum rotation to the Ewald sphere, DeltaPsi. Terminates refinement on achieved rmsd (or on intrisic convergence of the chosen minimiser)

__init__(experiments, predictor, reflection_manager, prediction_parameterisation, restraints_parameterisation, frac_binsize_cutoff=0.33333, absolute_cutoffs=None, gradient_calculation_blocksize=None)

achieved()

RMSD criterion for target achieved

predict_for_reflection_table(reflections, skip_derivatives=False)

perform prediction for all reflections in the supplied table

rmsd_names = ['RMSD_X', 'RMSD_Y', 'RMSD_DeltaPsi']

rmsd_units = ['mm', 'mm', 'rad']

class dials.algorithms.refinement.target_stills.LeastSquaresStillsResidualWithRmsdCutoffSparse(experiments, predictor, reflection_manager, prediction_parameterisation, restraints_parameterisation, frac_binsize_cutoff=0.33333, absolute_cutoffs=None, gradient_calculation_blocksize=None)

Bases: dials.algorithms.refinement.target.SparseGradientsMixin, dials.algorithms.refinement.target_stills.LeastSquaresStillsResidualWithRmsdCutoff

A version of the LeastSquaresStillsResidualWithRmsdCutoff Target that uses a sparse matrix data structure for memory efficiency when there are a large number of Experiments