a106: integration & scaling -...

Tim Grüne

Advanced Macromolecular Structure Determination

Integration & ScalingTim Grüne

Dept. of Structural Chemistry, University of Göttingen

March 2011

http://shelx.uni-ac.gwdg.de

[email protected]

A106: Advanced Macromolecular Crystallography 1/69

http://shelx.uni-ac.gwdg.de

Tim Grüne

Overview

• mathematical reminder: vectors, matrices

• Data Integration

– Determination of spot intensity

– Application of the Ewald sphere

– Determination of the resolution cut-off

• Data Scaling


Tim Grüne

Vectors: Scalar Product

There are two common products of vectors: the scalar (or inner) product and the cross (or

wedge) product.

a

b

γ

The scalar product

a · b = a1b1 + a2b2

= ‖a‖‖b‖ cos γ

of two vectors a and b is a number (and not a vec-

tor).


Tim Grüne


γb sin( )

b

1

If one of the two vectors has unit length, the scalar

product is the length of the orthogonal projection of

the other onto the unit vector.


Tim Grüne


b

e1

e2

The scalar product can be used to decompose a

vector into basis vectors: If e1 and e2 are mutually

orthogonal unit vectors, one can express b as

b = (b · e1)e1 + (b · e2)e2 (1)

Everything about the scalar product shown here in two dimensions can be generalised to

three and more dimensions.


Tim Grüne

Vectors: Cross Product

The cross (or wedge) product is specific to three dimensions.

The cross product between two vectors a and b results in a new

vector a× b = c, which is perpendicular to both a and b.


Tim Grüne

Vectors: Cross Product

The cross (or wedge) product is specific to three dimensions.

The length of c equals the area of the parallelogram between a

and b.


Tim Grüne

Vectors: Scalar and Cross Product

The (appropriate) combination of scalar and cross product equals the volume of the box

spanned by three vectors a, b, and c,

V = |c · (a× b)|

This is true, whether or not c is the cross product of a and b


Tim Grüne

Matrices

Objects have coordinates, i.e. numbers that act like an address. Coordinates are bound to a

coordinate system which consists of an origin and directions.

6

3Coordinates of X: 3 steps forwards,

6 steps to the right.


Tim Grüne

Matrices

As we changes the coordinate system, e.g. by turning left, the coordinates change, too.

φ

6.7

−0.4

Coordinates of X: -0.4 steps forwards,

6.7 steps to the right.


Tim Grüne

Matrices

We can calculate the new coordinates of X after we turned left by an angle of φ with the help

of a rotation matrix: Xforward, newXright, new

=

cosφ − sinφsinφ cosφ

Xforward, oldXright, old

in the previous example where φ = 30◦:−0.40

6.70

=

0.866 −0.50.5 0.866

36


Tim Grüne

Matrices

Matrix operations are the mathematical abstraction of e.g.

• rotations of observer or objects

• certain deformations

• projections (like virtual 3D landscapes onto the computer screen)

All symmetry operations can be expressed by a combination of a matrix multiplication and a

vector addition.


Tim Grüne

General Positions

The International Tables Volume A [3] list all 230 space groups. The symmetry operations for

each space group can be read off the list of general positions.

E.g. the four general positions of P41 are:

(1) x, y, z (2) x, y, z + 12 (3) y, x, z + 1

4 (4) y, x, z + 34

Let’s extract the matrix operation for the third general position.


Tim Grüne

General Positions

We are looking for a (“deformation”) matrix A =( a11 a12 a13a21 a22 a23a31 a32 a33

)and a translation vector T =(

t1t2t3

), such that

yx

z + 14

=

a11 a12 a13a21 a22 a23a31 a32 a33

xyz

+

t1t2t3

So the first entry reads

−y = a11x+ a12y + a13z + t1

and one sees immediately that

a11 = 0 a12 = −1 a13 = 0 t1 = 0


Tim Grüne

General Positions

The second entry reads

x = a21x+ a22y + a23z + t2

⇒ a21 = 1 a22 = 0 a23 = 0 t2 = 0

and finally for the last row

z +1

4= a31x+ a32y + a33z + t3

⇒ a31 = 0 a32 = 0 a33 = 1 t3 =1

4


Tim Grüne

General Positions

Putting it all together, the 41 screw axis in P41 can be described asx′

y′

z′

=

0 −1 01 0 00 0 1

xyz

+

0014

This means, for every atom in the crystal (with coordinates (x, y, z)), there is the same type

of atom (with the same chemical environment) at position A( xyz

)+ T =

yx

z+14


Tim Grüne

Two Notes

1. Applying this symmetry operator twice gives

A

yx

z+14

+ T =

xy

z+12

i.e. the second symmetry operator listed in the International Tables.

2. 0 = cos 90◦ and 1 = sin 90◦, i.e. the matrix A represents a rotation

of φ = 90◦ (about the z-axis) as one would expect for a four-fold screw

axis.


Tim Grüne

Data Processing


Tim Grüne

Data Processing

The refinement of an X-ray structure usually assumes that the data stem from and idealised

crystal and instrument.

Simulated idealised Insulin crystal with0.1◦ mosaicity; Courtesy Kay Diederichs;

The data stored from an X-ray experiment (the mtz-file used

in refinement) contains

h, k, l, F, σF

or

h, k, l, I, σI

but no information about the actual data collection experiment

(e.g. shape of crystal, wavelength, detector properties, . . . ).


Tim Grüne

Perfect Experiments

Properties of perfect crystals from a perfect experiment

• no background

• no disorder and distortions

• perfect spot shape


Tim Grüne

Data Processing

Misfocused beam Poorly diffracting crystal

Sources of Errors

• Beam/ beamline

• Crystal imperfections

• Detector (overloads,

noise)

Courtesy N. Sanshvili, S. Corcoran;

APS Chicago

The goal of data processing (aka data integration) and data scaling is to “convert” the exper-

imental measurement into an idealised experiment.


Tim Grüne

Ideal Experiment

The data of the idealised experiment are the reflections, consisting of

1. Miller-Indices (hkl)

2. Intensities I(hkl)

3. standard deviations σI(hkl)

The unit cell dimensions and the space group are parameters for the experiment.


Tim Grüne

Cell Parameters and Spot Prediction

Spot predictions of Lysozyme with

correct cell

a = b = 78.5Å c = 36.7Å


Tim Grüne

Cell Parameters and Spot Prediction

Spot predictions of Lysozyme with in-

correct cell (by 2 Å)

a = b = 76.5Å c = 34.7Å


Tim Grüne

Understanding a Diffraction Experiment

A reflection on a diffraction image carries two different kinds of information:

1. Diffraction Geometry: the position of the spots on the detector

2. Spot intensity.


Tim Grüne

Laue Conditions

The diffraction geometry is explained by the Laue Conditions.

Only the unit cell vectors ~a, ~b, ~c and the scattering vector ~S(hkl) = ~Sout(hkl) − ~Sin are

necessary to determine where on the detector reflections occur, namely if and only if there is

are integers h, k, l, so that the Laue Conditions are fulfilled:

~S(hkl) · ~a = h

~S(hkl) ·~b = k

~S(hkl) · ~c = l~Sin

~S out(hkl) ~S

(hkl) D

etec

tor


Tim Grüne

The Ewald Sphere Construction

The Ewald Sphere Construction is a geometric interpretation of the Laue Condition and a lot

easier to understand.


Tim Grüne

Ewald Sphere Construction

(0,0) (1,0)(−3,0)

(1,1)

~

-

-

-

-

-

-

-

-

-

-

-

-

X-r

ayso

urce

~a∗~b∗

Reciprocal Lattice:

~a∗ =~b× ~c

(~a×~b) · ~c~b∗ =

~c× ~a(~a×~b) · ~c

~c∗ =~a×~b

(~a×~b) · ~c

Lattice points at:

h~a∗+ k~b∗+ l~c∗


Tim Grüne


|~Sin| = 1/λ~

-

-

-

-

-

-

-

-

-

-

-

-

X-r

ayso

urce

Ewald Sphere:

Place a sphere with

radius 1/λ touching

lattice origin and its

centre aligned with

the X-ray source.


Tim Grüne


(0, −2)

(−1, 2)

(−5, −3)

(−7, −1)

~S(−1,2,0)~

-

-

-

-

-

-

-

-

-

-

-

-

X-r

ayso

urce

Laue Conditions:

All lattice points on

surface fulfil Laue

Equations.

The scattering vector

S belonging to a re-

flection (hkl) points

from the origin to the

lattice point

h~a∗+ k~b∗+ l~c∗

.A106: Advanced Macromolecular Crystallography 30/69

Tim Grüne


De

tecto

r

(0, −2)

(−1, 2)

(−5, −3)

2θ

2θ′

(−7, −1)

~

-

-

-

-

-

-

-

-

-

-

-

-

X-r

ayso

urce

Those reflections which hit the detector are recorded as spots.


Tim Grüne


Dete

cto

r~

-

-

-

-

-

-

-

-

-

-

-

-

X-r

ayso

urce

Crystal rotation = Lattice rotation = More spots recorded

(Rot. axis perpendicular to slide)


Tim Grüne

Indexing

The Laue Conditions and the Ewald Sphere Construction predict the spot positions on a

detector given the crystal unit cell and the crystal orientation.

The first step of an X-ray experiment has to invert this prediction:

Find the crystal unit cell dimension and crystal orientation based on the detected

diffraction pattern.

This is called indexing and is based on strong reflections which are detected easily and

reliably.


Tim Grüne

Indexing

Each spot on the detector corresponds to a reflection and thus a reciprocal lattice point ~p∗i .

The back-transfer from spot to reciprocal lattice point depends on [1]:

1. (Xi, Yi) coordinates of spot on the detector

2. (X0, Y0) origin of the detector

3. The X-ray wavelength λ

4. ∆ distance between crystal and detector

5. Orientation of the crystal when the spot was detected relative to its initial position

(rotation angle).


Tim Grüne

Indexing

A macromolecular crystal produces a few hundred strong reflections within 5–10 frames.

Indexing consists of:

1. Find strong spots

2. Calculate lattice points p∗ for each spot

3. Determine reciprocal unit cell vectors ~a∗,~b∗, and ~c∗ so that as

many of the p∗ can be expressed as

~p∗ = h~a∗+ k~b∗+ l~c∗

The fact that the Miller-indices (hkl) must be integers imposes a strong restriction on finding

the reciprocal unit cell. Otherwise ~a∗, ~b∗, and ~c∗ could not be determined reliably.


Tim Grüne

Indexing

The last three quantities, λ, ∆, and the crystal rotation angle, are usually know with high

accuracy.

The spot coordinates are determined by the integration software and depend mostly on the

crystal quality (at least for macromolecular crystals).

The detector origin is the most sensitive parameter and often the “culprit” when indexing fails.


Tim Grüne

Finding the Detector Origin

There are two simple tricks to determine the detector origin (X0, Y0):

1) ice rings collect an images without your crystal at the same settings with ice (water) in the

loop in order produce ice rings.

Image of a lysozyme crystal with ice rings (courtesy

T. Beck).

The origin of the beam corresponds to the centre of any of

the ice rings. E.g. the program mosflm allows to fit these

rings to determine their centre.


Tim Grüne

Finding the Detector Origin

There are two simple tricks to determine the detector origin (X0, Y0):

2) test crystal collect a data set from a test crystal (lysozyme or insulin) with known param-

eters and integrate the data.

It is actually good practice to do this first thing, whenever one collects data at a synchrotron

to make sure that one understands the system and that the beam line is properly set up.


Tim Grüne

Refinement of Experimental Parameters

The experimental parameters

• Unit cell dimensions and orientation

• Beam direction

• Crystal-to-detector distance

• Crystal rotation axis

• Detector origin

are refined during data processing with the determined spots.


Tim Grüne

Refinement of Experimental Parameters

Lattice symmetry reduces the number of parameters, e.g. in a tetragonal lattice |~a| = |~b|

The data processing step can therefore be stabilised by imposing the lattice symmetry during

integration.

Data integration programs only “look” at the detector positions where spots are predicted from

the experimental parameters. Their proper determination is therefore substantial for a good

data set.


Tim Grüne

Spot Intensities

The main purpose of data integration aka data processing: determination of the spot intensi-

ties.

There are two different classes of spots which are treated differently:

• Strong spots and

• Weak spots

The difference is mostly due to the presence of noise in the data.


Tim Grüne

Spot & Background Noise

Inte

ns

ity

Detector Pixel

noiseideal reflection

sum = data

• “ideal” spot: Gaussian curve

• even for “perfect” crystal: noise on the

detector

• measured: sum of noise and spot


Tim Grüne

Strong Spots

3D profile Numeric values per pixel

Dark: spot area

Light: background area

1. Sum up pixel values in spot area2. Calculate average background from flat area3. Substract background per counted pixel

Strong spots: real spot area large, background cancels completely → summation re-

sults in correct intensity.


Tim Grüne

Integration

Weak Spots

3D profile Numeric cross-section

520

540

560

580

600

620

640

0 10 20 30 40 50 60 70 80 90 100

Ph

oto

n C

ou

nts

Detector Pixel

Spot area difficult to distinguish from background

Small area: significant variation in background, summation gives incorrect result.

Solution: Profile Fitting


Tim Grüne

Profile Fitting

1. Calculate an expected spot shape from strong spots

2. Scale the shape at the calculated spot position to the pixel

values found there

3. Subtract the average background

The last item is the reason why there are measurements with negative intensity (so-called

counting errors). Even though they physically do not make sense, they contain important

information [2].


Tim Grüne

Profile Fitting — HKL2000

HKL2000

• 2D - process one frame after the other

• for each (predicted) spot, consider disc around spot

• spot profile from strong spots within circle

Main Window of HKL2000 with spot

predictions and radius for profile fitting


Tim Grüne

HKL2000 — adjusting integration box

Default Adjusted

• Too small spot area

• Background boxes overlap with other spots

• Rrim = 4.2%, I/σI = 42.2

• Increased spot size

• Decreased box size for background

• Rrim = 4.1%, I/σI = 47.7


Tim Grüne

Profile Fitting — Mosflm

Mosflm

• 2D - process one frame after the other

• divide frame into 3× 3 or 5× 5 squares

• Average strong spots within each square→ arbitrary profile


Tim Grüne

Profile Fitting — XDS

XDS

• 3D - collect spots over several frames

• divide frame into 3× 3

• Fit strong spots within each square to 3D Gaussian curve

• Fit weak spots to this Gaussian


Tim Grüne

XDS — Example of Cross Section***** RUN-AVERAGE OF PROFILE # 1 *****

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 5 10 5 0 0 0

0 0 0 0 0 0 0 0 0 0 0 0 2 4 2 0 0 0 0 0 0 12 23 11 2 0 0

0 0 0 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 6 10 5 0 0 0

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 0 0 0 0

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

0 0 0 0 3 2 0 0 0 0 0 0 2 5 3 0 0 0 0 0 0 0 3 2 0 0 0

0 0 2 15 28 13 2 0 0 0 0 3 23 43 19 3 0 0 0 0 2 16 29 13 2 0 0

0 0 4 36 67 30 5 0 0 0 0 6 54100 43 6 0 0 0 0 4 38 68 30 5 0 0

0 0 2 16 30 15 3 0 0 0 0 3 22 43 21 4 0 0 0 0 2 15 30 15 3 0 0

0 0 0 2 4 2 0 0 0 0 0 0 3 6 3 0 0 0 0 0 0 2 4 3 0 0 0

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

0 0 0 6 10 5 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0

0 0 0 13 23 11 2 0 0 0 0 0 2 4 2 0 0 0 0 0 0 0 0 0 0 0 0

0 0 0 5 10 6 0 0 0 0 0 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0


Tim Grüne

Resolution Cut-Off

One consequence of profile fitting: Processing programs will integrate noise when the reso-

lution cut-off is set too high.

Including the noise is going to negatively influence the “real” data.

Therefore it is important to set the resolution limit during the integration instead of e.g. cutting

the poor quality part after integration.


Tim Grüne

Data Scaling


Tim Grüne

Equivalent Reflections

Reflections which are theoretically identical are called equivalent reflections.

E.g. in space group P41 the reflections (123) and0 −1 01 0 00 0 1

123

=

−213

(with the matrix A from the beginning of this lecture) are equivalent.


Tim Grüne

Data Scaling

Reasons for deviations from equality:

• non-spherical crystal

• Detector properties

• Detector settings

I ∝ V

Ph

osp

ho

r L

aye

r

De

tecto

r

Syste

m


Tim Grüne

Merging

Scaling also usually merges the data:

1. Determine average intensity for equivalent reflections

2. Replace all equivalent reflections with average


Tim Grüne

Averaging Equivalent Reflections

I

Outlier, e.g obscured by beam stop

Intensities of a set of equivalent reflections.


Tim Grüne


I

Method 1: Weighted Mean,

〈I〉 =∑wiIi∑wi

where wi =1

σ2i (Ii)


Tim Grüne


I

Method 1: Weighted Mean,


where wi =1

σ2i (Ii)

〈I〉

Problem: Outlier with small I will have small σ(I) and

therefore large weight w and dominate 〈I〉


Tim Grüne


I

Method 2: Unweighted Mean,

〈I〉 =∑IiN


Tim Grüne


I

Method 2: Unweighted Mean,

〈I〉 =∑IiN

〈I〉

Better, but outliers still affect mean. After rejection of “out-

liers”, mean is falsified.


Tim Grüne


I

〈I〉

Method 3: Use median as mean (same number of obser-

vations above and below median).

Better. Robust method even with poor error estimates

σ(I).


Tim Grüne


I

〈I〉

Method 4: Iterative improvement of weights,


where (specific to SADABS)

wi =

(1−ti)2

σ2(Ii)+g·〈I〉2ti < 1

0 ti > 1 (outlier rejection)

and

ti =(Ii − 〈I〉)2

σ2(Ii) + 0.25 〈I〉2


Tim Grüne


I

〈I〉

Method 4: Iterative improvement of weights,


Robust & resistant to outliers: Itera-

tive weight determination is common

to most scaling programs. Outliers

(large ti) are strongly downweighted,

but not completely rejected. 0

1

Ii = <I> 1

ti

wi ∝ (1−ti)2


Tim Grüne

Integration and Scaling Programs

In order to use scaling programs which “don’t belong” to the corresponding integration pro-

gram, conversions are required. This gas become relatively simple recently. The following

scheme is not complete, and misses e.g. “d*trek”.

XDS HKL2000 SAINTMOSFLM

SCALA SCALEPACK SADABSCORRECT

(built−in to XDS)

x2sad

xds2sad

scalepack2mtzpointle

ss

Saint/Sadabs are only commercially available in conjunction with an X-ray machine from

Bruker-AXS. Unfortunately, Saint (in combination with Twinabs) is the only program to prop-

erly treat twinned data (to the lecturer’s knowledge).


Tim Grüne

Radiation Damage

Crystals exposed to X-rays suffer from radiation damage especially from (high intensity) syn-

chrotron radiation.

Radiation damage increases disorder, therefore high-resolution reflections are too weak com-

pared to low-resolution reflections.

This can be corrected for by amplifying the high-resolution reflections:

I → I ∗ eB sin2 θ/λ2with B = (1− x)B0 + xB1

x is proportional to the frame number (0 for the first image and 1 for the last).


Tim Grüne

Scaling as Error Diagnostics

Most scaling programs (notably SCALA and SADABS) print a lot of information about the

input data which allows to

• Recognize bad frames (e.g. underexposed frames)

⇒ exclude single frames

• Crystal slippage or poor centring

⇒ Re-collect data

• Radiation damage

⇒ cut data after a certain frame

• Bad detector areas

⇒ mask out area


Tim Grüne

Acknowledgement

The section about scaling was derived from a lecture kindly provided by G. M. Sheldrick


Tim Grüne

References

1. W. Kabsch, Integration, scaling, space-group assignment and post-refinement, Acta Crys-

tallogr. (2010), D66, pp. 133–144

2. S. French, K. Wilson, On the Treatment of Negative Intensity Observations, Acta Crystal-

logr. (1978), A34, pp. 517–525

3. T. Hahan (ed.), International Tables for Crystallography, Vol. A, Union of Crystallography


a106: integration & scaling -...

Documents