My Tweets

WADA: Why does EQAS not evaluate the laboratory z-scores (for a specific analyte) over time?

bgv — Fri, 25 Feb 2011 23:58:00 GMT

Test results of EQAS samples reported by a Laboratory are transformed into a z-score. Absolute z-scores below 2 are satisfactory, between 2 and 3 are questionable and over 3 are unsatisfactory (see my blog on z-scores). Under normal circumstances we expect that by natural variability scores will vary between -2 and 2 and that by exception a value below -2 or over +2 is found. However, also in the case that all z-scores obtained over time remain within the interval [-2,+2] there might still be a problem with the laboratory.

Indeed, it is very unlikely that a laboratory would generate a series of z-scores all with equal sign. This indicates the presence of a laboratory (+method) bias. This can be checked by evaluating the RSZ = Sum(z)/sqr(m) statistics (m = number of z-scores) (RSZ = Rescaled Sum of Z). Moreover it is very unlikely that all the time z-scores are found at one of the extremes of the [-2,+2] interval. This is evaluated by calculating the RSSZ statistics (Reduced Sum of Squares of Z) = Sum(z²)/m. Both RSZ and RSSZ test values can be evaluated by using the appropriate statistical test.

It is quite striking that several accreditation bodies (e.g. the Standards Concil of Canada, the European Department for the Quality of Medicines of the Council of Europe and the CANMET-MMSL ISO document ) incorporate this time aspect in their guidelines and WADA does not. This despite the fact that in doping testing the detection of systematic errors (or bias) is of great importance.

We should nevertheless warn against an over-interpretation of z-scores

- Comparing z-scores between rounds or between laboratories has to be done with great caution. A single laboratory operating consistently in line with the fitness for purpose criterion would typically produce z-scores in successive rounds covering the range –2 to +2: the following set [0.6, -0.8, 0.3, 1.7, 0.7, -0.1] would be typical. The small ups and downs between the scores do not indicate a change in performance – they arise by chance. So 1.7 is not ‘worse’ than 0.3: it does not indicate deterioration in performance!!

- Because of this ‘natural variation’ it is not sensible to make a ‘league table’ of laboratories (or to attribute points) based on their z-scores in a round. It is not valid to claim that a laboratory scoring 0.3 in a round is better than another scoring 1.7.

- Judgments based on a time-averaged z-scores require caution as well. Averages of z-scores obtained on a number of different analytes should not be used: they may well hide the fact that one of the analytes consistently gives a poor z-score. Averages of scores from the same analyte over several rounds need expert interpretation on a statistical basis, as we indicated above.

WADA Athlete Biological Passport Guideline allows manipulation of duplicate test results

bgv — Fri, 25 Feb 2011 10:29:00 GMT

The WADA Athlete Biological Passport Guidelines (Jan 2010) prescribe that each athletes' blood sample is analyzed in duplicate. Because of instrumental variation and for other reasons these duplicates will always give a different result. The Guideline defines a maximally accepted difference between duplicates for Hemoglobin (0.1 g/Dl) and for Reticulocytes Percentage (0.15 if first measurement < 1% and 0.25 if first measurement > 1%). In principle the accepted maximum difference between duplicates should be equal to the repeatability of the method. By imposing these values WADA requires a certain minimal performance. Several aspects need some more discussion.

First one should define what a duplicate analysis really is. From the Guideline it appears that WADA means a duplicate injection. Such (bad) practice only includes the instumental variation and disregards the effect of a poor sample preparation (e.g. homogenization). In analytical method validation practice all steps should be duplicated, including the sample preparation. I gues this also applies to clinical tests.

Second, both test results, the 1st one and the 2nd one have the same status. There is no statistically basis for deciding which test result to be reported. It is therefore unclear on which grounds WADA decided that the first result should be reported only and e.g. not the mean of these two results. Two analyses have been performed and it is logical to use all information obtained on the athletes blood sample.

Third, the Guideline is quite vague on what to do when a duplicate analysis fails the requirement of maximum difference. It mentions that a new duplicate analysis is necessary. This implicitely means that the results of the first duplicate are void. However, a too large value between duplicates in fact means that the method performs outside specifications. In principle this should be evaluated by the analysis of control samples. If the deviation persists, then instrument or other factors should be checked. It is only when the method is back within specifications that the measurement of samples can be continued. In this case it is logical that the results obtained with the second duplicate are reported. Without including such additional check with control samples and/or indication of an instrument malfunction, four (statistically) equivalent test results are collected on a single sample. Again there is no statistical evidence for only reporting the first one of the second duplicate. It would be better to consider all four results together and to check for a possible outlier.

The way it is now described by WADA, a test laboratory may repeat duplicate analysis until (by chance) good duplicates are obtained and then report the so called 'best' result. Such practice is unacceptable as it masks possibly large test variations.

Eight WADA accredited laboratories are in noncompliance with rule of maintaining proficiency by analyzing a minimum of 3000 Doping Control Samples

bgv — Thu, 10 Feb 2011 14:36:00 GMT

In 2009, eight of 35 WADA accredited laboratories did not comply to clause#2 of the letter of support when applying for accreditation. This letter of support should “Guarantee that, within two (2) years of obtaining accreditation, a minimum of 3000 samples from Code-compliant clients (as determined by WADA) will be provided annually to the laboratory for 3 years”. All eight laboratories were included in the statistics at least since 2006.

In addition WADA states that “in order to maintain proficiency, WADA accredited laboratories are required, within two (2) years of the effective date of the current version of the ISL (effective 1 January 2009), to analyze a minimum of 3000 Doping Control Samples provided annually by the Code-compliant Testing Authorities... If the number of samples falls below 3000 per year, WADA laboratory accreditation may be suspended or revoked...“. What happens when the 2010 statistics reveals a continuing violation of this rule by these eight laboratories? Last year the accreditation of one of these eight laboratories, the Malaysian laboratory, was revoked on June 18, 2010. What happened with their 35 Adverse Analytical Findings (mostly Anabolic Agents) reported in 2009?

World Anti Doping Agency (WADA) - Flaw #5: Quantification of Threshold Substances - Is the False Positive rate good enough?

bgv — Mon, 07 Feb 2011 21:49:00 GMT

In Technical document TD2010DL, WADA updated the Decision Limits (DL) of Threshold Substances. The finding of a value that exceeds the DL in a confirmatory analysis results in an Adverse Analytical Finding.

WADA accepts that the uncertainty associated to analytical findings may be different for different accredited laboratories. Therefore WADA formulated maximum limits for this uncertainty (called the Maximum Combined Standard Uncertainty, u_{c, Max}) at the Threshold level (T) for such substances present in the urine or blood. WADA distinguishes three levels (expressed as a percentage of T) of this uncertainty: 5, 10 and 15%. By participating in the EQAS programme, laboratories should demonstrate their compliance to this uncertainty requirement. We will discuss this aspect later on.

Based on u_{c, Max} WADA defines a guard band (g) which is a multiple of the Maximum Combined Standard Uncertainty. This guard band (g) should be added to the Threshold value in order to obtain the Decision Limt (DL).

DL= T + g

A value > DL is considered an Adverse Analytical Finding. A clean athlete with a value above DL is a false positive.

Question now is what the false positive rate is for clean athletes (thus also without involuntary intake of a Threshold Substance by self-medication or by using supplements). For that we need to make some assumptions. For instance, we assume a normal distribution (not evident) for the natural fluctuation of these substances in urine and/or blood. Furthermore we assume a normal distribution (not evident) for the test variability. We have also to assume that T is defined such that in only 2.5% (or perhaps 5%) of a large group of clean athletes a Threshold Substance may be present above T just by chance. The establishment of T is still a debate in literature and media.

For all substances that are measured with an allowed maximum uncertainty of 10% , WADA defined DL and the guard zone (g) such that (DL – T)/g = 2. For a normal distribution this gives a probability of 2, 27% of finding a value above DL when the true value equals T.

However, as we assumed that only 2.5% of clean athletes will have values above T (by chance), the probability that a false positive is found for a clean athlete is 2,5% of 2,27% equal to 0,0567%. This is 1 on 1764 samples! Is this good enough? It all depends on the consequences.

From the WADA statistics over 2008 we read that 274615 samples were analyzed with a total of 5523 adverse findings. If we assume no false positives in these findings, then there are 269092 clean samples and therefore expect a limit of 153 false positive samples (=0,0567% of 269092 clean samples) (this number can be refined by iteration). A total of 5523 adverse findings were reported over 2008. This means that maximally 2,77% of these findings may be a false positive.

It is questionable whether this is good enough to protect athletes against unfair allegations.

Ironically, the athlete is better off here when a laboratory with a high performance level (uncertainty less than u_{c, Max}) analyzes his/her samples as the guard band (g) is a multiple of u_{c, Max} !

So contrary to the situation with Prohibited Substances, better Laboratory Performance than required is now an advantage for the athlete in the situation of Threshold substances.

World Anti Doping Agency (WADA) - Flaw #4 - Different performance levels of laboratories cause arbitrariness in detecting prohibited substances

bgv — Sat, 05 Feb 2011 18:37:00 GMT

In their Technical Document on Minimum Required Performance Levels for the detection of prohibited substances, WADA recognizes that some accredited laboratories will be able to identify a wider range or lower concentrations of prohibited substances than other laboratories. Considering that such individual capabilities are encouraged by WADA, the need to define Minimum Required Performance Levels (MRPL) at which all accredited laboratories should minimally operate, was recognized by them. The MRPL is the level of a prohibited substance that an accredited laboratory should be able to routinely detect. For instance, all laboratories should be able to detect Clenbuterol at a level of 2ng/ml.

However, some accredited laboratories may have the capability to detect a prohibited substance such as Clenbuterol, at a much lower level. Indeed the level of Clenbuterol of 50 pg found for Alberto Contador is ~20 times lower than the MRPL defined by WADA!

According to WADA such a result below the MPRL should be considered to be an Adverse Analytical Finding leading to sanctions for the athlete.

This means that if the urine of Contador would have been analyzed by a different WADA accredited laboratory lacking the capability of a routine detection of Clenbuterol at a level lower than 2 ng/ml (the MRPL level), but complies to the WADA requirements, no Adverse Analytical Finding would have been obtained for Contador!

This arbitrariness caused by different performance levels of the laboratories that perform the analysis generates a devils dilemma.

One the one hand when a prohibited substance is detected the athlete should be sanctioned. On the other hand, from the point-of-view of the equality of rights, the conclusion that an athlete used a prohibited substance should not depend on the laboratory that did the analysis.

Indeed, from the reported WADA statistics we observe that for instance in 2008, the laboratory of Ghent reported 3% Adverse Analytical Findings, whereas the mean of all laboratories is about 1%. It is not excluded that this higher percentage of Adverse Analytical Findings is due to a better analytical capability of that laboratory. I guess that possible difficulties caused by different analytical capabilities of WADA accredited laboratories are avoided by their clause that “A” and “B” samples of an athlete are analyzed by the same analyst (and thus the same laboratory).

World Anti Doping Agency (WADA) External Quality Assurance Scheme (EQAS) - Flaw #3 – The analysis of the “B” sample

bgv — Wed, 02 Feb 2011 21:29:00 GMT

WADA states that:

Athlete samples (of urine or blood) are divided into an “A” and “B” sample. An adverse analytical finding obtained in an aliquot of sample “A” may be confirmed by analyzing another aliquot of the original “A” sample by a more sensitive or selective method. This confirmation testing may be repeated to obtain greater confidence in the adverse finding.

After the report of the adverse finding to the athlete, the “B“ sample is analyzed within 7 days, unless within that period the athlete waives his/her right for the analysis of the “B” sample and accepts the results of the “A” sample.

Initially WADA requested that a different analyst performs the “A” and “B” confirmation test. However, WADA states now in its Revision document that this requirement is considered unwarranted and technically unjustified due to the demonstration of satisfactory competence by a mandatory participation of the laboratory in the EQAS system.

This point-of-view is highly questionable as this procedure only includes a single source of variation: time. Other more important sources of variation are the analyst and laboratory. The assessment of the between-laboratory variation requires the set-up of collaborative studies. If the between-laboratory variation is an important factor, which I believe it is, the “B” sample should preferentially be analyzed by a second independent WADA accredited laboratory. As to some extent the between-laboratory variation may be estimated from the between-analyst variation, the analysis of the “B” sample by another analyst of the initial laboratory may be acceptable.

In any case a laboratory should estimate its between-analyst variation within the context of method validation. Indeed, it is highly improbable that a confirmation test will always be performed by the same analyst.

In my opinion, It should be the right of an athlete to request that the “B”sample is analyzed by a different WADA accredited laboratory than the “A” sample.