Association of Mexican-American Educators v. California

937 F.Supp. 1397 (1996)

The ASSOCIATION OF MEXICAN-AMERICAN EDUCATORS ("AMAE"), the California Association for Asian-Pacific Bilingual Education ("CAFABE"), Oakland Alliance of Black Educators, ("OABE"), on behalf of themselves, their members, and all others similarly situated; Sara MacNeil Boyd; Sam Genis; Marta LeClaire; Antoinette Williams; Diana Kwan; Toua Yang, Robert Williams; and Agnes Haynes, on behalf of themselves and all others similarly situated, Plaintiffs,
v.
The STATE OF CALIFORNIA and The California Commission on Teacher Credentialing, Defendants.

No. C-92-3874 WHO.

United States District Court, N.D. California.

September 17, 1996.

John T. Affeldt, Louis Salisbury, Armando M. Menocal, III, Public Advocates, Inc., San Francisco, CA, Brad Seligman, Mari Mayeda, The Impact Fund, Berkeley, CA, for plaintiffs.

Stephanie Wald, Daniel E. Lungren, California State Attorney General's Office, San Francisco, CA, Nancy E. Ryan, Charles A. Shanor, R. Lawrence Ashe, Paul, Hastings, Janofsky & Walker, Atlanta, GA, for defendants State of California and California Commission on Teacher Credentialing.

OPINION AND ORDER

ORRICK, District Judge.

To have the privilege of teaching in a public school in California, a person must pass a test in reading, writing, and mathematics known as the California Basic Educational Skills Test ("CBEST"), given by the California Commission on Teacher Credentialing ("CTC"). The CBEST was mandated by the California legislature in response to a public outcry about the perceived incompetence of many public school teachers.

Plaintiffs, representing a class of minority would-be teachers consisting of African-Americans, Latinos,^[1] and Asians, bring this action against the State of California ("State") and the CTC under Titles VI and VII of the Civil Rights Act of 1964, claiming that they are discriminated against by the insistence of defendants that they take and pass the CBEST before becoming public schoolteachers.

For the reasons set forth in this Opinion, which constitutes the Court's findings of fact and conclusions of law as required by Rule 52(a) of the Federal Rules of Civil Procedure, the Court rules in favor of defendants.^[2]

I.

Plaintiffs are the Association of Mexican-American Educators ("AMAE"), the California Association for Asian-Pacific Bilingual Education, the Oakland Alliance of Black Educators, and eight individuals. In this class action, they challenge the use of the CBEST as a requirement for certification to teach in the California public schools. Plaintiffs contend that the CBEST requirement violates Titles VI^[3] and VII of the Civil Rights Act of 1964 because it has a disparate impact on African-Americans, Latinos, and Asians. Defendants, while conceding that the CBEST results in some adverse impact on the plaintiff class, argue that the test is valid because it tests job-related skills and is justified by business necessity. This case began thirteen years ago, shortly after the CBEST was first administered in December 1982.^[4]

Effective February 1, 1983, the State legislature barred the CTC from issuing "any credential, permit, certificate, or renewal of an emergency credential to any person to serve in the public schools unless the person has demonstrated proficiency in basic reading, writing, and mathematics skills." Cal. Educ.Code § 44252(b). The legislature authorized the Superintendent of Public Instruction ("Superintendent") to "adopt an appropriate state test to measure proficiency in these basic skills." Id. § 44252(c). The CBEST was the result.^[5]

The CBEST is a pass-fail examination. It contains three sections — reading, writing, and mathematics. The reading and mathematics sections each contain 40 multiplechoice questions.^[6] The writing portion consists of two essay questions. The CBEST has undergone one major revision: The "higher order" math skills, such as geometry, were removed from the mathematics subtest prior to the first administration of the revised CBEST in August 1995.

A passing score on the CBEST is required for elementary school teachers, who hold multiple-subject credentials, and for secondary school teachers, who hold single-subject credentials in the areas of agriculture, art, business, English, foreign languages, health science, home economics, industrial and technology education, mathematics, music, physical education, science, and social science. See Cal.Educ.Code §§ 44256, 44257, 44259. The CBEST is also required for numerous nonteaching positions, including administrators, id. § 44270, school counselors or "pupil personnel services" positions, id. § 44266, librarians, id. § 44269, and school nurses, id. § 44267.5.

The CBEST is administered six times a year, and there is no limit on the number of times a candidate may sit for the examination. Furthermore, a candidate keeps his or her best score on any given section and need only retake the failed sections; once the candidate has accumulated a passing score on all three sections, the candidate has passed the CBEST. See id. § 44252.5(d).

In this case, the plaintiff class was certified by the Court as follows:

All Latinos, African Americans and Asians who have sought or are seeking California public school credentials and certificated positions who have been, are being, or will be adversely affected in their ability to obtain credentials and certificated positions by California Basic Educational Skills Test results.

(See Mem. Decision & Order filed July 19, 1994, at 22, as amended by Order filed Oct. 7, 1994.)

The eight individual plaintiffs in this lawsuit are all members of minority groups, seeking teaching or administrative credentials, who contend they were discriminated against as a result of the CBEST requirement. Each individual plaintiff has taken and failed the CBEST one or more times. Three of the plaintiffs eventually passed the CBEST.

Plaintiff Sara MacNeil Boyd ("Boyd"), an African-American woman, took and failed the CBEST four times. Boyd received her bachelor's degree in commerce from North Carolina Central University in 1955. She has credentials for secondary education and counseling/pupil personnel services. She completed both a master's degree in education and an administrative credentialing program at San Jose State University, but was unable to get an administrative credential because she could not pass the CBEST. By obtaining annual CBEST waivers from the CTC, however, Boyd was able to serve as a vice-principal from 1988 until her retirement in 1995.

Plaintiff Sam Genis ("Genis"), a Latino man, took and failed the CBEST four times. Genis earned his associate's degree from East Los Angeles College in 1976. He then obtained a bachelor's degree in Spanish from California State University, Los Angeles in 1979. From 1980 to 1983, he taught at Rio Vista Elementary School in the El Rancho Unified School District, but was unable to continue there because he had not passed the CBEST. He has since worked in private schools.

Plaintiff Agnes Haynes ("Haynes"), an African-American woman, took and failed the CBEST six times between 1991 and 1993, but subsequently passed. Haynes received a bachelor's degree in secondary education from Grambling College in Louisiana in 1964. She completed a master's degree in educational administration and an administrative credentialing program at San Francisco State University in 1995. By obtaining CBEST waivers, Haynes worked as an eighth-grade English and social studies teacher for two years in the Ravenswood City School District. She subsequently lost her teaching position because she had not passed the CBEST. She has since passed the test.

Plaintiff Diana Kwan ("Kwan"), an Asian woman, took and failed the CBEST four times. Kwan obtained an associate's degree from the City College of San Francisco in 1988 and a bachelor's degree in liberal studies from San Francisco State University in 1991. She has not entered a teacher preparation program because the program of her choice requires passage of the CBEST for admittance.^[7] Kwan currently works as a flight attendant for United Air Lines.

Plaintiff Marta Leclaire ("Leclaire"), who is Latina, took and failed the CBEST four times. Leclaire received an associate's degree from City College of San Francisco in 1972. She earned her bachelor's degree at San Francisco State University in developmental psychology in 1976. She completed a teacher credentialing program in multiple subjects/elementary education at San Francisco State University in 1978, but could not obtain a multiple-subject credential because she has not passed the CBEST. Leclaire does possess a general school services credential, which allows her to teach in child centers.

Plaintiff Antoinette Williams, an African-American woman, took the CBEST once in 1992 and failed it. She received her bachelor's degree in sociology from Fontbonne College in Missouri in 1979. She seeks a substitute teaching credential but has not been able to obtain one because she has not passed the CBEST.

Plaintiff Robert Williams ("Williams"), an African-American man, took the CBEST ten times and passed all three sections by August 1994. Williams obtained a bachelor's degree in physical education from Linfield College in Oregon in 1974. In 1975, he earned a master's degree in physical education from Stanford University and obtained a credential to teach physical education. Williams later completed an administrative credentialing program at California State University, Hayward, but was unable to obtain an administrative credential from the CTC until he passed the CBEST in 1994. Williams has worked in the San Leandro Unified School District since 1986, first as a physical education teacher, later as an assistant principal for two years, and most recently as a teacher on special assignment in human relations for the district.

Plaintiff Toua Yang ("Yang"), an Asian male, took and failed the CBEST seventeen times between 1991 and 1995. He has since passed the test. Yang received an associate's degree from Merced College in 1989. He earned a bachelor's degree in liberal studies from California State University, Sacramento in 1992. He also completed a teacher credentialing program at California State University, Sacramento, but was not credentialed because he did not pass the CBEST test (until 1995). Yang has served as a substitute teacher in the Sacramento City Unified School District since 1994.

The Court conducted the trial in two phases. The first phase consisted of the testimony of fact witnesses and took place over five days in February and March 1996. The second phase, during which expert testimony was presented, took place from June 3 to June 19, 1996.

II.

A.

1.

"The direction in which education starts a man will determine his future life." Plato, The Republic bk. IV, 425-B.

Teachers occupy a special position of trust in our society. They are entrusted with the education of our children, the importance of which one would be hard-pressed to exaggerate. A child's education is crucial not only to that child's individual prospects; in the aggregate, the education of all children has a profound effect on the future of the state, and indeed the country, in which we live. "A teacher affects eternity; he can never tell where his influence ends." Henry Brooks Adams, The Education of Henry Adams (1907).

As has often been observed, a teacher's job involves far more than simply instruction by rote. Teachers have the power to inspire in their students a love of learning and of knowledge, even a will to achieve and to fulfill their potential. "In shaping the students' experience to achieve educational goals, teachers by necessity have wide discretion over the way course material is communicated to students. They are responsible for presenting and explaining the subject matter in a way that is both comprehensible and inspiring." Ambach v. Norwick, 441 U.S. 68, 78, 99 S.Ct. 1589, 1595, 60 L.Ed.2d 49 (1979).

At the same time, however, the importance of the basic skills cannot be ignored. Teachers are also role models. Students learn not only what they are taught directly, but also what they observe. "Part of a teacher's responsibility is to set an example for his students and to act as a role model, a responsibility made necessary by the fact that students spend more time with their teachers than with any persons other than immediate family members and closest friends." Hoagland v. Mount Vernon Sch. Dist. No. 320, 23 Wash.App. 650, 597 P.2d 1376, 1382 (1979) (Dore, J., dissenting), aff'd, 95 Wash.2d 424, 623 P.2d 1156 (1981). Schoolteachers who use improper grammar or spelling, or who make mistakes in simple calculations, model that behavior to their students — much to the detriment of their education. The same can be said for school principals, librarians, and guidance counselors.

Given the significance of the teacher's role, the State has an obligation to the public "to maintain the highest standards of fitness and competence for the weighty task of educating young impressionable students." Id. As the Supreme Court has observed:

Our society grows increasingly complex, and our need for trained leaders increases correspondingly. Appellant's case represents, perhaps, the epitome of that need, for he is attempting to obtain an advanced degree in education, to become, by definition, a leader and trainer of others. Those who will come under his guidance and influence must be directly affected by the education he receives. Their own education and development will necessarily suffer to the extent that his own training is [inadequate].

McLaurin v. Oklahoma State Regents for Higher Educ., 339 U.S. 637, 641, 70 S.Ct. 851, 853, 94 L.Ed. 1149 (1950). What was true nearly fifty years ago remains true today, and has perhaps become even more urgent. Our economy has largely evolved from one driven by manual labor to one driven by mental labor, and literacy — of all kinds — has grown increasingly critical.

2.

"[I]f you improve a teacher, you improve a school." Willie v. Commissioner, 57 T.C. 383, 389, 1971 WL 2630 (1971).

Against this backdrop, the Court must consider the CBEST. The Court is called upon to decide whether teachers in California's public schools — all of whom have college degrees — should be required to pass a test of precollege level skills before they are allowed to teach.

Though the precise grade-level of the items on the CBEST is subject to some debate, it is nonetheless clear that it tests at most secondary-level, precollege skills. For example, the most difficult mathematics question on the August 1995 CBEST — judged the most difficult because the most examinees answered it incorrectly — was as follows:

How many students at a school can be served a half-pint of milk from 5 gallons of milk?^[8]

A. 80

B. 60

C. 40

D. 20

E. 10

The CBEST is self-evidently a test of basic skills in reading, writing, and mathematics.

As will be discussed in detail in this Opinion, the Court finds that plaintiffs have met their burden of proving that the CBEST has an adverse impact on the plaintiff class. Defendants, however, have successfully rebutted plaintiffs' case by showing that the CBEST is a valid, job-related test for the teaching and nonteaching positions in the public schools for which it is a requirement. In response, plaintiffs have failed to show the existence of an alternative selection device that would adequately replace the CBEST.

The CBEST is not a cure-all for the ills of California's public schools, but it is not meant to be. It is simply a threshold measure.^[9] The State is entitled to ensure that teachers and others who work in the public schools possess a minimal level of competency in basic reading, writing, and math skills before they are entrusted with the education of our children.

B.

Plaintiffs contend that the CBEST has an adverse impact on the minority groups represented in the plaintiff class: African-Americans, Latinos, and Asians. According to plaintiffs, the CBEST is not a valid, job-related measure of all the teaching and non-teaching jobs for which it is required.

"In enacting Title VII, Congress required ``the removal of artificial, arbitrary, and unnecessary barriers to employment when the barriers operate invidiously to discriminate on the basis of racial or other impermissible classification.'" Dothard v. Rawlinson, 433 U.S. 321, 328, 97 S.Ct. 2720, 2726, 53 L.Ed.2d 786 (1977) (quoting Griggs v. Duke Power Co., 401 U.S. 424, 431, 91 S.Ct. 849, 853, 28 L.Ed.2d 158 (1971)).^[10] Title VII "proscribes not only overt discrimination but also practices that are fair in form, but discriminatory in operation."^[11]Griggs, 401 U.S. at 431, 91 S.Ct. at 853. Thus, in a disparate impact case such as this one, plaintiffs need not show that defendants intended to discriminate against them; they need only prove that a facially neutral employment practice, viz., the CBEST, has had a significant adverse impact on groups protected by Title VII. Watson v. Fort Worth Bank & Trust, 487 U.S. 977, 986-87, 108 S.Ct. 2777, 2784-85, 101 L.Ed.2d 827 (1988). Nevertheless, "[n]othing in [Title VII] precludes the use of testing or measuring procedures; obviously they are useful." Griggs, 401 U.S. at 436, 91 S.Ct. at 856.^[12]

As a threshold matter, the Court must consider the appropriate standard for the parties' respective burdens of proof. Over the years, the burdens of proof applicable to Title VII disparate impact cases have changed.

Prior to the Supreme Court's decision in Wards Cove Packing Co. v. Atonio, 490 U.S. 642, 109 S.Ct. 2115, 104 L.Ed.2d 733 (1989), the allocation of proof in a disparate impact case such as this was described by the Ninth Circuit as follows.^[13] First, the plaintiff "must establish that a facially neutral employment practice produces a significant adverse impact on a protected class." Clady v. County of Los Angeles, 770 F.2d 1421, 1427 (9th Cir.1985), cert. denied, 475 U.S. 1109, 106 S.Ct. 1516, 89 L.Ed.2d 915 (1986). If the plaintiff meets that burden, then "the burden shifts to the employer to validate the selection device, that is, to show that it has ``a manifest relationship to the employment in question.'" Id. at 1427 (quoting Griggs, 401 U.S. at 432, 91 S.Ct. at 854).

If the employer fails to meet its burden, then the employer's "use of the selection device will be deemed a Title VII violation." Id. at 1428. If the employer succeeds in validating the selection device, however, the plaintiff may nonetheless "rebut the defendant's evidence by showing that although job-related, the test does not constitute a business necessity because an alternative selection device exists which would have comparable business utility and less adverse impact." Id.

In Wards Cove, the Supreme Court repudiated the widespread assumption that the burden of proof shifts entirely to the defendant during the second phase of a disparate impact case. Instead, the Court held that "the employer carries the burden of producing evidence of a business justification for his employment practice. The burden of persuasion, however, remains with the disparate-impact plaintiff." 490 U.S. at 659, 109 S.Ct. at 2126. In addition, the Court appeared to reduce the defendant's burden by requiring only a showing of "business justification," viz., that "a challenged practice serves, in a significant way, the legitimate employment goals of the employer," id. at 658-59, 109 S.Ct. at 2125-26 (emphasis added), rather than a showing of "business necessity" as required under Griggs. 401 U.S. at 431, 91 S.Ct. at 853 (emphasis added). The Court emphasized that "there is no requirement that the challenged practice be ``essential' or ``indispensable' to the employer's business for it to pass muster." Wards Cove, 490 U.S. at 659, 109 S.Ct. at 2126.

Though the Supreme Court characterized its decision as a mere clarification of existing law, see id. at 659-60, 109 S.Ct. at 2126, it came as a surprise to the lower federal courts and was widely viewed as increasing the plaintiff's burden in disparate impact cases. See, e.g., Graffam v. Scott Paper Co., 870 F.Supp. 389, 392 & n. 4 (D.Me.1994), aff'd, 60 F.3d 809 (1st Cir.1995); Stender v. Lucky Stores, Inc., 803 F.Supp. 259, 321 & n. 20 (N.D.Cal.1992); see also Wards Cove, 490 U.S. at 661, 109 S.Ct. at 2127 (Blackmun, J., dissenting); id. at 668-72, 109 S.Ct. at 2130-33 (Stevens, J., dissenting).

Partly in response to the Wards Cove decision, Congress passed the Civil Rights Act of 1991 ("1991 CRA"), which became effective on November 21, 1991. The 1991 CRA restored the proof allocation generally applied in disparate impact cases prior to Wards Cove. The statute describes the burdens of proof as follows:

An unlawful employment practice based on disparate impact is established ... only if —

(i) a complaining party demonstrates that a respondent uses a particular employment practice that causes a disparate impact on the basis of race, color, religion, sex, or national origin and the respondent fails to demonstrate that the challenged practice is job related for the position in question and consistent with business necessity; or

(ii) the complaining party [makes a showing of] an alternative employment practice and the respondent refuses to adopt such alternative employment practice.

42 U.S.C. § 2000e-2(k)(1)(A) (1994).

In this case, defendants' conduct, which began in late 1982 and continues into the present, spans all three eras: (1) pre-Wards Cove from 1982 to 1989, (2) Wards Cove, from June 5, 1989, to November 21, 1991, and (3) the 1991 CRA, from November 21, 1991, to the present. The case, however, was filed on September 23, 1992, well after the 1991 CRA's effective date. Thus, the Court must determine whether to apply the burdens of proof established by the 1991 CRA to this case.

Although the Supreme Court considered the issue of the 1991 CRA's retroactivity in Landgraf v. USI Film Products, 511 U.S. 244, 114 S.Ct. 1483, 128 L.Ed.2d 229 (1994), that case is distinguishable. The only provision at issue in Landgraf was § 102, which provides a right to recover compensatory and punitive damages and a right to jury trial. The Court held that those sections did not apply to a case that was on appeal when the 1991 CRA became effective.

Landgraf thus did not reach the issue of whether the burden of proof provisions of the 1991 CRA apply retroactively. The federal courts have divided on the issue. A few courts have refused to apply the 1991 CRA, reasoning that the burden of proof provisions "affect the liability of defendants" and therefore "cannot be retroactively applied in cases ... where alleged discriminatory conduct occurred prior to" the 1991 CRA's effective date. Matthews v. Runyon, 860 F.Supp. 1347, 1355 (E.D.Wis.1994); see also Houghton v. SIPCO, Inc., 38 F.3d 953, 959 (8th Cir.1994) (holding that burden of proof under Wards Cove, not 1991 CRA, applies to case filed before effective date of 1991 CRA).

One court even interpreted Landgraf to stand for the blanket proposition that no provision of the 1991 CRA applies retroactively. Jones v. Pepsi-Cola Metropolitan Bottling Co., 871 F.Supp. 305, 309 n. 11 (E.D.Mich.1994). Such a reading cannot be reconciled with the language of Landgraf itself, however, in which the Supreme Court explained:

[T]here is no special reason to think that all the diverse provisions of the Act must be treated uniformly for [retroactivity] purposes. To the contrary, we understand the instruction that the provisions are to "take effect upon enactment" to mean that courts should evaluate each provision of the Act in light of ordinary judicial principles concerning the application of new rules to pending cases and pre-enactment conduct.

511 U.S. at ___, 114 S.Ct. at 1505; see also id. at ___-___, 114 S.Ct. at 1494-95.

Other courts, including this Court, have concluded that the burden of proof provision does apply to cases that were pending at the time of or filed after the effective date of the 1991 CRA. See Graffam, 870 F.Supp. at 393-94; Housey v. Carini Lincoln-Mercury, 817 F.Supp. 762, 766-68 (E.D.Wis.1993); Stender, 803 F.Supp. at 321-22.

The Court finds that the allocation of proof provision of the 1991 CRA applies to this case because the provision effected a procedural change, rather than a change in substantive rights. As the Court explained in Landgraf:

Changes in procedural rules may often be applied in suits arising before their enactment without raising concerns about retroactivity.... Because rules of procedure regulate secondary rather than primary conduct, the fact that a new procedural rule was instituted after the conduct giving rise to the suit does not make application of the rule at trial retroactive.

511 U.S. at ___, 114 S.Ct. at 1502. The central policy underlying the presumption against statutory retroactivity — "the unfairness of imposing new burdens on persons after the fact" — is absent here. Id. at ___, 114 S.Ct. at 1500. This is not a case in which there exist "concerns of unfair surprise and upsetting expectations," id. at ___ n. 35, 114 S.Ct. at 1506 n. 35, or where "predictability and stability are of prime importance." Id. at ___, 114 S.Ct. at 1500. The applicable burden of proof does not affect parties' conduct prior to litigation. To the contrary, a burden of proof is implicated only at trial, long after the conduct has taken place.^[14] Nor does the burden of proof affect defendants' liability in a substantive way: "It does not make unlawful conduct that was lawful when it occurred," id. at ___, 114 S.Ct. at 1506, and it does not "impair rights a party possessed when he acted, increase a party's liability for past conduct, or impose new duties with respect to transactions already completed." Id. at ___, 114 S.Ct. at 1505. Therefore, the Court will follow the general rule that "a court is to apply the law in effect at the time it renders its decision." Id. at ___, 114 S.Ct. at 1496 (quoting Bradley v. Richmond Sch. Bd., 416 U.S. 696, 711, 94 S.Ct. 2006, 2016, 40 L.Ed.2d 476 (1974)).

In any event, it makes no difference which standard of proof the Court employs in this case. The Court's determination that the 1991 CRA burdens of proof apply has no effect on the outcome of the suit. Even under the heavier burden of proof imposed by the 1991 CRA, defendants prevail. See Fitzpatrick v. City of Atlanta, 2 F.3d 1112, 1117-18 (11th Cir.1993).

C.

Plaintiffs bear the burden of proving that the CBEST produces "a significant adverse impact" on the plaintiff class. Clady, 770 F.2d at 1427 (citing Teal, 457 U.S. at 446, 102 S.Ct. at 2530). Plaintiffs can meet this burden through reliable statistics. Id. The Court finds that plaintiffs have met their burden of showing adverse impact.

Both parties in this case have used the so-called "80-percent rule" prescribed by the Uniform Guidelines on Employee Selection Procedures, 29 C.F.R. pt. 1607 (1978) ("Uniform Guidelines"), which were promulgated jointly by the EEOC, the Civil Service Commission, the Department of Labor, and the Department of Justice. 29 C.F.R. § 1607.1(A). The Uniform Guidelines are not binding on the Court, but they do have some persuasive force. Bouman v. Block, 940 F.2d 1211, 1225 (9th Cir.), cert. denied, 502 U.S. 1005, 112 S.Ct. 640, 116 L.Ed.2d 658 (1991).

Under the Uniform Guidelines, "[a] selection rate for any race, sex, or ethnic group which is less than four-fifths ( 4/5 ) (or eighty percent) of the rate for the group with the highest rate" is considered a showing of adverse impact. 29 C.F.R. § 1607.4(D). Thus, under the 80-percent rule, plaintiffs must show a selection rate (or pass rate) for each of the groups represented by the plaintiff class that is less than 80 percent of the selection rate for non-Latino Caucasians ("whites"),^[15] who (as a group) have the highest pass rates on the CBEST. For example, suppose that ethnic group A, the highestscoring group on a test, has a pass rate of 50 percent. Applying the 80-percent rule, 80 percent of 50 percent (group A's selection rate) equals 40 percent. Thus, ethnic group B can show adverse impact if less than 40 percent of the group passes the test.

The undisputed evidence presented at trial by both parties showed that, under the 80-percent rule, an adverse impact exists with respect to first-time CBEST-takers who are grouped according to the class definition: Latinos, African-Americans, and Asians. Therefore, plaintiffs have made their prima facie case.

Defendants do not quarrel with the statistics; nevertheless, they argue that the CBEST does not have an adverse impact on all members of the plaintiff class. Defendants contend that a few subgroups, particularly English-fluent Asians, perform as well or better than whites on certain parts of the CBEST. Defendants also contend that cumulative, as opposed to first-time, pass rates should be used, and that these rates show no adverse impact for any group in the plaintiff class.

Plaintiffs object to defendants' approach to assessing adverse impact. They argue that all Asians are properly treated as a single group, whether they are fluent in the English language or not, and whether they are, for instance, Chinese, Filipino, Hmong, or Pacific Islander. According to plaintiffs, the appropriate analysis considers first-time pass rates of the CBEST as a whole by each of the three groups defined in the plaintiff class (African-Americans, Latinos, and Asians) as compared to whites. The Court agrees.

Defendants have cited no authority for the proposition that the plaintiff class should be subdivided differently from the way in which the groups are defined by the Court's order certifying the class, viz., African-Americans, Latinos, and Asians. What little guidance the Court could discover in this area supports grouping class members by race or ethnicity rather than by other characteristics (e.g., English fluency). For example, the Uniform Guidelines provide that employers should keep records on "the following races and ethnic groups: Blacks (Negroes), ... Asians (including Pacific Islanders), Hispanic (including persons of Mexican, Puerto Rican, Cuban, Central or South American, or other Spanish origin or culture regardless of race), [and] whites (Caucasians) other than Hispanic." 29 C.F.R. § 1607.4(B). The definition of the plaintiff class conforms to this standard.

Moreover, even if the Court were to adopt defendants' proposed subgroups, the Court would still find that the CBEST has an adverse impact on the entire plaintiff class. As discussed below, adverse impact is appropriately measured by the first time a candidate sits for the CBEST and fails it. Using that standard, defendants' proposed subgroups show an adverse impact under the 80-percent rule.

Defendants also argue that adverse impact is properly determined with respect to the pass rate of each subpart of the test, rather than pass rate on the CBEST as a whole. For this contention, defendants cite Connecticut v. Teal, 457 U.S. 440, 102 S.Ct. 2525, 73 L.Ed.2d 130 (1982). Teal does not support defendants' argument.

In Teal, the plaintiffs alleged that the defendant's written examination, which employees were required to pass in order to become permanent supervisors, had a disparate impact on African-Americans. In taking the next step and determining which of the employees who had passed the test should in fact be promoted, however, the defendant — on the eve of trial — adopted a kind of affirmative-action program and promoted a greater percentage of Blacks than whites. At trial, the defendant argued that "this ``bottom-line' result, more favorable to blacks than to whites, ... should be adjudged to be a complete defense" to the plaintiffs' disparate impact claim. 457 U.S. at 444, 102 S.Ct. at 2529.

The Supreme Court rejected the so-called "bottom-line" defense. In doing so, the Court emphasized that "Title VII prohibits ``procedures or testing mechanisms that operate as "built-in headwinds" for minority groups,'" and that "Congress' primary purpose was the prophylactic one of achieving equality of employment ``opportunities' and removing ``barriers' to such equality." Id. at 448-49, 102 S.Ct. at 2531. Under this standard, the Court concluded that "[t]he examination given to [the plaintiffs] in this case surely constituted such a practice and created such a barrier." Id. at 449, 102 S.Ct. at 2531. As such, it was actionable under Title VII.

Reviewing its precedents, the Supreme Court noted that it had "consistently focused on employment and promotion requirements that create a discriminatory bar to opportunities. This Court has never read [Title VII] as requiring the focus to be placed instead on the overall number of minority ... applicants actually hired or promoted." Id. at 450, 102 S.Ct. at 2532. "The suggestion that disparate impact should be measured only at the bottom line ignores the fact that Title VII guarantees these individual [plaintiffs] the opportunity to compete equally with white workers on the basis of job-related criteria." Id. at 451, 102 S.Ct. at 2532-33. The Supreme Court in Teal thus held that a defendant employer cannot excuse the use of a nonjob-related barrier by showing that the promotional process as a whole resulted in a proportionally greater number of minority promotions than white promotions.

Though Teal does not speak directly to the issue of whether each subpart of a test should be considered separately in analyzing disparate impact, Justice Brennan's opinion implied that a test should be viewed as a whole: "[The plaintiffs'] claim of disparate impact from the examination, a pass-fail barrier to employment opportunity, states a prima facie case of employment discrimination...." Id. at 452, 102 S.Ct. at 2533 (emphasis added).

Here, as in Teal, the barrier imposed by defendants is the requirement that plaintiffs take and pass all three sections of a pass-fail examination. One must pass the whole CBEST in order to be eligible for employment or promotion. Thus, the selection occurs when a candidate passes or fails the CBEST as a whole. Id. How different groups perform on each subpart of the examination is therefore not directly relevant to the issue of adverse impact.

The legislative history of the 1991 CRA also supports this finding:

When a decision-making process includes particular, functionally-integrated practices which are components of the same criterion, standard, method of administration, or test, ... the particular, functionally-integrated practices may be analyzed as one employment practice.

137 Cong.Rec. S15276 (daily ed. Oct. 25, 1991) (interpretive memorandum), reprinted in 1991 U.S.C.C.A.N. 549, 767.

It is true, as defendants argue, that a candidate retains her highest score on each subpart every time she sits for the CBEST, and that the candidate may retake the test an unlimited number of times, as often as six times a year. Nonetheless, a candidate cannot obtain a job unless and until she has passed the CBEST — and she does not pass the CBEST unless and until she passes not just one or two subtests, but all three sections. Each time that a candidate fails to pass the examination as a whole, that candidate is deprived of an employment or advancement opportunity. Kirkland v. New York State Dep't of Correctional Servs., 520 F.2d 420, 425 (2d Cir.1975), reh'g denied, 531 F.2d 5 (2d Cir.1975), cert. denied, 429 U.S. 823, 97 S.Ct. 73, 50 L.Ed.2d 84 (1976); Richardson v. Lamar County Bd. of Educ., 729 F.Supp. 806, 815 (M.D.Ala.1989), aff'd, 935 F.2d 1240 (11th Cir.1991); see also Teal, 457 U.S. at 452, 102 S.Ct. at 2533; Bouman, 940 F.2d at 1225 (analyzing pass rates on three-part written examination as a whole); Clady, 770 F.2d at 1429 (analyzing pass rates on written examination as a whole as "a discrete selection device").

Finally, defendants contend that adverse impact should be assessed in light of cumulative, rather than first-time, pass rates.^[16] As the Court has just explained, however, the harm to a candidate occurs, if at all, each and every time the candidate sits for the CBEST and fails it. Each time the candidate does not pass, she is barred from an opportunity to become a teacher or to advance, for instance, to an administrative position. As this Court has noted once before, passing the CBEST "is the sine qua non of employment in California's public schools." Association of Mexican-American Educators v. California, 836 F.Supp. at 1551 (hereinafter "AMAE").^[17] Therefore, it is basically irrelevant to the issue of adverse impact that some, most, or nearly all of those who take the CBEST eventually pass. The loss of an employment opportunity occurs each and every time a candidate fails the test. See Teal, 457 U.S. at 452, 102 S.Ct. at 2533.

The Court turns now to the evidence. Plaintiffs' expert, Dr. John Poggio, concluded that the first-time pass rates were as follows:

Asians                         53.0%
African-Americans              37.7
Latinos                        49.4
Whites                         73.4

(Poggio Direct Test. at 12.) Under his analysis, it is clear that all of the groups represented by the plaintiff class have a pass rate that is less than 80 percent of the white pass rate.

The analysis performed by defendants' expert, Dr. Joan Haworth, supports this finding. Her results with respect to first-time CBEST takers in the plaintiff class were as follows:

Asian                          59.9%
Black                          37.4
Hispanic/Latino                47.0
Filipino                       42.7
Puerto Rican                   45.8
Pacific Islander               49.9
White                          80.0

(See Ex. 1387, tbl. 2A, col. 5, Haworth Rep.) The only two categories in Dr. Haworth's analysis of the pass rates of first-time examinees to show no adverse impact under the 80-percent rule were "American Indian" (67.1 percent) and "Other" (65.6 percent), neither of which is a group included in the plaintiff class. (See id.) As discussed above, the categories used by Dr. Haworth should have been aggregated into the three categories represented by the plaintiff class. Thus, the "Asian" category should include both Filipinos and Pacific Islanders, and the "Hispanic/Latino" category should include Puerto Ricans.

The Court also notes that Dr. Haworth found no difference in the extent of adverse impact between the former version of the CBEST and the revised CBEST administered in August and October 1995.

Dr. Haworth conducted a multiple regression analysis in order to suggest some possible explanations for the comparatively poor performance of minority groups on the CBEST, such as lack of English fluency and lower level of education. The Court finds this analysis interesting, and ultimately encouraging, because it appears to show that preparation factors play a strong role in a candidate's performance on the CBEST, regardless of the candidate's race or ethnicity.

Nevertheless, this analysis is entirely irrelevant to the issue of adverse impact. It does not matter why the disparate impact exists. Defendants cannot escape liability by showing that the disparate impact is attributable to particular background factors. The Ninth Circuit has rejected "the proposition that a defendant need not validate an examination if the disparate impact of that examination correlates with some facially non-discriminatory factor or factors." Bouman, 940 F.2d at 1228. As the Court of Appeals explained:

[T]he whole point of a disparate impact challenge is that a facially non-discriminatory employment or promotion device — in this case an examination — has a discriminatory effect. It would be odd indeed if a defendant whose facially non-discriminatory examination which has a disparate impact could escape the obligation to validate the examination merely by pointing to some other facially non-discriminatory factor that correlates with the disparate impact. [The defendant's] failure to validate cannot be excused simply by the correlation between success on the examination and experience.

Id. (citation omitted).

This reasoning has been clear since Griggs. There, the Supreme Court observed that the plaintiffs, African-Americans, were less likely to meet the requirements of a high school diploma and a passing score on a general aptitude test because they had received "inferior education in segregated schools" and had lower high school graduation rates than whites. Griggs, 401 U.S. at 430 & n. 6, 91 S.Ct. at 853 & n. 6. Merely explaining the impediments to plaintiffs' ability to meet the job requirements did not, however, excuse the defendant from the burden of showing that the requirements were job-related. Indeed, the Court held that both requirements were invalid.

D.

The burden now shifts to defendants to validate the CBEST — in other words, "to show that it has ``a manifest relationship to the employment in question.'" Clady, 770 F.2d at 1427 (quoting Griggs, 401 U.S. at 432, 91 S.Ct. at 854). Where, as here, a scored test is involved, the Ninth Circuit requires a showing that the test is "job-related," that is, "that it actually measures skills, knowledge, or ability required for successful performance of the job."^[18]Contreras v. City of Los Angeles, 656 F.2d 1267, 1271 (9th Cir.1981); accord Clady, 770 F.2d at 1427-28.

The Ninth Circuit has explained the validation process as follows:

The employer must first specify the particular trait or characteristic which the selection device is being used to identify or measure. The employer must then determine that that particular trait or characteristic is an important element of work behavior. Finally, the employer must demonstrate by "professionally acceptable methods" that the selection device is "predictive of or significantly correlated" with the element of work behavior identified in the second step. Craig v. County of Los Angeles, 626 F.2d 659, 662 (9th Cir.1980), cert. denied, 450 U.S. 919, 101 S.Ct. 1364, 67 L.Ed.2d 345 (1981) (quoting Albemarle Paper Co. v. Moody, 422 U.S. 405, 431, 95 S.Ct. 2362, 2378, 45 L.Ed.2d 280 (1975)).

As discussed below, the Court finds that defendants have satisfied this standard. They have shown that basic skills in reading, writing, and mathematics are important elements in the jobs for which the CBEST is required and that the CBEST actually measures such basic skills. In this portion of the Opinion, the Court will discuss the job-relatedness of the skills tested by the items on the CBEST and the test's content validity for the jobs for which it is a requirement. Then, the State's determination of where to set the passing scores on the three parts of the CBEST will be addressed. Of necessity, the Court will analyze the validity evidence in some detail.

1.

As a preliminary matter, the Court will address plaintiffs' argument that, even assuming defendants have succeeded in proving that the CBEST is content valid, the CBEST still has not been shown to be job-related because of defendants' failure to show criterion-related validity and construct validity.

"Content validity" refers to the extent to which the items on the CBEST are representative of a defined universe or domain of content, in this case, the basic reading, writing, and mathematics skills relevant to the job of teaching. A classic illustration of a content-valid test is a typing test for a typist. This is the only kind of validity evidence presented by defendants in this case. "Criterion-related validity" refers to the extent to which an individual's score on the CBEST is predictive of some other criterion, usually job performance. A student's score on the SAT, for example, predicts to some extent that student's first-semester grades in college. "Construct validity" refers to the extent to which the CBEST is a measure of some hypothetical or psychological construct, such as logical reasoning ability.

The Court disagrees with plaintiffs' argument that the CBEST's validity cannot be shown absent construct and criterion-related validity evidence. Plaintiffs have provided no authority for the proposition that all three types of validity evidence must be shown for a test to be adequately validated. Indeed, the opposite appears to be true: The Uniform Guidelines, for instance, provide that defendants "may rely upon criterion-related validity studies, content validity studies or construct validity studies." 29 C.F.R. § 1607.5(A) (emphasis added).

Moreover, the Court would not expect to find the kind of evidence of criterion-related validity that plaintiffs argue is lacking. Plaintiffs argue that the CBEST is not valid because it does not predict a candidate's performance as a teacher. They cite both the Dick study (Ex. 205) and the Fresno study (Ex. 575 at 56-61) for this point.

The CBEST, however, does not purport, and was not designed, to predict a teacher candidate's performance on the job. Rather, as Dr. Mehrens' testimony emphasized, the CBEST is a measure of basic skills, a minimum threshold of competency that one would not expect to be positively correlated with job performance, any more than one would expect the written driver's examination to predict which candidates will be good drivers on the road. It is, therefore, not surprising that neither the Dick study nor the Fresno study showed any relationship between performance on the CBEST and successful performance as a teacher.^[19]

Plaintiffs further contend that content validation was inappropriate, and that construct validation was necessary, because the CBEST measures general mental aptitude rather than specific skills. The Uniform Guidelines provide as follows:

A selection procedure based upon inferences about mental processes cannot be supported solely or primarily on the basis of content validity. Thus, a content strategy is not appropriate for demonstrating the validity of selection procedures which purport to measure traits or constructs, such as intelligence, aptitude, personality, commonsense, judgment, leadership, and spatial ability.

29 C.F.R. § 1607.14(C). In support of this proposition, plaintiffs presented the testimony of Dr. Joel Lefkowitz.^[20] The Court rejects plaintiffs' argument.

As the Uniform Guidelines explain, a content validity strategy is only inappropriate where a selection device purports to measure a hypothetical construct or trait, such as leadership or spatial ability. The CBEST is not such a selection device; it does not purport to measure a candidate's general mental aptitude, intelligence, or any other construct. Rather, it is designed to measure specific, well-defined skills in reading, mathematics, and writing.

Moreover, there is no evidence to support the feasibility of conducting a construct validity study. "[C]onstruct validation is frequently impossible. Even the [Uniform] Guidelines acknowledge that construct validation requires ``an extensive and arduous effort.'" Guardians Ass'n v. Civil Serv. Comm'n, 630 F.2d 79, 92 (2d Cir.1980), cert. denied, 452 U.S. 940, 101 S.Ct. 3083, 69 L.Ed.2d 954 (1981) (quoting 29 C.F.R. § 1607.14(D)). The Court finds that not much has changed since 1978, when the authors of the Uniform Guidelines observed, "Construct validation is a relatively new and developing procedure in the employment field, and there is at present a lack of substantial literature extending the concept to employment practices." 29 C.F.R. § 1607.14(D).

The Court now turns to the validity evidence in the case. Defendants presented three validity studies of the CBEST: (1) the 1982 validity study conducted by Dr. Patricia Wheeler and Dr. Patricia Elias of the Educational Testing Service ("ETS"); (2) the 1985 Practitioners' Review conducted by Dr. Richard Watkins; and (3) the job analysis and content validity studies conducted by Dr. Kathleen Lundquist during 1994 and 1995.^[21] As discussed below, the Court concludes that this evidence amply demonstrates the job-relatedness of the items on the CBEST.

The Wheeler and Elias study was conducted in the fall of 1982, shortly before the CBEST was first administered in December 1982. It involved a total of 289 participants who reviewed the multiple-choice test items (math and reading). The participants included 277 teachers, administrators, and other nonteachers from thirteen school districts and teacher educators^[22] from twelve teacher-training institutions. The group included a fairly large sample of minority group members. In addition, 44 teachers and college faculty reviewed the essay topics and responses for the writing test. (See generally Ex. 120, Wheeler & Elias Report.)

The study participants (or "judges") conducted an item content review of the items on the reading and math subtests. A portion of the judges worked individually, and the remainder participated in discussion groups. All the judges were asked to rate the accuracy, fairness, and clarity of each test item and the relevance of each item to the job of teaching in California. The study employed a four-point scale for relevance: crucial, important, questionable, and not relevant for the job of teaching at any grade level in any subject area. Items that were marked questionable or not relevant by 40 percent or more of the judges were reviewed and a decision made to retain, revise, or discard them; however, no item was discarded solely for lack of relevance.

According to both plaintiffs' and defendants' experts, a 50-percent rule for an acceptably high judgment of relevance is professionally acceptable. That is, if more than half the participants rate an item as relevant, it should be retained. In the Wheeler and Elias study, 34 out of the 40 reading items and 24 of the 40 mathematics items were judged to be crucial or important by 70 percent or more of the judges. For mathematics, an additional 9 items were deemed crucial or important by more than 60 percent of the judges. Only one of the reading items and three of the math items were rated as crucial or important by less than 50 percent of the judges. The overall relevance ratings were 76 percent crucial or important for the reading examination, and 65 percent for mathematics.^[23] In addition, there were no major differences between the relevance ratings made by teachers and those made by nonteachers. (Ex. 120 at 34-35, tbls. 14 and 15.)

The second content validity study was the Practitioners' Review conducted by Dr. Watkins in 1985. (See generally Ex. 118, report.) This study involved 234 teachers, administrators, teacher educators, and other school employees. Thirty-six percent were members of minority groups.

The participants took part in nine review panels, in which they judged the relevance of both the skills assessed by the CBEST and the test items themselves. A four-point scale was used: Very relevant, moderately relevant, slightly relevant, and not relevant.

The percentage of panelists who rated the CBEST skills as very relevant or moderately relevant was as follows:

Reading
Literal comprehension           98%
Logical comprehension           97
Critical comprehension          94
Mathematics
Problem solving skills          95
Applied problem solution        89
Concepts and relationships      87
Writing                            99

(Ex. 118 at vi.) Thus, the vast majority of participants rated all the skills tested by the CBEST as either very or moderately relevant.^[24]

With respect to the test items, the participants were asked to judge the relevance and difficulty level of each item. For the reading subtest, the questions were judged to be very relevant or moderately relevant by 90 percent of the panel members, and 89 percent judged the questions to be easy or medium in difficulty. For the mathematics subtest, 87 percent judged the questions to be very or moderately relevant, and 84 percent judged the questions to be easy or medium in difficulty. Overall, only 3 percent of the panel members rated some math questions not relevant.

The third and most comprehensive series of validity studies was undertaken by Dr. Lundquist in 1994. Her work consisted of a job analysis survey and a content validation study. (See generally Ex. 1541, job analysis results; Ex. 1543, content validation report.)

The survey for the job analysis was developed as follows. First, a literature search was conducted on basic skill requirements for teachers at the kindergarten through twelfth grade levels. Then, 52 California public school teachers from different grade levels, geographic areas, and ethnic groups were interviewed regarding their job activities, their use of reading, writing, and mathematics skills, and other knowledge, skills, and abilities used on the job. Eighteen of these teachers were observed on the job. The information gathered from the interviews, observations, and literature review was used to draft a preliminary list of skills and activities used by teachers.

Next, panels of content experts reviewed the list of skills and abilities. In addition, the panels evaluated the skills required to use actual curricular materials, samples of which had been obtained from the interviewed teachers. The panels also linked the skill requirements to job activities. Most of the skills were determined to be required for using the curricular materials and for performing activities on the job. Dr. Lundquist concluded that there was "strong linkage of skills to [teacher job] activities." (Ex. 1541 at 139, job analysis results.)

Finally, the job analysis survey was created based on all of the collected information. The survey was then reviewed by the CTC and pilot-tested on a sample of 28 teachers, and revised accordingly. The final survey included 39 reading skills, 27 writing skills, and 37 math skills. It also surveyed 59 teacher job activities, and included questions regarding the use of non-English languages; reading, writing, and math abilities "representing the role of teachers as models of well-educated people"; and the candidate's background. (Lundquist Direct Test. at 20.) A version of the survey for administrators was also created to determine whether they shared a common set of skill requirements with classroom teachers.

The survey was distributed to more than 6,000 teachers and 1,100 administrators. Data from approximately 1,100 teachers and 230 administrators was collected. Each job activity was rated on importance, frequency, and time constraints;^[25] each skill was rated on importance, frequency, and whether it was required at entry.^[26]

To identify important activities and skills, Dr. Lundquist employed a very high standard. An activity or skill was considered important (and thus retained) only if at least 80 percent of the survey respondents rated the activity or skill as applicable to the job and the mean importance rating was 1.5 or higher on the importance scale, which ranged from 0 to 3: not applicable — 0, minor — 1, important — 2, or critical — 3. Activities and skills meeting the importance criteria were then subjected to subgroup analysis by ethnic group, credential category, and primary language subgroup. Items were dropped if they showed a statistically significant mean difference for the subgroup and the mean importance rating for any subgroup was below 1.5.

Of the 59 job activities, 13 were dropped under the importance criteria, and one additional activity was dropped after subgroup analysis. The results for skills were as follows: Of the 39 reading skills, one was dropped; of the 27 writing skills, two were dropped; and, most significantly, of the 37 math skills, 21 were dropped. After subgroup analyses, one additional reading skill and six additional math skills were dropped. Thus, a total of 37 reading skills,^[27] 25 writing skills,^[28] and only 10 math skills^[29] were retained as a result of the job analysis survey.

Based on her comparison of teachers' and administrators' responses, Dr. Lundquist concluded that "all of the skills identified as important for teachers were also important for administrators." (Ex. 1541 at 135.) In addition, Dr. Lundquist found "no meaningful differences" among ethnic groups in the skills that were deemed important, and no skills were dropped as a result of ethnic subgroup analysis. (Ex. 1541 at 131.) There was also "near unanimous agreement among teachers sampled that [basic] skills were critical for teachers in their roles as models of well-educated people." (Ex. 1541 at 137.)

The CTC was surprised, however, by the large number, and especially the types, of math skills that were dropped by the job analysis survey.^[30] Indeed, this result seemed absurd with respect to certain skills. For example, one of the skills that was dropped was "Perform arithmetic operations with basic statistical data related to test scores (e.g., averages, ratios, proportions, and percentile scores)." (Ex. 1541 at 259, job analysis results, ex. 5-20.) Obviously, teachers must know how to determine a student's grade for the term by calculating the average of the student's test scores. In addition, this skill just barely missed being retained; it was endorsed by 79.3 percent of the survey respondents. Only Dr. Lundquist's stringent 80 percent criterion resulted in its not being retained.

The CTC was reluctant to alter the CBEST's mathematics subtest in such a radical fashion without further investigation; therefore, the CTC commissioned Dr. Lundquist to conduct a content validity study to reexamine the math skills on the CBEST. (See generally Ex. 1543, content validation report.)

For the content validity study, 20 teacher educators participated in two focus groups in which they were presented with all 37 of the math skills from the original job analysis survey. As in the survey, the participants were asked to rate the applicability of each skill and whether the skill was required upon entry to the job. In addition, they were asked to rate both the current and future importance of each skill. Dr. Lundquist used roughly the same retention criteria as in the job survey analysis. For a skill to be retained, at least 80 percent of the teacher educators had to rate the skill as applicable and required at entry, and the skill had to have average current and future importance ratings equal to or greater than two (out of a three-point scale, one being "minor," and three being "critical"). Under these criteria, 26 out of the 37 math skills were retained. All of them were rated important or critical.

Those 26 math skills were then given to two groups: One composed of 26 content experts and another of 28 community members. The two groups were asked to rate the applicability of each skill and whether it was required at entry.^[31] If either group found that a particular skill was applicable and required at entry (using the 80 percent agreement criterion), then that skill was retained. The content expert panel retained 17 skills, the community group, 16 skills.

Overall, 19 of the 26 math skills were ultimately retained: the original 10 that had been retained after the job analysis survey, plus 9 additional skills.^[32]

Dr. Lundquist then formulated new test specifications for all three parts of the test, after conducting an additional study in which the skill factors and the skills that comprised them were reviewed by a group of teacher educators and a group of content experts. Although the test specifications for reading and writing were reformulated, they correspond to the former CBEST specifications (Ex. 138) for those sections. For mathematics, however, several skills in the old test specifications cannot be matched in the new specifications, as those skills were dropped as not job-related by the job analysis and content validity study. In comparing the new CBEST test specifications to test items on two forms of the CBEST, Dr. Lundquist was only able to match 40 percent of the test items to the new test specifications; the other 60 percent of the test items were no longer applicable because they related to nonjob-related skills no longer included in the CBEST test specifications. (Ex. 1543 at 4-6, 4-23, ex. 4-6).

The CBEST was revised before the August 1995 administration in response to Dr. Lundquist's content validation study and the revised test specifications. The math skills tested on the examination were limited to the 19 that had been confirmed by the content validity study.^[33] For both reading and mathematics, the skills were weighted proportionately to their criticality ratings in the job analysis survey.^[34]

Plaintiffs take issue with several aspects of Dr. Lundquist's work. Their most serious complaint is that the content validation study improperly negated the job analysis results by restoring a substantial portion of the mathematics skills, which would have been — and, according to plaintiffs, should have been — dropped as a result of the job analysis. In their view, the judgments of a small group of teachers were allowed to override the judgments of more than 1,100 teachers.

The Court, however, agrees that both the decision to reexamine the math skills and the process by which Dr. Lundquist conducted the content validity study reflect sound professional judgments in light of the questionable results of the job analysis survey. First, the Court agrees with the judgment of Dr. Mehrens that the criteria used by Dr. Lundquist for retaining skills in the job analysis were exceedingly conservative.^[35] Under less stringent criteria, several of the mathematics skills that were thrown out as a result of the job analysis, including the ability to use basic statistical data discussed supra, would have been retained. (Mehrens Direct Test. at 24.) Dr. Lundquist herself observed that if she had used a 70 percent criterion in lieu of 80 percent, 7 fewer math skills would have been dropped. (Ex. 1541 at 134, job analysis results.) As noted above, it is professionally acceptable to use a rule as low as 50 percent.

Second, the Court finds it beyond doubt that the skills that were restored to the CBEST as a result of the content validity study have a "manifest relationship" to the jobs for which the CBEST is required. Clady, 770 F.2d at 1427 (quoting Griggs, 401 U.S. at 432, 91 S.Ct. at 854). Indeed, as Dr. Mehrens pointed out, even some of the skills that were ultimately dropped from the CBEST appear to be central to teacher competence. For example, one of the skills that is not tested on the CBEST is the ability to "[i]nterpret the meaning of standardized test scores ... to determine how individuals performed relative to other students." This skill was endorsed as one required for teacher competence by two significant teacher organizations, the American Federation of Teachers ("AFT") and the National Education Association, as well as the National Council on Measurement in Education. See AFT, Standards for Teacher Competence in Educational Assessment of Students 31 (1990).

Plaintiffs also criticize the content validity study for the lack of representativeness of the two teacher educator focus groups and the content expert and community groups. These groups included very few ethnic minorities. Although the Court acknowledges that it would have been preferable to include more minorities, there is no evidence that such inclusion would have changed the outcome of the study. Indeed, the results of the job analysis survey, in which a substantial number of minorities did participate, indicates the opposite: There was little difference, overall, in the ethnic groups' ratings of the importance of the various math skills.

Plaintiffs also raise a number of more global issues with respect to the sufficiency of the validity evidence in this case.

Plaintiffs argue that defendants never conducted a job analysis of nonteaching jobs for which the CBEST is also a prerequisite, such as administrator, school nurse, and school librarian. The Court concludes, however, that the CBEST has been adequately validated with respect to nonteaching jobs.

The very nature of the CBEST makes evident its applicability to nonteachers who work with children in the public schools. The CBEST is a test of basic skills — skills that one would expect of an administrator or school librarian just as much as one would expect them of a classroom teacher. As Dr. Elias testified, it was not unreasonable to determine that nonteaching positions required basic skills, too.

Furthermore, the evidence shows that nonteachers generally rated the items on the CBEST to be as relevant to their jobs as did teachers. Both the Wheeler and Elias study^[36] and Dr. Lundquist's job analysis showed no significant difference between the importance ratings made by teachers and those made by administrators and other nonteachers.

Plaintiffs next argue that a test cannot be properly content validated without first conducting a job analysis to determine the tasks or skills important to the job. They contend that defendants did not conduct a job analysis until after this lawsuit was initiated. The Court disagrees with this argument. Although Dr. Lundquist's was the first formal job analysis, it is simply not true that no analysis of the relationship between the CBEST skills and the job of teaching was done before her study in 1995. Both the Wheeler and Elias study in 1982 and the Practitioners' Review in 1985 involved "the pooled judgments of informed persons such as ... job incumbents" about the relevance of the skills tested on the CBEST to the jobs for which it is required, an appropriate form of a job analysis under the professional standards of the time, particularly the then-current version of the SIOP Principles. See American Psychological Association, Division of Industrial-Organizational Psychology, Principles for the Validation and Use of Selection Procedures 13 (2d ed. 1980).

In addition, as Dr. Mehrens testified, the task of the test developers was to develop a test of basic skills, not to assess the job requirements of being a good teacher. To develop the CBEST, test development committees composed of experts worked with ETS specialists to develop the content specifications of the test. It is professionally acceptable to conduct job analysis surveys regarding the relevance of skills tested after the test has been developed rather than beforehand; after all, any employer wishing to adopt a test that has already been developed (a so-called "off-the-shelf" test), such as the National Teacher Examination, must validate it after it has already been created.

Plaintiffs have also complained that the validation studies failed to consider the requirements for beginning, rather than experienced, teachers and failed to include an adequate number of new teachers in their samples. These arguments are specious. The CBEST is not a beginning teacher test. Once a candidate has passed the CBEST, she is eligible to teach for the rest of her life. In addition, the CBEST teaches basic skills, which all teachers — including beginning teachers — are expected to know before they begin teaching. In other words, they are in no way skills that teachers learn on the job.

Finally, plaintiffs have argued that the CBEST is a "speeded" test, one in which the test-takers must work against the clock and in which being able to work through the problems quickly is a significant factor in passing the examination. As defendants contend, however, there is no persuasive evidence that the time limits for the CBEST subtests have affected the completion rates on those subtests or had a disproportionate impact on the plaintiff class.^[37]

In sum, the Court finds that the skills tested on the CBEST are job-related, and that the CBEST has been shown to be a valid measure of those basic skills.

Defendants' validity evidence is not perfect. But it is not required to be, under either legal or scientific standards.^[38] Validation studies "are by their nature difficult, expensive, time consuming and rarely, if ever, free of error." Cleghorn v. Herrington, 813 F.2d 992, 996 (9th Cir.1987). The Court finds that all the studies presented by defendants were conducted well within the bounds of professionally acceptable standards. Furthermore, the fact that the CBEST has been revised in light of Dr. Lundquist's studies does not mean that the prior version of the CBEST was invalid. The Court finds that it was properly validated in light of the scientific information available, and the professional standards applicable, in the early to mid-1980s. It would be fallacious to declare those judgments, viewed in hindsight and in the light of additional research, indefensible; to the contrary, they were entirely appropriate.^[39]

2.

The Court must now consider the validity of the passing score (also called a "cutoff" or "cut" score) for the CBEST. Even given that the CBEST is a valid measure of basic skills, the test is unjustified "if the particular cut-off score used ... to determine the eligibility of applicants is not itself a valid measure of the minimal ability necessary to become a competent teacher." Groves v. Alabama State Bd. of Educ., 776 F.Supp. 1518, 1530 (M.D.Ala.1991) (citations omitted). The Uniform Guidelines provide: "Where cutoff scores are used, they should normally be set so as to be reasonable and consistent with normal expectations of acceptable proficiency within the work force." 29 C.F.R. § 1607.5(H).

As discussed in this portion of the Opinion, the Court concludes that the passing scores on the CBEST reflect reasonable judgments about the minimum level of basic skills competence that should be required of teachers. The required passing scores do not violate plaintiffs' civil rights.

To pass the CBEST, a candidate must earn a total scaled score of 123. The passing score on each of the three sections is thus 41 (out of a score range of 20 to 80). A scaled score of 41 translates into a raw score of 28 out of 40 questions (70 percent) correct on the reading subtest, and 26 out of 40 (65 percent) correct on the math subtest. The CBEST is graded on a so-called "compensatory scoring" model, however, so that a candidate may pass with a score as low as 37 on one or two sections,^[40] provided that the total score is 123.

For the writing test, each of the two essays is graded by two readers, each of whom gives a raw score from one to four,^[41] so that the range of scores runs from a low score of four to a high score of 16. A raw score of 12 is the equivalent of a scaled score of 41 on the writing test, and, under the compensatory scoring model, one can pass with a raw score as low as 11.

Plaintiffs argue that the failure rates for the CBEST prove that the cut scores are set too high. According to Dr. Poggio's analysis, 45 percent of incumbent teachers and 25 percent of recent college graduates who take the CBEST fail it. (Poggio Supp.Direct Test. at 45-46; Haertel Direct Test. at 38-41.)

According to plaintiffs, the passing score on the CBEST was set for political reasons, without any regard for the available psychometric evidence. The cut-off score may not be set at an arbitrary level with no relationship to teacher competence. See Richardson, 729 F.Supp. at 822-23; Thomas v. City of Evanston, 610 F.Supp. 422, 431 (N.D.Ill. 1985). Cut scores must be set by some systematic process that reflects the goodfaith exercise of professional judgment. Richardson, 729 F.Supp. at 823.

Superintendent Louis (Bill) Honig ("Honig") had the ultimate responsibility for setting the CBEST's passing scores. He was assisted in his decision by the CTC and an advisory board created by the CBEST legislation ("Advisory Board"). See Cal.Educ. Code § 44252(c). In making his decision, Honig had before him the results of the Wheeler and Elias study, the results of the field test, and the recommendations of the CTC and the Advisory Board.

The Wheeler and Elias study used a technique called the Angoff method to produce cut scores for the CBEST. In the Angoff method, the judges are asked to estimate the expected performance level of a "borderline" or minimally competent group of teacher candidates. For each test item, the judges indicate what percentage of a group of borderline candidates would answer the item correctly. Then, an estimated passing score is computed by summing the percent correct estimation for each item and calculating the mean. In this case, the judges made Angoff ratings for each of the items on the mathematics and reading subtests of the CBEST.

To illustrate the borderline candidate, the description of a fictional teacher named "Dale" was read to all the judges.^[42] There was much hand-wringing on the part of defendants at trial over the prospect of such a singularly unimpressive candidate as Dale obtaining teaching credentials in California. The Court agrees with the opinion of Dr. Lundquist that Dale is a description of an incompetent, rather than minimally competent, teacher.^[43] If anything, the use of Dale in the Wheeler and Elias study tended to depress the judges' estimated performance scores, thereby lowering the recommended passing score for the multiple-choice portions of the CBEST.

The judges working independently and the judges working in discussion groups arrived at similar passing scores. For both groups the reading score was 23 out of 40 questions, and for the math test, roughly 19 correct out of 40.^[44]

For the writing test, each of the 14,000 essays obtained during a field test was read by two readers. They were scored on a four-point scale: one for not pass, two for marginal not pass, three for marginal pass, and four for pass. The score levels ranged from 8 to 13. The essays were sorted into groups by score and reread for scoring consistency. The readers unanimously agreed that the desirable passing score was 12, with an absolute minimum passing score of 11.

The CTC advised Honig to follow the Wheeler and Elias study's recommended passing scores. (Ex. 40.) The Advisory Board, however, recommended somewhat higher passing scores: 25 on the reading test and 23 on the mathematics test. (Ex. 179 at 2, tbl. 1.)

Honig set the passing scores higher still: 28 for the reading test (70 percent correct) and 26 for the math test (65 percent correct).^[45] (See Ex. 42, Honig letter to John Brown, Executive Secretary, California Teacher Preparation and Licensing.)

According to plaintiffs, Honig arbitrarily ignored the results of the Wheeler and Elias study and the advice of the CTC to follow the study's recommendations, and set passing score standards at unreasonably high levels.

The Court disagrees. Honig testified that he did consider both the results of the Wheeler and Elias study and the CTC's advice when he made his decision, though he ultimately disagreed with both recommendations. His decision was not arbitrary, but, as discussed further below, a reasonable exercise of judgment. Moreover, all the subsequent passing-standard studies recommended scores that were higher even than those set by Honig.

Dr. Lundquist conducted two separate standard-setting studies.^[46] As part of her content validity study, she conducted a passing score study for both the reading and mathematics sections of the CBEST, using both the old and new test specifications. (See generally Ex. 1543, content validation report.) A total of 81 teachers, drawn from a sample of approximately 2,000, attended three rating sessions. Ethnic minorities were overrepresented in the group. The teachers were asked to rate all test items on two forms of the CBEST math test (80 items) and one form of the CBEST reading test (40 items). Approximately 20 teachers were also asked to rate a national teacher basic skills examination called the Pre-Professional Skills Test ("PPST"). This was done to create a basis for comparing national skill standards to those tested on the CBEST.

The Angoff ratings on the math test items ranged from 59 percent to 90 percent on one form of the test, and from 66 percent to 88 percent on the other form of the test. The overall mean rating for each math test was thus roughly the same, approximately 78 percent. The result for the PPST was very similar, an overall mean of 81 percent on the math section.^[47] On the reading section, the Angoff ratings ranged from 73 percent to 88 percent, with an overall mean of 82 percent. Again, the result for the PPST was essentially the same, 83 percent.^[48]

Next, Dr. Lundquist calculated passing scores for the CBEST test forms with items weighted to represent their proportionate weights under the new CBEST specifications.^[49] The unweighted and weighted standards for both CBEST subtests were virtually the same. For the reading test, the weighted score was 81.5 percent and the unweighted score was 81.7 percent, a difference of only two-tenths of a percent. Surprisingly, the weighted passing score standard on the math test, which included only retained items, was quite similar to the unweighted standard, which included both retained and unretained items. The unweighted standard was approximately 78 percent, and the weighted standard was 77 percent.

Overall, the passing scores recommended by the teacher participants were higher than the actual passing scores on the CBEST. The recommended passing score for reading was 32, compared with the actual score of 28. The recommended passing score for math was 30, compared to the actual score of 26.

Dr. Lundquist undertook a second, supplemental passing score study of the revised CBEST before it was first implemented in August 1995. (See generally Ex. 1540, supplemental passing score study results.) From a sample of 2,300 teachers, a total of 164 teachers participated in five group meetings, each one held in a different location in California.^[50] At each meeting, the judges were given a brief description of the CBEST, and a discussion was held about minimum competence. A few practice ratings were performed by the teachers and discussed in the group.

The participants were then given the items chosen for the August 1995 administration of the revised CBEST, 50 items^[51] each in math and reading. The teachers were asked to make judgments about the percentage of a group of minimally competent examinees who would answer each math and reading item correctly.

The judges were given the correct answers to the items and, as a reality check, the "p-values" for every fifth item. The p-value is the percentage of examinees who have taken the CBEST who answered a particular item correctly. Thus, a p-value of 50 means that 50 percent of examinees answered that item correctly.

As in Dr. Lundquist's previous passing score study, the judges' recommended passing scores were higher than the actual passing scores. The overall mean passing score on the reading test was 77 percent, or a raw score of 31 out of 40 items correct. For the math test, the mean passing score was approximately 79 percent (using either the old or the new test specifications), or a raw score of 32 out of 40 items correct.

Dr. Lundquist conducted subgroup analyses to determine whether there were any differences in ratings among ethnic groups or between elementary and secondary school teachers. Her report shows that there were no significant differences among the judgments made by these groups.^[52]

Dr. Lundquist then lowered the proposed cut scores by one standard error of measurement to account for possible estimation errors by the judges who gave the Angoff ratings. In doing so, she acknowledged that errors in Angoff judgments can occur in both directions — i.e., the judges could just as easily been too lenient rather than too harsh — but she chose to err in the examinees' favor. (Lundquist Supp.Test. at 4 & n.)

A brief explanation of the term "standard error of measurement" is in order. An examinee's actual ("observed") score on one form of the CBEST might differ from that examinee's "true" score on the CBEST because the CBEST, like any other examination, is not a perfect measure; there is some random error inherent in all testing. In this context, the examinee's "true" score is defined as the average of the scores that the examinee would have earned, hypothetically speaking, if the examinee had taken a very large number of different forms of the CBEST. How much the examinee's observed score differs from the examinee's true score is estimated by an index called the "standard error of measurement." This index is a function of two factors: (1) the reliability of the test and (2) the standard deviation of the test scores across all examinees. The more reliable the test, the smaller the standard error of measurement will be because the examinee's observed score will be closer to the examinee's true score.

Thus, by lowering the recommended pass score set by the judges who gave the Angoff ratings, Dr. Lundquist did two things. She greatly increased the likelihood that an examinee would pass the CBEST if that examinee's true score met or exceeded the Angoff judges' recommended cut score. She also, however, greatly increased the odds that an examinee would pass even if that examinee's true score were well below the one recommended by the Angoff judges. As discussed below, this decision was questionable, but undoubtedly worked in plaintiffs' favor.

Dr. Lundquist's final recommended passing scores for the new CBEST are as follows: for reading, 28 out of 40 items (70 percent correct); for math, 29 out of 40 items (72.5 percent correct). Thus, in the final analysis, Dr. Lundquist has recommended that the reading score remain the same, whereas the recommended math score would rise by three points from 26 to 29.

Plaintiffs contend that Dr. Lundquist's Angoff procedure was flawed because multiple passes were not used and because participants were shown p-values for every fifth item from the outset, which, they claim, drove up the ratings. Neither of these arguments has merit.^[53]

Ordinarily, a group of judges in an Angoff procedure rates the same test items more than once or else rates more than one set of test items (multiple passes or iterations). Dr. Mehrens testified, however, that multiple passes do not usually change the Angoff judgment; they simply reduce the amount of variance in the judges' ratings. Furthermore, Dr. Mehrens questioned the usefulness of multiple passes in Dr. Lundquist's situation, where she had five different groups of judges and a very large number of judges overall for an Angoff procedure. Multiple passes seem especially uncalled for where, as here, the five groups independently came up with similar ratings.

With respect to the p-value issue, Dr. Lundquist demonstrated at trial that there was no difference in the Angoff ratings between items for which p-values were shown to the judges and items for which p-values were not shown. Plaintiffs' own expert, Dr. Poggio, ultimately conceded that this was the case.

The results of Dr. Lundquist's passing score study are strongly supported by the consistency of the Angoff judgments among the five separate panels, which tends to show that the instructions given were understandable, precise, and consistent across panels. Her methodology is further bolstered by a comparison of the Angoff judgments with the actual passing rates for the August 1995 CBEST. After the administration of the revised CBEST, Dr. Lundquist compared the actual first-time and historical passing rates for each item with the Angoff judgment for the item. Overall, she found a high correlation between the Angoff estimates and the first-time and historical p-values for the items on both the reading and math tests. This suggests that the raters accurately assessed the relative difficulty of the CBEST items. Dr. Lundquist also found that Angoff estimates were generally eight to ten percentage points below the pass rates for all examinees, which tends to show that the judges properly recognized that borderline candidates will perform less well than the group of all test-takers. (Lundquist Supp. Direct Test. at 8.)

The determination of where to set the passing scores is essentially a judgment call. The SIOP Principles provide:

Judgment is necessary in setting any critical or cutoff score. A fully defensible empirical basis for setting a critical score is seldom, if ever, available. The only justification that can be demanded is that critical scores be determined on the basis of a rationale which may include such factors as estimated cost-benefit ratio, number of openings and selection ratio, success ratio, social policies of the organization, or judgments as to required knowledge, skill, or ability on the job. If critical scores are used as a basis for rejecting applicants, their rationale or justification should be made known to users.

SIOP Principles at 32-33.

On a test like the CBEST, the cut score should be set at the level of basic skills knowledge that one believes the teacher candidates should possess. The evidence presented at trial showed that, if anything, the passing scores should have been raised rather than lowered. This is particularly true in light of the fact that many of the CBEST skills were judged to be crucial to the job of teaching; logically, therefore, an examinee would need to get such items correct to be considered minimally qualified. (Mehrens Direct Test. at 40.) Dr. Mehrens thus opined that the cut scores recommended by the Wheeler and Elias study were very low. A candidate would only be required to answer correctly 57.5 percent of the reading questions and 47.5 percent of the math questions.

Similarly, one can question the wisdom of Dr. Lundquist's decision to lower the recommended passing scores in her supplemental passing score study by a standard error. She could justifiably have left them where there were, or even raised them by a standard error, because measurement errors can run in both directions (i.e., to the test-taker's advantage or disadvantage). In other words, due to the random error inherent in all testing — no test is a perfect measure — one assumes that some people whose scores are close to the cut score will pass when they should have failed, and vice-versa.

As Dr. Mehrens testified, the decision to alter the cut score depends on the relative cost-benefit analysis of "false positives" versus "false negatives" on the CBEST. "False positives" are individuals who pass the CBEST "who are not truly at or above the judged minimum competence cut score but obtained passing scores because of positive error of measurement"; "false negatives" are individuals who fail the CBEST "who are not truly below the judged cut score but obtained failing scores due to negative error of measurement." (Mehrens Supp.Direct Test. at 32.)

Thus, a few candidates who fail the CBEST actually do possess the minimum level of competency in basic skills that the test is supposed to measure. Such a candidate is called a "false negative" because the candidate has been judged not to be competent in basic skills (has tested "negative"), when in fact the candidate possesses those skills (so the negative outcome is a "false" one). Likewise, some candidates who pass the CBEST in fact do not possess basic skills competency; they are called "false positives" because they have tested "positive," viz., judged to be competent in basic skills, when in fact they are not.

Here, the risk of false positives undoubtedly involves greater overall social costs than the risk of false negatives. A teacher candidate who lacks basic skills but passes the CBEST is certified as a teacher for life, whereas a candidate who possesses basic skills but fails the CBEST has the opportunity to retake the examination until she passes it. This weighs in favor of erring on the side of failing some qualified examinees by raising the cut score in order to prevent false positives, rather than lowering the cut score in order to prevent false negatives.

The Court finds that the cut scores on the CBEST, as set, represent professionally acceptable judgments about both the required knowledge, skills, and abilities for teaching jobs, and the estimated cost-benefit ratio, that is, the relative costs and benefits of false positives versus false negatives. Indeed, the CTC would be justified in raising the cut scores as herein discussed.

E.

Because defendants have succeeded in showing that the CBEST "does in fact measure job-related characteristics," Contreras, 656 F.2d at 1271, the CBEST does not violate Title VII unless plaintiffs can prove that "the test does not constitute a business necessity because an alternative selection device exists which would have comparable business utility and less adverse impact." Clady, 770 F.2d at 1428 (citation omitted).

As discussed in this final portion of the Opinion, the Court finds that plaintiffs have failed to meet their burden of showing the existence of an alternative to the CBEST. Moreover, they failed to produce evidence that any of the proffered alternatives would have a significantly smaller adverse impact on the members of the plaintiff class than does the CBEST.

Plaintiffs first argue that the existing credentialing requirements, aside from the CBEST, provide a sufficient assurance of minimum competency to make the CBEST unnecessary. Currently, these requirements include: (1) a bachelor's degree; (2) completion of, and recommendation for certification by, an accredited teacher preparation program; and (3) subject-matter certification, that is, either a passing score on a subject-matter examination or completion of a state-approved coursework program. Plaintiffs' expert Dr. Edward Haertel testified that these requirements are strong assurances of basic skills, making the CBEST unnecessary. But, it should be noted that a significant number of applicants apparently fail the CBEST despite satisfying these requirements. Consequently, these requirements are not an adequate substitute for the CBEST. Moreover, as defendants elicited on cross-examination, there are a number of other drawbacks to using these requirements as assurances of minimum competence.

First, Dr. Haertel contends that the reading, writing, and mathematics proficiencies required to earn a bachelor's degree clearly exceed the generic proficiencies required for teaching (presumably, the skills tested on the CBEST, which do not exceed the high school level). This proposition, however, is not without doubt. The requirements to obtain a bachelor's degree vary significantly from institution to institution. Simply possessing a B.A. does not mean that the candidate has taken any particular courses or majored in any particular subject or graduated with a good grade point average ("GPA").^[54] It is possible that a degreeholder managed to avoid taking a single class in a given subject, such as mathematics. Not all bachelor's degrees are created equal.

Furthermore, in any given year, anywhere from 35 to 45 percent of credential applicants come from outside California. For a candidate educated out of state, the CTC has no way of knowing whether the candidate's undergraduate institution was accredited and, if so, what that other state's accreditation standards are. Unlike the CBEST, the mere requirement of a bachelor's degree is not a uniform standard.

Second, Dr. Haertel stated that the accreditation process for teacher preparation programs is designed to ensure that each accredited program has verified the competencies of each candidate recommended for certification. In addition, he notes, the level of skills required to complete an accredited teacher program in California is much more advanced than a standard of minimal competency. The CTC, however, accredits teacher preparation programs only in California. For every would-be teacher who comes to California after obtaining a teaching certificate in another state, the CTC has no way of knowing whether or not the program attended by that person was accredited or what the accreditation standards of that other state are.

The same concern applies to the requirement that the average undergraduate GPA of students admitted to an accredited California teacher preparation program must exceed the average GPA of a comparable group of students (for instance, in the same major). Such a restriction does not apply to out-of-state teacher preparation programs.

Third, with respect to subject-matter proficiency, a candidate may either take a subject-matter examination or complete a subject-matter preparation program. Either option, according to Dr. Haertel, involves mastery of material that is substantially more challenging than that tested on the CBEST, at least as far as the reading level is concerned. As for writing and mathematics, Dr. Haertel suggests that such skills are tested to the extent they are needed for any particular subject matter. Again, however, out-of-state candidates may satisfy this requirement by presenting a transcript showing coursework in the subject area in an out-of-state teacher preparation program, the standards for which may not be as rigorous as those set by the CTC for California subject-matter programs, and the CTC has no way of knowing what those standards are. In addition, the math skills that were validated for the CBEST as job-related for all teaching jobs are not included in every subject-matter examination or coursework program.

Furthermore, Dr. Haertel concedes that assurances of teaching competence are weak for emergency permit holders. (Haertel Direct Test. at 19.) Emergency permits are substandard credentials that are issued when a fully credentialed teacher is unavailable in a given subject area or with a specific language skill. The CBEST is a requirement for an emergency credential. See Cal.Educ. Code § 44300(a)(2). An emergency permit may be issued where a person has a baccalaureate degree, a minimal number of units in the subject to be taught, and has passed the CBEST, and where the district requesting the permit certifies the person's subject-matter competence. Id. § 44300(a)(1)-(3). Though the emergency permitting process is intended as a last resort, approximately 2,000 permits were issued in 1989-90. Id.

Alternatively, plaintiffs propose the use of two other selection devices, either of which, they contend, can satisfy defendants' concerns about minimum competency in basic skills and has less adverse impact: (1) a GPA requirement; or (2) a coursework alternative to the CBEST.

These recommendations suffer from the same disadvantages as the previously discussed alternatives, however. First, use of a GPA standard does not take into account differences between institutions and in coursework completed. The Court disagrees with Dr. Poggio's rather preposterous testimony that college GPAs are largely fungible and of equal value no matter where they were earned, whether it be a prestigious, highly rigorous university or an underfunded and poorly regarded community college. (R.T. at 714-15.)

Second, use of a coursework alternative will allow out-of-state candidates to avoid taking the CBEST without adequate assurances about the quality and nature of the coursework that they performed at an out-of-state institution. As was demonstrated at trial, it can be difficult, even impossible, to make judgments about the nature and quality of the courses on an individual's transcript based solely on their titles.

The Court agrees wholeheartedly with Dr. Haertel's opinion that there are more effective ways than the CBEST to improve the quality of teaching in California's public schools: by raising teacher salaries, for example, or implementing a comprehensive support, evaluation, and assessment program such as the California New Teacher Project. Nonetheless, that is beside the point. California does not have unlimited resources to devote to costly improvements in teacher training and support — though certainly one might question the wisdom of foregoing such improvements to public education in favor of other legislative agendas. Regardless of other steps that could be taken, the CBEST remains an objective, cost-effective, and valid way to assure that teachers and others employed in the public schools possess basic skills. None of plaintiffs' proposed alternatives is an adequate substitute for the CBEST.

In sum, the Court holds that defendants' requirement that plaintiffs pass the CBEST in order to obtain employment in the California public schools does not violate plaintiffs' rights under Title VI or Title VII of the Civil Rights Act of 1964.

III.

Accordingly,

IT IS HEREBY ORDERED that:

1. Defendants' requirement that plaintiffs pass the CBEST in order to obtain employment in the California public schools does not violate plaintiffs' rights under Title VI or Title VII of the Civil Rights Act of 1964.

2. Judgment will be entered in favor of defendants.

NOTES

[1] The Court will use the term "Latino," which is employed in the class definition, rather than "Hispanic."

Though the appropriate nomenclature is still debated, a general preference appears to be developing for "Latino" over "Hispanic." Even the Supreme Court of the United States has used the term "Latino." See Hernandez v. New York, 500 U.S. 352, 111 S.Ct. 1859, 114 L.Ed.2d 395 (1991). "``Latino' is an abbreviated version of the Spanish word latinoamericano, or Latin American." "``Hispanic' is the English translation of hispano, a word commonly used to describe all peoples of Spanish-speaking origin." Manuel Perez-Rivas, Hispanic, Latino: Which?, N.Y. Newsday, Oct. 13, 1991, at 6. Though the two terms are roughly synonymous, many prefer the word "Latino" because "Hispanic" is thought to have colonial and assimilative overtones. Paul Brest & Miranda Oshige, Affirmative Action for Whom?, 47 Stan.L.Rev. 855, 883 n. 148 (1995). The term "Latino" is also suggestive of Latin America's indigenous culture apart from its Spanish origins. Deborah Ramirez, Multicultural Empowerment: It's Not Just Black and White Anymore, 47 Stan.L.Rev. 957, 959 n. 9 (1995).

[2] At the outset, it should be emphasized that this Opinion is not a statement of policy by the Court as to the qualifications California public schoolteachers must have, nor is it meant to be a criticism of the policies adopted by the State and the CTC or of the qualifications of the named plaintiffs for employment as teachers in the California public schools. It is simply a ruling that the State and the CTC do not, by insisting that every person who wishes to teach in California public schools first pass the CBEST, discriminate against the plaintiff class in violation of the Civil Rights Act of 1964.

[3] Title VI provides:

No person in the United States shall, on the ground of race, color, or national origin, be excluded from participation in, be denied the benefits of, or be subjected to discrimination under any program or activity receiving Federal financial assistance.

42 U.S.C. § 2000d. The Supreme Court has concluded that Title VI itself prohibits only intentional discrimination, but that disparate impact claims may nonetheless be redressed through agency regulations designed to implement Title VI. See Alexander v. Choate, 469 U.S. 287, 292-93, 105 S.Ct. 712, 715-16, 83 L.Ed.2d 661 (1985) (explaining Guardians Ass'n v. Civil Serv. Comm'n, 463 U.S. 582, 103 S.Ct. 3221, 77 L.Ed.2d 866 (1983)).

Courts have generally applied the standards applicable to disparate impact cases under Title VII to disparate impact cases arising under Title VI. Larry P. v. Riles, 793 F.2d 969, 982 & n. 9 (9th Cir.1984); accord New York Urban League, Inc. v. New York, 71 F.3d 1031, 1036 (2d Cir. 1995); Elston v. Talladega County Bd. of Educ., 997 F.2d 1394, 1407 & n. 14 (11th Cir.1993); Groves v. Alabama State Bd. of Educ., 776 F.Supp. 1518, 1523 (M.D.Ala.1991). Indeed, the Title VI standard arguably imposes a lower burden on defendants, who need only show that "a substantial legitimate justification" exists for the challenged practice. New York Urban League, 71 F.3d at 1036 (quoting Georgia State Conference of Branches of NAACP v. Georgia, 775 F.2d 1403, 1417 (11th Cir.1985)); Elston, 997 F.2d at 1407.

Because the standards are so similar, the Court will not address the merits of plaintiffs' Title VI claims separately.

[4] In July 1983, AMAE filed discrimination charges with the Equal Employment Opportunity Commission ("EEOC") in Los Angeles. Nearly seven years later, in May 1990, the EEOC issued a Letter of Determination. The EEOC notified AMAE in December 1990 that it had referred the case to the Department of Justice.

In May 1992, two years after receiving the Letter of Determination, AMAE, through its counsel, requested a right-to-sue letter from the EEOC. The right-to-sue letter was issued on June 17, 1992.

AMAE then filed a second charge of discrimination with the EEOC on September 8, 1992. A right-to-sue letter on the second charge issued in October 1992. AMAE filed suit in this Court on September 23, 1992.

On August 3, 1995, defendants sought summary judgment on the grounds that plaintiffs' action was barred by the doctrine of laches. The Court denied the motion in a Memorandum Decision and Order filed August 25, 1995, holding that genuine issues of material fact remained regarding the reasons for plaintiffs' delay in bringing suit, including the extent to which the EEOC's actions were to blame.

At trial, defendants did not present any testimony or other evidence on the laches defense. Accordingly, the Court will not consider it.

[5] For a more detailed history surrounding the CBEST's implementation, see Association of Mexican-American Educators v. California, 836 F.Supp. 1534, 1537-39 (N.D.Cal.1993).

[6] There are actually 50 questions in each section, 10 of which are not scored and are used both to field-test new items and to allow the test to be equated with earlier forms of the CBEST.

[7] Many teacher preparation programs in California require applicants to pass the CBEST as a prerequisite to admission, despite the fact that the State expressly discourages using the examination for admissions purposes. See Cal.Educ. Code § 44252(f) ("It is the intent of the Legislature that applicants for admission to teacher preparation programs not be denied admission on the basis of state basic skills proficiency test results.").

[8] The correct answer is A. There are two half-pints in a pint; two pints in a quart; four quarts in a gallon; multiplied by five gallons. Thus, 2 × 2 × 4 × 5 = 80.

[9] It should be noted that not every person teaching in the public schools in California has passed the CBEST. Thousands of teachers were "grandfathered" in at the time the CBEST legislation was passed, and others have received waivers from the CTC.

[10] Congress removed the exemptions for state and municipal employers in the 1972 amendments to Title VII. See H.R.Rep. No. 238, 92d Cong., 2d Sess. 1, reprinted in 1972 U.S.C.C.A.N. 2137, 2137, 2152-54.

[11] Title VII provides, in pertinent part:

It shall be an unlawful employment practice for an employer —

(1) to fail or refuse to hire or to discharge any individual, or otherwise to discriminate against any individual with respect to his compensation, terms, conditions, or privileges of employment, because of such individual's race, color, religion, sex, or national origin; or

(2) to limit, segregate, or classify his employees or applicants for employment in any way which would deprive or tend to deprive any individual of employment opportunities or otherwise adversely affect his status as an employee, because of such individual's race, color, religion, sex, or national origin.

42 U.S.C. § 2000e-2(a).

[12] Title VII further provides as follows:

Notwithstanding any other provision of this subchapter, it shall not be an unlawful employment practice for an employer ... to give and to act upon the results of any professionally developed ability test provided that such test, its administration or action upon the results is not designed, intended or used to discriminate because of race, color, religion, sex or national origin.

42 U.S.C. § 2000e-2(h) (1994).

[13] See also Connecticut v. Teal, 457 U.S. 440, 446-47, 102 S.Ct. 2525, 2530-31, 73 L.Ed.2d 130 (1982); Dothard, 433 U.S. at 329, 97 S.Ct. at 2726-27; Albemarle Paper Co. v. Moody, 422 U.S. 405, 425, 95 S.Ct. 2362, 2375, 45 L.Ed.2d 280 (1975); Griggs, 401 U.S. 424, 91 S.Ct. 849.

[14] Courts have held that burden of proof provisions in noncriminal statutes are procedural and apply retroactively. E.g., Lavespere v. Niagara Mach. & Tool Works, Inc., 920 F.2d 259 (5th Cir.1990), cert. denied, 510 U.S. 859, 114 S.Ct. 171, 126 L.Ed.2d 131 (1993); Lieberman-Sack v. Harvard Community Health Plan, 882 F.Supp. 249 (D.R.I.1995). This is consistent with the general practice that a judicial decision altering the burden of proof in a civil action applies retroactively to cases pending at the time of the decision. See, e.g., Melton v. Moore, 964 F.2d 880 (9th Cir.1992) (discussing James B. Beam Distilling Co. v. Georgia, 501 U.S. 529, 111 S.Ct. 2439, 115 L.Ed.2d 481 (1991)).

[15] The Court recognizes, of course, that many Latinos are also Caucasian. For simplicity's sake, however, the Court will use the term "whites" to mean non-Latino Caucasians.

[16] Plaintiffs have also argued that cumulative pass rates are inappropriate because minority applicants who fail the CBEST are deterred from retaking the test at greater rates than nonminority applicants. See Dothard v. Rawlinson, 433 U.S. 321, 330, 97 S.Ct. 2720, 2727, 53 L.Ed.2d 786 (1977). The CTC undertook a study of this issue in 1990. (Ex. 116.) The data showed that Asians and African Americans who failed the CBEST repeated the test in smaller numbers than Latinos and whites. The reasons for this disparity, however, are unknown. Plaintiffs produced no evidence of any deterrent effect other than the testimony of Dr. Jew, which the Court finds to be of little weight due to its unscientific, anecdotal nature.

Notwithstanding this finding, the Court notes that the information provided in the CTC pamphlet distributed to those who fail the CBEST is unfortunate. (See Ex. 60.) The pamphlet provides a somewhat misleading table showing the likelihood of passing each section of the CBEST upon retaking it, based on one's current score. As the testimony of defendants' expert, Dr. William Mehrens, illustrated, the table can easily be misinterpreted to reduce the candidate's real likelihood of success by a significant measure. The pamphlet also fails to emphasize that studying for the examination can greatly increase a candidate's ability to pass the CBEST on retaking.

[17] Defendants argued that the CBEST is a licensing test, but it is in fact an employment test. See AMAE, 836 F.Supp. at 1550.

[18] An employment test is not required to test every skill required to perform the job. Nevertheless, under professional testing standards, "the investigator should describe the whole job as part of the job analysis, indicate what is included in the domain and explain why certain parts were or were not included." Society for Industrial and Organizational Psychology, Principles for the Validation and Use of Personnel Selection Procedures 20 (3d ed. 1987) ("SIOP Principles").

[19] In addition, both studies are of questionable value because of their serious methodological flaws. The Dick study involved a very small sample size, thirty-two teachers, and it suffered from an extreme range restriction problem: Only two teachers in the sample had failed the CBEST; the other thirty had passed. The Fresno study also had an extreme range restriction problem because 98 percent of the teachers in the sample received the same rating, "satisfactory," which was the highest available rating on a three-point scale.

[20] The Court finds Dr. Lefkowitz lacking in credibility. Other courts have also made note of his sloppy analysis, which has included using the incorrect formulae and making basic errors in computation. E.g., Police Officers for Equal Rights v. City of Columbus, 916 F.2d 1092, 1102-03 (6th Cir.1990).

[21] In addition, defendants introduced the Curriculum Matching Project, in which two ETS employees, Ave Maria Merritt and Michael Ponisciak, matched CBEST test specifications to material found in textbooks purportedly used in the California public schools. Due to the unscientific nature of the study, the Court did not find it particularly helpful; however, the study did support the overall conclusion that the kinds of skills tested on the CBEST can be found in elementary and secondary school textbooks.

[22] The term "teacher educator" sounds redundant, but it means a person who teaches would-be teachers.

[23] Plaintiffs made much of the fact that the data recording form used in the study instructed the judge as follows: "For each item, check the level of relevance for the CBEST program." (Ex. 120, App. A.) It is true, as plaintiffs argue, that the relationship of an item to "the CBEST program" is not the proper inquiry. Even defendants' expert Dr. William Mehrens had to concede as much. The Wheeler and Elias report, however, clearly indicates throughout that the judges were instructed to consider the relevance of items to the job of teaching, not to "the CBEST program." The phrasing on the form was sloppy, but it is a minor mistake and does not negate the study's results.

[24] The judges were also asked whether the relative weights assigned to the reading and math skills were right and, if not, what weight they should be given. Here, the results were more mixed. More than half of the judges concluded that the weights given the reading skills were about right. Of those who did not, most thought that literal comprehension should be given more weight and logical comprehension, less weight. The percentage of judges who thought that the weights assigned to the math skills were about right was much lower; those results were as follows:

Arithmetic             (40%)     46%
Algebra                (40%)     35
Geometry               (20%)     59

Generally speaking, the judges concluded that more weight should be given to arithmetic and less to algebra, though overall there was significant disagreement on the weighting of skills.

As Dr. Mehrens explained, however, the relative weight assigned to each of the math subtests is unimportant where they are highly correlated, which they are here. (Mehrens Direct Test. at 30-31.) Altering the weights would have "only very minimal effects" on test-takers' CBEST scores. (Id. at 31.)

[25] The time constraints data was considered by Dr. Lundquist to be uninterpretable because of nonsensical results: For instance, high ratings were given to the activity "Establish rapport with all students." In addition, she received telephone calls from teachers expressing confusion over the category. Thus, no analysis of the data was conducted. (Ex. 1541 at 126, job analysis report.)

[26] No analysis was performed on the "required at entry" data because of concerns about the data. The category was supposed to describe skills that a teacher should know upon entering the job, as opposed to skills that would be learned on the job. Again, nonsensical results suggested that the participants did not understand the scale. For example, the skill "Use the table of contents ... of a book to locate information" was rated as required upon entry by less than 80 percent of the survey respondents. In addition, as with the "time constraints" category for activities, Dr. Lundquist received phone calls from confused participants. (Ex. 1541 at 130, job analysis results.)

[27] The two dropped reading skills were (1) "Understanding material in another language" and (2) "Recognizing the style or manner of expression used by the author." (Ex. 1967 at 24.)

[28] The two dropped writing skills were (1) "Using informal language when writing" and (2) "Writing at an appropriate level for professional publication." (Ex. 1967 at 25.)

[29] The ten retained math skills were:

1. Add, subtract, multiply, and divide with whole numbers.

2. Add, subtract, multiply, and divide with fractions, decimals, and percentages.

3. Determine and perform necessary arithmetic operations to solve a practical math problem (e.g., determine the total invoice cost for ordered supplies by multiplying quantity by unit price, summing all items).

4. Solve mathematical problems using a calculator.

5. Understand and use standard units of length, temperature, weight, and capacity in the U.S. measurement system.

6. Measure length, perimeter, area, and volume.

7. Understand and use estimates of time to plan and achieve work-related objectives.

8. Estimate the results of problems involving addition, subtraction, multiplication, and division prior to computation.

9. Determine whether enough information is given to solve a problem; identify the facts given in a problem.

10. Use numerical information contained in tables and various kinds of graphs (e.g., bar, line, circle) to solve math problems.

(Ex. 1541, ex. 5-20, job analysis results.)

[30] The twenty-seven dropped math skills were:

1. Add, subtract, multiply, and divide with positive and negative numbers.

2. Perform arithmetic operations that involve powers, roots, and scientific notation.

3. Perform arithmetic operations with basic statistical data related to test scores (e.g., averages, ratios, proportions, and percentile scores).

4. Recognize relationships in numerical data (e.g., compute a percentage change from one year to the next).

5. Solve simple algebraic problems (e.g., equations with one unknown).

6. Solve complex algebraic problems (e.g., solving equations with multiple unknowns, factoring algebraic expressions).

7. Understand and use standard units of length, temperature, weight, and capacity in the international metric system.

8. Use a formula to compute perimeter, area, and volume.

9. Solve problems that involve angles, triangles, parallel lines, and perpendicular lines using geometric principles and theorems.

10. Understand and solve problems using simple coordinate geometry (i.e., plot points on an x-y axis, solve problems involving the interpretation of points on an x-y axis).

11. Understand basic principles of probability and predict likely outcomes based on data provided (e.g., estimate the likelihood that an event will occur).

12. Apply properties of exponential and logarithmic functions (e.g., growth and decay, Richter scales, pH scales, am/fm radio dials).

13. Translate a math equation into a verbal problem.

14. Write equations using variables to represent situations presented in words.

15. Recognize alternative mathematical methods of solving a problem.

16. Recognize the position of numbers in relation to each other (e.g., 1/3 is between ¼ and ½ - 7

17. Use the relations less than, greater than, or equal to, and their associated symbols to express a numerical relationship.

18. Identify numbers, formulas, and mathematical expressions that are mathematically equivalent (e.g., 2/4 = ½, [pi]d = 2[pi]r).

19. Recognize patterns and spatial relationships through observations of geometric figures (e.g., seeing the right triangles in a cube).

20. Understand and use rounding rules when solving problems.

21. Understand and apply the meaning of logical connectives (e.g., and, or, if-then) and quantifiers (e.g., some, all, none).

22. Identify or specify a missing entry from a table of data (e.g., subtotal).

23. Use graphs of discrete and continuous functions (e.g., bell curves, scattergrams).

24. Read and interpret schematic diagrams, such as flowcharts, electrical wiring diagrams, and scale drawings.

25. Interpret the meaning of mean, median, and mode.

26. Interpret the meaning of variance and standard deviation.

27. Interpret the meaning of standardized test scores (e.g., stanine scores, percentiles) to determine how individuals performed relative to other students.

(Ex. 1541 ex. 5-20 (job analysis results).)

[31] In addition, the two groups were asked to judge what percentage of minimally qualified teachers would be proficient in the skill today and what percentage would be proficient in the skill in three to five years' time.

[32] The nine additional skills were:

1. Perform arithmetic operations with basic statistical data related to test scores, such as averages, ratios, proportions, and percentile scores.

2. Recognize relationships in numerical data, for example, compute a percentage change from one year to the next.

3. Solve simple algebraic problems, for example, equations with one unknown.

4. Understand basic principles of probability and predict likely outcomes based on data provided. For example, estimate the likelihood that an event will occur.

5. Recognize alternative mathematical methods of solving a problem.

6. Recognize the position of numbers in relationship to each other. For example, knowing that one-third is between one-fourth and one-half, or that minus seven is less than minus four.

7. Understand and use rounding rules when solving a problem.

8. Understand and apply the meaning of logical connectives, the words "and," "or," "if, then," and quantifier words like "some," "all" or "none."

9. Identify or specify a missing entry from a table of data, for example, a subtotal.

(See R.T. at 1299-1301.)

[33] Plaintiffs have suggested that the revised CBEST improperly includes skills that should have been eliminated from the test specifications as a result of Dr. Lundquist's job analysis and content validity study, but they have shown no evidence of this.

In particular, plaintiff's counsel argued repeatedly that the revised CBEST improperly included an item that he characterized as a "multiple unknown" problem, even though solving algebraic equations with multiple unknowns is a skill that was not retained.

The Court was surprised to hear counsel make this argument. To set the record straight, the problem did not involve multiple unknowns; it involved a single unknown. The problem reads as follows:

According to her college transcript, Emilia received 6 more A's than B's. If she received a total of 36 A's and B's, how many A's did she receive?

A. 12
B. 21
C. 24
D. 27
E. 30

(Ex. 642 (problem 70).)

The most straightforward way to solve the problem is thus:

36 = x + (x + 6)
36 = 2x + 6
30 = 2x
15 = x

"X" represents the number of B's and "x = 6" represents the number of A's. Therefore, Emilia received 21 A's and the correct answer is B. There is but one unknown: x.

Moreover, as Dr. Lundquist testified, this is not the only way to solve the problem. One could also eyeball it — that is, divide 36 by two to get 18, then count up by three to get 21. Or, one could use trial and error by simply subtracting six from each answer choice until one figured out which answer plus the answer minus six equals 36. None of these methods involves using more than one unknown.

According to defendants, those unretained math skill items that remain on the revised CBEST are used only for equating purposes; they are not scored.

[34] The CBEST passing scores remained unchanged.

[35] Plaintiffs argue that Dr. Lundquist's criteria for retention of skills in the job analysis were too lenient because (1) she did not make use of the "required at entry" data and (2) she used a 1.5 mean rather than a 2.0 mean for the importance scale, which plaintiffs consider too low.

A 1.5, which is the midpoint of Dr. Lundquist's three-point scale, falls between "minor" and "important." Plaintiffs argue that this was insufficient to guarantee that the skill so adjudged was important, which is required by the Uniform Guidelines and prevailing professional standards. The Uniform Guidelines state:

There should be a job analysis which includes an analysis of the important work behavior(s) required for successful performance and their relative importance.... The work behavior(s) selected for measurement should be critical work behavior(s) and/or important work behavior(s) constituting most of the job.

Uniform Guidelines, 29 C.F.R. § 1607.14(C)(2).

The Court finds that both of Dr. Lundquist's decisions reflect manifestly reasonable professional judgments. As discussed supra, Dr. Lundquist explained that she did not analyze the "required at entry" data because of concerns that the respondents had not understood what they were being asked. With respect to the 1.5 mean, as Dr. Lundquist testified at trial, a 1.5 rounds up to 2.0. It must be remembered that the mean rating of 1.5 was coupled with an 80 percent endorsement criterion, which is quite stringent.

[36] Seventeen percent of the 234 panelists in Dr. Watkins' Practitioners' Review in 1985 were nonteachers. (Ex. 118 at 4, Practitioners' Review.) By examining the data from that study, it is possible to determine whether the responses of nonteachers differed from the responses of teachers. They did not. Approximately 89 percent of nonteachers judged the reading questions to be either moderately or very relevant, compared with 90 percent overall. On the math test, 91 percent of nonteachers compared with 87 percent overall rated the items to be moderately or very relevant.

[37] During the first administration of the CBEST, examinees had 60 minutes for each of the three sections. From 1982 to 1984, the time limit for the mathematics section was 65 minutes. In 1984, the time limits were increased to 65 minutes on reading, 70 minutes on mathematics; writing remained at 60 minutes.

In June 1995, the time limit was changed to four hours to complete either the entire test or any portion(s) of the test that the examinee has previously failed; thus, the examinee has four hours in which to complete one, two, or all three sections of the CBEST.

[38] "Evaluating the acceptability of a test or test application does not rest on the literal satisfaction of every primary standard in this document, and acceptability cannot be determined by using a checklist[.]" American Psychological Association, et al., Standards for Educational and Psychological Testing 2 (1985) ("APA Standards").

[39] Indeed, the professional testing standards recognize that periodically revisiting the validity of a selection device is an appropriate undertaking. See Uniform Guidelines, 29 C.F.R. § 1607.5(K); APA Standards at 27, 29, standards 3.10, 3.18.

[40] A raw score of 37 represents approximately 65 percent correct on the reading test and 60 percent correct on the math test.

[41] The raw score scale is defined as follows: one, clear fail; two, marginal fail; three, marginal pass; four, clear pass.

[42] Plaintiffs have suggested that the definition of the minimally competent candidate was not limited to Dale and was open-ended and vague, but there is no evidence to support this. It is clear from the record that Dale was the touchstone for the borderline candidate in the Wheeler and Elias study.

[43] The Dale description is as follows:

— Dale was an average high school student that [sic] wanted to go into education to work with children.

— In taking the SAT for entrance into college, Dale scored 360 on the verbal section ([mean] = 426) and 380 on the quantitative section ([mean] = 467) (these scores are well below the average SAT performance).

— Throughout the undergraduate program, Dale received B's and C's and an occasional A in the liberal arts teacher education curriculum.

— While student teaching in the sixth grade, the supervising teacher noted in her evaluation, that Dale's lesson plans and progress notes were poorly organized and contained numerous grammatical errors.

— The supervising teacher also noted that Dale reads aloud rather poorly.

— Dale can, of course, read the daily newspaper but only comprehends and retains superficial knowledge.

— In student teaching, Dale is personable and truly enjoys working with children.

— Dale can give clear directions for assignments, but often times [sic] doesn't answer questions from students, instead referring them to the librarian or science teacher, as appropriate.

— The supervising teacher found that her lesson plans have not been followed by Dale to the letter and that mistakes were made in the grade books.

— The principal and department head have also done several evaluations [sic] of Dale's work.

— The department head finds that Dale doesn't seem to read well enough to understand the curriculum guides of the district. Furthermore, in grading students [sic] essays, Dale fails to detect grammatical errors in student's [sic] papers.

— Notes home to parents occasionally [sic] have spelling mistakes and exhibit a rather haphazard organization of the information.

— The principal finds that Dale has difficulty keeping track of lunch money, field trip expenses and attendance records.

— The principal doesn't know whether Dale lacks attention to detail [sic] or is simply deficient in some reading and math skills.

(Ex. 120, App. E.)

[44] The independent judges' figure was 19.5 and the discussion group's, 17.8.

[45] The writing score was set where recommended, at a raw score of 12.

[46] In addition, the 1985 Watkins study included a passing score study. The Court finds this study less helpful than Dr. Lundquist's studies because of the way in which the participants were instructed. The judges were asked to estimate the percentage of "entry-level credential applicants" who should be expected to answer each test item. This task is quite different from the Wheeler and Elias approach, in which the judges were asked to make judgments regarding borderline, rather than all entry-level, candidates. Also, the Ebel method was employed, which, though similar to the Angoff method, tends to result in higher passing standards.

Thus, as one would expect, the recommended passing scores in the Watkins study were higher than those obtained in the 1982 study. Taking the mean of the recommended passing scores, the passing score for the reading test was 29, the same as the actual passing score set by Honig, and the passing score for the mathematics test was also 29, two points higher than the actual passing score.

[47] It is worth noting that a passing score of 81 percent correct on the PPST would result in only a one-third passing rate on the examination, whereas a comparable passing score on the CBEST would result in a substantially higher pass rate, 65 percent.

[48] With such a passing score, only 40 percent of PPST examinees would pass the reading test, whereas 62 percent of CBEST examinees would pass.

[49] As noted above, many of the math items were not retained under the new CBEST specifications. Thus, for her analysis, a total of only 33 items from both forms of the math test were retained, which she combined as though they were one form of the test.

[50] The meetings were held in Fresno (18 participants), Los Angeles (54 participants), Sacramento (33 participants), San Diego (22 participants), and San Jose (37 participants).

[51] Again, this includes the 40 scored items and the 10 nonscored items on each subtest.

[52] Although Dr. Lundquist found that African-Americans established lower passing standards than other ethnic groups, she determined that the cause of this was a single participant whose ratings were extremely low — more than three standard deviations below the group mean. Though she chose not to do so, Dr. Lundquist clearly would have been justified in discarding these outlier ratings.

[53] Dr. Poggio attempted to challenge the results of the supplemental passing score study by conducting a telephone survey of study participants. The results of that survey purported to show that the participants in Dr. Lundquist's passing score study did not have a clear understanding of the task they were asked to complete, thereby raising concerns about the study's conclusions.

The Court, however, finds that the telephone survey is fatally flawed methodologically and wholly unreliable. It was neither blind nor double-blind; there is no assurance that the interviews were conducted in any kind of consistent or standardized manner; the script (such as it was) contained biasing language; the interviews were conducted as much as three months after the interviewee had participated in the passing score study; and, perhaps most important, the participants' responses to questions about how they had been instructed at the passing score study were not even recorded, despite the fact that the telephone survey is supposed to support Dr. Poggio's main assertion that Dr. Lundquist's instructions to the Angoff panels had been inadequate.

[54] Table 3 in Dr. Haworth's report shows that over half of those who take the CBEST are either "seeking credentials" (48 percent) or fulfilling a "requirement for employment" (6 percent), and Tables 13A and 13B show that only 83 percent of these applicants eventually pass. Thus, as Dr. Mehrens pointed out, some or all of the 17 percent who failed probably would have been credentialed were it not for the CBEST, i.e., they probably satisfied Dr. Haertel's requirements even though they lack the basic skills that are needed to pass the CBEST. No one rebutted Mehrens' testimony on this point.