In Re Proportionality Review Project

The opinion of the Court was delivered by

O’HERN, J.

When the United States Supreme Court restored the constitutionality of the death penalty, it imposed a concomitant obligation on states to provide “the further safeguard of meaningful appellate review” of every death sentence. Gregg v. Georgia, 428 U.S. 153, 195, 96 S.Ct. 2909, 2935, 49 L.Ed.2d 859, 887 (1976). This matter arises out of our exercise of that function and concerns specifically our system for proportionality review of death sentences. By that we mean not the review of any legal error in the imposition of the sentence but, rather, the review of the sentence itself. We seek to determine “[w]hether the sentence of death is excessive or disproportionate to the penalty imposed in similar cases, considering both the crime and the defendant.” Id. at 167, 96 S.Ct. at 2922, 49 L.Ed.2d at 871. Proportionality review seeks “to ensure that the death penalty is being administered in a rational, non-arbitrary, and evenhanded manner, fairly and with reasonable consistency.” State v. Marshall, 130 N.J. 109, 131, 613 A.2d 1059 (1992), cert. denied, 507 U.S. 929, 113 S.Ct. 1306, 122 L.Ed.2d 694 (1993) (Marshall II). That review serves as “a means through which to monitor the imposition of death sentences and thereby to prevent any impermissible discrimination in imposing the death penalty.” State v. Ramseur, 106 N.J. 123, 327, 524 A.2d 188 (1987), denial of habeas corpus aff'd. sub nom., Ramseur v. Beyer, 983 F.2d 1215 (3d Cir.1992), cert. denied, 508 U.S. 947, 113 S.Ct. 2433, 124 L.Ed.2d 653 (1993). “Proportionality review seeks to determine only whether a particular death sentence is aberrational, not whether it compares perfectly with other sentences.” State v. Bey, 137 N.J. 334, 352, 645 A.2d 685 (1994) (Bey IV), cert. denied, 513 U.S. 1164, 115 S.Ct. 1131, 130 L.Ed.2d 1093 (1995) (citing Marshall II, supra, 130 N.J. at 131, 613 A.2d 1059). Our goal is to prevent the death penalty from being imposed “capriciously or in a freakish manner.” Gregg, supra, 428 U.S. at 195, 96 S.Ct. at 2935, 49 L.Ed.2d at 887. Our dissenting member argues that we should insist that death sentences be “generally imposed” in similar cases for a sentence of death to be found to be proportional. Post at 105-06, 735 A.2d at 547-48.

Because New Jersey jurors have been sparing in their imposition of the death sentence, it will never be the case that death would be “generally received” or “received in a defined preponderance of cases.” Because juries impose death infrequently, we have recognized that “death need not be normal or general to be a licit sentence.”

[State v. Loftin, 157 N.J. 253, 322, 724 A.2d 129 (1999) (Loftin II) (quoting Marshall II, supra, 130 N.J. at 153, 613 A.2d 1059).]

Our dissenting member asks for more. He “would have us find that death is the normal sentence [for similar cases] when that can never be so.” Ibid.

In Loftin II, supra, 157 N.J. at 279, 724 A.2d 129, we considered our existing system of proportionality review and the effect of N.J.S.A. 2C:11-3e, which limited proportionality review “to a comparison of similar eases in which a sentence of death has been imposed.” The details of our system of proportionality review are fully described in that opinion and need not be repeated here in any detail. We give this brief summary.

The system consists of two parts. The first part is frequency analysis, a statistical measure of the numerical frequency with which similar cases have resulted in sentences of death. The second part is precedent-seeking review, a traditional judicial way of comparing the files in similar cases to determine whether a defendant’s death sentence is freakish or aberrational or the result of impermissible influences.

Until recently, similar cases were identified for purposes of frequency analysis by: (1) their salient factors (for example, cases involving prior murders or a sexual assault); (2) the raw numbers of statutory aggravating and mitigating factors (aggravating factors are those that make a murder death-eligible, such as the murder of a public official or murder in the course of committing a felony, and mitigating factors are those that may be weighed by a jury in determining whether to impose a sentence of death, such as extreme mental disturbance); and (3) an index of outcomes, a composite statistical test incorporating various statutory and non-statutory factors (such as motive or extent of premeditation) that sought to rank cases by the presence or absence of factors that appear to influence prosecutorial and jury decision-making.

These three statistical methods were applied to those cases that were clearly death-eligible, including cases in which the State had not sought the death penalty or defendants had obtained non-capital pleas. We refer to this as the universe of similar cases for purposes of comparison. The Administrative Office of the Courts (AOC) collects and organizes the data. Thus, each proportionality review involves an examination of prior cases through frequency analysis and precedent-seeking review. In Loftin II, supra, we found that because the raw numbers failed to account for the qualitative nature of aggravating and mitigating factors, the numerical preponderance test had not contributed to the Court’s proportionality reviews and in light of its inherent flaws could not be expected to do so in the future. 157 N.J. at 295, 724 A.2d 129. We determined that test should be abandoned. Ibid.

In Loftin II we also expressed concern that the statistical methods used in the index-of-outcomes test to predict the probability of any defendant receiving a sentence may lack sufficient reliability. Id. at 295-96, 724 A.2d 129. We also examined the data in support of a contention that there had been “impermissible discrimination in imposing the death penalty.” Loftin II, supra, 157 N.J. at 275, 724 A.2d 129 (quoting Ramseur, supra, 106 N.J. at 327, 524 A.2d 188). We referred to this as an inquiry into systemic proportionality review as opposed to individual proportionality review. In connection with that inquiry, we appointed retired Appellate Division Judge Richard S. Cohen to conduct a review and make findings and recommendations concerning whether a defendant’s race or the race of the victim possibly affected prosecutorial decisions to seek, and jury decisions to impose, the death penalty. Following the receipt of Special Master Cohen’s report and supplemental report, we concluded that the data did not demonstrate racial disparity in the imposition of the death penalty. Nevertheless, because many questions had been raised by the parties, by Special Master Cohen, and by the AOC, about our present systems of both individual and systemic proportionality review, we decided to remand the contested issues to a Special Master appointed to hear and take testimony and to report to the Court on the efficacy of the system.

Finally, in Loftin II we considered, but did not decide, whether N.J.S.A. 2C:ll-3e would impermissibly infringe on our exercise of appellate review. We summed up our review of the system of proportionality review as follows:

Our experience teaches us that the proportionality review methodologies we use are not without substantial shortcomings and, accordingly, warrant careful reconsideration. Our goal is to retain those elements of the present system that provide useful information, to refine and improve that which we retain, if appropriate, and to reject methods that have proved unhelpful. We seek a practical approach that ensures every defendant before us a rigorous and complete review of his or her sentence of death.

Our reconsideration extends to four discrete areas of concern: the size of the universe of comparison cases; particular issues in respect of individual proportionality review; questions relating to the statistical models used in both individual and systemic proportionality review, and the status of proportionality review as a separate proceeding in death penalty appeals____ Because these issues, with one exception, cannot be resolved on the record before us, we are appointing a Special Master to conduct additional fact-finding and make recommendations to the Court. On our receipt of his report, we will be in a position to determine whether the statutory limitation on the proportionality review universe prevents meaningful appellate review.

[157 N.J. at 286-87, 724 A.2d 129 (footnote omitted).]

On February 1, 1999, this Court appointed Superior Court Judge David S. Baime, a Presiding Judge in the Appellate Division, as Special Master to evaluate the Court’s proportionality review methodology, which was modeled after a proposal by the first Special Master, see David C. Baldus, Death Penalty Proportionality Review Project: Final Report to the New Jersey Supreme Court (Sept. 24,1991), and first used in Marshall II, supra. See Loftin II, supra, 157 N.J. at 453-57, 724 A.2d 129. Generally, the Special Master was assigned to conduct a review, perform analyses, and make findings and recommendations relating to the discrete areas of concern that we had identified. Id. at 454-55, 724 A.2d 129.

Specifically, the Court ordered the following:

(1) The Special Master shall conduct additional fact-finding concerning the proper scope of the proportionality review universe. The Special Master shall make an independent evaluation of the deathworthiness of a sample of cases previously classified by the Administrative Office of the Courts (AOC) as either death-eligible or death-ineligible. The “provability” of the selected cases and the presence or absence of aggravating and mitigating factors shall be considered and the results compared to the data-coding decisions made by the AOC. If there is a variance between the survey results and the AOC data-coding decisions, possible causes of the variance shall be identified along with recommendations for improved data-coding procedures. The Special Master shall consider whether a questionnaire should be filled out by the judge in each ease and used to improve both the data-collection and data-coding process. Alternatively, if the Special Master determines that the intrinsic difficulties and ambiguities of data-coding death-eligible cases cannot be overcome, the Special Master shall consider the impact of anticipated coding errors on the AOC models;

(2) The Special Master shall review data-coding generally and make recommendations for improvements if appropriate;

(3) The Special Master shall attempt to determine, based on projections about the size of the database over time and other relevant considerations, how long it will take before frequency review results can attain a level of statistical reliability;

(4) The Special Master shall undertake a review of both the strengths and weaknesses of the index-of-outcomes test and make recommendations whether the statistical models can be modified and improved or whether the index-of-outcomes test should be eliminated;

(5) The Special Master shall consider methods by which to select a representative number of cases within the group of similar cases for consideration and comparison to the defendant’s case in the salient-factors test and precedent seeking review. The Special Master shall examine alternate case sorting approaches that account for mitigating factors. The Special Master shall assess whether some reduction in the number of case classifications is possible without compromising the principle that only similar cases be compared;

(6) The Special Master shall attempt to develop parsimonious statistical models for more reliable regression studies of race effect and shall consider whether the process of purging, ie., the removal of the indirect effects of race from variables that appear to be unrelated to race, produces results that are useful;

(7) The Special Master shall consider Special Master Cohen’s recommendation, submitted in State v. Loftin, supra [Loftin II], that the Court appoint a panel of judges to perform periodic assessments of penalty-trial outcomes, along with the composition and mandate of such an independent judicial panel, as independent verification of the culpability ratings derived from the models;

(8) The Special Master shall develop a factual record and issue findings concerning the desirability of maintaining proportionality review as a separate proceeding or, alternatively, conducting proportionality review in connection with a capital defendant’s direct appeal____

[Id. at 455-56, 724 A.2d 129.]

Pursuant to that order, Judge Baime interviewed all personnel of the AOC responsible for screening cases, and gathered all protocols pertaining to AOC procedures. See David S. Baime, Report to the New Jersey Supreme Court: Proportionality Review Project at 18 (Apr. 28,1999) (Baime Report). He conducted an independent screening of 105 AOC files for death-eligibility and enlisted retired Appellate Division Judge Charles Villanueva to screen an additional 105 files. See ibid. He asked the Attorney General and the New Jersey County Prosecutors’ Association to present evidence concerning the extent to which prosecutors decided not to seek the death penalty, and specifically sought information from prosecutors regarding cases that were not capitally prosecuted but that Judge Baime found to be “clearly death-eligible.” See ibid.

With respect to data coding, Judge Baime directed two law clerks to simulate the process in accordance with AOC protocols for twenty-four case files each and to compare their results with the AOC’s. See id. at 42. With the aid of John P. McCarthy, Jr., Esq., Director of the Office of Trial Court Services, and Joseph J. Barraco, Esq., Assistant Director of Criminal Practice, Judge Baime considered all 433 death-eligible homicides in an attempt to discern meaningful case characteristics and the effect of mitigating factors on deathworthiness. See id. at 52-53. Further, Judge Baime retained statisticians Dr. David Weisburd and Dr. Joseph Naus as consultants to study the performance of the index-of-outeomes test. See id. at 4.

The Special Master’s report recommended the following:

(1) retention of the dearly death-eligible universe of cases for proportionality review with the following improvements in the AOC’s methodology;

(a) appointment of a retired judge to serve as Standing Master to supervise AOC screening, data collecting and data-coding functions, and to preside over hearings pertaining to these subjects;

(b) adoption of a protocol or rule mandating that all hearings before the Standing Master be kept confidential, that all information divulged by counsel during such proceedings may not be used for any purpose other than proportionality review, and that all transcripts and records of such proceedings be sealed;

(c) adoption of a protocol or rule, requiring that all criminal division managers forward to the AOC additional sources of information for data-coding and screening, including the judgment of conviction, presentenee report, notice of aggravating factor(s), any order resulting from a motion to dismiss, decisions pertaining to evidentiary questions, the requisite plea form, indictment or accusation, all defendant statements provided in discovery, all witness statements provided in discovery, all investigative reports from any law enforcement agency, defense notice of mitigating factors, autopsy and medical examiner reports, psychiatric evaluations and psychological reports;

(d) adoption of a protocol on an experimental or pilot program basis requiring trial judges to complete questionnaires in all potentially death-eligible homicide cases;

(e) modification of the AOC’s evidence typology to require that evidence of capital murder elements must be “overwhelming” as a prerequisite for inclusion of a case in the death-eligible universe;

(2) adoption of a protocol requiring “double coding” and editing by individual members of the AOC staff, memorialization of all interpretive rules respecting key variables, and periodic updating of the data base to reflect changes in data-coding rules;

(3) adoption of a modified salient factors test that accounts for mitigation and contains fewer categories;

(4) abandonment of the index-of-outcomes test and logistic multiple-regression analysis for individual proportionality review because of the instability of current models and the relatively limited projected increase in New Jersey’s data base;

(5) continued experimentation toward the creation of a reliable statistical model for the purpose of systemic proportionality review;

(6) adoption of a model charge reminding jurors of their duty to consider each case fairly and without regal'd to race, religion, national origin or gender; and

(7) continuation of bifurcated proceedings until the proportionality review process is streamlined and the Court can assess the feasibility of consolidation.

[Baime Report, supra, at 6-7.]

On May 18, 1999, the Attorney General, the Public Defender and amici curiae, the Association of Criminal Defense Lawyers of New Jersey (ACDL) and the New Jersey State Conference of NAACP Branches (NAACP), submitted briefs expressing their views on the Baime Report. (To distinguish this report from prior reports we refer to the Baime Report or Baime I, in anticipation of its second part.) Each of the parties also filed a response brief on May 26, 1999. We have now reviewed Judge Baime’s Report concerning the statistical models used in individual proportionality review. As noted, we have not. yet received Judge Baime’s report concerning the statistical models used to examine systemic proportionality review. Although the size of the universe, and the effect of N.J.S.A. 2C:11-3e, “cannot be considered apart from those other questions[,]” Loftin II, supra, 157 N.J. at 285, 724 A.2d 129, rather than await the second half of the Baime Report, we proceed on a step-by-step basis to determine “those elements of the present system that provide useful information, to refine and improve that which we retain, if appropriate, and to reject methods that have proved unhelpful.” Id. at 286, 724 A.2d 129. Specifically, we will determine whether the “practical difficulties attendant to data collection and analysis of noncapital cases,” id. at 289, 724 A.2d 129, would counsel that we accord comity to the expressed preference of the Legislature and the Attorney General for a smaller universe consisting of either the group of death-sentenced cases or, in the alternative, cases in which a penalty trial has occurred or a prosecutor has served notice of capital aggravating factors. Ultimately, we must decide whether the statute would impermissibly infringe on the exercise of the judicial function, which is reserved exclusively to the Court. Id. at 264, 724 A.2d 129. For now, in the discharge of that function we shall continue to use the larger universe to conduct proportionality review. We do not anticipate that the revisions of the system now or in the future will call into question any previously-conducted proportionality reviews.

For convenience, we will follow generally the format of Judge Baime’s report. It is our intention in this opinion to express our conceptual disposition of the major topics covered in Baime I. It is not our intention, nor is it within our capacity, to perform the actual computer programming that is necessary to store and sort the data. We have always believed that proportionality review need not be a mysterious exercise. The computer is used to compile, sort, and electronically store a list of case characteristics as we would have done a generation ago on a legal pad or on index cards. Once the cases are sorted, we hope to see how often death is the penalty for similarly-sorted eases. ‘We seek a practical approach that ensures every defendant before us a rigorous and complete review of his or her sentence of death.” Id. at 286, 724 A.2d 129.

The initial proportionality review report submitted to the Court in Marshall II was largely in narrative form and, including its brief descriptions of comparable cases, did not exceed eighty pages: forty pages dedicated to frequency analysis and forty pages dedicated to precedent-seeking review. Over time, proportionality review reports have taken on an arcane style vexing to judges and lawyers untrained in statistical analysis. The Chew/Harvey/Cooper Report, largely tabular in form, contains approximately 800 pages consisting of proportionality review tables and appendices.

One of Judge Baime’s recommendations is the appointment of a retired Superior Court judge to serve as a Standing Master to assist in proportionality review. We expect that such a judge, experienced in criminal matters, would be able to assist us in achieving a practical approach, while ensuring a rigorous and complete review of each sentence. If this opinion provides insufficient guidance to implement the revised system of proportionality review, we invite further inquiry. In the meantime, Judge Baime will assist the Court further during this transitional phase. He will continue as the Court’s Special Master while developing and implementing the system that will be used by a Standing Master who will be appointed by the Court at a future date.

I

THE UNIVERSE OF CASES

As a matter of abstract logic, we agree with Judge Baime that a universe limited to cases in which the death-penalty sentence has been imposed cannot support a coherent proportionality system. This is so because “[wjithout knowledge of the life-sentenced cases, [a court] would be unable to determine whether there is a ‘meaningful basis’ for distinguishing the death sentences it reviews from the ‘many eases’ in which lesser sentences are imposed.” Baime Report, supra, at 10 (citation omitted).

What Judge Baime has identified for us are the limits of logic. He has observed that the validity of the inclusion of death-eligible cases in the universe depends substantially on “questions of feasibility.” Id. at 16. We are satisfied, as was Judge Baime, that the studies that he conducted with the assistance of retired Judge Villanueva, law clerks and AOC staff,

clearly indicate that reasonably consistent and accurate screening decisions can be made based upon the presentenee reports and judgments of conviction. Indeed, many of these decisions can fairly be characterized as mechanical____ Of the 2104 cases that have been screened since the beginning of the proportionality review process, only 433 homicides have been classified as clearly death-eligible, approximately twenty-one percent.

[Id. at 28.]

If past practice remains constant, approximately thirty to thirty-five cases would have to be classified for determination each year. Nonetheless, because Judge Baime recognized that the database is only as good as the data that goes in, he recommended a series of steps to improve the AOC’s methodology for classifying cases.

We specifically approve of recommendations (l)(a), (b), (c), and (d), calling respectively for: the appointment of a retired judge as a Standing Master to supervise the AOC’s screening, data collecting, and data coding function, and to preside over confidential hearings pertaining to these subjects; a requirement that Criminal Case Managers forward to the AOC additional available sources of information; and a pilot program under which trial judges would complete questionnaires in potentially death-eligible homicide cases.

A primary objective of those recommendations is to assist this Court in conducting precedent-seeking review, particularly with respect to those homicide defendants whose ultimate sentences were imposed as a result of a guilty plea or a conviction following a non-penalty phase trial. As we observed in State v. Cooper, 159 N.J. 55, 97, 731 A.2d 1000 (1999):

The difficulty inherent in the process of precedent-seeking review is exacerbated when the Court attempts to compare defendant’s ease to the cases of other defendants whose ultimate sentences were imposed as a result of a guilty plea or a conviction following a non-penalty phase trial. In some cases, the AOC’s summary is sufficiently detailed to permit the Court to deduce by inference what considerations may have persuaded the prosecutor to forego a penalty trial. In other cases, the reasons why the prosecutor elected to forego a capital prosecution are less apparent. The lack of a contemporaneous and reliable summary by the prosecutors of the various factors that were considered in arriving at the decision to forego capital prosecution diminishes the effectiveness and reliability of our precedent-seeking review.

We are certain, with the reservations noted hereafter, that judicial questionnaires will provide a reliable measure, at least to a detached judicial observer, of the considerations that may have led to non-capital disposition of murder charges that were on their face death-eligible (as, for example, in the case of murder in the course of sexual assault).

Concerning judicial questionnaires, we ask that they be reviewed by the Trial Judges’ Committee on Capital Causes to determine both their usefulness and their potential to encroach on judicial autonomy and resources. Although both judges and the Standing Master would have the power to compel the State and a defendant to supply discoverable materials helpful to the process of classifying and coding cases, we will not require trial judges or the parties to express their personal views about the weight of the evidence involved, nor will we ask trial judges to require prosecutors or defense attorneys to do so. We acknowledge that cases at a trial level are far from over. Prosecutors and defense attorneys would be justifiably hesitant to furnish subjective evaluations of the weight of the evidence or the reliability of witnesses for either side. Once disclosed, that information could become part of future proceedings. Given the reasonably reliable results of the existing coding process, we believe that with the additional provision of all discoverable materials and a suitable questionnaire from trial judges, the Standing Master will be able to resolve remaining disputes over which cases are clearly death-eligible. There is no need to revisit ease summaries prepared under the existing protocols. Specific ease summaries have been challenged by the parties from time to time and the net product is sufficiently reliable to move forward.

We do not adopt recommendation (l)(e), which would require, as a prerequisite for inclusion of a case in the death-eligible universe, that evidence of capital-murder elements be overwhelming. Under that rubric, the universe might not contain some “clearly death-eligible homicides.” Marshall II, supra, 130 N.J. at 135, 613 A.2d 1059. We leave to the Standing Master and the parties the further resolution of this issue. We will keep this docket open to resolve any remaining disputes. We trust that we may continue to call on the expertise of Judge Baime.

We adopt Judge Baime’s recommendation 2 for administrative changes related to the improvement of the data-collection process by introducing a system of “double coding,” whereby “[t]wo AOC staff members ... independently code all of the cases, and then discuss and resolve their differences.” Baime I at 48. We also adopt the recommendation that the AOC draft and maintain specific protocols for data-coding. Ibid. We agree that the database should be updated periodically to keep the information “current and accurate.” Id. at 49.

In connection with that work, we ask the AOC to consider, in consultation with Judge Baime, the argument of the Attorney General that it is inconsistent that capitally-charged eases not resulting in a capital indictment by the grand jury are included for proportionality review, whereas eases rejected by the trial jury are not. Both petit and grand juries are the ultimate arbiters of what constitutes a death-eligible case.

II

MODIFICATION OF THE SALIENT-FACTORS TEST

A.

The Organization of Cases by Categories

Judge Baime makes several recommendations for improvement to the salient-factors test. With the exception of a recommendation concerning the incorporation of mitigating factors, we adopt his recommendations. He finds that the organization of the eases by statutory aggravating factors makes good sense, but recommends generally that the subcategories be dissolved. Baime Report, supra, at 56. After reviewing 433 death-eligible cases, Judge Baime, John McCarthy and Joseph Barraco found that the subcategories have little relevance as predictors of sentencing outcomes. Id. at 56-57. By way of example, Judge Baime notes that the prior murder conviction category contains three subcategories based on the number of aggravating factors present. Id. at 57. He found, however, that death-sentencing rates in cases with two or more aggravating factors were lower than for those cases with one aggravating factor. Ibid. This finding and others made him conclude that the “subcategories did not have any effect on deathworthiness.” Ibid.

Judge Baime recommends retention of a few discrete subeatego-ries. First, he would leave the subcategory denominated as “with particular violence or terror” in the sexual assault classification because defendants so classified seem to be viewed by prosecutors and juries as particularly deathworthy. Id. at 58. He recommends that strict guidelines be developed to avoid the inherent subjectivity in defining this subcategory, and specifically suggests that the subcategory include multiple stabbings, gunshot wounds and mutilations, as well as cases involving children. Id. at 58-59.

Judge Baime recommends that the robbery category be subdivided into “residential, business and other,” eliminating the numerous categories that currently exist. Id. at 59. The creation of the three categories is presumably an attempt to reduce the large number of eases in the robbery-murder category for purposes of review.

Finally, Judge Baime would retain subcategories in the multiple victims category. Id. at 59. He believes that because many of those cases involve intrafamily and rage killings, and prosecutors have not ordinarily capitally prosecuted such eases, those cases should be distinguished. Ibid. In addition, he notes that cases involving drug transactions between the victim and the defendant have “rarely resulted in capital prosecutions and death sentences.” Ibid. With the exception of those instances, Judge Baime observes that defendants who kill multiple victims in the course of the commission of another crime are viewed as particularly death-worthy. Id. at 59-60. He therefore recommends breaking the category into two subcategories — aggravated and non-aggravated cases. Id. at 60. He would denote multiple killings in the course of other felonies as “aggravated,” excluding multiple killings in the course of drug crimes and intrafamily and rage killings. Id. at 59-60.

B.

The Principle of Unique Assignment

Judge Baime recommends that the Court retain this principle. Ibid. Briefly stated, the principle is that even though a case may contain multiple identifying factors, e.g., killing a public official and robbing or torturing the official, the case is assigned to one category for salient-factor review. Judge Baime is concerned with emphasizing the quality of aggravating factors rather than the quantity, recognizing that one factor may be decisive in a jury’s decision to sentence a defendant to death. Id. at 61 (noting particular strength of factors like public office of victim, prior murder convictions of defendant, multiple victims, etc.). Judge Baime acknowledges, however, that unique assignment is not ideal. Ibid. The Public Defender states that the concept of giving cases a unique assignment is “a sensible system that seems to reflect reality to a great degree,” but has expressed concern over Judge Baime’s application of unique assignment to the process of ranking the salient-factor categories. The Public Defender notes that under the proposed system, once a ease is used to calculate the sentencing rate for one category (e.g. killing a public servant), the ease is unavailable for calculating the sentencing rate for a lower category found in the same case (e.g. torturing the victim). The Public Defender suggests ranking the categories “with replacement,” by which each case would be available for purposes of calculating the death-sentencing rate in each category to which the case applies. In this way, the hierarchy would be created using the replacement method, rather than unique assignment, but for comparison purposes a case would remain uniquely assigned to one category. As we understand the issue, we find no intrinsic problem with the use of the replacement method to create the hierarchy and the unique assignment method for running the salient-factors test for purposes of producing the AOC reports. We decline to order that relief now. Following the implementation of the new protocols, and after the first series of capital case reviews,the Public Defender may present the issue again to Judge Baime or his successor as Standing Master for possible reconsideration.

C.

The Role of Mitigating Factors in the Analysis

Third, Judge Baime recommends abandoning Professor Bal-dus’s hierarchical structure of assigning eases according to the level of aggravation. Id. at 62. Judge Baime notes that it has become clear that the current structure does not accurately represent the hierarchy of death-sentencing frequencies. He proposes an alternative structure, ranking salient factors in descending order based upon the death-sentencing rates among all death-eligible defendants in a category. Id. at 63. Again, he urges a flexible approach with room for exceptions. Id. at 63-64.

Judge Baime proposes that mitigating factors be introduced into the salient-factors test. Id. at 64-65. Acknowledging the difficulty in accurately representing the mitigating factors given that a unanimous jury need not find a factor in order for it to be present, as well as the fact that juries weigh mitigating factors differently, he nevertheless recommends that the salient-factors category be divided into “low” mitigation and “high” mitigation subcategories. Ibid. As originally contemplated, the salient-factors test was expected to include mitigation in that analysis. Marshall II, swpra, 130 N.J. at 146, 613 A.2d 1059.

In order to channel the salient-factors analysis into a simpler reading of the data, we reject for now the incorporation of mitigating factors into the analysis. Judge Baime has recommended maintaining the principle of unique assignment because “the number of aggravating factors is not particularly relevant in assessing deathworthiness____” Baime Report, supra, at 60. The same may be true for mitigating factors as predictors of the lack of culpability. Precedent-seeking review is better suited to handle the presence of mitigating factors.

D.

Use of the SalienP-Factors Test in Selection of Comparison Cases for PrecedenP-Seeking Review

In attempting to reduce the number of cases the Court is obliged to examine in precedent-seeking review, Judge Baime recommends that the Court leave that responsibility to the parties, under the supervision of a Standing Master. Id. at 74. Because the parties are in the best position to make arguments for including similar cases in the Court’s review, we accept this approach, and note that it can serve to correct potential errors made in the salient-factors analysis. If some of the categories are split into subcategories, it is important that the Court remain open to recommendations from both parties concerning eases that may have been erroneously excluded from or included in the reviewed category. Judge Baime recommends that the Standing Master consider the arguments advanced by the parties and provide the Court with a recommendation. Ibid. The Court will make the ultimate decision concerning which cases are to be considered for precedent-seeking review.

Ill

INDEX-OF-OUTCOMES TEST

Judge Baime recommends abandoning the index-of-outcomes test due to the instability of the regression models.¹ Baime I, supra, at 77. He notes that the main problem is the lack of parsimony in the models, which means that the computer models are using the data from a relatively small number of cases to explain the effect of too many different factors on the likelihood of receiving the death penalty. While finding that the increase in the number of cases over time has had a positive effect on the stability of the models, “this fact does not itself mean that the models are now stable, but rather that they have met a very minimal requirement of stability.” Id. at 90.

In examining the size of the coefficients produced in schedules over time, Judge Baime found some settling of the factors, but cautions that “the coefficients are still large overall for logistic regression analyses and tend to suggest substantial model instability.” ² Id. at 92. In addition, Judge Baime notes that although an examination of the culpability estimates for each defendant over time shows some overall consistency, the levels for certain defendants have undergone substantial changes with the addition of new cases each year. Id. at 93-94.

Judge Baime identifies the following possible causes of instability in the models: (1) if an important predictor of a death sentence is left out of the model, the culpability predictions will likely be unreliable and will challenge the validity of the individual case rankings for proportionality review; (2) if an excluded variable is related to a factor in the model, the measurement of the included variable would be biased by the exclusion of the independent variable; and (3) there remains a persistent lack of parsimony in the models. Id. at 96-101. The problem with the index-of-outcomes test is that there are too many independent variables (degree of victimization, extent of premeditation, nature of offense) in relationship to the relatively few dependent variables (death verdicts) to reach a reliable conclusion about the effect of the independent variables. This in turn has led to unreliability in ranking the cases in terms of overall culpability or deathworthiness. In the end, Judge Baime concludes that any attempts to overcome these problems will not produce meaningful results.

For an analogy, return to the example of the effect of gender on pay scales for workers described in Loftin II, where we said that regression analysis

is used in employment discrimination cases when it is claimed that a class of employees has been denied promotion or accorded differential compensation based on factors such as race or gender. Salary would be the dependent variable to be explained whereas gender and race would be the explanatory or independent variables. EEOC v. Sears Roebuck & Co., 839 F.2d 302, 325 (7th Cir.1988).

[157 N.J. at 295 n. 8, 724 A.2d 129.]

A study might produce results that a statistician would find reliable if there were a manageable number of independent variables such as age, education, or experience on the job, in relationship to the dependent variable of salary. But if an expanded number of independent variables were added to the mix, such as socio-economic status, membership in clubs, status of marriage partner, or manner of dress, depending on their number, their effect on the dependent variable of salary might not be measurable. In Marshall II, we were perhaps too sanguine in assessing the prospects that the index-of-outcomes test could produce reliable information. Its early measures of deathworthiness conformed to our intuition and experience in that the persons whom we perceived to be highly deathworthy remained so on the index. But there remain obvious problems with the test, and there is no advocate for the test in its present form.

Statistical modeling certainly will be needed to examine systemic disproportionality. We must remember, however, that when Professor Baldus created the index-of-outcomes test, he had not been asked to undertake an analysis of discrimination in New Jersey’s capital punishment system. Baime I at 78-79. Although “[m]uch of the conceptual framework of proportionality review devised by Professor Baldus in 1991 had, and continues to have, great efficacy,” id. at 107, the Court’s consultants have observed that the present schedules do suggest “that some specific variables have some impact (of a size difficult to determine),” Weisburd & Naus, Report: Assessment of the Index of Outcomes Approach for Use in Proportionality Review, Apr. 1999, at 24 (Appendix A, Baime I), which may indeed indicate model instability.

Judge Baime recommends that his consultants continue to attempt to create more reliable models to achieve the goals of systemic proportionality review. He will work with Professors Naus and Weisburd, and plans to report the results by October 1999. Baime I, supra, at 108 n. 13.

Experience gleaned from studies elsewhere may refine the statistical methods that enable one reasonably to conclude whether race is or is not an impermissible factor influencing capital-sentencing decisions. See Kent S. Miller & Michael L. Radelet, Executing the Mentally III: The Criminal Justice System and the Case of Alvin Ford, 128-29 (1993) (noting 1990 General Accounting Office study showing “that, other things being equal, those who murdered whites were more likely to be sentenced to death than those who murdered blacks” and that over “half of the studies reviewed found that race of the defendant also influenced the likelihood of ... receiving the death penalty”); David C. Baldus, George Woodworth, David Zuckerman, Neil Alan Weiner & Barbara Broffitt, Racial Discrimination and the Death Penalty in the Post-Furman Era: An Empirical and Legal Overview, With Recent Findings From Philadelphia, 83 Cornell L.Rev. 1638 (1998) (describing studies of racial discrimination in Philadelphia capital-death sentencing).

In the meantime, Judge Baime recommends that a model jury instruction be given that reminds jurors that they are not to consider race, color, religious beliefs, national origin, or sex of the victim, and that the jury is “not to return a sentence of death unless it has concluded that it would return the same verdict no matter what the race, color, religious beliefs, national origin or sex of the defendant or the victim might be.” Baime Report, supra, at 109 (citing 21 U.S.C.A. § 848(o)(l) (Supp.1998)). The Special Master recommends that the instruction be given only at the request of the defendant and only “in appropriate cases.” Ibid.

While cautioning that a true bigot would certainly ignore the charge, Judge Baime contends that it might well have an impact on some who “attach an irrational significance to race or another nongermane criteria that is outside their awareness.” Ibid. He has faith in a jury’s ability to pay close attention to instructions and to apply them diligently. Id. at 110 (citing State v. Breakiron, 210 N.J.Super. 442, 468-69, 510 A.2d 80 (App.Div.1986), (Baime, J.A.D., dissenting), rev’d in part, 108 N.J. 591, 532 A.2d 199 (1987)). We share that faith.

We recently approved a Model Criminal Charge instructing juries to consider, when relevant, the possible effect of race on a witness’s ability to make a correct identification. State v. Cromedy, 158 N.J. 112, 727 A.2d 457 (1999). Perhaps, when relevant, sexual orientation should be included in the proposed charge, in accordance with the Legislature’s hate crime statute. See N.J.S.A. 2C:44-3e. We leave the formulation of the charge to the Committee on Capital Causes. For now, trial courts should, when appropriate, give an instruction generally in the form suggested by Judge Baime.

TV

PROPORTIONALITY REVIEW AS A SEPARATE PROCEEDING

The last question to be addressed is whether proportionality review should continue to be conducted as a separate proceeding following a defendant’s direct appeal. We share the concern expressed in the July 1998 Report of the Governor’s Study Commission on the Implementation of the Death Penalty, that excessive delay in the prosecution of capital appeals “undermines the deterrent effect of capital punishment, promotes disrespect for the criminal justice system and prolongs the suffering of victims’ families.” Michael Booth, Death Penalty Panel Urges Limits on Trial and Appellate Remedies, 153 N.J.L.J. 241 (July 20, 1998). The task of the criminal justice system is to identify any sources of excessive delay in the system and to seek to ameliorate them.³ Because we had not yet developed a system for proportionality review at the time we decided Ramseur, supra, we stated that proportionality review would be conducted in a separate proceeding. It was also thought that bifurcated proceedings would “conserve resources because a proportionality review would not occur if the defendant’s direct appeal was successful.” Loftin II, supra, 157 N.J. at 316, 724 A.2d 129. Judge Baime has concluded:

That practice made sense in an era in which capital punishment jurisprudence was unsettled and the likelihood of a reversal great. Although death penalty cases remain extremely complex, many difficult issues have now been resolved and the original rationale for bifurcation is perhaps less compelling. Moreover, “this practice ... exacts a cost by drawing out the appeals process when a death sentence is affirmed.”

[Baime Report, supra, at 111 (quoting Loftin II, supra, 157 N.J. at 316, 724 A.2d 129) (footnote omitted).]

Judge Baime wrote:

I favor consolidation. Whatever the deterrent value of capital punishment — an issue upon which reasonable persons can and do differ — it is surely diminished with the passage of time caused by endless appeals. I believe that the process can be streamlined if my recommendations are accepted. As I noted earlier, much of the attention and time of the Comí and counsel have been devoted to the fruitless endeavor of attempting to find meaning in the results yielded by the index of outcomes test. Eliminating the index of outcomes test will streamline the system at virtually no cost to the parties. The rights of the defendant will be fully protected by relying on the salient factors test and precedent seeking review. On balance, I believe that consolidation will conserve more resources than it will cost in the relatively few cases that are reversed on direct appeal.

[Id. at 112-13.]

We are in basic agreement with his views. To our knowledge, other jurisdictions that conduct proportionality review do not conduct bifurcated proceedings. Consistent with our intention to “streamline the system” pursuant to the recommendations of Judge Baime, we direct that the next scheduled proportionality reviews shall be conducted in accordance with the revised format recommended herein.

The next stage is for the AOC, with the guidance of Judge Baime and, ultimately, the Standing Master to be appointed by the Court, to establish the procedures for individual proportionality review in accordance with the directives herein. That will require recalculation of the data and the production of revised reports. The Clerk of the Court will discuss with the parties briefing and argument schedules that reflect the new procedures. Systemic proportionality review will have to be conducted under the existing methods of analysis. In conjunction with those proceedings, we shall determine when and how to conduct consolidated proportionality review. Consolidation will demonstrate the principle that a responsible judicial system can be both just and efficient.

We recognize that consolidating proportionality review will be an added burden to the already burdened prosecution and defense teams, and we will work with them to integrate the new system for proportionality review into the direct appeal process. In connection with the next proportionality reviews conducted under the revised format, the parties should submit their views concerning procedures for implementing consolidation. We shall request that the parties confer with the Standing Master and the Clerk of the Court before submitting their views to us as part of those appeals.

V

To sum up, one of the participants has observed that conducting proportionality review is not like completing the Human Genome Project. We are identifying and then sorting, by very familiar characteristics, about thirty to thirty-five cases per year. We have permitted technical debate to obscure substantive meaning. Our task is to ensure that technical problems with issues such as confidence intervals, model convergence, and more (or less) parsimonious models, be translated into an understandable legal format that we and the parties can apply. We expect, with the help of Judge Baime and the Standing Master, AOC staff, the parties, and amici, that we can achieve this goal. It is to them and those who have assisted them, including our own clerks, that the Court expresses gratitude for their efforts to build on and improve the system of proportionality review recommended by Special Master David Baldus. All participants approached the task without effort at obstruction. Their cooperation in recognizing past problems when evident and pointing out potential future problems has been of great assistance to Judge Baime and to the Court. We all share one goal, “to ensure that the death penalty is being administered in a rational, non-arbitrary, and evenhanded manner, fairly and with reasonable consistency.” Marshall II, supra, 130 N.J. at 131, 613 A.2d 1059. We shall continue that pursuit.

1 As the Court explained in Loftin II, "Multiple regression analysis is a statistical tool used to describe the relationship between one or more independent variables (e.g., prior murder) and a dependent variable (e.g., the death penalty)." 157 N.J. at 295 n. 8, 724 A.2d 129. A statistical model attempts, in a numerical way, to express the effect of each of several independent variables on the dependent variable. (The computer converts the data being analyzed into an electronic format and performs the calculations that would otherwise be done manually.) To return to the example of the effect of gender on pay scales for workers described in Loftin II, supra, 157 N.J. at 295 n. 8, 724 A.2d 129, the model attempts to determine (without the presence of gender) the effect or "coefficient” of variables such as education or experience on pay scales. For example, if 1000 employees with MBAs earned an average of $130,000, the statistician estimates the effect that an MBA will have on a worker's ability to attain that range of pay. Once a coefficient has been determined for each independent variable, the resulting "model" can be used to make predictions in future cases.

2 A coefficient is a number describing the average change in the dependent variable when an independent variable increases by one and all other independent variables are held constant. If the coefficient for prior murder is large, then the model reveals that a small increase in the number of prior murders (two instead of one) correlates with a large increase in the likelihood of the death penalty. However, a poorly designed model may produce a coefficient that is larger than its true value (for example, a coefficient that indicates that the presence of the prior murders aggravating factor correlates with a chance of receiving the death penalty four hundred times that of someone with no prior murders). Coefficients that are too large and overstate the relationship between the variables thus indicate that there is a problem with the model.

3 For example, we agree with, and would not await legislative concurrence for, the acceleration of the preparation of transcripts in capital cases in order to enable a more expeditious scheduling of appeal-management conferences after the verdict. Real-time computer-assisted preparation of transcripts can help to reduce the course of delay. Obviously, added resources would assist in this effort.