Mark v. Lang

Seymour Bieber,

Spec. Bef. Pursuant to stipulation of counsel for the respective parties, this matter was referred to me, to hear and determine.

Petitioners, lieutenants of the Police Department of the City of New York, failed, by relatively small margins, Examination No. 9961 for promotion to captain. Upon institution of the subject article 78 CPLR proceeding, Special Term, citing the opinion of the Court of Appeals in Matter of Acosta v. Lang (13 N Y 2d 1079), found that there were “ apparent defects in the ratings ” of petitioners’ answers to Part II of said examination and, accordingly, directed the matter be set down for trial (Matter of Mark v. Lang, N. Y. L. J., July 14, 1966, p. 8, col. 4 [Sup. Ct., N. Y., Spec. Term, Lupiano, J.]).

Petitioners, in substance, allege that their answers to the six essay questions here involved are “ as good as or better ” than the illustrative key answers prepared by respondents Civil Service Commission and the Department of Personnel (hereinafter referred to as “ respondents ”). These essay questions, each of which will be separately considered hereinafter, posed various problems concerning an evaluation of proposals for a civilian review board, the factors to be considered if evidence obtained by an investigating officer is to be used in court, steps to be taken to maintain precinct morale by a newly appointed police captain with a prior reputation as a “ hard boiled ’ ’ lieutenant, a comparison of the present 20-squad system with a proposed plan for its modification, the course of action to be followed by a precinct captain in the event of a mass demonstration at the station house, and arguments for and against the fingerprinting of juveniles.

Petitioners’ answers to the six subject questions were graded by two examiners and, purs ant to rule 4.5.1 of the Rules of the New York City Civil Service Commission, the final rating given to each answer represented the average of their judgment. For example, if one examiner gave 10 points for an answer while his associate rated the answer at only 5 points, a petitioner’s final grade for the answer was fixed at 7.5 points. Thus, the pleadings in this proceeding set forth, at very great length and in very considerable detail, the opposing authorities and reasoning of the respective parties as to whether the items contained in petitioners’ answers to the essay questions, for which they received only partial or no credit, were as good as or better than the specific items selected by respondents as valid answers, entitled to full credit.

The credible testimony and documentary evidence adduced by petitioners upon the trial of this matter clearly and overwhelmingly support their contentions with respect to each of the essay questions. As more fully discussed hereinafter, it appears that the rating keys used by respondents’ examiners to grade the answers here involved omitted and failed to give credit for responses which were separate from and as significant as other items set forth in such keys. Prior to examining each of the six questions and the respective answers thereto, however, it is necessary first to consider the nature of the examination with which this litigation is concerned, and to note, in general, the potential hazards and defects regarding the preparation of an essay test, the method of grading it and, in particular, the specific limitations which surround the use of this type of question in competitive civil service testing.

Many of the difficulties inherent in an essay written test have long been recognized by those associated with public personnel examination. As a matter of fact, the documentary proof adduced before me shows that respondents’ principal witness, one Harry Reiner, Chief of the Law Enforcement and Social Services Examining Division of the Bureau of Examinations, New York City Department of Personnel, had coauthored a publication of the Public Personnel Association -which clearly delineated the problems and pitfalls attendant to the use of the essay test (Personnel Report No. 611, Public Personnel Association, “The Essay Test in Public Personnel Selection”.). Unfortunately, however, the record of the instant proceeding establishes that respondents and those who assisted in preparing and grading the subject examination either disregarded or failed to heed many of the guidelines set forth in this publication.

It is fundamental that the applicable laws of New York require civil service examinations, so far as practicable, to be objective as well as competitive (N. Y. Const., art. V, § 6; Civil Service Law, § 50). Thus, a civil service examination must employ an objective standard or measure which is capable of being challenged and reviewed by other examiners of equal ability and experience (Matter of Fink v. Finegan, 270 N. Y. 356, 361-363). To this end, the so-called “ rating key ”, generally, is employed to give a factual, accurate and objective grade with respect to an applicant’s qualifications for the position or promotion he seeks. In the instant matter, however, as more fully discussed hereinafter, the use of essay questions which called for “ evaluations ”, and “ arguments for and against ” certain proposals resulted in the establishment of rating keys which left room for subjective crediting, particularly in so-called “ gray areas ”. In other words, these questions gave rise to key answers which could not be classified as “black or white”, “yes or no”, “ right or wrong ”. There remained an area of discretion, a place for argument and opinion, which in turn, and of necessity, permitted the examiner and rater to inject his personal opinion and subjective motivation in grading the answers of petitioners. This defect was singularly evident with respect to the testimony concerning Question Five, the proposed civilian review board, still a debatable proposal at the time of this 1964 examination. As noted at Special Term, ‘ ‘ The questions all entailed the exercise of value judgments and the answers thereto are not susceptible to confinement within a rigid prearranged formula. * * * Moreover, many of the ratings appear to be based on verbal rather than substantive grounds ” (Lupiano, J., supra).

It is noteworthy that Bernard Berger, another coauthor of the afore-mentioned publication and, at the time of its publication, the Principal Personnel Examiner in the City’s Bureau of Examinations, states therein that the objective test has “ a decided weakness * * * in providing a full picture of the abilities and understanding of the candidate ”. On the other hand, however, Kenneth 0. Warner, Director of the Publishing-Personnel Association, notes in the foreword that “Psychometricians usually cast an uneasy eye at the reliability and sampling of the essay test; they shun the essay test for its more objective ally — the multiple-choice test ”. (Emphasis added.) Regardless of which position one advocates, however, respondents must bear in mind, that under controlling constitutional, statutory and case law, as above noted, they are obligated, despite any attendant and well-recognized difficulties, to assure the selection of public personnel by competitive and objective testing. The use of essay questions can in no way serve to alter or modify this legal requirement.

If the use of essay testing is to continue, the weaknesses inherent therein, fully analyzed in an article referred to on page 2 of the afore-mentioned publication (i.e., Powell, “ Improving Civil Service Essay Test ”), must be recognized and cured by respondents. As noted therein, no rating key to an essay question can possibly enumerate and evaluate all variations in responses. Thus, the varying grades of answers require high-level judgment and broad knowledge of the particular field in question for adequate appraisal by skilled and qualified examiners or raters. The examiner must be intimately familiar with and thoroughly know the exact subjects included in the test. Without in any way intending to deprecate the knowledge or ability of the raters involved in this proceeding, it is nevertheless obvious from the evidence before me that in many of the areas knowledge, skill and training required to evaluate properly the with which this test was concerned, they lacked the technical respective answers given by petitioners. For example, Question Six involved the use of documentary evidence in court. Many of the answers, as hereinafter noted, discussed technical legal problems of relevancy, materiality and admissibility, yet, significantly, none of the raters was an attorney or law school graduate who would thereby be presumed to be capable of carefully analyzing and weighing the propriety of the answers relating to technical matters of law. The mere fact that assistance may have been obtained from qualified attorneys or other legal experts in the preparation of the rating key for this question is not sufficient, inasmuch, as above indicated, the variations in responses not anticipated and, therefore, not set forth in the rating key would still necessitate a person qualified in the field of law for the proper evaluation thereof and its application to the credit structure.

Moreover, the absence of such specialized knowledge or skill by a rater undoubtedly leads to much of the subjectivity which occurs in the grading of an essay test. Unless and until the rater is able to employ actual technical knowledge and ability, essay questions will almost always fail to realize their potential as a reliable, objective testing device.

Furthermore, in order to make the essay examination a reliable testing device and its rating key an objective evaluation, as required by law, a consistent and equitable approach is needed to the construction as well as the use of the rating key. It is not sufficient to be satisfied with a rating key which, while permitting independent examiners to agree “ substantially ”, as in the instant matter, nevertheless, perpetuates the element of subjectivity in their grading. (See Powell, “ Personnel Administration in Government ”, [Prentice-Hall], pg. 249; Purst, “ Constructing Evaluation Instruments ”, p. 296 et seq.) In this proceeding, therefore, petitioners raised objection to respondents’ use of two raters in grading each answer. They contend that in every instance of a disagreement between the raters, which, under applicable Civil Service rule 4.5.1, then resulted in an averaging of their respective credit allotments, a candidate is entitled, as a matter of law, to the higher of the two ratings rather than the average mark of both grades. Inasmuch as the Rules of the Civil Service Commission, insofar as is here relevant, now have the force and effect of binding law, petitioners’ argument is without merit at this time.

However, when a two-rater system is used, as it is by the Department of Personnel of the City of New York, there are natural hazards present in the application of a rating key to essay questions of which, in view of the proof adduced before me, respondents’ examiners were not aware or failed to take into consideration. Thus, as noted in the Reiner publication, it is of paramount importance in the application of the key for both raters to understand the key answer and the rating scale fully before the start of actual grading.

“ Objectivity must necessarily be a by-product of such an approach ” (Personnel Report No. 611, supra, pp. 16-17). Documentary evidence indicating numerous instances of wide disagreement between the two raters here involved and the injection of subjective opinion in the afore-mentioned “ gray areas ” of answers clearly warrant the conclusion that respondents’ raters, in effect, had failed to agree on the key or its use, and the prerequisite conclusion that equivalent statements by the candidates would be fully accepted. The mere provision in the rating keys for credit to be given for “ other” answers is not sufficient, particularly inasmuch as (1) an unpredicted “ other” answer may be “ as good as or better ” than a fully credited key answer and, therefore, be entitled to more points than the key credits allow for a contemplated “ other ” answer; (2) the rater may not have the technical knowledge to recognize or to appreciate the true value of an unforeseen “ other ” answer; and (3) subjectivity rather than objectivity is the by-product of permissible credit under the so-called “ other ” category.

Another recurring problem area in the two-rater system concerns those answers to essay questions which, while not completely erroneous, contain irrelevant material, slight errors of fact or mere mistakes of judgment. Unless the Department of Personnel adopts a specific policy one way or the other, or unless both raters agree in advance, there is absolutely no guideline in this type of examination rating as to whether and to what extent a candidate should be penalized for such minor errors or irrelevancies which may tend only to reduce the total value of other correct and proper parts of his answer. Obviously, if one rater ignores such deficiencies while the second rater penalizes the candidate therefor, a distinct inequity will result. This fault is not entirely ameliorated by a rating key provision which allows ‘1 plus or minus ten points ’ ’. Such provision is merely another subjective device for additional credit or penalty. This particular problem, however, can be readily overcome by appropriate policy directives of the city examination agencies involved or by provision in the key for the specific assessment of penalties for irrelevant material or for minor errors of fact or judgment.

There are many other difficulties attendant to the use of the essay examination in the selection of public personnel which, while noted in the cited publication (supra) do not, in my opinion, warrant further discussion here. From the foregoing, however, it is evident that respondents must seriously consider whether present techniques employed by them in the construction and use of essay question rating keys permit the essay examination to be, under applicable law, a workable, effective, systematic and objective method of civil service testing.

Each of the six essay questions of Examination No. 9961, respondents’ rating keys, petitioners’ answers, their proposed rating keys and their requests for additional credits will now be considered:

Question No. 1:

The twenty-squad system is used at present to distribute patrolmen individually among the three shifts on a rotating basis. It has been proposed, however, to change the patrolmen’s duty chart so that five squads would perform the same tour together as a unit, while retaining the twenty-squad breakdown. Sergeants’ schedules would he similarly arranged so that the same sergeant would remain with each unit. Compare the present system with the proposed one, giving the relative advantages and disadvantages of each.

Respondents’ established rating key for this question provided for a maximum of 40 points for answers relating to the advantages of the present system and a maximum of 75 points for listed advantages of the proposed system. In addition thereto, the raters were permitted a leeway of plus or minus 10 points in consideration of the ‘ ‘ character, scope and level of discussion ’ The key answers were listed as follows:

A. ADVANTAGES OF PRESENT SYSTEM:

1. Flexibility ■ — • can more easily provide increased patrol at peak load periods.

2. Broadens scope of training and increases potential for growth since men are exposed to a variety of superiors (not restricted ito single supervisor who may be less capable).

3. If we assume -that proposed plan means that unit always remains on same tour, present system provides equitability in rotating hours of work.

4. Less likelihood of collusion among officers, superiors, and public.

B. ADVANTAGES OF PROPOSED SYSTEM:

1. Identification with constant work group — increased sense of responsibility; feeling of achievement; increased morale in general.

2. Fixing responsibility is facilitated.

3. Closer and more continuous supervision possible.

4. Greater recognition of training needs.

5. Meaningful comparisons between units.

6. If we assume that proposed plan means that unit always remains on same tour, proposed system more likely to provide continuous attention to police hazards.

7. Other (More meaningful personnel records; facilitates development of specialist skills, etc.)

The disadvantages of one system are basically the opposites of the advantages of the other system.

no credit :

1. Mentioning equitability without stating conditions in A3.

2. Mentioning better attention to police hazards without stating conditions in B6.

3. Mentioning better police service.

4. Present system already in operation.

Petitioners’ proposed rating key for Question No. 1 contains all but items A3 and B6 of respondents’ key, plus certain additional key answers (infra). Petitioners’ expert witness’ testimony conclusively established the propriety and necessity of excluding respondents’ key answers A3 and B6 inasmuch as it was clearly demonstrated that it would be impossible to create the subject duty chart based on a 40-hour week. The credible evidence also established that the following additional answers, not contained in respondents ’ key, warrant credit to a petitioner who has so responded to Question No. 1:

A. UNDER ADVANTAGES OF PRESENT SYSTEM :

1. Provides a broader training for Sergeants (Petitioners’ key 4A).

2. Superiors can better evaluate performance of subordinates because of a broader comparison base (Petitioners’ key 5A).

3. Commanding officer can more accurately evaluate Sergeants as they all supervise the same personnel (Petitioners’ key 6A).

4. Difficulty in designing the proposed chart under the guide lines set forth in the question so there would be no coverage gaps (Petitioners’ key 7A).

5. Police service not such that troups can always remain intact so that -a great deal of the proposed system benefits would go by the board (Petitioners’ key 8A).

While respondents have argued that Petitioners’ key 5A (subd. A2, supra) is untrue because of a purported “lack of depth ” in the present personnel duty chart, the testimony of Michael G. Birmingham, former Police Department Chief of Staff, a recognized expert with broad police and law-enforcement experience, clearly supports petitioners’ key answer in this respect. I find, therefore, that such answer warrants credit to a petitioner so responding.

B. UNDER ADVANTAGES OE PROPOSED SYSTEM :

1. More efficient evaluation of patrolmen performance possible because of closer relationship between superior and subordinate in work unit (Petitioners’ key B6).

2. Permit Sergeants to make more effective assignments (Petitioners’ key B8).

3. Simplify supervision and permit Sergeants to devote more time to other police duties (Petitioners’ key B9).

Contrary to respondents’ contention, petitioners’ key answer B6 (subd. B1, supra) is different than respondents’ key answer B-2, 3 and 4 (supra). Obviously, “ evaluation” and “ supervision ” are wholly different concepts, each of which is entitled to credit under a proper rating key. On the other hand, I find that petitioners’ suggested key answer in their Exhibit No. 11 (Petitioners’ key B7), to the effect that the proposed system would “ permit the assignment of patrolmen to steady post or assignment such as Radio Motor Patrol car”, is not a valid point, inasmuch as the evidence establishes that this fact is the same in both the present and proposed system. No credit, therefore, has been allowed for such answer.

Significantly, both the respondents’ key (supra) and petitioners’ proposed key to Question No. 1 contain the same maximum points with regard to parts A and B, namely, 40 and 75 points, respectively. While such credit structure has been shown to be proper as a whole, I find that petitioners’ key answer A8 is not worth the allotted 10 points but has a maximum value of 5 points only. Similarly, as noted above, petitioners’ key B7 is disallowed as a valid answer and, therefore, the key’s 10 points for such answer is eliminated from the credit structure of answers to Question No. 1.

Giving due consideration to the foregoing, petitioners’ claims for additional credits with respect to their answers to Question No. 1 are disposed of as follows: [Charts of additional credits not printed].

Question No. 2:

A newly appointed captain, who had built up a reputation as a ‘ hard boiled ’ lieutenant, has been placed in command of a precinct with which he has had little previous contact. His superior, when assigning him, tells him, ‘ You have a good precinct and morale there is high. By maintaining the morale of your men at this level, you will have no disciplinary problems.’ What steps can the captain take in order to keep up the morale of the rank-and-file officers under his command?

Respondents’ established rating key for this question permitted a candidate to accumulate a total of 130 points. Similarly to Question No. 1, the raters, in effect, were allowed a subjective leeway of “ plus or minus ten points ”, based on the “ character, scope and level of discussion”. Of the aforesaid 130 points, 10 were allotted to responses concerning familiarity with personal or work problems (key answers A), 60 to “ staff related ” answers (key answer B) and 60 to “ job related ” answers (key answer C), to wit:

respondents’ key to question 2:

A. Become informed as to any existing precinct problems and past actions with respect to them (personal or work problems).

B. Staff Related:

1. Foster good supervision (by self and by superiors under your command).

2. Bet good personal example.

3. Bhow sincere interest in their welfare.

4. Discipline where necessary (require adherence to high standards).

5. Use discipline as a corrective measure, not as a punitive tool.

6. Recognize good work performance.

a. Reward, commend, special assignment.

b. Public recognition (rolleall, bulletin board, etc.).

7. Have an open-door policy.

8. Sponsor social affairs for members of command.

9. Other.

C. Job Related:

1. Inform men of what is expected of them.

a. Responsibilities and quality of performance needed.

b. Clear instructions (concise, understandable).

e. To whom accountable.

2. Encourage staff participation in solving problems, planning work and procedures.

a. Stimulate through conferences.

b. Promptly consider suggestions and justify rejections.

3. Explain and justify changes in policy and operations (avoid confusion or misinterpretation of new or changed orders).

4. Attempt to provide the best possible tools of the trade: communications, automotive, weapons and to keep them in the finest condition possible.

5. Encourage professional development (PA courses, college, professional society).

6. Other.

It is to be noted that pursuant to respondents ’ key, a candidate, under subdivision B thereof, would be required to give each and every key answer in order to obtain a perfect score. The credible evidence adduced upon the trial, however, indicates that this far exceeds normal requirements and customary methods of grading civil service examinations.

In addition to accepting respondents’ key answers, testimony of competent and expert witnesses warrants the conclusion that the following answers, set forth in petitioners ’ suggested key to Question No. 2 and adopted herein, are entitled to credit:

1. Analyze reasons for reputation as “ hard boiled ” and take steps to correct image (Petitioners’ key B).

2. Be fair and impartial; avoid favoritism; guard against personal bias (Petitioners’ key C8).

3. Set up machinery for ascertaining the existence of grievances (gripes) and for properly handling them (Petitioners’ key C9).

4. Avoid promises or representations that cannot be kept (Petitioners’ key CIO).

5. Back up subordinates when they are right [don’t “ pass the buck ”; “ stick neck out ” where necessary] (Petitioners’ key Cll).

6. Endeavor to assign subordinates to jobs for which they are best qualified and have an interest or liking (Petitioners’ key C12).

7. Whenever possible, reprimand or criticize subordinates privately (Petitioners’ key 013).

8. Avoid hasty or snap decisions or judgments (Petitioners’ key 014).

9. Where possible, keep subordinates informed concerning matters in which they are interested (Petitioners’ key 015).

10. Follow the principles of chain of command (Petitioners’ key D5).

11. Make assignments of subordinates reasonable and capable of good performance (Petitioners’ key D6).

12. Give subordinates sufficient authority to accomplish their responsibilities (Petitioners’ key D7).

13. Maintain a continuing knowledge of the status quo and take immediate remedial action where required (Petitioners’ key D8).

Respondents argue that existing machinery in the Police Department serves to invalidate petitioners’ suggested key 09 (subd. 3, supra) as a proper answer. However, petitioners’ expert witnesses and their cited documentary authority for this answer clearly establish that the 11 machinery ’ ’ contemplated in this response is not the existing formal procedures of the Police Department nor the so-called “ open door ” policy set forth in respondents’ key answer B7 (supra). Thus, contrary to respondents ’ contention, I find such response is properly entitled to additional credit.

Parenthetically, respondents’ expert witness, a Police Department Inspector, conceded that petitioners’ key answer D8 (subd. 13, supra) is a separate and valid response to Question No. 2. lie made a similar though more reluctant concession regarding the following additional answer to this question:

14. Make subordinates feel that they are members of a .team [develop in them a sense of belonging; encourage in them good attitudes toward their job] (Petitioners’ key D9).

Contrary to respondents’ objection to this last answer (subd. 14, supra), I find that it is relevant to the “ hard-boiled ” reputation problem involved in the subject question and, therefore, is properly within a complete key answer thereto.

The maximum credit obtainable under respondents’ rating key to Question No. 2 is 180 points. As will be discussed more fully hereinafter, this, in and of itself, serves to refute respondents’ argument that 160 points, as recommended by petitioners’ rating key to Question No. 4 (infra) is excessive. In any event, I find the recommended credit structure set forth in petitioners’ Exhibit 19 to be proper, valid and suitable with respect to Question No. 2.

Nothing in the proof adduced before me supports Reiner’s testimony that petitioners ’ proposed ‘ ‘ staff related ’ ’ answers should be limited to 5 points each. Likewise, as noted by petitioners’ expert witness, I find respondents’ key answer B8 to be invalid. It certainly is not a proper police function for a Precinct Captain, “ hard-boiled ” or otherwise, to sponsor social affairs for members of his command.

Giving due consideration to the foregoing, petitioners’ claims for additional credits with respect to their answers to Question No. 2 are disposed of as follows: [Charts of additional credits not printed].

Question No. 3:

There has been considerable controversy as to whether juveniles who are arrested should be fingerprinted. Police officials generally favor this practice while social workers are generally opposed. Briefly discuss the arguments for and against this practice.

Respondents’ established rating key, in accordance with which petitioners’ answers were graded, permitted a total of 50 points to be obtained by a candidate for arguments in favor of fingerprinting and a maximum of 60 points for arguments against such procedure. This rating key includes credit for the following responses only:

A. In favor of fingerprinting:

1. It is most accurate method of identification known (full record).

2. It may help protect the innocent (or may help solve juvenile crimes).

3. Prior record is valuable for judicial sentencing or treatment.

4. First duty of law enforcement is to protect society (welfare of juvenile offenders only secondary).

5. Mere taking of prints of juveniles may deter would-be offenders and recidivists.

6. Other.

B. Against fingerprinting:

1. It is closely allied in public mind with criminals and criminal processes (raises question of stigma).

2. It violates or reverses the spirit and/or procedure of juvenile court (which is not criminal in aims and methods).

3. Records other than fingerprints taken on a juvenile offender are usually sufficient identification in eases of subsequent offenses by a youth.

4. a. It may be harmful in that association of fingerprinting with criminal procedure in youth’s mind may mar his outlook for life.

b. Taking of prints of so-called “ tough-guy ” may give him a feeling of importance.

c. This would also have an effect on his associates, leading them to emulate him by committing petty or serious crimes in order to be fingerprinted.

d. The feeling of importance of being a member of a gang outweighs any deterrent effect fingerprinting may have.

5. Other.

Petitioners’ suggested rating key allots a maximum of 55 points for arguments in favor of fingerprinting and maintains respondents’ 60 points’ allocation to arguments against it. However, the expert testimony of Professor Alexander B. Smith and the documentary evidence adduced in support thereof establish that respondents improperly failed to give credit for the following answers which were not included in their rating key:

1. Fingerprinting juveniles enables the FBI to include nation-wide juvenile delinquency statistics in their Uniform Crime Report (Petitioners’ key A6).

2. Fingerprinting which has become commonplace for juveniles is no more a stigma for juveniles than being in a detention home, before the courts, under supervision of a probation officer, or in an institution; they do not therefore constitute a psychological identification of the child with criminals generally (Petitioners’ key A7).

3. Fingerprinting juveniles is part of the trend toward universal fingerprinting «(Petitioners’ key A8).

4. The nonpolice use of juvenile fingerprint files is no more detrimental to the juvenile than the nonpolice use of all other police and court records in a particular case (Petitioners’ key A9).

5. Fingerprinting creates an attitude of hostility in a child or his parents, especially if they are sensitive (Petitioners’ key B5).

6. The act of fingerprinting shows a lack of faith in the juvenile on the part of society that may block his rehabilitation and reintegration into society (Petitioners’ key B6).

While petitioners’ key answer B5 (supra) is similar to respondents’ key B4a, I find that a careful analysis thereof reveals its critical distinction, especially with respect to the attitude of the parents. If a candidate has sufficient knowledge to set forth this last fact in his answer, I hold that he would then be entitled to additional credit of at least 2.5 points.

With the addition of the foregoing answers to a proper key, I find that a maximum of 50 points should be allowed with regard to arguments for fingerprinting; one answer will be entitled to 15 points, and all others will be credited with 10 points each. The key answer credits .suggested by Professor Smith for responses to the arguments against fingerprinting, similarly, are adopted here. Thus, giving due consideration to the foregoing, petitioners’ claims for additional credits with respect to their answers to Question No. 3 are disposed of as follows: [Charts of additional credits not printed].

Question No. 4:

There is a growing .tendency on the part of some citizens to form into large groups and to take their grievances against the police directly to the local station houses. As the captain of a local precinct, what actions would you take if your station house was the object of such a mass demonstration.

In response to this question, respondents’ rating key set forth the following answers for which credit was allowed as follows: a maximum of 60 points was allotted for those items dealing with “ immediate action ”, A1 being valued at ten points, any one answer “ dealing with group ” similarly valued at 10 points and the other responses at 5 points each; likewise, a total of 50 points was allocated to answers dealing with “ Frozen Areas and Assignments ” (infra), the rating key fixing one answer at 10 points and all others at 5 points each. It is to be noted, however, that the rating key in this respect must be deemed to be in error, for if a candidate answered all eight parts of this subdivision, he would receive only 45 points instead of the allotted 50. Significantly, in an effort to cure this apparent defect, respondents’ witness Meltzer, one of the raters of petitioners’ subject examination papers, testified that the “ mob control” answer indicated in B8 of Question No. 4 (infra) was 10 points rather than the 5 ascribed to it by the rating key.

In addition to the answers of “ A ” and “ B ” (infra) totalling' 110 points maximum credit, the rating key, under subdivision “ C ” provided for so-called “ Other ” answers which would warrant another 30 points .maximum. Thus, respondents’ rating key, for the following key answers, made allowance for a purported maximum of 140 credits:

A. Immediate action:

1. Captain of precinct, if present, or other superior in charge should station himself at entrance with loud speaker device so that he can communicate with group.

2. Dealing with group:

a. Ascertain as quickly as possible the identity of those involved, .the number in the group, and the nature of their grievance.

b. Determine who the leaders are.

e. Attempt to communicate with leaders or invite inside to discuss grievance.

d. Be sure that the whole group knows that you are anxious to hear and discuss the grievance and settle it if possible (Good public relations).

e. If the leaders and the group are not willing to discuss the grievance, then tell them they will have to disperse.

f. Allow peaceful picketing.

g. Otherwise give order to men to disperse the group. Direct them to do this without the necessity of arrest, if this is possible.

h. No brutality (caution officers to be firm but avoid brutality, violence).

3. Notify Communications Bureau of the borough so that all concerned may be informed.

4. Send for community leaders who may assist in controlling the group.

5. Tell of other redress open.

B. Frozen Area and Assignments:

1. Establish frozen area in immediate vicinity of station house (with zones & posts).

2. If more than one entrance into station house, cover each one by assignment of men.

3. Sufficient men in front of main entrance so that group does not “ storm ” station house and take it over.

4. Rooftops to be covered by patrolmen.

5. Outline access route for responding vehicles to use in getting to station house, if needed.

6. Vehicles and pedestrians to be excluded.

7. If transit facilities within frozen area, assign men so that confusion, etc. does not occur.

8. Other (If can’t disperse peacefully, use mob control tactics).

C. Other:

1. If precinct personnel cannot handle, send for assistance under Rapid Mobilization Plan; increase as needed.

2. Use precinct squad detectives to infiltrate crowd so as to learn temper of group.

3. Cooperate with press so that the police “story” is presented accurately.

4. Particularly close supervision needed in this situation.

5. Other.

In apposition to respondents’ rating key, petitioners’ suggested key significantly omitted item A2e. As noted by petitioners’ expert witness, this answer, as set forth in respondents’ key, erroneously directs the dispersal of peaceful picketing. Not only is such direction improper and illegal, but such an answer would serve to conflict with respondents’ key answer A2f of the same key (supra). This is perhaps but another example of the difficulty respondents will continue to encounter if they persist in the practice of having non-lawyers prepare questions and key answers which necessarily deal with technical matters of law and legal procedure.

Petitioners’ suggested rating key added another item to section A of respondents ’ key, to wit, 1 ‘ Protect rights of those not involved in demonstration to lawful use of streets and highways There can be no meritorious dispute regarding the validity of this item. In addition thereto, petitioners also suggest a new section D to the rating key, as follows:

D. Concluding and follow-up action:

1. Maintain assignments until conditions return to normal.

2. Submit reports on action taken, with evaluation and comments thereon.

3. Maintain liaison with protesting group, reduce tensions, establish better relations, process grievance as required.

Respondents, significantly, offered no expert or police authority to dispute any of the substantive aspects of these additional rating key items which, accordingly, are hereby found to be proper answers, entitled to credit.

With regard to the credit structure for Question No. 4, respondents’ witness Reiner stated that while it was proper for respondents’ raters to allow a total of 140 points under respondents’ key, the suggested 160 points in petitioners’ key would be excessive. I disagree. Nevertheless, I find that an allowance of 20 points for the new section D (supra) as suggested by petitioners, is not warranted by the contents and significance of the answers required thereby.

As hereinabove indicated and in refutation of Reiner’s argument regarding anything beyond 140 points as excessive, it is to be noted (and Reiner so conceded) that respondents’ credit structure for the key to Question No. 2 (supra) could result in a maximum scoring of 180 points, or a total of 40 points beyond the limitation Reiner seeks here. Thus, upon the evidence before me, I find that no satisfactory reason was established by respondents to limit the answers to Question No. 4 to 140 points, or on the other hand, to allot to section D more than a total of 10 points. The maximum points for the rating key to this question, therefore, is fixed at 150 points.

Giving due consideration to the foregoing, petitioners’ claims for additional credits with respect to their answers to Question No. 4 are disposed of as follows: [Charts of additional credits not printed].

Contrary to their claims upon the trial of this matter, I find that neither Morell nor Carlson is entitled to any additional credits on Question No. 4 for the alleged scope, depth, organization and character of their respective answers.

Question No. 5:

A plan has been proposed whereby review boards would be established in New York City to deal with charges of police misconduct made by private citizens. Evaluate this proposal, giving reasons.

This question, concerning a much debated proposal at the time the subject examination was given, clearly revealed the fundamental fault underlying the use of an essay type test for the selection of public personnel, namely, subjectivity. An analysis of the ratings and papers submitted by respondents in support thereof indicates that in many instances regarding this question, the personal opinions of the examiners seem to prevail over expert opinion and authorities offered by petitioners to corroborate the arguments set forth in their answers. Thus, it frequently appears, as the allowance of additional credits will hereinafter demonstrate, that respondents’ raters failed to give credit for a valid answer simply because they, personally, were of a different opinion.

Likewise, I find unwarranted the raters’ practice of failing to give credit for an answer simply because a candidate lists it in one place as an advantage and in another part of the examination as a disadvantage. This, too, is the fault of the way an essay question is phrased. Questions such as this one, which, in effect, call for “ advantages and disadvantages ” or “ arguments for and against [‘evaluate’]” necessarily lead to the subject problem. Thus, conceivably, some people felt that civilians on a review board were an advantage (impartial review, etc.) while at the same time believed their lack of “ professional ” knowledge of police work would be a disadvantage. Both such answers, obviously, are entitled to credit under a question so phrased.

The following rating key, each part valued at 501 points, was established by respondents for Question No. 5:

A. Advantages.

1. Provides for impartial outside review (or: police will not be judge of their own actions).

2. Tends to limit police to legal and constitutional bounds and actions (or: less likelihood of overzealous or improper law enforcement.

3. Arrests or other actions will tend to be more carefully made — actions more likely to be successfully upheld; less likelihood of civilian suits.

4. Significantly increases public confidence in police or improves relationship with police.

5. Other.

B. Disadvantages.

1. Threatens good law enforcement in that police may fear to act, especially in borderline cases.

2. May result in increase in unjustified charges or be used as a forum for challenging or criticising police activity.

3. Difficulty of selecting qualified impartial citizens, willing to serve (or: More likely to be influenced by social and political forces and pressure groups).

4. May lower police morale (or: police tend to lack confidence in civilians).

5. Other.

No -Credit: a. May set precedent for outside review boards in other agencies.

b. Taking both sides of same issue without adequate explanation.

At least 14 of the 16 petitioners here involved set forth alleged advantages and disadvantages of the proposal which were not credited under the above rating key fixed by respondents. The fact that there were so many responses other than those in such key should have alerted respondents’ raters to the fact that their key, at best, was incomplete. Nothing in the record in this respect warrants the conclusion, as asserted by Reiner, that all of the new items (hereinafter set forth in the revised rating key) came within the intended scope of the items of respondents’ key, inasmuch as they were deemed by the examiners to be incorrect and, therefore, entitled to and given no credit.

The revised rating key adopted here was prepared by one of the authors of the article cited and relied upon by respondents in the preparation of the question and answers here involved. While such author, an expert witness called in behalf of petitioners, had no past experience in the formulation of examinations for the selection of public personnel, his testimony on the subject matter of Question No. 5 was not challenged by an expert as to the validity of the new items or as to his opinion that each such new item was separate from and equal in significance to each item in respondents’ rating hey. In view of such evidence, I find each such new item (infra) is entitled to the same value as that assigned to answers in respondents’ key.

The revised key, therefore, adds three items to section A and three to .section B of respondents’ key, as follows:

1. A-5:

Provides for the application of civilian judgment in balancing the interests of law enforcement and individual rights ”.

Contrary to respondents’ contention, I find this to be a valid answer separate and apart from their key answer Al.

2. A-6:

Citizens would not he inhibited from lodging complaints or giving testimony because of fear of police or distrust of departmental review procedures ”.

Contrary to respondents ’ contention, I find this to be a valid answer separate and apart from their key answer A4.

3. A-7:

Provides an avenue of communication through which the public can suggest to the police general reforms that eliminate improper or irritating practices”.

Contrary to respondents’ contention, I find this to be a valid answer separate and apart from their key answer A2.

Each of these three new key items is valued at a maximum of ten points.

1. B-5:

Civilian board members would not be experts in police work and would tend to be unfamiliar with problems and techniques of law enforcement ”.

Contrary to respondents’ contention, I find this to be a valid answer separate and apart from their key answer B3.

2. B-6:

Review of police misconduct should be by officials who are in a position to effectuate changes and reforms; collateral review by an outside body would tend to interfere with the authority of those primarily responsible for the administration of the police department”.

In view of the fact at least 8 petitioners gave this last new item as a response under “ disadvantages ”, respondents’ principal witness reluctantly conceded on cross-examination that it should properly be a part of the rating key.

3. B-7:

Recourse against police misconduct is available through existing departmental review and discipline procedures and through judicial action ”.

While Reiner conceded this response would constitute a new item, he stated it was not an answer to the question. I disagree, inasmuch as such a response indicates that the proposed board is ‘ ‘ unnecessary ” or 11 not needed ’ ’, which, quite properly, would entitle such an answer to credit under," disadvantages ”.

Each of these three new "disadvantages” is valued at a maximum of 10 points.

Giving due consideration to the foregoing, petitioners’ claims for additional credits with respect to their answers to Question No. 5 are disposed of as follows: [Charts of additional credits not printed].

Question No. 6:

“As a result of his investigation of a crime, an investigating officer has discovered certain items, including documents, which may have value as evidence. State and briefly discuss the principal factors that must be considered if this evidence is to be used in court.”

Respondents’ key answer:

A. Admissibility:

1. Matter must have been obtained in a legal fashion (or: without violating a law, or a right protected by the Constitution, or by force, duress or threats).

2. Matter must be relevant.

3. .Conclusions and opinions, no matter how well supported, are inadmissible (but facts which give rise to these e’s & o’s must be established and introduced as evidence).

4. Other.

B. Reliability (or Weight as Evidence):

1. Continuity of chain of possession must be preserved.

2. Protection of documents and other items from destruction or contamination to preserve their evidential value.

3. Must be clearly marked permanently and distinctly, for ready identification.

4. Is it a document of original entry (if not, on what other documents or records is it based?) (or: Original document best evidence (if original lost, destroyed, etc., or outside court’s jurisdiction, can introduce secondary evidence in form of photos, photostats, copies) by authenticity of copy must be certified) (or: What are the possibilities of transcription errors, or of error in the basic documents?).

5. Other (Would the person supplying document have any logical reason, or personal reason to falsify it or gain an advantage through falsification? For what purpose was document prepared?).

a. For express purpose of recording the desired information.

b. For another purpose, and desired information was entered only coincidentally.

Police Jan. Feb. 1962, using documents as evidence (p. 19 et seq.).

Respondents ’ rating key assigned a total of 50 points to the four answers under subdivision A (20, 15, 10 and 5 points, respectively) and a maximum of 70 points to answers under subdivision B (20, 20, 20, 10 and 10 points, respectively, for all five answers, respectively). Again indicating the weakness of having a non-lawyer prepare a rating key dealing with a technical legal problem, respondents’ item A3 is clearly erroneous and should not have been included as part of the rating key’s answers. Expert opinion is not only admissible but, most frequently, vital and needed.

This error was apparently appreciated by petitioners’ expert witness on this question who prepared two alternative rating keys. His second alternative key omits item A3, while his first alternative key omits all of respondents’ answers under subdivision A, “Admissibility”. In my opinion, the second alternative key, as suggested by this outstanding police authority, lawyer and examiner, is superior to his first alternative and to the rating key used by respondents. Such alternative rating key, therefore, is adopted here. It includes two new key answers not set forth in respondents’ key despite the fact that at least 13 of the 16 petitioners here involved had responses in their examination papers which should have alerted the raters to the probable propriety of such answers. These two items relate to:

1. Delivery for examination or analysis to laboratory or technician. (Petitioners’ key B4).

2. Availability of reproductions in the form of photographs, sketches, copies, or molds for proper presentation of evidence in court. (Petitioners’ key B5).

Significantly, the police expert called by respondents failed to contradict the validity of either of the foregoing two new key answers and, in effect, confirmed the fact that the mobile laboratory and main laboratories of the city Police Department perform the examinations and analysis referred to in item B4 (supra).

A proper credit structure for the new rating key which includes the two new items under subdivision B would, in my opinion, allow 20 points for each item thereunder, with a maximum of 80 points. As noted by petitioners, while respondents’ senior examiner agreed with the 80 points ’ value for subdivision B, he would reduce the value of each new item by assigning 20 points to any one item thereunder and 10 points to each other item. I find such contention to be wholly without merit. If adopted, this suggested rating key would require 8 of 9 answers for a perfect score of 80 points. On the other hand, it is to be noted that respondents’ rating key had three items in subdivision B at 20 points each and at least two other items at 10 points each (supra). Thus, a perfect score of 70 points on respondents’ rating key could be obtained by four of a possible nine answers. I find, therefore, that the extensive and exhaustive scope of the revised key (the second alternative [supra]), adopted here as the proper rating key, coupled with the limited time available to a candidate on this examination, justified petitioners’ expert opinion that each item was worth 20 points. I hold, therefore, that an answer which fully and correctly sets forth four of the items in subdivision B, as adopted here, is entitled to the full 80 points.

Q-iving due consideration to the foregoing, petitioners’ claims for additional credits with respect to their answers to Question No. 6 are disposed of as follows: [Charts of additional credits not printed].

With regard to the respective specific items for which credit was sought by petitioners under the key answers above listed for each of the six essay questions involved, such items are set forth fully in petitioners’ Exhibits 9, 11, 15, 17 and 19, their suggested rating keys. It will serve no useful purpose, therefore, to repeat herein each and every item of petitioners’ answers in greater detail than already indicated here.

It should be noted, however, that for the most part, the credit structure adopted here for each question accommodated, insofar as practicable, all new items (which were not included in respondents’ rating keys) to the credit structures they had used for the marking of the subject test. This obviated the possibility that an answer common to respondents’ key and the one here adopted in lieu thereof would receive different credits when marked on the original rating or when marked as a result of this proceeding.

In most instances, as the foregoing rating keys and credit structures indicate, respondents’ proposed credit allowances for the revised rating keys were untenable. The value assigned by respondents to each new item was almost always less than that suggested by petitioners, although respondents’ witnesses were frequently unable to state a valid reason therefor. Many times, as above stated, respondents’ assertions that the new items were “the same” or “closely related” to their key items were wholly without foundation and unwarranted. Reiner’s testimony to the effect that each of respondents’ rating keys contemplated the allowance of credit for 1 ‘ other ’ ’ items not specified therein does not support these assertions (supra). It is to be noted that nothing in respondents’ credit structures indicate that the value of a specified item would be reduced by the allowance of credit for a so-called “ other ” item.

An analysis of respondents’ proposed credit structures (as compared to those adopted here) also establishes that to use those recommended by respondents, in effect, would cause petitioners to be entitled and to receive less credit for an answer than was allowed by the raters in the original marking of identical items in papers of other candidates not involved in the litigation. Thus, even if it were to be assumed that respondents’ proposed credit allowances would be appropriate as an original rating system, it is obviously inappropriate as a method of rating at this time. The resulting prejudice in the grading of petitioners’ answers as compared with that of others not before the court cannot be condoned. 'Contrary to the position advocated, in substance, by respondents, it would be wholly unjustifiable at this stage of regrading answers to cause any petitioner to receive less credit for any item in his examination paper than respondents have already allowed for an identical response to all other candidates on this test.

In recapitulation, therefore, the following three tables set forth statistically the results of the regrading of petitioners’ answers to Part II of Examination No. 9961, as indicated in detail hereinabove. The first table lists the additional credits scored by each of the 16 petitioners on each of the six questions; the second table lists the old and new grades fixed for each question and the resulting new average of each petitioner on Part II of the subject examination; the third tabulation sets forth the new final grades obtained by each of the petitioners on the subject examination as a whole. [Tables not printed.]'

Accordingly, the petition is granted and judgment is rendered in favor of petitioners as hereinabove indicated.

Settle final judgment providing for the regrading of petitioners’ answers to Part II of Examination No. 9961 and the regrading of their final averages on said examination in accordance herewith.