An Investigation of the Substantive Process Validity of Multistate Bar Examination Items through Verbal Protocol Analysis

Persistent Link:
http://hdl.handle.net/10150/194980
Title:
An Investigation of the Substantive Process Validity of Multistate Bar Examination Items through Verbal Protocol Analysis
Author:
Bonner, Sarah
Issue Date:
2005
Publisher:
The University of Arizona.
Rights:
Copyright © is held by the author. Digital access to this material is made possible by the University Libraries, University of Arizona. Further transmission, reproduction or presentation (such as public display or performance) of protected items is prohibited except with permission of the author.
Abstract:
The dissertation describes a think-aloud study that investigated the internal processes of 25 participants responding to selected items from the Multistate Bar Exam (MBE). The MBE is a nationally administered high-stakes licensure test used in 53 U.S. jurisdictions. The following questions were addressed by the study: Which mental processes are most frequently used in answering the selected items? What is the relative effect on performance of reasoning from legal principles as compared to general reasoning or use of testwiseness principles? When variance in performance due to similarity of responses to expert problem-solving models is accounted for, do divergent mental processes add to prediction? Do minority ethnic examinees use different cognitive processes or commit different types of errors when answering items in selected MBE content areas as compared to majority ethnic examinees?The dissertation discusses the construct of legal reasoning, the method of verbal protocol analysis, and potential method effects of reactivity and veridicality. The study method, including the development and reliability of the rating systems used to quantify the verbal transcripts for analysis are also described.Findings of the study are presented. Using legal principles and rehearsing facts were the most commonly used mental processes. Participants used a number of other mental processes which are described. Using legal principles and avoiding extraneous inferences were associated with increased performance. Drawing premature conclusions and making errors in legal principles negatively affected performance. The effects of different error types and construct-irrelevant thinking such as cue-using strategies are also be reported. Similarity of responses to expert problem-solving models predicted performance on verbalized items, and adding divergent mental processes as predictors did not significantly increase the amount of variance accounted for. No evidence was found in the verbal responses that minority ethnic examinees used different cognitive processes when answering items in selected MBE content areas as compared to majority ethnic examinees; however, a method effect appeared for the minority examinees in the study that complicates interpretation. Some evidence was found of differences in response patterns for minority examinees that could not be explained by the verbal responses.
Type:
text; Electronic Dissertation
Degree Name:
PhD
Degree Level:
doctoral
Degree Program:
Educational Psychology; Graduate College
Degree Grantor:
University of Arizona
Advisor:
D'Agostino, Jerome V
Committee Chair:
D'Agostino, Jerome V

Full metadata record

DC FieldValue Language
dc.language.isoENen_US
dc.titleAn Investigation of the Substantive Process Validity of Multistate Bar Examination Items through Verbal Protocol Analysisen_US
dc.creatorBonner, Sarahen_US
dc.contributor.authorBonner, Sarahen_US
dc.date.issued2005en_US
dc.publisherThe University of Arizona.en_US
dc.rightsCopyright © is held by the author. Digital access to this material is made possible by the University Libraries, University of Arizona. Further transmission, reproduction or presentation (such as public display or performance) of protected items is prohibited except with permission of the author.en_US
dc.description.abstractThe dissertation describes a think-aloud study that investigated the internal processes of 25 participants responding to selected items from the Multistate Bar Exam (MBE). The MBE is a nationally administered high-stakes licensure test used in 53 U.S. jurisdictions. The following questions were addressed by the study: Which mental processes are most frequently used in answering the selected items? What is the relative effect on performance of reasoning from legal principles as compared to general reasoning or use of testwiseness principles? When variance in performance due to similarity of responses to expert problem-solving models is accounted for, do divergent mental processes add to prediction? Do minority ethnic examinees use different cognitive processes or commit different types of errors when answering items in selected MBE content areas as compared to majority ethnic examinees?The dissertation discusses the construct of legal reasoning, the method of verbal protocol analysis, and potential method effects of reactivity and veridicality. The study method, including the development and reliability of the rating systems used to quantify the verbal transcripts for analysis are also described.Findings of the study are presented. Using legal principles and rehearsing facts were the most commonly used mental processes. Participants used a number of other mental processes which are described. Using legal principles and avoiding extraneous inferences were associated with increased performance. Drawing premature conclusions and making errors in legal principles negatively affected performance. The effects of different error types and construct-irrelevant thinking such as cue-using strategies are also be reported. Similarity of responses to expert problem-solving models predicted performance on verbalized items, and adding divergent mental processes as predictors did not significantly increase the amount of variance accounted for. No evidence was found in the verbal responses that minority ethnic examinees used different cognitive processes when answering items in selected MBE content areas as compared to majority ethnic examinees; however, a method effect appeared for the minority examinees in the study that complicates interpretation. Some evidence was found of differences in response patterns for minority examinees that could not be explained by the verbal responses.en_US
dc.typetexten_US
dc.typeElectronic Dissertationen_US
thesis.degree.namePhDen_US
thesis.degree.leveldoctoralen_US
thesis.degree.disciplineEducational Psychologyen_US
thesis.degree.disciplineGraduate Collegeen_US
thesis.degree.grantorUniversity of Arizonaen_US
dc.contributor.advisorD'Agostino, Jerome Ven_US
dc.contributor.chairD'Agostino, Jerome Ven_US
dc.contributor.committeememberSabers, Darrell L.en_US
dc.contributor.committeememberNitko, Anthony J.en_US
dc.contributor.committeememberSechrest, Leeen_US
dc.contributor.committeememberMcKnight, Patricken_US
dc.identifier.proquest1187en_US
dc.identifier.oclc137354276en_US
All Items in UA Campus Repository are protected by copyright, with all rights reserved, unless otherwise indicated.