Supplemental Test Development Information

Music

	Number
Committee Participants: Total	11

Public School Educators	9
Higher Education Faculty	2

Gender
Female	7
Male	4

Ethnicity
American Indian or Alaska Native	0
Black (not of Hispanic origin)	0
Asian or Pacific Islander	2
Hispanic	0
White	4
Other	0
Unspecified	5

Music

	Number
Committee Participants: Total	5

Public School Educators	4
Higher Education Faculty	1

Gender
Female	2
Male	3

Ethnicity
American Indian or Alaska Native	0
Black (not of Hispanic origin)	2
Asian or Pacific Islander	1
Hispanic	0
White	0
Other	1
Unspecified	1

Deaf/Hard of Hearing
Yes	0
No	5

Music

	Number
Committee Participants: Total	15

Public School Educators	12
Higher Education Faculty	3

Gender
Female	7
Male	8

Ethnicity
American Indian or Alaska Native	0
Black (not of Hispanic origin)	0
Asian or Pacific Islander	2
Hispanic	0
White	7
Other	0
Unspecified	6

Music

INTACT CLASSROOM SESSIONS 2015
04/11/2015	Gordon College
04/27/2015	Salem State University
05/02/2015	University of Massachusetts - Amherst
05/05/2015	Westfield State University
05/09/2015	Gordon College

STAND ALONE EVENTS AT MTEL OPERATIONAL TEST ADMINISTRATIONS

03/07/2015
05/09/2015
07/11/2015

Music

Multiple-Choice Items (MCQs)

Number of MCQs Prepared	323
Number of MCQs Piloted*	176
Number of MCQs Identified for Further Review	31
Number of MCQs Deleted Post Pilot Test Review	3

Open-Response Items (ORIs)

Number of ORIs Developed	19
Number of ORIs Piloted*	7
Number of ORIs Identified for Further Review	4
Number of ORIs Deleted Post Pilot Test Review	none

*This chart includes information regarding items piloted prior to the first operational administration and may be a subset of all items developed. Additional multiple-choice items are piloted as non-scorable on operational test forms, until all items are piloted.

Music

Number of Qualifying Score Participants: 15

Number of Participants Providing Ratings: 15 (100%)

Question [Rating Scale: 1 (not at all well) to 5 (very well)]	Mean Rating
How well do you feel you understood the training that you received for making your qualifying score judgments?	4.8
How confident are you of the qualifying score judgments you made?	4.7
How satisfied are you that you were able to complete the work you were asked to do in the time that was provided?	5.0
How successful were the coordination and logistics of the qualifying score conference?	5.0
Overall, how satisfied are you with the qualifying score process in which you participated?	4.7

German

	Number
Committee Participants: Total	8

Public School Educators	8
Higher Education Faculty	0

Gender
Female	6
Male	2

Ethnicity
American Indian or Alaska Native	0
Black (not of Hispanic origin)	0
Asian or Pacific Islander	0
Hispanic	0
White	5
Other	1
Unspecified	2

German

	Number
Committee Participants: Total	13

Public School Educators	11
Higher Education Faculty	2

Gender
Female	9
Male	4

Ethnicity
American Indian or Alaska Native	0
Black (not of Hispanic origin)	4
Asian or Pacific Islander	1
Hispanic	2
White	1
Other	5
Unspecified	0

Deaf/Hard of Hearing
Yes	1
No	12

German

	Number
Committee Participants: Total	7

Public School Educators	7
Higher Education Faculty	0

Gender
Female	5
Male	2

Ethnicity
American Indian or Alaska Native	0
Black (not of Hispanic origin)	0
Asian or Pacific Islander	0
Hispanic	0
White	4
Other	1
Unspecified	2

German

Multiple-Choice Items (MCQs)

NOTE: As described in the "Pilot Testing" section of the Technical Manual, in addition to the review of test items by expert judges (members of the BRC and CACs), when permitted by sufficient candidate populations, empirical data were collected about statistical and qualitative characteristics of the new items through pilot testing. For this field there exists an insufficient candidate population for collecting statistical information through pilot testing. Therefore, pilot test information for multiple-choice questions is unavailable for the field.

Open-Response Items (ORIs)

Number of ORIs Developed	20
Number of ORIs Piloted*	4
Number of ORIs Identified for Further Review	none
Number of ORIs Deleted Post Pilot Test Review	none

*This chart includes information regarding items piloted prior to the first operational administration and may be a subset of all items developed. Additional multiple-choice items are piloted as non-scorable on operational test forms, until all items are piloted.

German

Number of Qualifying Score Participants: 7

Number of Participants Providing Ratings: 7 (100%)

Question [Rating Scale: 1 (not at all well) to 5 (very well)]	Mean Rating
How well do you feel you understood the training that you received for making your qualifying score judgments?	4.7
How confident are you of the qualifying score judgments you made?	4.7
How satisfied are you that you were able to complete the work you were asked to do in the time that was provided?	5.0
How successful were the coordination and logistics of the qualifying score conference?	4.9
Overall, how satisfied are you with the qualifying score process in which you participated?	4.4

Chinese (Mandarin)

	Number
Committee Participants: Total	8

Public School Educators	6
Higher Education Faculty	2

Gender
Female	7
Male	1

Ethnicity
American Indian or Alaska Native	0
Black (not of Hispanic origin)	0
Asian or Pacific Islander	6
Hispanic	0
White	2
Other	0
Unspecified	0

Chinese (Mandarin)

	Number
Committee Participants: Total	13

Public School Educators	11
Higher Education Faculty	2

Gender
Female	9
Male	4

Ethnicity
American Indian or Alaska Native	0
Black (not of Hispanic origin)	4
Asian or Pacific Islander	1
Hispanic	2
White	1
Other	5
Unspecified	0

Deaf/Hard of Hearing
Yes	1
No	12

Chinese (Mandarin)

	Number
Committee Participants: Total	8

Public School Educators	5
Higher Education Faculty	3

Gender
Female	7
Male	1

Ethnicity
American Indian or Alaska Native	0
Black (not of Hispanic origin)	0
Asian or Pacific Islander	6
Hispanic	0
White	1
Other	0
Unspecified	1

Chinese (Mandarin)

INTACT CLASSROOM SESSIONS 2013–2015
10/04/2013	Brandeis University
10/17/2013	University of Massachusetts - Amherst
04/29/2015	Tufts University
05/02/2015	Smith College
05/09/2015	Gordon College

STAND ALONE EVENTS AT MTEL OPERATIONAL TEST ADMINISTRATIONS

10/26/2013
03/01/2014
05/10/2014
07/12/2014
10/25/2014
03/07/2015
05/09/2015
07/11/2015

STAND ALONE EVENT(S) IN STATE(S) OTHER THAN MASSACHUSETTS

11/16/2013

Chinese (Mandarin)

Multiple-Choice Items (MCQs) and Short-Answer (SAs) items

Number of MCQs and SAs Prepared	170
Number of MCQs and SAs Piloted*	74
Number of MCQs/SAs Identified for Further Review	17
Number of MCQs/SAs Deleted Post Pilot Test Review	1

Open-Response Items (ORIs)

Number of ORIs Developed	23
Number of ORIs Piloted*	4
Number of ORIs Identified for Further Review	none
Number of ORIs Deleted Post Pilot Test Review	none

*This chart includes information regarding items piloted prior to the first operational administration and may be a subset of all items developed. Additional multiple-choice items are piloted as non-scorable on operational test forms, until all items are piloted.

Chinese (Mandarin)

Number of Qualifying Score Participants: 8

Number of Participants Providing Ratings: 8 (100%)

Question [Rating Scale: 1 (not at all well) to 5 (very well)]	Mean Rating
How well do you feel you understood the training that you received for making your qualifying score judgments?	4.8
How confident are you of the qualifying score judgments you made?	4.5
How satisfied are you that you were able to complete the work you were asked to do in the time that was provided?	4.9
How successful were the coordination and logistics of the qualifying score conference?	5.0
Overall, how satisfied are you with the qualifying score process in which you participated?	5.0

Italian

	Number
Committee Participants: Total	5

Public School Educators	3
Higher Education Faculty	2

Gender
Female	4
Male	1

Ethnicity
American Indian or Alaska Native	0
Black (not of Hispanic origin)	0
Asian or Pacific Islander	0
Hispanic	0
White	3
Other	0
Unspecified	2

Italian

	Number
Committee Participants: Total	16

Public School Educators	14
Higher Education Faculty	2

Gender
Female	11
Male	5

Ethnicity
American Indian or Alaska Native	0
Black (not of Hispanic origin)	3
Asian or Pacific Islander	2
Hispanic	2
White	3
Other	6
Unspecified	0

Deaf/Hard of Hearing
Yes	3
No	13

Italian

	Number
Committee Participants: Total	7

Public School Educators	5
Higher Education Faculty	2

Gender
Female	6
Male	1

Ethnicity
American Indian or Alaska Native	0
Black (not of Hispanic origin)	0
Asian or Pacific Islander	0
Hispanic	0
White	4
Other	0
Unspecified	3

Italian

INTACT CLASSROOM SESSIONS 2013–2015
10/23/2013	Brandeis University

STAND ALONE EVENTS AT MTEL OPERATIONAL TEST ADMINISTRATIONS

10/26/2013
03/01/2014
05/10/2014
07/12/2014
10/25/2014

STAND ALONE EVENT(S) IN STATE(S) OTHER THAN MASSACHUSETTS

11/16/2013

Italian

Multiple-Choice Items (MCQs)

NOTE: As described in the "Pilot Testing" section of the Technical Manual, in addition to the review of test items by expert judges (members of the BRC and CACs), when permitted by sufficient candidate populations, empirical data were collected about statistical and qualitative characteristics of the new items through pilot testing. For this field there exists an insufficient candidate population for collecting statistical information through pilot testing. Therefore, pilot test information for multiple-choice questions is unavailable for the field.

Open-Response Items (ORIs)

Number of ORIs Developed	20
Number of ORIs Piloted*	8
Number of ORIs Identified for Further Review	none
Number of ORIs Deleted Post Pilot Test Review	none

*This chart includes information regarding items piloted prior to the first operational administration and may be a subset of all items developed. Additional multiple-choice items are piloted as non-scorable on operational test forms, until all items are piloted.

Italian

Number of Qualifying Score Participants: 7

Number of Participants Providing Ratings: 7 (100%)

Question [Rating Scale: 1 (not at all well) to 5 (very well)]	Mean Rating
How well do you feel you understood the training that you received for making your qualifying score judgments?	4.4
How confident are you of the qualifying score judgments you made?	4.3
How satisfied are you that you were able to complete the work you were asked to do in the time that was provided?	4.4
How successful were the coordination and logistics of the qualifying score conference?	4.6
Overall, how satisfied are you with the qualifying score process in which you participated?	4.7

Russian

	Number
Committee Participants: Total	7

Public School Educators	3
Higher Education Faculty	4

Gender
Female	6
Male	1

Ethnicity
American Indian or Alaska Native	0
Black (not of Hispanic origin)	0
Asian or Pacific Islander	0
Hispanic	0
White	5
Other	0
Unspecified	2

Russian

	Number
Committee Participants: Total	15

Public School Educators	12
Higher Education Faculty	3

Gender
Female	10
Male	5

Ethnicity
American Indian or Alaska Native	0
Black (not of Hispanic origin)	5
Asian or Pacific Islander	1
Hispanic	2
White	2
Other	5
Unspecified	0

Deaf/Hard of Hearing
Yes	2
No	13

Russian

	Number
Committee Participants: Total	3

Public School Educators	2
Higher Education Faculty	1

Gender
Female	3
Male	0

Ethnicity
American Indian or Alaska Native	0
Black (not of Hispanic origin)	0
Asian or Pacific Islander	0
Hispanic	0
White	1
Other	0
Unspecified	2

Russian

Multiple-Choice Items (MCQs)

NOTE: As described in the "Pilot Testing" section of the Technical Manual, in addition to the review of test items by expert judges (members of the BRC and CACs), when permitted by sufficient candidate populations, empirical data were collected about statistical and qualitative characteristics of the new items through pilot testing. For this field there exists an insufficient candidate population for collecting statistical information through pilot testing. Therefore, pilot test information for multiple-choice questions is unavailable for the field.

Open-Response Items (ORIs)

Number of ORIs Developed	23
Number of ORIs Piloted*	4
Number of ORIs Identified for Further Review	none
Number of ORIs Deleted Post Pilot Test Review	none

*This chart includes information regarding items piloted prior to the first operational administration and may be a subset of all items developed. Additional multiple-choice items are piloted as non-scorable on operational test forms, until all items are piloted.

Russian

Number of Qualifying Score Participants: 3

Number of Participants Providing Ratings: 3 (100%)

Question [Rating Scale: 1 (not at all well) to 5 (very well)]	Mean Rating
How well do you feel you understood the training that you received for making your qualifying score judgments?	4.7
How confident are you of the qualifying score judgments you made?	4.7
How satisfied are you that you were able to complete the work you were asked to do in the time that was provided?	5.0
How successful were the coordination and logistics of the qualifying score conference?	5.0
Overall, how satisfied are you with the qualifying score process in which you participated?	5.0

Portuguese

	Number
Committee Participants: Total	9

Public School Educators	8
Higher Education Faculty	1

Gender
Female	7
Male	2

Ethnicity
American Indian or Alaska Native	0
Black (not of Hispanic origin)	0
Asian or Pacific Islander	0
Hispanic	1
White	2
Other	1
Unspecified	5

Portuguese

	Number
Committee Participants: Total	16

Public School Educators	14
Higher Education Faculty	2

Gender
Female	11
Male	5

Ethnicity
American Indian or Alaska Native	0
Black (not of Hispanic origin)	3
Asian or Pacific Islander	2
Hispanic	2
White	3
Other	6
Unspecified	0

Deaf/Hard of Hearing
Yes	3
No	13

Portuguese

	Number
Committee Participants: Total	6

Public School Educators	5
Higher Education Faculty	1

Gender
Female	6
Male	0

Ethnicity
American Indian or Alaska Native	0
Black (not of Hispanic origin)	0
Asian or Pacific Islander	0
Hispanic	1
White	2
Other	1
Unspecified	2

Portuguese

Multiple-Choice Items (MCQs)

NOTE: As described in the "Pilot Testing" section of the Technical Manual, in addition to the review of test items by expert judges (members of the BRC and CACs), when permitted by sufficient candidate populations, empirical data were collected about statistical and qualitative characteristics of the new items through pilot testing. For this field there exists an insufficient candidate population for collecting statistical information through pilot testing. Therefore, pilot test information for multiple-choice questions is unavailable for the field.

Open-Response Items (ORIs)

Number of ORIs Developed	20
Number of ORIs Piloted*	6
Number of ORIs Identified for Further Review	none
Number of ORIs Deleted Post Pilot Test Review	none

*This chart includes information regarding items piloted prior to the first operational administration and may be a subset of all items developed. Additional multiple-choice items are piloted as non-scorable on operational test forms, until all items are piloted.

Portuguese

Number of Qualifying Score Participants: 6

Number of Participants Providing Ratings: 6 (100%)

Question [Rating Scale: 1 (not at all well) to 5 (very well)]	Mean Rating
How well do you feel you understood the training that you received for making your qualifying score judgments?	4.8
How confident are you of the qualifying score judgments you made?	5.0
How satisfied are you that you were able to complete the work you were asked to do in the time that was provided?	4.8
How successful were the coordination and logistics of the qualifying score conference?	4.7
Overall, how satisfied are you with the qualifying score process in which you participated?	4.5

Theater

	Number
Committee Participants: Total	10

Public School Educators	8
Higher Education Faculty	2

Gender
Female	6
Male	4

Ethnicity
American Indian or Alaska Native	0
Black (not of Hispanic origin)	0
Asian or Pacific Islander	0
Hispanic	0
White	6
Other	1
Unspecified	3

Theater

	Number
Committee Participants: Total	5

Public School Educators	4
Higher Education Faculty	1

Gender
Female	2
Male	3

Ethnicity
American Indian or Alaska Native	0
Black (not of Hispanic origin)	2
Asian or Pacific Islander	1
Hispanic	0
White	0
Other	1
Unspecified	1

Deaf/Hard of Hearing
Yes	0
No	5

Theater

	Number
Committee Participants: Total	8

Public School Educators	6
Higher Education Faculty	2

Gender
Female	7
Male	1

Ethnicity
American Indian or Alaska Native	0
Black (not of Hispanic origin)	1
Asian or Pacific Islander	0
Hispanic	0
White	5
Other	0
Unspecified	2

Theater

INTACT CLASSROOM SESSIONS 2015
04/11/2015	Emerson College
04/17/2015	Emerson College
04/27/2015	Salem State University

STAND ALONE EVENTS AT MTEL OPERATIONAL TEST ADMINISTRATIONS

03/07/2015
05/09/2015
07/11/2015

COMPUTER-BASED PILOT TESTING

02/01/2016–04/01/2016

Theater

Multiple-Choice Items (MCQs)

Number of MCQs Prepared	272
Number of MCQs Piloted*	269
Number of MCQs Identified for Further Review	32
Number of MCQs Deleted Post Pilot Test Review	none

Open-Response Items (ORIs)

Number of ORIs Developed	14
Number of ORIs Piloted*	5
Number of ORIs Identified for Further Review	none
Number of ORIs Deleted Post Pilot Test Review	none

*This chart includes information regarding items piloted prior to the first operational administration and may be a subset of all items developed. Additional multiple-choice items are piloted as non-scorable on operational test forms, until all items are piloted.

Theater

Number of Qualifying Score Participants: 8

Number of Participants Providing Ratings: 8 (100%)

Question [Rating Scale: 1 (not at all well) to 5 (very well)]	Mean Rating
How well do you feel you understood the training that you received for making your qualifying score judgments?	4.8
How confident are you of the qualifying score judgments you made?	4.6
How satisfied are you that you were able to complete the work you were asked to do in the time that was provided?	5.0
How successful were the coordination and logistics of the qualifying score conference?	4.9
Overall, how satisfied are you with the qualifying score process in which you participated?	4.8

Dance

	Number
Committee Participants: Total	7

Public School Educators	5
Higher Education Faculty	2

Gender
Female	6
Male	1

Ethnicity
American Indian or Alaska Native	0
Black (not of Hispanic origin)	0
Asian or Pacific Islander	0
Hispanic	1
White	5
Other	0
Unspecified	1

Dance

	Number
Committee Participants: Total	13

Public School Educators	11
Higher Education Faculty	2

Gender
Female	9
Male	4

Ethnicity
American Indian or Alaska Native	0
Black (not of Hispanic origin)	4
Asian or Pacific Islander	1
Hispanic	2
White	1
Other	5
Unspecified	0

Deaf/Hard of Hearing
Yes	1
No	12

Dance

	Number
Committee Participants: Total	6

Public School Educators	4
Higher Education Faculty	2

Gender
Female	5
Male	1

Ethnicity
American Indian or Alaska Native	0
Black (not of Hispanic origin)	0
Asian or Pacific Islander	0
Hispanic	0
White	4
Other	0
Unspecified	2

Dance

INTACT CLASSROOM SESSIONS 2013–2015
10/02/2013	University of Massachusetts - Amherst
10/08/2013	Bridgewater State University
02/05/2014	Salem State University
02/20/2014	Bridgewater State University

STAND ALONE EVENTS AT MTEL OPERATIONAL TEST ADMINISTRATIONS

07/13/2013
10/26/2013

STAND ALONE EVENT(S) IN STATE(S) OTHER THAN MASSACHUSETTS

11/16/2013

Dance

Multiple-Choice Items (MCQs)

NOTE: As described in the "Pilot Testing" section of the Technical Manual, in addition to the review of test items by expert judges (members of the BRC and CACs), when permitted by sufficient candidate populations, empirical data were collected about statistical and qualitative characteristics of the new items through pilot testing. For this field there exists an insufficient candidate population for collecting statistical information through pilot testing. Therefore, pilot test information for multiple-choice questions is unavailable for the field.

Open-Response Items (ORIs)

Number of ORIs Developed	9
Number of ORIs Piloted*	9
Number of ORIs Identified for Further Review	none
Number of ORIs Deleted Post Pilot Test Review	none

*This chart includes information regarding items piloted prior to the first operational administration and may be a subset of all items developed. Additional multiple-choice items are piloted as non-scorable on operational test forms, until all items are piloted.

Dance

Number of Qualifying Score Participants: 6

Number of Participants Providing Ratings: 6 (100%)

Question [Rating Scale: 1 (not at all well) to 5 (very well)]	Mean Rating
How well do you feel you understood the training that you received for making your qualifying score judgments?	4.8
How confident are you of the qualifying score judgments you made?	4.7
How satisfied are you that you were able to complete the work you were asked to do in the time that was provided?	4.8
How successful were the coordination and logistics of the qualifying score conference?	4.8
Overall, how satisfied are you with the qualifying score process in which you participated?	4.7

English as a Second Language (ESL)

	Number
Committee Participants: Total	24

Public School Educators	19
Higher Education Faculty	5

Gender
Female	24
Male	0

Ethnicity
American Indian or Alaska Native	0
Black (not of Hispanic origin)	0
Asian or Pacific Islander	1
Hispanic	5
White	12
Other	1
Unspecified	5

English as a Second Language (ESL)

	Number
Committee Participants: Total	16

Public School Educators	12
Higher Education Faculty	4

Gender
Female	11
Male	5

Ethnicity
American Indian or Alaska Native	0
Black (not of Hispanic origin)	4
Asian or Pacific Islander	1
Hispanic	3
White	3
Other	5
Unspecified	0

Deaf/Hard of Hearing
Yes	3
No	13

English as a Second Language (ESL)

	Number
Committee Participants: Total	12

Public School Educators	8
Higher Education Faculty	4

Gender
Female	11
Male	1

Ethnicity
American Indian or Alaska Native	0
Black (not of Hispanic origin)	1
Asian or Pacific Islander	0
Hispanic	1
White	7
Other	0
Unspecified	3

English as a Second Language

INTACT CLASSROOM SESSIONS 2013–2015
10/01/2013	Lincoln School (Brookline Public Schools)
10/05/2013	Simmons College Gordon College Salem State University
10/08/2013	Wheaton College
10/18/2013	University of Massachusetts - Boston
10/31/2013	Wheelock College
01/30/2014	Wheaton College
02/01/2014	Gordon College
02/04/2014	Elms College
02/05/2014	Salem State University
02/06/2014	University of Massachusetts - Amherst
02/11/2014	Lincoln School (Brookline Public Schools)
02/14/2014	Wheelock College
02/19/2014	Worcester State University
04/25/2015	Cambridge College
04/27/2015	Salem State University
05/02/2015	Simmons College Framingham State University
05/05/2015	Westfield State University
05/09/2015	Gordon College

STAND ALONE EVENTS AT MTEL OPERATIONAL TEST ADMINISTRATIONS

10/26/2013
03/01/2014
05/10/2014
07/12/2014
10/25/2014

English as a Second Language

Multiple-Choice Items (MCQs)

Number of MCQs Prepared	336
Number of MCQs Piloted*	209
Number of MCQs Identified for Further Review	40
Number of MCQs Deleted Post Pilot Test Review	11

Open-Response Items (ORIs)

Number of ORIs Developed	23
Number of ORIs Piloted*	18
Number of ORIs Identified for Further Review	1
Number of ORIs Deleted Post Pilot Test Review	1

*This chart includes information regarding items piloted prior to the first operational administration and may be a subset of all items developed. Additional multiple-choice items are piloted as non-scorable on operational test forms, until all items are piloted.

English as a Second Language

Number of Qualifying Score Participants: 12

Number of Participants Providing Ratings: 12 (100%)

Question [Rating Scale: 1 (not at all well) to 5 (very well)]	Mean Rating
How well do you feel you understood the training that you received for making your qualifying score judgments?	4.9
How confident are you of the qualifying score judgments you made?	4.7
How satisfied are you that you were able to complete the work you were asked to do in the time that was provided?	4.8
How successful were the coordination and logistics of the qualifying score conference?	4.6
Overall, how satisfied are you with the qualifying score process in which you participated?	4.8

Sheltered English Immersion

	Number
Committee Participants: Total	27

Public School Educators	21
Higher Education Faculty	6

Gender
Female	26
Male	1

Ethnicity
American Indian or Alaska Native	0
Black (not of Hispanic origin)	1
Asian or Pacific Islander	2
Hispanic	3
White	9
Other	1
Unspecified	11

Sheltered English Immersion

	Number
Committee Participants: Total	18

Public School Educators	14
Higher Education Faculty	4

Gender
Female	12
Male	6

Ethnicity
American Indian or Alaska Native	0
Black (not of Hispanic origin)	6
Asian or Pacific Islander	1
Hispanic	3
White	3
Other	5
Unspecified	0

Deaf/Hard of Hearing
Yes	3
No	15

Sheltered English Immersion

	Number
Committee Participants: Total	13

Public School Educators	11
Higher Education Faculty	2

Gender
Female	13
Male	0

Ethnicity
American Indian or Alaska Native	0
Black (not of Hispanic origin)	0
Asian or Pacific Islander	2
Hispanic	1
White	5
Other	0
Unspecified	5

Sheltered English Immersion

INTACT CLASSROOM SESSIONS 2013–2014
10/05/2013	Gordon College
10/18/2013	University of Massachusetts - Boston
10/19/2013	Brookline School District
10/31/2013	Wheelock College
02/17/2014	Clarion Hotel, Northampton
04/02–14/2014	Evaluation Systems, Hadley
07/12, 07/16–18/2014	Evaluation Systems, Hadley

STAND ALONE EVENTS AT MTEL OPERATIONAL TEST ADMINISTRATIONS

10/26/2013
03/01/2014

Sheltered English Immersion

Multiple-Choice Items (MCQs)

Number of MCQs Prepared	185
Number of MCQs Piloted*	117
Number of MCQs Identified for Further Review	22
Number of MCQs Deleted Post Pilot Test Review	3

*This chart includes information regarding multiple-choice items piloted prior to the first operational administration and may be a subset of all items developed. Additional items are piloted as non-scorable on operational test forms, until all items are piloted.

Open-Response Items

In consultation with the Massachusetts Department of Education and the Department of Justice review, the open-response item for the Sheltered English Immersion test was designed as a single, five-part open-response item requiring candidates to show their knowledge of sheltering content. Candidates use one of 10 mentor texts for the basis of their response. Each mentor text is an informational text that is representative of the kind of content and academic language features a student might encounter in texts in a given content area. Candidates are expected to spend about 20–30 minutes to prepare an adequate response to each of the five parts of the assignment, for a total of about 2–2 ½ hours for the section. The multiple-choice section counts for 60 percent and the open-response section counts for 40 percent of the candidate's total test score.

Piloting of the new open-response item type focused on evaluating the item format. Pilot test responses to the open- response item format were reviewed for the following characteristics:

Candidate ability to complete the assignment
Testing time required to complete the assignment
Administration process and computer screen layout
Application of the proposed score scale to the responses

A review of the data and item comments collected indicated that the candidates were able to appropriately and adequately respond to the new open-response item type and were able to produce a response within the targeted response time of 2 ½ hours, leaving sufficient time to respond to the multiple-choice questions. In addition, candidates were able to move throughout the various open-response item components with relative ease.

Sheltered English Immersion

Number of Qualifying Score Participants: 13

Number of Participants Providing Ratings: 13 (100%)

Question [Rating Scale: 1 (not at all well) to 5 (very well)]	Mean Rating
How well do you feel you understood the training that you received for making your qualifying score judgments?	5.0
How confident are you of the qualifying score judgments you made?	4.4
How satisfied are you that you were able to complete the work you were asked to do in the time that was provided?	4.7
How successful were the coordination and logistics of the qualifying score conference?	4.8
Overall, how satisfied are you with the qualifying score process in which you participated?	4.9