» Accreditation » National Accreditation » Appendix
The goal of the Internal Audit (Appendix D) is to provide a look at the real-world impact of policies and procedures set forth in Appendices A, B and C. In order, the content of Appendices A, B and C are:
As the admissions and monitoring processes, efforts to support completers, and dimensions of program capacity have already been described in the preceding appendices, the goal of the Internal Audit is to describe the internal quality control processes used to evaluate and improve program quality. To audit these processes, a purposeful sample of candidates was taken from completers/candidates from different academic programs and with different experiences and outcomes. Policies outlined in Appendices A-C were reviewed in light of this sample of students and discrepancies discussed. This analysis was used to provide feedback for program improvement during a Quality Assurance Team meeting.
Participants engaged in this process both in planning and action phases. The planning process was mainly completed by the Dean and School Director of the School of Education, the Assistant Director of Teacher Education, and the Associate Director of Accreditation and Assessment under the office of Institutional Research and Effectiveness (IRE). The main discussion centered on how to choose a sample that was representative of a broad variety of programs and also individual candidates who would have encountered issues with various policies and procedures. Once programs and procedures were chosen, individual candidates were chosen for the sample with collaboration between the Associate Director of Accreditation and Assessment, the Director of Field Experience, the Graduate Coordinator of Programs, and the Teacher Education Student Services Coordinator. These individuals cooperated to identify candidates for the sample and gather data and information regarding their experiences.
The resources needed for this analysis included data analysis skill, time and effort, and a functioning quality assurance system with trustworthy and reliable data. The Northwest School of Education is blessed to have these resources.
Once candidates were chosen for the sample, the Associate Director of Accreditation and Assessment sought data from several sources. These included data recorded and submitted to the Department of Elementary and Secondary Education (DESE) in the form of the Annual Performance Report (APR). The APR outlines data points for teacher and leader program completers, including the Missouri Educator Evaluation System (MEES), average GPAs, content course GPAs (in the case of candidates seeking secondary and K-12 teaching certifications) and the certifications the candidates are seeking. Beyond this, records from candidate interactions with the Teacher Education Admissions Committee (TEAC) had to be requested for some students from the Teacher Education Student Services Coordinator, and records from the Teacher Education Guidance Committee (TEGC) were requested from the Director of Field Experience. Finally, the Associate Director of Accreditation and Assessment accessed placement data from DESE program completers to indicate whether candidates were currently employed in Missouri public education.
Inputs for this process included sampled candidates, their performance data points from the APR, assessments required for admissions to their educator preparation program, and for entry into student teaching. Also, if relevant, their interactions with Teacher Education Admissions Committee (TEAC) and the Teacher Education Guidance Committee (TEGC), as needed, were included. Finally, placement data was provided by the Missouri Department of Elementary and Secondary Education, who track graduates employed by public schools in Missouri.
The process activities included the Associate Director of Accreditation and Assessment collecting data and information and then reviewing how the candidate was assessed, coached and treated at each gateway of program admittance and/or retention. Outputs indicated that candidates were effectively guided through the program and that the data used came from available sources. At times, the data was challenging to acquire and the program should explore process improvements to expedite data access and acquisition time.
While the process was implemented mainly by the Associate Director of Accreditation and Assessment, the process was reviewed by the Quality Assurance Team (QAT). The QAT provided feedback on the validity and fidelity of the process utilized. Also, the QAT gave feedback on how this process could be modified in the future. Finally, combined feedback from both the QAT and the Dean and School Director of the School of Education was passed back to programs and committees on how their processes could be modified to ensure fairness and how discrepancies should be handled.
Monitors and measures were used to indicate effectiveness. These included monitoring by TEAC to ensure that candidates who were admitted conditionally into the teacher education program did eventually pass all requirements needed for official admittance. All education students who wished to be admitted to teacher education had to fulfill all criteria as outlined in Appendix A. Since 2002, Northwest has used the TEAC committee to support student retention, success, and decision quality for admittance and continuation- the TEAC committee represents a strong component of a robust quality assurance system.
All candidates who applied for student teaching submit an application which includes a copy of their Degree Audit and resume. This application was reviewed by their advisor, the registrar’s office, the Teacher Education Student Services Coordinator and the Director of Field Experiences. Also, the Teacher Education Guidance Committee (TEGC) monitored candidates who experienced issues in clinical practice, including student teaching. TEGC ensured that candidates received Educator Improvement Plans (EIP), if needed, and that candidates who did not meet the expectations put forth by an EIP were asked to leave student teaching.
The sample chosen included 10 candidates. Eight of these candidates were seeking a teacher certification and two were seeking principal certifications. These included candidates who were seeking certification at the levels and content areas of: Early Childhood, Elementary, Middle School, K-12, Secondary Education, Special Education, and Educational Leadership. One of the candidates was seeking three certifications, including Elementary Education, Early Childhood, and Early Childhood Special Education. One teacher education candidate was initially admitted in the alternative certification program. Of the two Educational Leadership candidates, one completed the “standard” program and another completed the “expedited” online program that allowed them to complete their degree within a year.
Of these students, seven encountered issues upon admission to teacher education and were therefore referred to the Teacher Education Admissions Committee (TEAC). All of these candidates were eventually admitted to teacher education. Two other candidates had issues during their student teaching experiences and encountered the Teacher Education Guidance Committee (TEGC). Of these two candidates, both were asked to leave student teaching. One returned the following semester and had a successful outcome and completed student teaching. One did not.
Of all of these candidates, seven of them were placed successfully according to DESE in a Missouri public school in the content area they sought. This data did not include candidates who were education professionals either out of state or in a private institution. The three not included in placement data in their content areas were: the candidate who encountered TEGC and did not successfully complete their student teaching, and the two candidates who completed their Educational Leadership programs. Regarding Educational Leadership, this is often typical. Many times a candidate seeks these degrees in preparation for a future, potential position in leadership.
Regarding diversity data of placements, Northwest identifies the following as diverse placements, based on the demographics of Missouri school buildings where candidates have student taught prior to 17-18 academic year. Using DESE demographic data, these buildings were rated according to the percentage of: non-White students; English Language Learners; students with an IEP; and students eligible for free or reduced lunch. Based on those rankings, school buildings in the highest quartile for each measure were designated as “diverse” in that area. For instance, the following designations for diverse placements were recognized:
The results of the Internal Audit based on all 10 candidates are listed below.
Candidate 1 was an Elementary Education major with minors in Early Childhood and Early Childhood Special Education. Candidate 1 was admitted to teacher education in the Fall 2016 semester, and completed student teaching in the Fall 2017 semester. According to DESE placement data, Candidate 1 has been teaching at the Elementary level for two years.
Candidate 1 submitted an appeal to TEAC after applying for admission to teacher education in the Spring 2016 semester. Candidate 1 applied without having passed the Missouri General Education Assessment (MoGEA) or completing the Missouri Educator Profile (MEP). The MoGEA is a general education assessment comprised of four subtests covering reading, writing, math, social studies and science. The MEP, although no longer used, at the time was an assessment used to determine whether a candidate met certain workstyles inventory scores that were normalized by experienced teachers. Both of these were part of the Missouri Educator Gateway Assessment (MEGA) suite of assessments required by DESE during different milestones of candidate educational experiences. Candidate 1 did, in fact, complete and pass the MoGEA and the MEP and was approved by the TEAC committee for admittance into teacher education. For admission to teacher education, the candidate passed every requirement for the Professional Education Unit Handbook as outlined as 2016-2017 or older.
Regarding advising, right before admission to teacher education in Fall 2016, Candidate 1 was advised to take classes specifically related to the major of Early Childhood, such as Early Childhood Assessment and Curriculum and Methods in Early Childhood. After admission, Candidate 1 took practicum courses in two consecutive semesters (Spring and Summer 2017), before completing student teaching in Fall 2017. All courses were taken at the Maryville campus.
Criteria for gateways were followed throughout this candidate’s experiences between admission to teacher education and admission to student teaching. While the diversity of field experiences was not tracked at that time, the candidate did engage in 33 diversity hours. These included seven hours of tutoring ELL students.
The criteria for admission to student teaching include: a 2.75 cumulative GPA; a 3.00 content course GPA; a 3.00 professional education course GPA; candidate must have attempted the Missouri Content Assessment in their content area prior to student teaching; and the candidate must have completed a student teaching application that was reviewed and approved by their advisor, the registrar, the Teacher Education Student Services Coordinator and the Director of Field Experience. Candidate 1 met all requirements.
Regarding the student teaching experience itself, this student had two placements. One was at Alfred Renner Elementary School in Park Hill, which is diverse regarding special education. The second was at Park Hill High School, which is considered diverse regarding race and ethnicity. The supervisor was Dee Moore.
Candidate 1 met all graduation requirements. These included: ensuring that all delayed grades were removed; completing 30 diversity hours; applying for graduation one semester prior to graduation in the Registrar’s Office; and maintaining a 2.75 GPA, a 3.0 content course GPA, and 3.0 professional education course GPA.
Candidate 1 is currently teaching at the elementary level in the North Kansas City school district. She has taught for two years in a school that is identified as being diverse regarding SES and percent of students with an IEP.
Candidate 2 was a Middle School Education candidate who graduated in Spring 2018. The candidate was admitted to teacher education in June, 2017 after having originally applying in the Fall 2016 semester. At the time of initial application, the candidate was not admitted due to having not passed the MoGEA social studies/science subsection and having a content area GPA below 3.00. In this case we were able to enact admission and monitoring processes linked to candidate success through TEAC. After petitioning to TEAC to request admittance, the candidate was admitted to teacher education.
While Candidate 2 was a Middle School major, their content area was math. This candidate completed math content courses in the spring of 2017, prior to admission. These included Math Methods for Middle School Teachers and Algebra and Geometry for the Elementary and Middle School Teacher. Once admitted, in fall 2017 the candidate took more courses focused on teaching, including Teaching Classroom Management and Discipline and Multiculturalism in Education. The student completed 34 diversity hours.
Finally, the candidate was admitted for student teaching in Spring 2018. The candidate met all criteria for admission to student teaching.
During student teaching, the candidate had one placement in Underwood Middle School, Iowa. Using the current designations for diverse placements, this placement was not identified as diverse regarding race or ethnicity, ELL status, or IEP. Free and reduced lunch statistics for Iowa schools is not currently available. The supervisor for this placement was Trudy Evans.
Candidate 2 is currently employed as a middle school mathematics teacher at Bode Middle School. This school is considered diverse according to student race/ethnicity and English Language Learners.
Candidate 3 was a double major in English and Spanish Education. This candidate also had issues while applying for admission into the teacher preparation program. Candidate 3 originally applied in the Spring 2016 semester. However, the candidate had not completed Practicum I or Computers and Information Technology. In this case the program was again able to demonstrate admission and monitoring processes linked to candidate success supported by strong advisement and processes, including TEAC, which played a vital role in the student's success. Candidate 3 was approved for entry by TEAC, and then officially admitted in the Spring 2017 semester.
This candidate received their instruction at the Maryville campus. This candidate proceeded quickly after admittance, completing 17 credit hours in the Fall 2017 semester. These included Multiculturalism in Education and Secondary Teaching Practicum II. The candidate’s GPA from that semester was a 4.0. These were accomplished quickly so that student teaching could be completed in the Spring 2018 semester. This candidate also completed a total of 33 diversity hours before student teaching.
The candidate did meet criteria for student teaching, including having attempted (but not passing at the time) the Spanish Missouri Education Content Assessment. During student teaching the university supervisor was Pam Raffurty. The candidate completed student teaching at Lee’s Summit West High School, which is considered diverse regarding race and ethnicity. The final assessment during student teaching this semester was still the Missouri Pre-Service Teacher Assessment (MoPTA). Candidate 3 did pass this with a score of 49 (passing being 37 or above).
Candidate 3 graduated in the Spring 2018 semester and passed both the English Education and Spanish Education Missouri Content Assessments (although both took two attempts to pass). The candidate is currently teaching at Lafayette High School in St. Joseph, Missouri. This school is considered diverse according to race/ethnicity of students, English Language Learners, and students with an IEP.
Candidate 4 was a major in Biology Education. This candidate petitioned for admission to teacher education in Spring 2016 as an alternative admission. This was done because the candidate had not completed a Computers and Information Technology course. TEAC denied this alternative admission, so the candidate petitioned for a blocked class petition, which was approved. This candidate was accepted officially at the beginning of Fall 2016.
Regarding assessments, this candidate performed well. The MoGEA was passed with exceedingly high scores. All subtests were passed with scores at least 20 points above the suggested cut scores of 220 for each assessment.
The candidate began teacher education in Fall 2016 and completed 15 hours successfully. These included Methods in Secondary School Science, and Secondary Teaching Practicum II. The semester was a success academically, with 13 hours of A and two hours of B. Candidate 4 also completed 33 diversity hours before student teaching.
The candidate completed all requirements for student teaching successfully. This included passing the Biology Education Missouri Content Assessment on the first attempt with a score of 243 (220 required for passing). However, after student teaching began in Spring 2017, problems occurred.
The cooperating teacher in Spring 2017 from Raymore-Peculiar High School reported serious issues to the university supervisor. These included having lesson plans unprepared, not knowing content, and incorrectly performing dissections during a biology course. The issues were so severe that the cooperating teacher had to intervene. The content supervisor observed the candidate and confirmed these issues. These issues were referred to the Teacher Education Guidance Committee (TEGC). The TEGC met and reported to the candidate that they needed to make significant improvements to instruction quickly or face removal from the program. Candidate 4 responded by text that they were leaving student teaching and withdrawing from the program.
However, in the Fall 2017 semester, the principal at Albany High School contacted Northwest about Candidate 4. The Albany principal wanted to hire the candidate as a teacher of record. The TEGC met again and discussed the matter. The candidate was given an opportunity to restart the student teaching process again with some caveats:
The improvements were dramatic. The following is an excerpt from an email sent by Candidate 4’s university supervisor.
"I wanted to update both of you on (Candidate 4). He is doing quite well and continues to improve and get stronger with each observation. I have observed him 3 times this semester; the last observation on Thursday, November 16 with his Environmental Science class." the letter continued:
"Each time I have observed him, I check to see how he is doing on completing his Educator Improvement Plan that we wrote together last August. I feel very comfortable with signing off on this EIP at the end of this semester."
EIP development and implementation demonstrates support for completer entry into and/or continuation in their professional role, as appropriate to the credential or degree being earned, and thus supports AAQEP standard 4.3.
Following up on the issue with Candidate 4’s university supervisor and continued monitoring after completion demonstrates a programmatic ability to investigate available and trustworthy evidence regarding completer placement, effectiveness, and retention in the profession, supporting AAQEP standard 4.4
In Spring 2018, Candidate 4 successfully passed the MoPTA with a score of 48 (passing score of 37). This candidate has completed two years of employment as a biology teacher at Albany High School, with the first year including student teaching. This school is currently considered diverse in regards to SES (students eligible for free or reduced lunch).
Candidate 5 was an example of someone who encountered TEGC with an unsuccessful result. In contrast to Candidate 4, this candidate left the program and did not return. Candidate 5 was admitted to teacher education in the Fall 2014 semester. While the candidate had to retake the MoGEA Math and English subsections, all criteria for admission were followed.
From that point, the candidate continued through the teacher preparation program. Candidate 5 received A’s and B’s, except for a D in Math Methods for Elementary Teachers. Perhaps this was a sign of issues to come. Candidate 5 retook Math Methods and received an incomplete, and finally a B. This student completed 31 diversity hours. These included teaching students with disabilities how to play soccer.
The candidate was them admitted into student teaching for the Fall 2015 semester. The candidate attempted the Elementary Education Missouri Content Assessments and all other student teaching application criteria were met as well. The candidate was placed in Richardson Elementary, which is not considered diverse using any metric (race/ethnicity, ELL, SES or IEP).
The second formative assessment by the university supervisor included the following notes (Candidate’s name replaced by “Candidate 5”).
Mrs. Mathews reports that (Candidate 5) is making a little bit of progress. I observed a few times when I could definitely tell when (Candidate 5) was trying to use more expression when she was reading aloud, but then would soon revert back to a flat tone and expression. Continued work is needed here as well as taking opportunity to provide more visual representation during instruction and making sure instructions are clear and explicit. Mrs. Mathews reports there are still times when she has to intervene and correct misteaching. An example, yesterday (Candidate 5) charted short vowel sound words and included we, ride and bike. I am very concerned about (Candidate 5)'s ability to provide correct and effective instruction outside the co-teaching structure.
After three formative assessments with major issues reported related to instruction and content, and an Educator Improvement Plan that was developed between the candidate and the supervisor, the case was given to the TEGC. The TEGC met, and discussed the situation with the candidate. The candidate was dismissed from student teaching by the TEGC. The Field Experience Director reported that Candidate 5 was not surprised by the decision. The candidate was offered an opportunity to take a grade of D for student teaching and still graduate without the ability to be recommended for teacher certification. The candidate decided to take this opportunity. Candidate 5 is not currently placed as a teacher in a Missouri public school and has not received a certification to teach anywhere else in Missouri.
The Director of Field Experience indicated that this was an issue that could have been dealt prior to student teaching through appropriate assessments of dispositions. He reported that this candidate just did not have the disposition to be an educator. Northwest hopes that we have addressed that issue sufficiently.
Candidate 6 was a Special Education candidate who graduated after the Fall 2017 semester. This candidate was admitted in the Spring 2016 semester with all admission criteria being followed. Following admission, Candidate 6 took 18 credit hours in the Fall 2016 and 16 credit hours in the Spring 2017. These included Assessment in Special Education and Practicum in Special Education, in which the candidate earned an A and B, respectively. In addition, this candidate completed 33 diversity hours. These included visits to Fairmount Elementary, which is diverse in the areas of race/ethnicity, English Language Learners, and free or reduced lunch.
This candidate met all criteria for admission to student teaching for the Fall 2017 semester. Candidate 6 was placed in West Platte Elementary, which was not diverse according to any metric. On the MEES Summative, this candidate scored the highest level of performance on 10 out of the 16 items as assessed by the cooperating teacher, and on 15 of the 16 MEES items as scored by the university supervisor. The supervisor was Kim Siela. Candidate 6 also received a passing score of 48 on the MoPTA.
Following student teaching, the candidate graduated while meeting all criteria. Candidate 6 has taught for two years as a parent educator at the same school where student teaching took place. The candidate has passed the MoCA in the areas of elementary education and special education.
Candidate 7 was a STEM teacher candidate focusing on Physics and Chemistry, two high needs areas. This candidate was admitted to teacher education in Fall 2016, meeting all criteria and with no interaction with the TEAC committee. Assessment-wise, this candidate performed extremely well. In the MoGEA, all subtests were scored within 30 points or less of the perfect score of 300.
After admission, this candidate completed 18 credit hours in the Spring 2017 semester and 16 in Fall 2017. All courses these two semester were completed with a perfect 4.0 GPA. Candidate 7 completed 36 diversity hours and was admitted to student teaching in the Spring 2018 semester while meeting all criteria.
During student teaching, the candidate completed one placement at Van Horn High School. This school was considered diverse regarding student race and ethnicity, English Language Learners, and free and reduced lunch. The university supervisor was Pam Raffurty and Dr. Rena Smith was the content supervisor. Candidate 7 passed the MoPTA with a score of 47 (37 required for passing).
Candidate 7 passed the Missouri Content Assessments for Physics and Chemistry Education, for which Northwest prepared the candidate, and the Mathematics Education Content Assessment as well. The candidate was hired at Van Horn High School, where student teaching was completed. For two years the candidate has taught chemistry and AP chemistry at this school which is considered diverse according to student race and ethnicity, English Language Learners, and free and reduced lunch.
Candidate 8 was an alternative certification candidate who was seeking a degree in Vocal Music Education. This candidate petitioned for admission through TEAC during the Spring 2014 semester. A petition was required because a GRE score was not on file. The candidate’s petition was approved the same semester of Spring 2014. Because this candidate was considered an alternative certification candidate, no MoGEA scores were required.
Following a unique schedule, this candidate did not immediately begin taking classes in Summer 2014 or Fall 2014. However, Candidate 8 began taking alternative certification coursework in Fall 2016, through Spring 2017 and then Fall 2017. This candidate acted as the teacher of record for two years in a diverse school, completing 30 diversity hours in that manner.
Candidate 8 finished two years of teaching and coursework during the Fall 2017 semester. Two years of teaching and coursework are required of all alternative certification candidates. Candidate 8 passed the MopTA with a score of 43, while 37 was required to pass. This candidate also passed the Missouri Content Assessment for Instrumental and Vocal Music. For two years Candidate 8 has taught chorus, choir and vocal music at Savannah Middle School. This school is considered diverse regarding the percentage of students with an IEP.
Candidate 9 is an Educational Leadership program completer who finished a standard, two-year sequence. This candidate began graduate school in the Fall 2017 semester. In the Fall 2017 semester the candidate competed six credit hours, then 12 in the Spring 2018 semester, and finally six in the Summer 2018 semester. Coursework included Managerial School Leadership and Data Literacy for Education. Throughout coursework the candidate maintained a 4.0 GPA.
This candidate graduated in the Summer 2018 semester after completing all graduation criteria. Candidate 9 then took the Missouri Content Assessment for Building Administrators (again, a certification requirement), and passed on the second attempt. While the passing score is 220 or above, the candidate scored a 215 on a first attempt and 242 on the second in September, 2019. During the 2018-2019 academic year (before passing the certification assessment), Candidate 9 was still employed as a science teacher in Bode Middle School. This school is considered diverse in terms of race/ethnicity and students who are English Language Learners.
Candidate 10 is an Educational Leadership program completer who finished the expedited program. This candidate began graduate program coursework in Summer 2018 semester and completed in the Spring 2019 semester. Candidate 10 completed 12 hours in the Summer 2018 semester, 12 in Fall 2018 and six in Spring 2019. Throughout this expedited process the candidate maintained a 4.0 GPA. Courses included Innovative School Leadership and School Law.
In May 2019, Candidate 10 took and passed the Missouri Content Assessment for Building Leaders on the first attempt. The candidate achieved a score of 229 (a passing score is 220). Since the latest placement data is only available from the 2018-2019 academic year, DESE does not have placement data for this candidate yet.
Using a case study approach enabled a rich analysis of the interaction of our quality assurance systems and processes in light of candidate success, retention and quality. Of the 10 candidates selected, all were eventually admitted to a Northwest professional education program. All but one completed successfully. Placement data exists for seven of these candidates. Placement data was not found for the candidate that did not complete or for the Educational Leadership candidates who completed their programs in 2019, as we do not yet have that information.
Analysis of the data allowed several themes to emerge from this audit. The Teacher Education Admissions Committee functioned quite well. TEAC serves as an important support system. The appeals process offer candidates feedback on how to improve their performance before challenges become too advanced, or before the student falls too far behind to catch up academically, with standardized examinations, or in reference to professional dispositions and/or teaching practices. Candidates who were admitted, completed. The Teacher Education Guidance Committee, in addition, operated well in both its functions: removing two candidates who should not have been student teaching; and reinstating a candidate whose issues had been resolved and demonstrated the behaviors necessary for success. The major failing in regards to the candidate who had an unsuccessful TEGC outcome was the lack of a dispositions process prior to student teaching. With additional disposition checks and a new process as of 2019, Northwest administration and faculty hope this known program limitation has been remediated.
Moving forward, the data analysis provided two insights. First of all, there seemed to be a wealth of available data— perhaps an overabundance. This data covered everything from initial program acceptance, through student teaching, course performance and certification assessment scores. The addition of placement data for completers from DESE is a valuable addition, although we recognize its limitations (includes only in-state, public school placements) and seek complementary evidence for all completers. The one major limitation of this process seemed to be the number of data storage systems that needed to be accessed to collect all of this information. The assessment professional had to toggle between multiple excel sheets and several data warehousing or assessment programs (Banner, TK-20, Canvas, and several proprietary data systems) in order to gather all requisite data to complete the Internal Audit. In the future, we must simplify data storage, sharing, use, and analysis.
