Follow-up on Assessment of Student Outcomes by Senior-Year Design Project and Continuing to Improve by Performance Indicator Breakdown-Based Assessment

The study is a follow-up on rubric-based assessment of level of attainment of Student Outcomes (SOs) in Environmental Engineering undergraduate education by the senior-year Graduation Design Project (GDP): the focal points are the process of and results from incorporation of additional assessment tools and implementation of “Performance Indicator (PI)-breakdown” approach to continue improving the SO Assessment and Evaluation (A&E) process. For several consecutive cycles, A&E to define attainment level of total of seven SOs by the GDP gave results below the set thresholds for some of them (SO1,5,8), which indicated a discrepancy and a need for improvement not only in the assessment tools, but also in the A&E processes. Accordingly, two remedial actions were undertaken to meet those needs and the SOs were re-assessed. As the first remedial action, alternative assessment tools were incorporated to the SO A&E process. Assessment results revealed a clear progress in SO attainment, from 2014-15 to 2015-16. However, those lump-sum results still gave a general sense of students’ performance at SO-level. Therefore, some additional assessment tools were added and the “PI-breakdown”-based approach was implemented as the second remedial action. Those implementations enabled obtaining more realistic, detailed, and informative results and facilitated further fine tuning of the SO A&E process. Keywords—assessment and evaluation (A&E), engineering education, graduation design project, performance indicator (PI), problem/design-based learning, student outcome (SO), rubric


Introduction
Environmental Engineering Undergraduate Program (EEUP) [1] is among the 23 UPs of Istanbul Technical University (ITU) currently accredited by the ABET EAC (Engineering Accreditation Commission) [2].More than 50% of the total credits (minimum requirement for graduation) offered by the curriculum is comprised of those from the "Engineering Science" and "Engineering Design" courses, mainly given in the junior-and senior-years.Compulsory "Engineering Design" courses have a particular significance in the curriculum, since those are meant to serve for improving student attributes related to "Problem-Based Learning (PBL)", in addition to conveying program/content-specific knowledge to the students, as well as develop and sharpen their engineering design skills.
"Graduation Design Project (GDP)" offered in the senior-year, is the final engineering design course of the curriculum, and is designated for summative enhancement of the expected sum of gradually accumulated knowledge and skill-sets of the senior-year students right before graduation.Accordingly, educational objectives of this course span over a wide spectrum including but not limited to helping students improve their critical problem-solving skills and decision-making abilities, engage in active and collaborative/cooperative learning and develop self-learning strategies, engage in team-work, structure solutions to real-life problems etc.; all linked to PBL [3].Those objectives are closely related with the ABET EAC Student Outcomes (SOs) as well, and thus GDP has a significant role in assessing the level of attainment of several SOs by the EEUP.
In order to measure students' performance in the GDP and help determine the effectiveness of the education in developing and/or improving targeted student abilities and learning attributes, a detailed and comprehensive grading rubric was designed by the course coordinators in 2010-11 Spring, and has been in use since then with some modifications [4].Details of the GDP-specific grading rubric and the overall grading system are given elsewhere [5].Results of the extensive assessment process run in 8 consecutive terms between 2010-11 Spring and 2014-15 Fall, as well as recommendations for changes in the GDP-Rubric were also reported previously [5 and 6, respectively].As stated in those studies, the A&E process executed for several consecutive review cycles to define the level of attainment of SOs addressed by the GDP gave results below the set thresholds for some of them (e.g., SO1, SO5, SO8).Those results indicated a discrepancy and called for a new outlook.
Accordingly, the current study is a follow-up on rubric-based assessment of related SOs by the senior-year GDP, and includes (i) overall assessment results for level of attainment of SO3, SO5, SO8, and SO11 for 6.5 years and 13 consecutive cycles between 2010-11 and 2016-17 Spring, (ii) assessment results from the subsequent 2 runs in 2014-15 Spring and 2015-16 Fall with performance vector details for all seven SOs addressed by the GDP, (iii) the first remedial action executed by incorporating the newly introduced "Environmental Management Considerations (EMC)" into the GDP-specific grading rubric to improve the assessment tool (2015-16 Fall) and the obtained results, (iv) the second remedial action undertaken to continue improving the process through implementation of the "Performance Indicator (PI)-breakdown" based assessment to the SO A&E process and the obtained results (2015-16 Spring, 2016-17 Fall, and 2016-17 Spring), and (v) concluding remarks on evaluating the assessment results for further fine-tuning the SO A&E process and how those support improvement at course/program level.

2
Tools and actions

Student outcomes (SOs) addressed by the GDP
Student outcomes (SOs) addressed by the GDP at "[(3-full)]: Emphasized (Assessed and Evaluated)" level of contribution are listed below together with the corresponding ABET EAC notation (letters in brackets).
• SO1 (a) -An ability to apply knowledge of mathematics, science, and engineering • SO3 (c) -An ability to design a system, component, or process to meet desired needs within realistic constraints such as economic, environmental, social, political, ethical, health and safety, manufacturability, and sustainability • SO4 (d) -An ability to function on multidisciplinary teams • SO5 (e) -An ability to identify, formulate, and solve engineering problems • SO7 (g) -An ability to communicate effectively • SO8 (h) -The broad education necessary to understand the impact of engineering solutions in a global, economic, environmental, and societal context • SO11 (k) -An ability to use the techniques, skills, and modern engineering tools necessary for engineering practice

Assessment tools: period between 2010-11 and 2014-15 Spring terms
This period covers 9 consecutive terms during which abilities of 272 senior-year students (210 and 62 in spring and fall terms, respectively) were assessed and evaluated based on their individual-and/or team-performances regarding their GDP assignments.Tools used during that long period for assessing the level of attainment of the SOs addressed by the GDP were the same as those listed in Table 1 for 2014-15 Spring and are briefly described below: • Tools for SO1, SO5, SO8 A&E: OBEx (Outcome-Based Exam): specific questions addressing those SOs, which were prepared and asked in the "technical exam" given by the end of each term • Tool for SO3 A&E: GDP-Rubric: in the first four A&E runs, results from OBEx questions were used, whereas in the next five runs, assessment was carried out by using scores from the relevant parts of the GDP-Rubric • Tool for SO4 A&E: SO4-Rubric: an analytic rubric comprised of 4 PIs, which was designed by the assigned faculty (SO4-coordinator) for assessing students' performances in team-work • Tool for SO7 A&E: two analytic rubrics (W and O): each comprised of 6-8 PIs, which were designed by the assigned faculty (SO7-coordinator) for assessing students' abilities in written (W) and oral (O) communication, respectively • Tool for SO11 A&E: Drawings: scores collected by the student-teams from the "Technical drawings" chapters of their GDP final reports

Assessment tools: GDP-specific grading rubric
As stated above, a detailed and comprehensive grading rubric was structured and implemented by the GDP-coordinators in 2010-11 Spring, specifically to assess student performance on all features of the GDP assignment.Details of the rubric and the overall grading system are given elsewhere [5].Briefly, main sections of the rubric were related with "content quality and technicalities (18%)", "process and system design (60%)", "cost analysis (18%)", and "time and project management (4%)".GDP final reports, including detailed entries at each of those main-and related subsections, were assessed by the team-supervisors, the GDP-coordinators and the jury members.Additionally, individual student performances (e.g., contribution to progress of project and team-work, presentation skills during defense, scores in technical exam, etc.) were also assessed [6].Various sub-sections of the GDP-Rubric included queries related with the SOs addressed by the GDP, hence scores from those sections were used in the SO A&E.Accordingly, the GDP-Rubric was used both for grading and for SO A&E in 9 consecutive terms between 2010-11 and 2014-15 Spring.

1st remedial action: improving the assessment tools (2015-16 Fall)
ABET EAC's Program Criteria for "Environmental and similarly named engineering programs" have been revised recently (effective starting from 2015-16 and further) by addition of, e.g., the requirement that "the curriculum must prepare graduates to design environmental engineering systems that include considerations of risk, uncertainty, sustainability, life-cycle principles, and environmental impacts" [7].Those new entries of "Environmental Management Considerations (EMC)" were rapidly incorporated into the ITU EEUP curriculum in compliance with the accreditation requirements and included in the GDP assignments as of 2014-15 Spring term [8].The student-teams included their inputs on EMC in their final reports [8], yet their performance on completion of those new tasks were not assessed explicitly since entries related with the EMC were not added to the GDP-Rubric the same year [4,5,6,8].
The GDP-Rubric was initially designed as comprehensive as possible to meet the needs [4], however the abovementioned additions to the GDP assignment called for improving the rubric as well by inclusion of the relevant assessment tools (performance indicators and vectors) addressing assessment of students' performances regarding the new EMC titles.Hence, an improved version of the rubric (GDP-iRubric) was structured as a remedial action and recommended to be used for grading, as well as for SO A&E in the next run [6].Accordingly, the GDP-iRubric was implemented in 2015-16 Fall.

2nd remedial action: improving the assessment process (2015-16 Spring)
To provide a better insight to the SO A&E and enable determining at which particular aspects the senior-year students have strengths and weaknesses, it was required to provide a more elaborate and informative A&E process.Accordingly, a second reme-dial action -namely the "PI-breakdown" based assessment-was first implemented in 2015-16 Spring term and applied at the following cycles in 2016-17 Fall and Spring terms as well: supervisors of the student-teams were asked to assess students' performances both by using the GDP-iRubric and the other assessment tools, as well as by using the detailed analytic rubrics specifically designed for each SO addressed by the GDP.Data collected from those reports were used to obtain the detailed PIbreakdown based assessment outputs.

Results
Assessment results obtained from 13 consecutive review cycles in 2010-2017 and showing the overall level of attainment of some of the SOs addressed by the GDP are presented in Figure 1.For the period between 2010-11 and 2014-15 Spring terms, several SOs were determined as being attained successfully at levels significantly above the set thresholds (e.g., SO3 and SO11 (Figure 1)), whereas the A&E process at that period produced results below the set thresholds for some other SOs (e.g., SO5 and SO8 (Figure 1), SO1).However, when the features, objectives, content, and operation of the GDP, as well as the final product -the report prepared by the student teams-were considered, obtaining student performances below the set thresholds in those SOs seemed contradictory.Thus, some other assessment tools were included in the A&E process in 2015-16 Fall term (Table 1) and the improved version of the GDP-specific grading rubric (GDP-iRubric) was used in the assessment as well.
Performance vector details of the results attained by the previous [4,5] and the improved [4,6] SO A&E process for all seven SOs addressed by the GDP are exemplified in Table 1 through the data obtained in 2014-15 Spring and 2015-16 Fall terms, respectively.Assessment tools used/added in those review cycles are also included in the table to enable a better comparison.Similar to the case stated above, attainment levels of all seven SOs and performance vector details of several of them showed improvement in the 2015-16 Fall term upon selection and use of additional/alternative assessment tools.
Yet, those assessment results were still giving a general sense of students' performances at SO-level: even with the previous and improved assessment tools and results in hand (Figure 1 and Table 1), it was still not possible to identify at which particular dimensions, in other words -Performance Indicators (PIs)-comprising the explicit details of the SO A&E scheme, that the senior-year students performed below expectations.The lump-sum nature of those assessments were prone to be masking the points of weakness and strengths of the students, hence weakening the chances of taking appropriate corrective actions and verifying good practices.
Overall assessment of attainment level (at/above "satisfactory") of SO3 (a), SO5 (b), SO11 (c), and SO8 (d) by the GDP in 13 consecutive review cycles between 2010-11 and 2016-17 Spring terms.Horizontal lines show the set thresholds.See Table 1

14-15S
15-16F 14-15S • Tools used (before and) in 14-15S: "OBEx"-Outcome Based Exam, "GDP-Rubric"-Grading rubric for GDP, "SO4 and SO7"-rubrics specific for the related SOs, "Drawings"-Technical drawings chapter of GDP final report; • Scores given to each student by the Advisory Team in the SO4-Rubric; • Additional/improved tools used in 15-16F: "GDP-iRubric"-based on overall grades; • Additional/improved tools used in 15-16F: sum of scores obtained from (i) question asked in OBEx (60%) and (ii) technical drawings chapter of GDP final report graded in the GDP-iRubric (40%); • Students' perspective of their own performance regarding SO4: data from a mini survey (4 questions) given to senior-year students at OBEx; • Additional/improved tools used in 15-16F: sum of scores obtained from "Environmental Management Considerations" and "Cost Analysis" chapters of GDP final report (graded in the GDP-iRubric); • Additional/improved tools recently recommended and used in 15-16S, 16-17F, and 16-17 S: rubric-based assessment of all relevant SOs improved by the PIbreakdown based approach Accordingly, it became apparent that the aggregative measures of students' performances were required to be broken down to address individual SO-related PIs.Moreover, a genuine picture of the relevance, accuracy, and utility of the selected assessment tools was essential.
To overcome the shortcomings due to the lump-sum nature of the applied assessment procedure, the remedial action decision of applying "PI-breakdown" based assessment in the SO A&E process was implemented as of 2015-16 Spring term: the A&E system included detailed assessment of the students' abilities and attributes corresponding to each specific PI of each SO, assessed in accordance with four performance vectors and specific performance descriptors -together comprising the detailed analytic rubrics designed for each SO (Table 1).Detailed data were collected at 3 consecutive cycles in 2015-16 Spring, 2016-17 Fall, and 2016-17 Spring terms.Informative results obtained by applying the PI-breakdown based approach in SO A&E addressed by the GDP are presented in Figure 2 for SO3 and SO5, and in Figure 3 for SO8 and SO11.
The recently recommended and implemented "PI-breakdown" based assessment facilitated attainment of more realistic and meaningful results.Comparative evaluation of the A&E results showed that the overall attainment levels (at/above satisfactory) of the related SOs were well above the set thresholds in 2015-16 Spring, 2016-17 Fall, and 2016-17 Spring terms as well (Figure 2 and 3).More importantly, the PIbreakdown enabled determining the particular performance indicators at which students' abilities might be improved further: (i) there was a gradual enhancement in successful attainment of PI-2 (formulate problem) and a marked improvement in PI-3 (solve problem) of SO5, implying a good practice (Figure 3); whereas there was still some room for improving the students' abilities and attributes related to (ii) PI-3 (solutions) of SO3 (Figure 2), (iii) PI-2 (demonstrate impacts) of SO8 (Figure 3), and (iv) Conclusions and recommendations The presented long-term study exemplifies the Continuous Improvement (CI) Strategy of the ITU EEUP through the educational quality assurance actions taken in the recent years in the senior-year Graduation Design Project (GDP) course.SO A&E results from several early assessment cycles indicated a discrepancy and called for a new outlook.Accordingly, appropriate remedial action decisions were successfully implemented in the fall and spring terms of the 2015-16 and 2016-17 academic years and the SOs addressed by the GDP were re-assessed.Implementation of the "PIbreakdown" based assessment facilitated further fine tuning of the SO A&E process and made a much comprehensive and informative evaluation possible.
The 2016-17 academic year was determined by ITU's chief administration as the year for conducting university-wide curriculum design activities and revision of course plans.In addition to those efforts to develop integrated revisions applying to all engineering programs, program-specific curriculum design activities and revisions were also executed by each engineering program, including the ITU EEUP.Accordingly, "suggestion for changes" offered by the GDP A&E team, as well as "recommendation for changes" outlined by the SO-coordinators based on the informative and comprehensive results of the PI-breakdown based assessments were communicated to the related responsible bodies (i.e., Department Administration, Curriculum Development Committee, Accreditation Coordination Committee, etc.) for further discussion and evaluation prior to be directly implemented in the next run (course-level remedial action decisions), and/or to be communicated to the higher administrative units for discussion and approval (program/curriculum-level remedial action decisions).By this way, the informative outputs of the SO A&E process run through the GDP course were meant to contribute to the revision and improvement of the senioryear design project course and the EEUP curriculum.