Development of a Mechanical Engineering Test Item Bank to promote learning outcomes-based education in Japanese and Indonesian higher education institutions

Jeffrey S. Cross, Estiyanti Ekawati, Satoko Fukahori, Shinnosuke Obi, Yugo Saito, Nathanael P. Tandian, Farid Triawan

Abstract


Following on the 2008-2012 OECD Assessment of Higher Education Learning Outcomes (AHELO) feasibility study of civil engineering, in Japan a mechanical engineering learning outcomes assessment working group was established within the National Institute of Education Research (NIER), which became the Tuning National Center for Japan. The purpose of the project is to develop among engineering faculty members, common understandings of engineering learning outcomes, through the collaborative process of test item development, scoring, and sharing of results. By substantiating abstract level learning outcomes into concrete level learning outcomes that are attainable and assessable, and through measuring and comparing the students’ achievement of learning outcomes, it is anticipated that faculty members will be able to draw practical implications for educational improvement at the program and course levels. The development of a mechanical engineering test item bank began with test item development workshops, which led to a series of trial tests, and then to a large scale test implementation in 2016 of 348 first semester master’s students in 9 institutions in Japan, using both multiple choice questions designed to measure the mastery of basic and engineering sciences, and a constructive response task designed to measure “how well students can think like an engineer.” The same set of test items were translated from Japanese into to English and Indonesian, and used to measure achievement of learning outcomes at Indonesia’s Institut Teknologi Bandung (ITB) on 37 rising fourth year undergraduate students. This paper highlights how learning outcomes assessment can effectively facilitate learning outcomes-based education, by documenting the experience of Japanese and Indonesian mechanical engineering faculty members engaged in the NIER Test Item Bank project.

First published online: 30 November 2017


Keywords


learning outcomes; quality assurance; educational improvement; mechanical engineering; Japan; Indonesia; test item bank

Full Text:

PDF

References


Arum, Richard, Josipa Roksa, and Amanda Cook. Improving Quality in American Higher Education: Learning Outcomes and Assessments for the 21st Century. San Francisco: Jossey-Bass, 2016.

Anderson, Lorin W., and David Krathwhol. A Taxonomy for Learning, Teaching, and Assessing. New York: Addison Wesley Longman, Inc., 2001.

Association of American Colleges and Universities. “LEAP Essential Learning Outcomes.” https://www.aacu.org/leap/essential-learning-outcomes.

Birtwistle, Tim, Courtney Brown, and Robert Wagenaar. “A long way to go…A study on the implementation of the learning-outcomes based approach in the EU and USA.” Tuning Journal for Higher Education 3, no. 2 (2016): 429-463. doi: 10.18543/the-3(2)-2016pp429-463.

European Network for the Accreditation of Engineering Education. “EUR-ACE Framework Standards and Guidelines (EAFSG).” http://www.enaee.eu/wp-assets-enaee/uploads/2012/02/EAFSG-Doc-Full-status-8-Sept-15-on-web-fm- Denis.pdf.

Group of National Experts on the AHELO Feasibility Study. “Engineering Assessment Framework.” (AHELO Feasibility Study 8th meeting of the AHELO GNE Paris, 18-19 November 2011). http://search.oecd.org/officialdocuments/displaydocumentpdf/?cote=edu/imhe/ahelo/gne(2011)19/ANN5/FINAL&doclanguage=en

Fukahori, Satoko, ed. “An Analytical Report of the Results of AHELO Feasibility Study” (Tokyo: National Institute for Educational Policy Research), original title in Japanese:「AHELO調査結果の分析に関する研究会(研究成果報告書)」(pronounced as follows:“AHELO Chosa-kekka-no-bunseki-ni-kansuru-kenkyukai (Kenkyu-seika-hokokusho)”). http://www.nier.go.jp/koutou/ahelo/2014/report.pdf.

Fukahori, Satoko. “Competence-based Quality Assurance of University Education - Lessons Learnt from the OECD-AHELO Feasibility Study.” Educational Studies in Japan: International Yearbook 8 (2014):9-61.

Hirota, Teruyuki. “Why Front Line Academics Reject Reform – Towards Utilizing Subject Specific Benchmark Statements.” University Evaluation Studies 15 (2016): 37-46.

International Engineering Alliance. “Graduate Attributes and Professional Competencies.” http://www.ieagreements.org/assets/Uploads/Documents/Policy/Graduate-Attributes-and-Professional-Competencies.pdf.

Japan Accreditation Board of Engineering Education. http://www.jabee.org/english/.

Lumina Foundation. “Degree Qualifications Profile.” https://www.luminafoundation.org/files/resources/dqp.pdf.

Ministry of Education, Culture, Sports, Science and Technology (MEXT) Central Council for Education. “Towards the Construction of Bachelor Degree Programs” (Tokyo: MEXT, 2008); original title in Japanese:「学士課程教育の構築に向けて(答申)」(pronounced as follows:“Gakushikatei-kyoiku-no-kochiku-ni-mukete (Toshin)”). http://www.mext.go.jp/b_menu/shingi/chukyo/chukyo0/toushin/1217067.htm.

National Institute for Academic Degrees and University Evaluation (NIAD-UE). “A Research on the Quality Assurance of Disciplinary Education in Japanese Universities” (Tokyo: NIAD-UE, 2016); original title in Japanese 「我が国における大学教育の分野別質保証の在り方に関する調査研究報告書」(pronounced as: “Wagakuni-niokeru-daigaku-kyoiku-no-bunyabetsu-sanshokijun-no-arikata-ni-kansuru-chosa-kenkyu”). http://www.niad.ac.jp/n_shuppan/project/__icsFiles/afieldfile/2016/07/14/no09_nr16-0714.pdf.

National Institute for Educational Policy Research (NIER). “Tuning Test Item Bank Project.” http://www.nier.go.jp/tuning/centre/kikai.html.

_____. “The National Institute for Educational Policy Research International Symposium for Educational Reform 2013: The TUNING-AHELO Global Quality Assurance through Sharing Competence Frameworks and Degree Level Specifications” (Tokyo: NIER, 2013). http://www.nier.go.jp/06_jigyou/symposium/i_sympo25/.

National Research Council. Engineering Tasks for the New Century: Japanese and U.S. Perspectives (Washington, DC: The National Academies Press, 1999), https://doi.org/10.17226/9624.

Organization of Economic Cooperation and Development (OECD). “A Tuning- AHELO Conceptual Framework of Expected Desired/Learning Outcomes in Engineering.” OECD Education Working Papers, 60 (2011), OECD Publishing, Paris. http://dx.doi.org/10.1787/5kghtchn8mbn-en.

_____. “OECD-AHELO Feasibility Study Report Volume 2 – Data Analysis and National Experiences.” (Paris: OECD, 2013). http://www.oecd.org/edu/skills-beyond-school/AHELOFSReportVolume2.pdf.

Quality Assurance Agency (QAA). “The UK Quality Code for Higher Education Subject Benchmark Statements.” http://www.qaa.ac.uk/assuring-standards-and-quality/the-quality-code/subject-benchmark-statements.

Science Council of Japan. “Benchmark Statements for Curriculum Design for Subject- Specific Quality Assurance in University Education.” (Tokyo: Science Council of Japan, 2012); original title in Japanese「大学教育の分野別質保証のための教育課程編成上の参照基準について」(pronounced as “Daigaku-kyoiku-no-bunyabetsu-shituhosho-no-tame-no-kyoiku-katei-henseijono-sanshokijun-ni-tsuite”). http://www.scj.go.jp/ja/member/iinkai/daigakuhosyo/daigakuhosyo.html.

Tremblay, Karine, Dianne Lalancette, and Deborah Roseveare. “OECD-AHELO Feasibility Study Report Volume 1 - Design and Implementation.” (Paris: OECD, 2012). http://www.oecd.org/edu/skills-beyond-school/AHELOFSReportVolume1.pdf.

Tuning Educational Structures in Europe. “Subject Areas.” http://www.unideusto.org/tuningeu/subject-areas.html.

Wiggins, Grant, and Jay McTighe. Understanding by Design, expanded 2th ed. (Alexander, Virginia: Association for Supervision & Curriculum, 2005).




DOI: http://dx.doi.org/10.18543/tjhe-5(1)-2017pp41-73

Refbacks

  • There are currently no refbacks.