Development of a Mechanical Engineering Test Item Bank to promote learning outcomes-based education in Japanese and Indonesian higher education institutions
Abstract
Following on the 2008-2012 OECD Assessment of Higher Education Learning Outcomes (AHELO) feasibility study of civil engineering, in Japan a mechanical engineering learning outcomes assessment working group was established within the National Institute of Education Research (NIER), which became the Tuning National Center for Japan. The purpose of the project is to develop among engineering faculty members, common understandings of engineering learning outcomes, through the collaborative process of test item development, scoring, and sharing of results. By substantiating abstract level learning outcomes into concrete level learning outcomes that are attainable and assessable, and through measuring and comparing the students’ achievement of learning outcomes, it is anticipated that faculty members will be able to draw practical implications for educational improvement at the program and course levels. The development of a mechanical engineering test item bank began with test item development workshops, which led to a series of trial tests, and then to a large scale test implementation in 2016 of 348 first semester master’s students in 9 institutions in Japan, using both multiple choice questions designed to measure the mastery of basic and engineering sciences, and a constructive response task designed to measure “how well students can think like an engineer.” The same set of test items were translated from Japanese into to English and Indonesian, and used to measure achievement of learning outcomes at Indonesia’s Institut Teknologi Bandung (ITB) on 37 risingfourth yearundergraduate students. This paper highlights how learning outcomes assessment can effectively facilitate learning outcomes-based education, by documenting the experience of Japanese and Indonesian mechanical engineering faculty members engaged in the NIER Test Item Bank project.
Published online: 30 November 2017
Downloads
References
Arum, Richard, Josipa Roksa, and Amanda Cook. Improving Quality in American Higher Education: Learning Outcomes and Assessments for the 21st Century. San Francisco: Jossey-Bass, 2016.
Anderson, Lorin W., and David Krathwhol. A Taxonomy for Learning, Teaching, and Assessing. New York: Addison Wesley Longman, Inc., 2001.
Association of American Colleges and Universities. “LEAP Essential Learning Outcomes.” https://www.aacu.org/leap/essential-learning-outcomes.
Birtwistle, Tim, Courtney Brown, and Robert Wagenaar. “A long way to go…A study on the implementation of the learning-outcomes based approach in the EU and USA.” Tuning Journal for Higher Education 3, no. 2 (2016): 429-463. doi: 10.18543/the-3(2)-2016pp429-463.
European Network for the Accreditation of Engineering Education. “EUR-ACE Framework Standards and Guidelines (EAFSG).” http://www.enaee.eu/wp-assets-enaee/uploads/2012/02/EAFSG-Doc-Full-status-8-Sept-15-on-web-fm- Denis.pdf.
Group of National Experts on the AHELO Feasibility Study. “Engineering Assessment Framework.” (AHELO Feasibility Study 8th meeting of the AHELO GNE Paris, 18-19 November 2011). http://search.oecd.org/officialdocuments/displaydocumentpdf/?cote=edu/imhe/ahelo/gne(2011)19/ANN5/FINAL&doclanguage=en
Fukahori, Satoko, ed. “An Analytical Report of the Results of AHELO Feasibility Study” (Tokyo: National Institute for Educational Policy Research), original title in Japanese:「AHELO調査結果の分析に関する研究会（研究成果報告書）」(pronounced as follows:“AHELO Chosa-kekka-no-bunseki-ni-kansuru-kenkyukai (Kenkyu-seika-hokokusho)”). http://www.nier.go.jp/koutou/ahelo/2014/report.pdf.
Fukahori, Satoko. “Competence-based Quality Assurance of University Education - Lessons Learnt from the OECD-AHELO Feasibility Study.” Educational Studies in Japan: International Yearbook 8 (2014):9-61.
Hirota, Teruyuki. “Why Front Line Academics Reject Reform – Towards Utilizing Subject Specific Benchmark Statements.” University Evaluation Studies 15 (2016): 37-46.
International Engineering Alliance. “Graduate Attributes and Professional Competencies.” http://www.ieagreements.org/assets/Uploads/Documents/Policy/Graduate-Attributes-and-Professional-Competencies.pdf.
Japan Accreditation Board of Engineering Education. http://www.jabee.org/english/.
Lumina Foundation. “Degree Qualifications Profile.” https://www.luminafoundation.org/files/resources/dqp.pdf.
Ministry of Education, Culture, Sports, Science and Technology (MEXT) Central Council for Education. “Towards the Construction of Bachelor Degree Programs” (Tokyo: MEXT, 2008); original title in Japanese:「学士課程教育の構築に向けて（答申）」(pronounced as follows:“Gakushikatei-kyoiku-no-kochiku-ni-mukete (Toshin)”). http://www.mext.go.jp/b_menu/shingi/chukyo/chukyo0/toushin/1217067.htm.
National Institute for Academic Degrees and University Evaluation (NIAD-UE). “A Research on the Quality Assurance of Disciplinary Education in Japanese Universities” (Tokyo: NIAD-UE, 2016); original title in Japanese 「我が国における大学教育の分野別質保証の在り方に関する調査研究報告書」(pronounced as: “Wagakuni-niokeru-daigaku-kyoiku-no-bunyabetsu-sanshokijun-no-arikata-ni-kansuru-chosa-kenkyu”). http://www.niad.ac.jp/n_shuppan/project/__icsFiles/afieldfile/2016/07/14/no09_nr16-0714.pdf.
National Institute for Educational Policy Research (NIER). “Tuning Test Item Bank Project.” http://www.nier.go.jp/tuning/centre/kikai.html.
_____. “The National Institute for Educational Policy Research International Symposium for Educational Reform 2013: The TUNING-AHELO Global Quality Assurance through Sharing Competence Frameworks and Degree Level Specifications” (Tokyo: NIER, 2013). http://www.nier.go.jp/06_jigyou/symposium/i_sympo25/.
National Research Council. Engineering Tasks for the New Century: Japanese and U.S. Perspectives (Washington, DC: The National Academies Press, 1999), https://doi.org/10.17226/9624.
Organization of Economic Cooperation and Development (OECD). “A Tuning- AHELO Conceptual Framework of Expected Desired/Learning Outcomes in Engineering.” OECD Education Working Papers, 60 (2011), OECD Publishing, Paris. http://dx.doi.org/10.1787/5kghtchn8mbn-en.
_____. “OECD-AHELO Feasibility Study Report Volume 2 – Data Analysis and National Experiences.” (Paris: OECD, 2013). http://www.oecd.org/edu/skills-beyond-school/AHELOFSReportVolume2.pdf.
Quality Assurance Agency (QAA). “The UK Quality Code for Higher Education Subject Benchmark Statements.” http://www.qaa.ac.uk/assuring-standards-and-quality/the-quality-code/subject-benchmark-statements.
Science Council of Japan. “Benchmark Statements for Curriculum Design for Subject- Specific Quality Assurance in University Education.” (Tokyo: Science Council of Japan, 2012); original title in Japanese「大学教育の分野別質保証のための教育課程編成上の参照基準について」(pronounced as “Daigaku-kyoiku-no-bunyabetsu-shituhosho-no-tame-no-kyoiku-katei-henseijono-sanshokijun-ni-tsuite”). http://www.scj.go.jp/ja/member/iinkai/daigakuhosyo/daigakuhosyo.html.
Tremblay, Karine, Dianne Lalancette, and Deborah Roseveare. “OECD-AHELO Feasibility Study Report Volume 1 - Design and Implementation.” (Paris: OECD, 2012). http://www.oecd.org/edu/skills-beyond-school/AHELOFSReportVolume1.pdf.
Tuning Educational Structures in Europe. “Subject Areas.” http://www.unideusto.org/tuningeu/subject-areas.html.
Wiggins, Grant, and Jay McTighe. Understanding by Design, expanded 2th ed. (Alexander, Virginia: Association for Supervision & Curriculum, 2005).
Authors are required to sign and submit a copyright transfer agreement after acceptance but before publication of their manuscript. To that effect, they receive, from the Managing Editor of Tuning Journal for Higher Education, a standard copyright assignment form designed along the following lines:
1. Authorship:
The author who signs the copyright transfer agreement must be the sole creator of the work or legally acting on behalf of and with the full agreement of all the contributing authors.
2. Copyright and Code of conduct:
a) Authors warrant that their work is original; has not been previously copyrighted or published in any form; is not under consideration for publication elsewhere; its submission and publication do not violate TJHE Ethical Guidelines for Publication and any codes (of conduct), privacy and confidentiality agreements, laws or any rights of any third party; and no publication payment by the Publisher (University of Deusto) is required.
b) Authors are solely liable for the consequences that may arise from third parties’ complaints about the submitted manuscript and its publication in Tuning Journal for Higher Education (TJHE).
c) Authors grant to the Publisher the worldwide, sub-licensable, and royalty-free right to exploit the work in all forms and media of expression, now known or developed in the future, for educational and scholarly purposes.
d) Authors retain the right to archive, present, display, distribute, develop, and republish their work (publisher's version) to progress their scientific career provided the original publication source (Tuning Journal) is acknowledged properly and in a way that does not suggest the Publisher endorses them or their use of the wortk.
e) Authors warrant that no permissions or licences of any kind will be granted that might infringe the rights granted to the Publisher.
3. Users:
Tuning Journal for Higher Education is an Open Access publication. Its content is free for full and immediate access, reading, search, download, distribution and reuse in any medium or format only for non-commercial purposes and in compliance with any applicable copyright legislation, without prior permission from the Publisher or the author(s). In any case, proper acknowledgement of the original publication source must be made and any changes to the original work must be indicated clearly and in a manner that does not suggest the author’s and or Publisher’s endorsement whatsoever. Any other use of its content in any medium or format, now known or developed in the future, requires prior written permission of the copyright holder.
1.jpg)
1.jpg)
.jpg)
1.jpg)
.jpg)
.jpg)
