Development of a Mechanical Engineering Test Item Bank to promote learning outcomes-based education in Japanese and Indonesian higher education institutions

  • Jeffrey S. Cross Tokyo Tech’s School of Environment and Society http://orcid.org/0000-0002-9672-2512
  • Estiyanti Ekawati Institut Teknologi Bandung
  • Satoko Fukahori National Institute for Educational Policy Research (NIER)
  • Shinnosuke Obi Keio University
  • Yugo Saito Kyoto University
  • Nathanael P. Tandian Institut Teknologi Bandung
  • Farid Triawan Tokyo Tech’s School of Environment and Society
Keywords: learning outcomes, quality assurance, educational improvement, mechanical engineering, Japan, Indonesia, test item bank

Abstract

Following on the 2008-2012 OECD Assessment of Higher Education Learning Outcomes (AHELO) feasibility study of civil engineering, in Japan a mechanical engineering learning outcomes assessment working group was established within the National Institute of Education Research (NIER), which became the Tuning National Center for Japan. The purpose of the project is to develop among engineering faculty members, common understandings of engineering learning outcomes, through the collaborative process of test item development, scoring, and sharing of results. By substantiating abstract level learning outcomes into concrete level learning outcomes that are attainable and assessable, and through measuring and comparing the students’ achievement of learning outcomes, it is anticipated that faculty members will be able to draw practical implications for educational improvement at the program and course levels. The development of a mechanical engineering test item bank began with test item development workshops, which led to a series of trial tests, and then to a large scale test implementation in 2016 of 348 first semester master’s students in 9 institutions in Japan, using both multiple choice questions designed to measure the mastery of basic and engineering sciences, and a constructive response task designed to measure “how well students can think like an engineer.” The same set of test items were translated from Japanese into to English and Indonesian, and used to measure achievement of learning outcomes at Indonesia’s Institut Teknologi Bandung (ITB) on 37 risingfourth yearundergraduate students. This paper highlights how learning outcomes assessment can effectively facilitate learning outcomes-based education, by documenting the experience of Japanese and Indonesian mechanical engineering faculty members engaged in the NIER Test Item Bank project.


Published online
: 30 November 2017

Downloads

Download data is not yet available.

Author Biographies

Jeffrey S. Cross, Tokyo Tech’s School of Environment and Society
JEFFREY S. CROSS (cross.j.aa@m.titech.ac.jp) received a Ph.D. in chemical engineering from Iowa State University in 1992. He has worked in Japan at Fujitsu Lab/Fujitsu Ltd., National Institute for Research in Inorganics Materials (now NIMS), and at the Tokyo Institute of Technology (Tokyo Tech) for over 24 years. Jeffrey is a Professor in the Tokyo Tech’s School of Environment and Society, Dept. of Transdisciplinary Science and Engineering and a graduate coordinate for the Energy Science and Engineering Major. He teaches online courses on academic writing and on education technology, researches biofuels, educational technology and analyzes Japan’s energy policy. He is the founder and general manager the Tokyo Tech Online Education Development Office, which develops MOOCs on edX. He was an observer in the Japan Team that participated in the OECD-AHELO project on civil engineering competence assessment. He currently serves as the secretariat for the UK-Japan Engineering Education League.
Estiyanti Ekawati, Institut Teknologi Bandung
ESTIYANTI EKAWATI (esti@spm.itb.ac.id) received a Ph.D. in engineering from Murdoch University, Australia in 2004. She is an Associate Professor at the Institut Teknologi Bandung, Indonesia, Faculty of Industrial Technology. She teaches courses on engineering designs, probability and statistics, dynamic modeling, controls and optimizations. She is a member of IEEE Control Systems Society and IEEE Education Society. She currently serves as the secretary of Institut Teknologi Bandung Quality Assurance Unit.
Satoko Fukahori, National Institute for Educational Policy Research (NIER)
SATOKO FUKAHORI (fukahori@nier.go.jp) is the Research Director of the Department for Higher Education Research, National Institute for Educational Policy Research (NIER), the research arm of the Ministry of Education, Culture, Sports, Science and Technology (MEXT) of Japan. Her research topics include quality assurance of higher education, program design, assessment of higher education learning outcomes, etc. She was the principal national coordinator for the 2008-2012 design and implementation of the Organization for Economic Cooperation and Development (OECD) Assessment of Higher Education Learning Outcomes (AHELO), to which Japan participated in the engineering strand. From 2014, Dr. Fukahori has been leading the Japanese follow-up program “The Global Quality Assurance of Higher Education through the Development of a Tuning Test Item Bank,” working with a team of engineers seeking to develop a shared understanding of expected learning outcomes in the field of mechanical engineering. She is also the Director of the Tuning Japan National Center, the Japanese contact point for Tuning activities. Dr. Fukahori completed her B.A. (1991) and M.A. (1993) degrees in Comparative Education at Kyoto University, and received her Ph.D. (2000) at Columbia University Graduate School of Arts and Sciences in Sociology of Education.
Shinnosuke Obi, Keio University
SHINNOSUKE OBI (obsn@mech.keio.ac.jp) received Dr.-Ing. from the University of Erlangen-Nuremberg, Germany, in 1991. He is a professor in the Department of Mechanical Engineering, Keio University, Japan, and Fellow of the Japan Society of Mechanical Engineers, member of Japan Society of Fluid Mechanics, Heat Transfer Society of Japan, Gas Turbine Society of Japan, and the Japan Society of Civil Engineers. From October 2017 he is appointed to the Dean of the International Center of Keio University where he supervises the operation of various international programs offered to the students of Keio University and its partner institutions in the world. He serves also as a member of various committees at the Ministry of Education, Culture, Sports, Science and Technology of Japan, related to scholarship programs for international students, and human resource development project at JICA, Japan International Cooperation Agency, through AUN/SEED-Net Program.
Yugo Saito, Kyoto University
YUGO SAITO (ugo.saito@gmail.com) is a program-specific assistant professor at Kyoto University’s Center for the Promotion of Excellence in Higher Education. He uses statistical methods to explore assessing learning outcomes in higher education.
Nathanael P. Tandian, Institut Teknologi Bandung
NATHANAEL P. TANDIAN (n4th4n.t4nd14n@gmail.com, n4th4n@termo.itb.ac.id) completed his studies in Institut Teknologi Bandung - ITB, Indonesia (bachelor in 1983), University of Minnesota, Twin Cities, USA (MSME in 1991 and Ph.D. in 1994); all are in Mechanical Engineering. He has been being a faculty member in the Mechanical Engineering Department, ITB since 1984 and teaches thermodynamics, mechatronics, and applied mathematics. He is also a member of the Energy Conversion Research Group and the Center for New and Renewable Energy in ITB, with research interest in the area of refrigeration, heat transfer, and computational fluid dynamics. He actively serves as a member of the ad hoc committee for curriculum, and a member of the quality assurance unit in the Mechanical, Aeronautic and Aerospace Engineering Faculty, ITB. Currently he is also a fellow of the Institution of Engineers Indonesia.
Farid Triawan, Tokyo Tech’s School of Environment and Society
FARID TRIAWAN (triawan.f.aa@m.titech.ac.jp, farid.triawan@gmail.com) earned his bachelor degree in mechanical engineering from Bandung Institute of Technology, Indonesia in 2005, and then received his master (in 2009) and doctoral degrees (in 2012) in mechanical engineering both from Tokyo Institute of Technology (Tokyo Tech), Japan. He worked at Torishima Pump Mfg. Co., Ltd. in Japan for 3.5 years as a mechanical engineer and was assigned as the leader of materials research group of R&D Department. Currently, Farid is a specially appointed Associate Professor (Lecturer) in the Tokyo Tech School of Environment and Society, Dept. of Transdisciplinary Science and Engineering. He teaches several courses related to materials mechanics and structural engineering, and has been actively managing the newly opened International Undergraduate Program (so-called GSEP) at Tokyo Tech. He is also a member of Japanese Society of Mechanical Engineers (JSME).

References

Arum, Richard, Josipa Roksa, and Amanda Cook. Improving Quality in American Higher Education: Learning Outcomes and Assessments for the 21st Century. San Francisco: Jossey-Bass, 2016.

Anderson, Lorin W., and David Krathwhol. A Taxonomy for Learning, Teaching, and Assessing. New York: Addison Wesley Longman, Inc., 2001.

Association of American Colleges and Universities. “LEAP Essential Learning Outcomes.” https://www.aacu.org/leap/essential-learning-outcomes.

Birtwistle, Tim, Courtney Brown, and Robert Wagenaar. “A long way to go…A study on the implementation of the learning-outcomes based approach in the EU and USA.” Tuning Journal for Higher Education 3, no. 2 (2016): 429-463. doi: 10.18543/the-3(2)-2016pp429-463.

European Network for the Accreditation of Engineering Education. “EUR-ACE Framework Standards and Guidelines (EAFSG).” http://www.enaee.eu/wp-assets-enaee/uploads/2012/02/EAFSG-Doc-Full-status-8-Sept-15-on-web-fm- Denis.pdf.

Group of National Experts on the AHELO Feasibility Study. “Engineering Assessment Framework.” (AHELO Feasibility Study 8th meeting of the AHELO GNE Paris, 18-19 November 2011). http://search.oecd.org/officialdocuments/displaydocumentpdf/?cote=edu/imhe/ahelo/gne(2011)19/ANN5/FINAL&doclanguage=en

Fukahori, Satoko, ed. “An Analytical Report of the Results of AHELO Feasibility Study” (Tokyo: National Institute for Educational Policy Research), original title in Japanese:「AHELO調査結果の分析に関する研究会(研究成果報告書)」(pronounced as follows:“AHELO Chosa-kekka-no-bunseki-ni-kansuru-kenkyukai (Kenkyu-seika-hokokusho)”). http://www.nier.go.jp/koutou/ahelo/2014/report.pdf.

Fukahori, Satoko. “Competence-based Quality Assurance of University Education - Lessons Learnt from the OECD-AHELO Feasibility Study.” Educational Studies in Japan: International Yearbook 8 (2014):9-61.

Hirota, Teruyuki. “Why Front Line Academics Reject Reform – Towards Utilizing Subject Specific Benchmark Statements.” University Evaluation Studies 15 (2016): 37-46.

International Engineering Alliance. “Graduate Attributes and Professional Competencies.” http://www.ieagreements.org/assets/Uploads/Documents/Policy/Graduate-Attributes-and-Professional-Competencies.pdf.

Japan Accreditation Board of Engineering Education. http://www.jabee.org/english/.

Lumina Foundation. “Degree Qualifications Profile.” https://www.luminafoundation.org/files/resources/dqp.pdf.

Ministry of Education, Culture, Sports, Science and Technology (MEXT) Central Council for Education. “Towards the Construction of Bachelor Degree Programs” (Tokyo: MEXT, 2008); original title in Japanese:「学士課程教育の構築に向けて(答申)」(pronounced as follows:“Gakushikatei-kyoiku-no-kochiku-ni-mukete (Toshin)”). http://www.mext.go.jp/b_menu/shingi/chukyo/chukyo0/toushin/1217067.htm.

National Institute for Academic Degrees and University Evaluation (NIAD-UE). “A Research on the Quality Assurance of Disciplinary Education in Japanese Universities” (Tokyo: NIAD-UE, 2016); original title in Japanese 「我が国における大学教育の分野別質保証の在り方に関する調査研究報告書」(pronounced as: “Wagakuni-niokeru-daigaku-kyoiku-no-bunyabetsu-sanshokijun-no-arikata-ni-kansuru-chosa-kenkyu”). http://www.niad.ac.jp/n_shuppan/project/__icsFiles/afieldfile/2016/07/14/no09_nr16-0714.pdf.

National Institute for Educational Policy Research (NIER). “Tuning Test Item Bank Project.” http://www.nier.go.jp/tuning/centre/kikai.html.

_____. “The National Institute for Educational Policy Research International Symposium for Educational Reform 2013: The TUNING-AHELO Global Quality Assurance through Sharing Competence Frameworks and Degree Level Specifications” (Tokyo: NIER, 2013). http://www.nier.go.jp/06_jigyou/symposium/i_sympo25/.

National Research Council. Engineering Tasks for the New Century: Japanese and U.S. Perspectives (Washington, DC: The National Academies Press, 1999), https://doi.org/10.17226/9624.

Organization of Economic Cooperation and Development (OECD). “A Tuning- AHELO Conceptual Framework of Expected Desired/Learning Outcomes in Engineering.” OECD Education Working Papers, 60 (2011), OECD Publishing, Paris. http://dx.doi.org/10.1787/5kghtchn8mbn-en.

_____. “OECD-AHELO Feasibility Study Report Volume 2 – Data Analysis and National Experiences.” (Paris: OECD, 2013). http://www.oecd.org/edu/skills-beyond-school/AHELOFSReportVolume2.pdf.

Quality Assurance Agency (QAA). “The UK Quality Code for Higher Education Subject Benchmark Statements.” http://www.qaa.ac.uk/assuring-standards-and-quality/the-quality-code/subject-benchmark-statements.

Science Council of Japan. “Benchmark Statements for Curriculum Design for Subject- Specific Quality Assurance in University Education.” (Tokyo: Science Council of Japan, 2012); original title in Japanese「大学教育の分野別質保証のための教育課程編成上の参照基準について」(pronounced as “Daigaku-kyoiku-no-bunyabetsu-shituhosho-no-tame-no-kyoiku-katei-henseijono-sanshokijun-ni-tsuite”). http://www.scj.go.jp/ja/member/iinkai/daigakuhosyo/daigakuhosyo.html.

Tremblay, Karine, Dianne Lalancette, and Deborah Roseveare. “OECD-AHELO Feasibility Study Report Volume 1 - Design and Implementation.” (Paris: OECD, 2012). http://www.oecd.org/edu/skills-beyond-school/AHELOFSReportVolume1.pdf.

Tuning Educational Structures in Europe. “Subject Areas.” http://www.unideusto.org/tuningeu/subject-areas.html.

Wiggins, Grant, and Jay McTighe. Understanding by Design, expanded 2th ed. (Alexander, Virginia: Association for Supervision & Curriculum, 2005).

Published
2017-11-30
How to Cite
Cross, Jeffrey S., Estiyanti Ekawati, Satoko Fukahori, Shinnosuke Obi, Yugo Saito, Nathanael P. Tandian, and Farid Triawan. 2017. “Development of a Mechanical Engineering Test Item Bank to Promote Learning Outcomes-Based Education in Japanese and Indonesian Higher Education Institutions”. Tuning Journal for Higher Education 5 (1), 41-73. https://doi.org/10.18543/tjhe-5(1)-2017pp41-73.