11 Key Strategies in Using AI Chat GPT to Develop HOTS-Based Entrepreneurship Questionnaires

Authors

  • Almaida Vebibina Department of Educational technology, Faculty of Education, Universitas Negeri Surabaya, Surabaya 60213, Indonesia; Department of Cosmetology Education, Faculty of Engineering, Universitas Negeri Medan, Medan 20222, Indonesia.
  • Rusijono Department of Educational technology, Faculty of Education, Universitas Negeri Surabaya, Surabaya 60213, Indonesia;
  • Andi Mariono Department of Educational technology, Faculty of Education, Universitas Negeri Surabaya, Surabaya 60213, Indonesia;
  • Ahsan Muafa Department of Educational technology, Faculty of Education, Universitas Negeri Surabaya, Surabaya 60213, Indonesia;
  • Dydik Kurniawan Department of Educational technology, Faculty of Education, Universitas Negeri Surabaya, Surabaya 60213, Indonesia;
  • Ragil Sugeng Dewantoro Department of Educational technology, Faculty of Education, Universitas Negeri Surabaya, Surabaya 60213, Indonesia;
  • Dian Noer Asyari Department of Educational technology, Faculty of Education, Universitas Negeri Surabaya, Surabaya 60213, Indonesia;
  • Sussi Widiastuti Department of Educational technology, Faculty of Education, Universitas Negeri Surabaya, Surabaya 60213, Indonesia;
  • Jarot Suseno Department of Educational technology, Faculty of Education, Universitas Negeri Surabaya, Surabaya 60213, Indonesia;

DOI:

https://doi.org/10.48161/qaj.v5n1a1134

Abstract

Developing entrepreneurship assessment questions that measure higher-order thinking skills (HOTS) is a challenge that requires time and in-depth analysis. Artificial intelligence such as Chat GPT is here to simplify the process. However, its use in creating HOTS-based questions in entrepreneurship still faces obstacles. This research is proposed based on the background of various obstacles in the use of Chat GPT to develop an effective questionnaire that meets the learning objectives. To bridge the gap, this study explores the potential of Chat GPT in crafting assessment questions that focus on higher-order thinking by utilizing existing strategies to enhance critical, analytical, creative and problem-solving thinking in entrepreneurship education. A total of 20 entrepreneurship experts at Surabaya State University were involved in this research. We developed LLM (Large Language Model) using the approach to create HOTS-based multiple-choice questions (MCQs) from an advanced course context. Subsequently, we evaluated 470 MCQs generated by Chat GPT-4 and 337 MCQs generated by Chat GPT-3.5 as well as 595 MCQs corresponding to higher-order cognitive levels. An assessment instrument was used to evaluate the quality of questions generated through 23 items adopted from previous studies. To test the difference in performance, the Mann-Whitney Test was analysed with a p value < 0.05. The results showed that ChatGPT-4 was able to generate questions with clear language usage, providing a single correct answer, as well as a high-quality selection of distractors. In addition, the questions developed by ChatGPT-4 are in accordance with HOTS aspects. This finding can be utilized by teachers as the latest generative model to increase the bank of diverse multiple-choice questions without replacing human expertise.

Downloads

Download data is not yet available.

References

Zhou, Y., Gan, L., Chen, J., Wijaya, T. T., & Li, Y. (2023). Development and validation of a higher-order thinking skills assessment scale for pre-service teachers. Thinking Skills and Creativity, 48(March), 101272.

Maxnun, L., Kristiani, K., & Sulistyaningrum, C. D. (2024). Development of HOTS-based cognitive assessment instruments: ADDIE model. Journal of Education and Learning, 18(2), 489–498.

Häkkinen, P., Järvelä, S., Mäkitalo-Siegl, K., Ahonen, A., Näykki, P., & Valtonen, T. (2017). Preparing teacher-students for twenty-first-century learning practices (PREP 21): A framework for enhancing collaborative problem-solving and strategic learning. Teaching and Teacher Education: Theory and Practice, 23(1), 25–41.

Tyas, M. A., Nurkamto, J., Marmanto, S., & Laksani, H. (2019). Developing higher-order thinking skills (HOTS)–based questions: Indonesian EFL teachers’ challenges. Journal of Advanced Research in Dynamical and Control Systems, 2(1), 52–63.

Kim How, R. P. T., Zulnaidi, H., & Rahim, S. S. A. (2022). HOTS in quadratic equations: Teaching style preferences and challenges faced by Malaysian teachers. European Journal of Science and Mathematics Education, 10(1), 15–33.

Ilgun Dibek, M., Sahin Kursad, M., & Erdogan, T. (2024). Influence of artificial intelligence tools on higher-order thinking skills: A meta-analysis. Interactive Learning Environments, 1–23.

Yalcinalp, S., Türkoğlu, H., Koc, S. E., & Halil, E. (2024). Utilization of artificial intelligence in education: A perspective on learning strategies. IntechOpen, 13. Retrieved from http://dx.doi.org/10.1039/C7RA00172J.

Choi, J. H., Hickman, K. E., Monahan, A. B., & Schwarcz, D. (2022). ChatGPT goes to law school. Journal of Legal Education, 71(3), 387–400.

Terwiesch, C. (2023). Would Chat GPT get a Wharton MBA? A prediction based on its performance in the operations management course. Mack Institute for Innovation Management, 1–26. Retrieved from https://mackinstitute.wharton.upenn.edu/wp-content/uploads/2023/01/Christian-Terwiesch-Chat-GTP-1.24.pdf.

Eulerich, M., Sanatizadeh, A., Vakilzadeh, H., & Wood, D. A. (2023). Can artificial intelligence pass accounting certification exams? ChatGPT: CPA, CMA, CIA, and EA? SSRN Electronic Journal, 1–26.

Abeysekera, I. (2024). ChatGPT and academia on accounting assessments. Journal of Open Innovation: Technology, Market, and Complexity, 10(1), 100213.

Vecchiarini, M., & Somià, T. (2023). Redefining entrepreneurship education in the age of artificial intelligence: An explorative analysis. International Journal of Management Education, 21(3), 1–11.

Short, C. E., & Short, J. C. (2023). The artificially intelligent entrepreneur: ChatGPT, prompt engineering, and entrepreneurial rhetoric creation. Journal of Business Venturing Insights, 19(February), e00388.

Bui, H. N., & Duong, C. D. (2024). ChatGPT adoption in entrepreneurship and digital entrepreneurial intention: A moderated mediation model of technostress and digital entrepreneurial self-efficacy. Entrepreneurship Quarterly, 19(2).

Gupta, P., Mahajan, R., Badhera, U., & Kushwaha, P. S. (2024). Integrating generative AI in management education: A mixed-methods study using social construction of technology theory. International Journal of Management Education, 22(3), 101017.

Nah, F. F.-H., Zheng, R., Cai, J., Siau, K., & Chen, L. (2023). Generative AI and ChatGPT: Applications, challenges, and AI-human collaboration. Journal of Information Technology Case and Application Research, 25(2), 277–304.

Liu, Y., et al. (2023). Summary of ChatGPT-related research and perspective towards the future of large language models. Meta-Radiology, 1(2), 100017.

Javaid, M., Haleem, A., Singh, R. P., Khan, S., & Khan, I. H. (2023). Unlocking the opportunities through ChatGPT tool towards ameliorating the education system. BenchCouncil Transactions on Benchmarks, Standards and Evaluation, 3(2), 100115.

Parker, L., Carter, C., Karakas, A., Loper, A. J., & Sokkar, A. (2024). Graduate instructors navigating the AI frontier: The role of ChatGPT in higher education. Computers and Education: Open, 6(July 2023), 100166.

George-Reyes, C. E., Vilhunen, E., Avello-Martínez, R., & López-Caudana, E. (2024). Developing scientific entrepreneurship and complex thinking skills: Creating narrative scripts using ChatGPT. Frontiers in Education, 9.

Chen, L., Ifenthaler, D., & Yau, J. Y.-K. (2021). Online and blended entrepreneurship education: A systematic review of applied educational technologies (Vol. 4, No. 2). Springer Singapore.

Luan, L., Lin, X., & Li, W. (2023). Exploring the cognitive dynamics of artificial intelligence in the post-COVID-19 and learning 3.0 era: A case study of ChatGPT. arXiv Preprint arXiv:2302.04818., 2023, p. 9501.

Dwivedi, Y. K., et al. (2023). "So what if ChatGPT wrote it?" Multidisciplinary perspectives on opportunities, challenges, and implications of generative conversational AI for research, practice, and policy. International Journal of Information Management, 2023, 71(March).

Mavlutova, I., Lesinskis, K., Liogys, M., & Hermanis, J. (2020). Innovative teaching techniques for entrepreneurship education in the era of digitalisation. WSEAS Transactions on Environment and Development, 2020, 16, 725–733.

Jatmika, R. T. D., Ratnasari, V., & Nadlifatin, R. (2024). Empowering micro-entrepreneurs through artificial intelligence: A conceptual framework for AI-based marketing. Procedia Computer Science, 2024, 234, 1087–1094.

Chalmers, D., MacKenzie, N. G., & Carter, S. (2021). Artificial intelligence and entrepreneurship: Implications for venture creation in the fourth industrial revolution. Entrepreneurship Theory and Practice, 2021, 45(5), 1028–1053.

Upadhyay, N., et al. (2022). Influence of digital entrepreneurship and entrepreneurial orientation on the intention of family businesses to adopt artificial intelligence: Examining mediating role of business innovativeness. 2022, 1–65.

Xiao, P., Chen, Y., & Bao, W. (2023). Waiting, banning, and embracing: An empirical analysis of adapting policies for generative AI in higher education. arXiv Preprint arXiv:2305.18617., 2023, 1–33.

Tayan, O., Hassan, A., Khankan, K., & Askool, S. (2024). Considerations for adapting higher education technology courses for AI large language models: A critical review of the impact of ChatGPT. Machine Learning with Applications, 2024, 15(May), 100513.

Kwangmuang, P., Jarutkamolpong, S., Sangboonraung, W., & Daungtod, S. (2021). The development of learning innovation to enhance higher-order thinking skills for students in Thailand junior high schools. Heliyon, 2021, 7(6), e07309.

Lewis, A., & Smith, D. (1993). Defining higher-order thinking. Theory into Practice, 1993, 32(3), 131–137.

Kaminski, J. C., & Hopp, C. (2020). Predicting outcomes in crowdfunding campaigns with textual, visual, and linguistic signals. Small Business Economics, 2020, 55(3), 627–649.

Prüfer, J., & Prüfer, P. (2020). Data science for entrepreneurship research: Studying demand dynamics for entrepreneurial skills in the Netherlands. Small Business Economics, 2020, 55(3), 651–672.

Moore, S., Nguyen, H. A., Chen, T., & Stamper, J. (2023). Assessing the quality of multiple-choice questions using GPT-4 and rule-based methods. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2023, 14200 LNCS, 229–245.

Doughty, J., et al. (2024). A comparative study of AI-generated (GPT-4) and human-crafted MCQs in programming education. Proceedings of the 26th Australasian Computing Education Conference, 2024, 114–123.

Schaper, N. (2024). Using ChatGPT to create constructively aligned assessment tasks and criteria in the context of higher education teaching.

Hwang, K., Challagundla, S., Alomair, M. M., Chen, L. K., & Choa, F.-S. (2023). Towards AI-assisted multiple choice question generation and quality evaluation at scale: Aligning with Bloom’s taxonomy. 2023. Available at: https://gaied.org/neurips2023/files/17/17_paper.pdf

Mellor, R. (2008). Entrepreneurship for everyone: A student textbook. SAGE Publications.

Anderson, L. W. (2005). Objectives, evaluation, and the improvement of education. Studies in Educational Evaluation, 2005, 31(2), 102–113.

Krathwohl, D. R. (2002). A revision of Bloom’s taxonomy: An overview. Theory and Practice, 2002, 41(4), 212–218.

Hopson, M. H., Simms, R. L., & Knezek, G. A. (2001). Using a technology-enriched environment to improve higher-order thinking skills. Journal of Research on Technology in Education, 2001, 34(2), 109–119.

Zohar, A., & Dori, Y. J. (2003). Higher order thinking skills and low-achieving students: Are they mutually exclusive? Journal of the Learning Sciences, 2003, 12(2), 145–181.

Eulerich, M., Sanatizadeh, A., Vakilzadeh, H., & Wood, D. A. (2024). Is it all hype? ChatGPT’s performance and disruptive potential in the accounting and auditing industries. Springer US, 2024.

Ashrafimoghari, V., & Gürkan, N. (2024). Evaluating large language models on the GMAT: Implications for the future of business education. SSRN Electronic Journal, 2024, 1–30.

Ouyang, L., et al. (2022). Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 2022, 35(NeurIPS).

Bendeck, Y., & Voelker, T. A. (2024). ChatGPT’s performance on case discussion questions. 2024 (April).

Ghosh, A., & Bir, A. (2023). Evaluating ChatGPT’s ability to solve higher-order questions on the competency-based medical education curriculum in medical biochemistry. Cureus, 2023, 15(4).

Sinha, R. K., Roy, A. D., Kumar, N., & Mondal, H. (2023). Applicability of ChatGPT in assisting to solve higher-order problems in pathology. Cureus, 2023, 15(2).

Lu, K. (2023). Can ChatGPT help college instructors generate high-quality quiz questions? Human-Computer Interaction and Emerging Technologies (IHIET-AI 2023): Artificial Intelligence for Future Applications, 2023, 70(Ihiet), 311–318.

Doughty, J., et al. (2024). A comparative study of AI-generated (GPT-4) and human-crafted MCQs in programming education. 2024. https://doi.org/10.1145/3636243.3636256

Bhandari, S., Liu, Y., Kwak, Y., & Pardos, Z. A. (2024). Evaluating the psychometric properties of ChatGPT-generated questions. Computers and Education: Artificial Intelligence, 2024, 7(August), 100284.

Indran, I. R., Paramanathan, P., Gupta, N., & Mustafa, N. (2023). Twelve tips to leverage AI for efficient and effective medical question generation: A guide for educators using ChatGPT. Medical Teacher, 2023, 0(0), 1–6.

Published

2025-01-22

How to Cite

Vebibina, A., Rusijono , R. ., Mariono , A. ., Muafa , A. ., Kurniawan , D. ., Sugeng Dewantoro, R. ., Noer Asyari , D. ., Widiastuti , S. ., & Suseno , J. . (2025). 11 Key Strategies in Using AI Chat GPT to Develop HOTS-Based Entrepreneurship Questionnaires . Qubahan Academic Journal, 5(1), 33–62. https://doi.org/10.48161/qaj.v5n1a1134

Issue

Section

Articles