


When ChatGPT took part in China's college entrance examination and fed it the national A and B papers, it turned out to be seriously biased!
Xi Xiaoyao Technology Talk Original
Author | PythonChatGPT, as an intelligent human-machine conversation application, quickly became popular all over the world after its launch. In just one month, its number of users has exceeded 100 million. People also use ChatGPT to test many exam items, such as SAT, AP, GRE, etc. However, what would it be like if ChatGPT were allowed to participate in China’s college entrance examination? Will he be biased? Can ordinary people like us pass ChatGPT? Let’s take a look at the comments brought to you by students from Fudan University and East China Normal University.
Paper title:
Evaluating the Performance of Large Language Models on GAOKAO Benchmark
Paper link:https://www.php.cn/link/de66dbc30377a1f7908db45298324aa3
How to use ChatGPT to answer college entrance examination questions?
This paper uses a zero-supervised prompt method to convert test questions into ChatGPT input, as shown in the figure below. Different inquiry methods are designed for different subjects and question types. For math questions, convert the formula into latex input.
College Entrance Examination Data Set
The test in this article uses the national A paper and the national B paper from 2010 to 2022, a total of 13 years. Each set of test papers contains 10 subjects Subjects include Chinese, Mathematics, English, Physics, Chemistry, Life, History, and Geography. Mathematics is divided into science mathematics and liberal arts mathematics.
The data set contains a total of 2811 test questions. The specific question types will not be discussed here. I believe readers are very familiar with the college entrance examination questions.
During the evaluation, high school teachers from Caoyang No. 2 Middle School in Shanghai were hired to mark the subjective questions.
Experiment and Analysis
The scores obtained by ChatGPT in the college entrance examination over the past years are shown in the figure below. Since each subject is normalized to 100 points when calculating scores, this score cannot be directly compared with your and my college entrance examination scores. But it can also be seen that this score is not ideal, and it is estimated that neither Fudan nor East China Normal University will be admitted. Why is this?
The above picture shows the performance of ChatGPT in various disciplines, as well as subjective and objective topics. Blue is objective questions and yellow is subjective questions. The analysis found that ChatGPT performed better on objective questions, especially English reading comprehension, multiple choice, and cloze, achieving 88.3%, 78.1%, and 73.8% accuracy respectively. But even for objective questions, the accuracy rate in science and mathematics is less than 40%. Mathematics is really difficult~
ChatGPT’s performance on subjective questions is poor. In the subjects of physics, chemistry, biology and mathematics, the performance of subjective questions is obviously not as good as objective questions. The scores on objective questions in science subjects are also poor. Maybe ChatGPT is biased towards liberal arts subjects? According to the reviewers' comments, the main shortcomings of ChatGPT are: 1. The complex equations in mathematical problems are difficult to solve correctly, and wrong formulas are used in the problem solving process. 2. Inadequate ability to understand and summarize when reading longer materials.
Summary
ChatGPT may not have used Chinese college entrance examination question data during the training process, so its performance is not affected by data leaks and has high credibility.
Observation results show that compared with foreign exams, ChatGPT’s performance on Chinese college entrance examination questions is slightly inferior. Therefore, domestic students do not need to worry too much about not being able to pass the ChatGPT exam for the time being. However, the long text summarization capabilities mentioned in the article have been significantly improved in GPT4-32K, and the domestic large model has also been further optimized on Chinese data. Therefore, we can expect more impressive performance on large-model college entrance examination questions in the future.
In addition, the idea of using ChatGPT to solve college entrance examination questions may be able to answer the debate among netizens about which province’s test questions are more difficult?
The above is the detailed content of When ChatGPT took part in China's college entrance examination and fed it the national A and B papers, it turned out to be seriously biased!. For more information, please follow other related articles on the PHP Chinese website!

Hot AI Tools

Undresser.AI Undress
AI-powered app for creating realistic nude photos

AI Clothes Remover
Online AI tool for removing clothes from photos.

Undress AI Tool
Undress images for free

Clothoff.io
AI clothes remover

Video Face Swap
Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Article

Hot Tools

Notepad++7.3.1
Easy-to-use and free code editor

SublimeText3 Chinese version
Chinese version, very easy to use

Zend Studio 13.0.1
Powerful PHP integrated development environment

Dreamweaver CS6
Visual web development tools

SublimeText3 Mac version
God-level code editing software (SublimeText3)

Hot Topics











Meta's Llama 3.2: A Leap Forward in Multimodal and Mobile AI Meta recently unveiled Llama 3.2, a significant advancement in AI featuring powerful vision capabilities and lightweight text models optimized for mobile devices. Building on the success o

Hey there, Coding ninja! What coding-related tasks do you have planned for the day? Before you dive further into this blog, I want you to think about all your coding-related woes—better list those down. Done? – Let’

This week's AI landscape: A whirlwind of advancements, ethical considerations, and regulatory debates. Major players like OpenAI, Google, Meta, and Microsoft have unleashed a torrent of updates, from groundbreaking new models to crucial shifts in le

Shopify CEO Tobi Lütke's recent memo boldly declares AI proficiency a fundamental expectation for every employee, marking a significant cultural shift within the company. This isn't a fleeting trend; it's a new operational paradigm integrated into p

Introduction OpenAI has released its new model based on the much-anticipated “strawberry” architecture. This innovative model, known as o1, enhances reasoning capabilities, allowing it to think through problems mor

Introduction Imagine walking through an art gallery, surrounded by vivid paintings and sculptures. Now, what if you could ask each piece a question and get a meaningful answer? You might ask, “What story are you telling?

For those of you who might be new to my column, I broadly explore the latest advances in AI across the board, including topics such as embodied AI, AI reasoning, high-tech breakthroughs in AI, prompt engineering, training of AI, fielding of AI, AI re

SQL's ALTER TABLE Statement: Dynamically Adding Columns to Your Database In data management, SQL's adaptability is crucial. Need to adjust your database structure on the fly? The ALTER TABLE statement is your solution. This guide details adding colu
