Table of Contents
The accuracy of GPT-4 is only 33%
The method of subject selection has been questioned
The result was that GPT-4v seemed to understand graphics "completely differently from humans."
Home Technology peripherals AI GPT-4 performs poorly in graph inference? Even after 'releasing water', the accuracy rate is only 33%

GPT-4 performs poorly in graph inference? Even after 'releasing water', the accuracy rate is only 33%

Nov 21, 2023 am 08:38 AM
ai reasoning

GPT-4’s graphical reasoning ability is less than half that of humans?

A study by the Santa Fe Research Institute in the United States shows that GPT-4’s accuracy in graphical reasoning questions is only 33%.

GPT-4v has multi-modal capabilities, but its performance is relatively poor. It can only answer 25% of the questions correctly

GPT-4 performs poorly in graph inference? Even after releasing water, the accuracy rate is only 33%

△Dotted line indicates Average performance of 16 tasks

As soon as the results of this experiment were released, they immediately caused extensive discussion on YC

Some netizens who supported this result said that GPT is indeed good at processing abstract graphics. Poor performance, and it is more difficult to understand concepts such as "position" and "rotation"

GPT-4 performs poorly in graph inference? Even after releasing water, the accuracy rate is only 33%

However, some netizens expressed doubts about this conclusion, and their views can be simply summarized as :

Although this view cannot be said to be wrong, it cannot be completely convincing

GPT-4 performs poorly in graph inference? Even after releasing water, the accuracy rate is only 33%

As for the specific reasons, we Read on.

The accuracy of GPT-4 is only 33%

In order to evaluate the performance of humans and GPT-4 on these graphics problems, the researchers used the ConceptARC data set launched in May this year

ConceptARC includes a total of 16 subcategories of graphical reasoning questions, with 30 questions in each category, for a total of 480 questions.

GPT-4 performs poorly in graph inference? Even after releasing water, the accuracy rate is only 33%

These 16 subcategories include positional relationships, shapes, operations, comparisons, etc.

Specifically, these questions are composed of Made up of pixel blocks. Humans and GPT need to find patterns based on given examples and analyze the results of images processed in the same way

The author specifically shows examples of these 16 subcategories in the paper, one for each category.

GPT-4 performs poorly in graph inference? Even after releasing water, the accuracy rate is only 33%
GPT-4 performs poorly in graph inference? Even after releasing water, the accuracy rate is only 33%
GPT-4 performs poorly in graph inference? Even after releasing water, the accuracy rate is only 33%

The results showed that the average accuracy rate of 451 human subjects was no less than 83% in each sub-item. Taking the average of 16 tasks, it reaches 91%.

In the case where a question can be tried three times (if it is correct once, it is correct), the highest accuracy of GPT-4 (single sample) does not exceed 60%, and the average is only 33%

GPT-4 performs poorly in graph inference? Even after releasing water, the accuracy rate is only 33%

Earlier, the author of ConceptARC Benchmark involved in this experiment also conducted a similar experiment, but in GPT-4 it was zero sample Test, the average accuracy rate of 16 tasks is only 19%.

GPT-4 performs poorly in graph inference? Even after releasing water, the accuracy rate is only 33%

GPT-4v is a multi-modal model, but its accuracy is low. On a small-scale ConceptARC data set consisting of 48 questions, the accuracy rates of zero-sample testing and single-sample testing were only 25% and 23% respectively.

GPT-4 performs poorly in graph inference? Even after releasing water, the accuracy rate is only 33%

And the researchers After further analyzing the wrong answers, it was found that some human errors seemed likely to be caused by "carelessness", while GPT did not understand the rules in the questions at all.

GPT-4 performs poorly in graph inference? Even after releasing water, the accuracy rate is only 33%

Most netizens have no doubts about these data, but what has caused this experiment to be questioned is the group of subjects recruited and the input method provided to GPT

The method of subject selection has been questioned

Initially, the researchers recruited subjects on an Amazon crowdsourcing platform.

The researcher extracted some simple questions from the data set as an introductory test. The subjectsneed to answer at least two of the three random questions correctly before entering the formal test .

The results found by the researchers show that some people only take the entrance test for the purpose of greed for money, but do not complete the questions as required

As a last resort, the researcherswill The threshold for participating in the test has been raised to the point where no less than 2,000 tasks have been completed on the platform, and the pass rate must reach 99%. However, although the author uses the pass rate to screen people, in terms of specific abilities, in addition to requiring subjects to speak English, there are "no special requirements" for

other professional abilities such as graphics . To achieve data diversity, the researchers moved recruitment efforts to another crowdsourcing platform later in the experiment. In the end, a total of 415 subjects participated in this experiment

Despite this, some people still questioned that the samples in the experiment were "

not random enough".

GPT-4 performs poorly in graph inference? Even after releasing water, the accuracy rate is only 33%Some netizens pointed out that on the Amazon crowdsourcing platform used by researchers to recruit subjects,

there were large models posing as humans .

GPT-4 performs poorly in graph inference? Even after releasing water, the accuracy rate is only 33%The operation of the multi-modal version of GPT is relatively simple. You only need to directly input the image and use the corresponding prompt word

GPT-4 performs poorly in graph inference? Even after releasing water, the accuracy rate is only 33%In the zero-sample test, just delete the corresponding EXAMPLE part

But for the plain text version of GPT-4 (0613) without multi-modality, you need to convert the image into grid points ,

Use numbers instead of colors.

GPT-4 performs poorly in graph inference? Even after releasing water, the accuracy rate is only 33%Some people disagree with this operation:

After converting the image into a digital matrix, the concept completely changes

, even humans, looking at the "graphics" represented by numbers, may not be able to understand

One More Thing

GPT-4 performs poorly in graph inference? Even after releasing water, the accuracy rate is only 33%Coincidentally , Joy Hsu, a Chinese doctoral student at Stanford University, also tested the graph understanding ability of GPT-4v on a geometric data set. Last year, a data set was released to test the understanding of Euclidean geometry by large models. After GPT-4v was opened, Hsu used the data set to test it again.

The result was that GPT-4v seemed to understand graphics "completely differently from humans."

In terms of data, GPT-4v is significantly inferior to humans in answering these geometric questions

GPT-4 performs poorly in graph inference? Even after releasing water, the accuracy rate is only 33%

Paper Address:

[1]https://arxiv.org/abs/2305.07141

[2]https://arxiv.org/abs/2311.09247GPT-4 performs poorly in graph inference? Even after releasing water, the accuracy rate is only 33%

The above is the detailed content of GPT-4 performs poorly in graph inference? Even after 'releasing water', the accuracy rate is only 33%. For more information, please follow other related articles on the PHP Chinese website!

Statement of this Website
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

Video Face Swap

Video Face Swap

Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Tools

Notepad++7.3.1

Notepad++7.3.1

Easy-to-use and free code editor

SublimeText3 Chinese version

SublimeText3 Chinese version

Chinese version, very easy to use

Zend Studio 13.0.1

Zend Studio 13.0.1

Powerful PHP integrated development environment

Dreamweaver CS6

Dreamweaver CS6

Visual web development tools

SublimeText3 Mac version

SublimeText3 Mac version

God-level code editing software (SublimeText3)

Hot Topics

Java Tutorial
1664
14
PHP Tutorial
1268
29
C# Tutorial
1240
24
How to understand DMA operations in C? How to understand DMA operations in C? Apr 28, 2025 pm 10:09 PM

DMA in C refers to DirectMemoryAccess, a direct memory access technology, allowing hardware devices to directly transmit data to memory without CPU intervention. 1) DMA operation is highly dependent on hardware devices and drivers, and the implementation method varies from system to system. 2) Direct access to memory may bring security risks, and the correctness and security of the code must be ensured. 3) DMA can improve performance, but improper use may lead to degradation of system performance. Through practice and learning, we can master the skills of using DMA and maximize its effectiveness in scenarios such as high-speed data transmission and real-time signal processing.

How to use the chrono library in C? How to use the chrono library in C? Apr 28, 2025 pm 10:18 PM

Using the chrono library in C can allow you to control time and time intervals more accurately. Let's explore the charm of this library. C's chrono library is part of the standard library, which provides a modern way to deal with time and time intervals. For programmers who have suffered from time.h and ctime, chrono is undoubtedly a boon. It not only improves the readability and maintainability of the code, but also provides higher accuracy and flexibility. Let's start with the basics. The chrono library mainly includes the following key components: std::chrono::system_clock: represents the system clock, used to obtain the current time. std::chron

Quantitative Exchange Ranking 2025 Top 10 Recommendations for Digital Currency Quantitative Trading APPs Quantitative Exchange Ranking 2025 Top 10 Recommendations for Digital Currency Quantitative Trading APPs Apr 30, 2025 pm 07:24 PM

The built-in quantization tools on the exchange include: 1. Binance: Provides Binance Futures quantitative module, low handling fees, and supports AI-assisted transactions. 2. OKX (Ouyi): Supports multi-account management and intelligent order routing, and provides institutional-level risk control. The independent quantitative strategy platforms include: 3. 3Commas: drag-and-drop strategy generator, suitable for multi-platform hedging arbitrage. 4. Quadency: Professional-level algorithm strategy library, supporting customized risk thresholds. 5. Pionex: Built-in 16 preset strategy, low transaction fee. Vertical domain tools include: 6. Cryptohopper: cloud-based quantitative platform, supporting 150 technical indicators. 7. Bitsgap:

How to handle high DPI display in C? How to handle high DPI display in C? Apr 28, 2025 pm 09:57 PM

Handling high DPI display in C can be achieved through the following steps: 1) Understand DPI and scaling, use the operating system API to obtain DPI information and adjust the graphics output; 2) Handle cross-platform compatibility, use cross-platform graphics libraries such as SDL or Qt; 3) Perform performance optimization, improve performance through cache, hardware acceleration, and dynamic adjustment of the details level; 4) Solve common problems, such as blurred text and interface elements are too small, and solve by correctly applying DPI scaling.

What is real-time operating system programming in C? What is real-time operating system programming in C? Apr 28, 2025 pm 10:15 PM

C performs well in real-time operating system (RTOS) programming, providing efficient execution efficiency and precise time management. 1) C Meet the needs of RTOS through direct operation of hardware resources and efficient memory management. 2) Using object-oriented features, C can design a flexible task scheduling system. 3) C supports efficient interrupt processing, but dynamic memory allocation and exception processing must be avoided to ensure real-time. 4) Template programming and inline functions help in performance optimization. 5) In practical applications, C can be used to implement an efficient logging system.

How to measure thread performance in C? How to measure thread performance in C? Apr 28, 2025 pm 10:21 PM

Measuring thread performance in C can use the timing tools, performance analysis tools, and custom timers in the standard library. 1. Use the library to measure execution time. 2. Use gprof for performance analysis. The steps include adding the -pg option during compilation, running the program to generate a gmon.out file, and generating a performance report. 3. Use Valgrind's Callgrind module to perform more detailed analysis. The steps include running the program to generate the callgrind.out file and viewing the results using kcachegrind. 4. Custom timers can flexibly measure the execution time of a specific code segment. These methods help to fully understand thread performance and optimize code.

An efficient way to batch insert data in MySQL An efficient way to batch insert data in MySQL Apr 29, 2025 pm 04:18 PM

Efficient methods for batch inserting data in MySQL include: 1. Using INSERTINTO...VALUES syntax, 2. Using LOADDATAINFILE command, 3. Using transaction processing, 4. Adjust batch size, 5. Disable indexing, 6. Using INSERTIGNORE or INSERT...ONDUPLICATEKEYUPDATE, these methods can significantly improve database operation efficiency.

How to use string streams in C? How to use string streams in C? Apr 28, 2025 pm 09:12 PM

The main steps and precautions for using string streams in C are as follows: 1. Create an output string stream and convert data, such as converting integers into strings. 2. Apply to serialization of complex data structures, such as converting vector into strings. 3. Pay attention to performance issues and avoid frequent use of string streams when processing large amounts of data. You can consider using the append method of std::string. 4. Pay attention to memory management and avoid frequent creation and destruction of string stream objects. You can reuse or use std::stringstream.

See all articles