A brief introduction to string encoding issues in Python programming

This article introduces Python string programming, We have already said that String is also a data type, but characters What's special about string is that there is an encoding problem.

Because computers can only process numbers, if you want to process text, you must first convert the text into numbers before processing. The earliest computers were designed using 8 bits as a byte. Therefore, the largest integer that can be represented by a byte is 255 (binary 11111111 = decimal 255). If you want to represent a larger integer, You must use more bytes. For example, the maximum integer that can be represented by two bytes is 65535, and the maximum integer that can be represented by 4 bytes is 4294967295.

Since the computer was invented by Americans, only 127 characters were encoded into the computer at first, that is, uppercase and lowercase English letters, numbers and some symbols. This encoding table is called ASCII encoding, such as uppercase letters The code for the letter A is 65, and the code for the lowercase letter z is 122.

But to process Chinese, one byte is obviously not enough. At least two bytes are needed, and it cannot conflict with ASCII encoding. Therefore, China has formulated the GB2312 encoding to encode Chinese.

What you can imagine is that there are hundreds of languages ​​​​in the world. Japan has compiled Japanese into Shift_JIS, and South Korea has compiled Korean into Euc-kr. Each country has its own standards, and it will inevitably appear. The result of conflict is that in multi-language mixed text, garbled characters will appear when displayed.

Therefore, Unicode came into being. Unicode unifies all languages ​​into a set of encodings, so there will no longer be garbled code problems.

The Unicode standard is also constantly evolving, but the most commonly used one is to use two bytes to represent a character (if you want to use very remote characters, you need 4 bytes). Modern operating systems and most programming languages ​​support Unicode directly.

Now, let’s take a look at the difference between ASCII encoding and Unicode encoding: ASCII encoding is 1 byte, while Unicode encoding is usually 2 bytes.

The letter A encoded in ASCII is decimal 65, binary 01000001;

The character 0 encoded in ASCII is decimal 48, binary 00110000. Note that the character '0' is different from the integer 0 ;

Chinese characters have exceeded the range of ASCII encoding. The Unicode encoding is 20013 in decimal and 01001110 00101101 in binary.

You can guess that if you use Unicode encoding for ASCII encoding, you only need to add 0 in front. Therefore, the Unicode encoding of A is 00000000 01000001.

A new problem arises: if unified into Unicode encoding, the problem of garbled characters will disappear. However, if the text you write is basically all in English, Unicode encoding requires twice as much storage space as ASCII encoding, which is very uneconomical in terms of storage and transmission.

So, in the spirit of conservation, UTF-8 encoding that converts Unicode encoding into "variable length encoding" appeared. UTF-8 encoding encodes a Unicode character into 1-6 bytes according to different number sizes. Commonly used English letters are encoded into 1 byte, and Chinese characters are usually 3 bytes. Only very rare characters will be encoded. Encoded into 4-6 bytes. If the text you want to transmit contains a large number of English characters, using UTF-8 encoding can save space:

So you will see that the source code of many web pages will have something like information indicates that the web page uses UTF-8 encoding.

The above is the detailed content of A brief introduction to string encoding issues in Python programming. For more information, please follow other related articles on the PHP Chinese website!