Detailed explanation of string operations and encoding Unicode in Python

This article mainly introduces you to some knowledge about string operations and encoding Unicode in Python. I won’t say much below. Friends who need it can come and learn together.

String type

str: Unicode string. Strings constructed using '' or r'' are all str, and single quotes can be replaced with double quotes or triple quotes. Regardless of how you specify it, there is no difference when it is stored internally in Python.

bytes: Binary string. Since files in other formats such as jpg cannot be displayed using str, they are represented by bytes. Each byte of bytes is a number from 0 to 255. When printing, Python will display the part that can be expressed in ASCII as ASCII, which makes it easier to read. bytes supports almost all str methods except formatting, even including the re module

bytearray() : Binary string that can be changed in place.

utf-8 encoding range

##RangeNumber of bytesStorage format##0x0000~0x007F (0 ~ 127)0x0080~0x07FF(128 ~ 2047)##0x0800~FFFF(2048 ~ 65535) 3 bytes1110xxxx 10xxxxxx 10xxxxxx4 bytes##0x2000000~0x3FFFFFF5 bytes111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 0x4000000~0x7FFFFFFF) 6 bytes1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxxByte order mark BOM

1 byte	0xxxxxxx
2 bytes	110xxxxx 10xxxxxx
		##0x10000~1FFFFFF(65536 ~ 2097152)
11110xxx 10xxxxxx 10xxxxxx 10xxxxxx

BOM is the abbreviation of byte order marker,
Specifies the rules when encoding is written

Python is using 'utf-8 'Encoding will not write a BOM header when writing a file, but specifying encoding 'utf-8-sig' will force Python to write a BOM header.

Using 'utf-16-be' will not write a BOM header, but using 'utf-16' will write a BOM header.

>>> open('h.txt','w',encoding='utf-8-sig').write('aaa')
3
>>> open('h.txt','rb').read()
b'\xef\xbb\xbfaaa'
>>> open('h.txt','w',encoding='utf-16').write('bbb')
3
>>> open('h.txt','rb').read()
b'\xff\xfeb\x00b\x00b\x00'
>>> open('hh.txt','w',encoding='utf-16-be').write('ccc')
3
>>> open('hh.txt','rb').read()
b'\x00c\x00c\x00c'
>>> open('h.txt','w',encoding='utf-8').write('ddd')
3
>>> open('h.txt','rb').read()
b'ddd'

Rules when reading

If the correct encoding is specified, the BOM will be ignored, otherwise the BOM It will be displayed as garbled characters or an exception will be returned.

>>> open('h.txt','r').read()
'锘縟dd'
>>> open('h.txt','r',encoding='utf-8-sig').read()
'ddd'

Encoding and decoding

chr and ord

• >>> ord('中') #20013
  >>> chr(20013) #'中'

Hardcode Unicode into the string.

• '\xhh': Use 2-digit hexadecimal to represent a character

'\uhhhh': Use 4-digit hexadecimal to represent a character Represent a character:

'\Uhhhhhhhh': Use 8-digit hexadecimal to represent a character

>>> s = ' py\x74h\u4e2don' #'pyth in on'

str and bytes, bytearray conversion

str.encode( encoding='utf-8')

bytes(s,encoding='utf-8')<br/>

bytes.decode(encoding='utf-8')<br/>

str(B, encoding='utf-8')<br/>

bytearray(string, encoding='utf-8')

bytearray(bytes)

Document encoding statement

Python uses utf-8 encoding by default.

# -*- coding: latin-1 -*-

: Indicates that the declaration document is latin-1 encoding.

Helper functions

sys.platform  #'win32'
sys.getdefaultencoding() # 'utf-8'
sys.byteorder  #'little'
s.isalnum()  #s表示字符串
s.isalpha()
s.isdecimal
s.isdigit()
s.isnumeric()
s.isprintable()
s.isspace()
s.isidentifier() #如果字符串可以用作变量名，那么返回True
s.islower()
s.isupper()
s.istitle()

More in Python Please pay attention to the PHP Chinese website for related articles on string operations and encoding Unicode!