PHP automatically determines the character set and transcodes_PHP tutorial

The principle is very simple, because gb2312/gbk is Chinese two bytes, these two bytes have a value range, while Chinese characters in utf-8 are three bytes, and each byte also has a value range. Value range. Regardless of the encoding situation, English is less than 128 and only occupies one byte (except full-width).

If it is a file encoding check, you can also directly check the BOM information of utf-8. Without further ado, let’s go directly to the function. This function is used to check and transcode strings.

About BOM

Byte-order mark (BOM) is the Unicode character ("zero-width non-breaking whitespace") located at the code point U+FEFF. This character is used to indicate the byte order when encoding a string of UCS/Unicode characters as UTF-16 or UTF-32. It is often used as a marker to indicate that a file is encoded in UTF-8, UTF-16, or UTF-32.

In most character encodings, byte order mark is a pattern that is unlikely to appear in other files (it usually looks like an obfuscated sequence of control codes). If the byte order mark is misinterpreted as a real character in a Unicode file, it will not be visible because it is a zero-width non-breaking whitespace. In Unicode 3.2, the use of U+FEFF for non-byte order tokens has been dropped (instead, U+2060 is used to represent this use), allowing U+FEFF to be used only for byte order The semantics of the token.

In UTF-16, a byte order mark is placed as the first character of a file or string stream to indicate the number of all sixteen-bit characters in the file or string stream. Endianness (byte order).

• If the hexadecimal unit is represented in big-endian order, this byte order mark character will appear in the sequence as 0xFE, followed by 0xFF (the 0x is used to indicate hexadecimal).
• If the sixteen-bit unit uses little-endian order, this byte sequence is 0xFF, followed by 0xFE.

In Unicode, a code point with a value of U+FFFE is guaranteed not to be designated as a Unicode character. This means that 0xFF and 0xFE will only be interpreted as U+FEFF in little-endian order (because it cannot be U+FFFE in big-endian order).

UTF-8 has no byte order issues. UTF-8 encoded byte order marks are used to indicate that it is a UTF-8 file. It is only used to mark a UTF-8 file, not to specify the byte order. [1] Many Windows programs (including Notepad) add byte order marks to UTF-8 files. However, in Unix-like systems (which make heavy use of en:text files for file formats and for inter-process communication), this practice is not recommended. Because it will prevent the correct processing of some important codes such as en:Shebang at the beginning of the interpreter script. It also affects programming languages ​​that don't recognize it. For example, gcc will report that there are unrecognizable characters at the beginning of the source file. In PHP, if output buffering is not enabled, it will cause the page content to start being sent to the browser (ie: the user header file has been sent), which makes the PHP script unable to specify the user header file (HTTP Header). The byte order mark is represented in UTF-8 as the sequence EF BB BF. For most text editors and web browsers that are not ready to handle UTF-8, in the ISO-8859-1 environment, it will display ???.

Although Byte Order Notation can also be used for UTF-32, this encoding is rarely used for transmission and its rules are the same as UTF-16. For the IANA-registered character sets UTF-16BE, UTF-16LE, UTF-32BE, and UTF-32LE, byte order notation cannot be used. U+FEFF at the beginning of a document is interpreted as a (discarded) "zero-width non-breaking whitespace" because the name of these character sets determines their byte order. For the registered character sets UTF-16 and UTF-32, a leading U+FEFF is used to indicate byte order.