Parsing Converter 3: Lexical part of handwritten PHP to Python compiler-Python Tutorial-php.cn

Home

Backend Development

Python Tutorial

Parsing Converter 3: Lexical part of handwritten PHP to Python compiler

高洛峰

Mar 12, 2017 am 10:17 AM

这篇文章解析转换器3：手写PHP转Python编译器的词法部分

一时技痒，自然而然地想搞个大家伙，把整个PHP程序转成Python。不比模板，可以用正则匹配偷懒，这次非写一个Php编译器不可。

上网搜了一下，发现大部分Python to xxx的transpile都是直接基于AST，省略了最重要的Tokenizer，Parser。直接写个Visitor了事。要不然就是基于Antlr之类的生成器，搞一大堆代码，看得令人心烦。

既然大家都不想做这个苦力，我就来试试，手工写一个Php编译器。分Tokenizer，Parser，Visitor三个部分来实现。

翻出《龙书》《虎书》做参考，仔细学了一回PHP，不学不知道，原来PHP有那么多特性，做个编译器真心累人。

词法部分很简单，就是一个自动机。设计了一个结构存放自动机，然后简单粗暴地在自动机上编程，也顾不上什么性能了，就是个一锤子买卖。

写得还算快，调试不是很顺，不过我是不会说的，哈

自动机不复杂，发上来大家看看，敬请指正。

self.statemachine = {
            &#39;current&#39;: {
                &#39;state&#39;: &#39;default&#39;, &#39;content&#39;: &#39;&#39;, &#39;line&#39;: 0},
            &#39;default&#39;: [
                {&#39;name&#39;: &#39;open&#39;, &#39;next&#39;: &#39;php&#39;, &#39;extra&#39;: 0, &#39;start&#39;: 0, &#39;end&#39;: 0, &#39;cache&#39;: &#39;&#39;,
                 &#39;token&#39;: r&#39;<\?&#39;},
                {&#39;name&#39;: &#39;open&#39;, &#39;next&#39;: &#39;php&#39;, &#39;extra&#39;: 0, &#39;start&#39;: 0, &#39;end&#39;: 0, &#39;cache&#39;: &#39;&#39;,
                 &#39;token&#39;: r&#39;<\?php&#39;}],
            &#39;php&#39;: [
                {&#39;name&#39;: &#39;close&#39;, &#39;next&#39;: &#39;default&#39;, &#39;extra&#39;: 0,
                 &#39;token&#39;: r&#39;\?>&#39;, &#39;start&#39;: 0, &#39;end&#39;: 0, &#39;cache&#39;: &#39;&#39;},
                {&#39;name&#39;: &#39;lnum&#39;, &#39;next&#39;: &#39;&#39;, &#39;extra&#39;: 0, &#39;start&#39;: 0, &#39;end&#39;: 0, &#39;cache&#39;: &#39;&#39;,
                 &#39;token&#39;: r&#39;[0-9]+&#39;},
                {&#39;name&#39;: &#39;dnum&#39;, &#39;next&#39;: &#39;&#39;, &#39;extra&#39;: 0, &#39;start&#39;: 0, &#39;end&#39;: 0, &#39;cache&#39;: &#39;&#39;,
                 &#39;token&#39;: r&#39;([0-9]*\.[0-9]+)|([0-9]+\.[0-9]*)&#39;},
                {&#39;name&#39;: &#39;exponent&#39;, &#39;next&#39;: &#39;&#39;, &#39;extra&#39;: 0, &#39;start&#39;: 0, &#39;end&#39;: 0, &#39;cache&#39;: &#39;&#39;,
                 &#39;token&#39;: r&#39;(([0-9]+|([0-9]*\.[0-9]+)|([0-9]+\.[0-9]*))[eE][+-]?[0-9]+)&#39;},
                {&#39;name&#39;: &#39;hnum&#39;, &#39;next&#39;: &#39;&#39;, &#39;extra&#39;: 0, &#39;start&#39;: 0, &#39;end&#39;: 0, &#39;cache&#39;: &#39;&#39;,
                 &#39;token&#39;: r&#39;0x[0-9a-fA-F]+&#39;},
                {&#39;name&#39;: &#39;bnum&#39;, &#39;next&#39;: &#39;&#39;, &#39;extra&#39;: 0, &#39;start&#39;: 0, &#39;end&#39;: 0, &#39;cache&#39;: &#39;&#39;,
                 &#39;token&#39;: r&#39;0b[01]+&#39;},
                {&#39;name&#39;: &#39;label&#39;, &#39;next&#39;: &#39;&#39;, &#39;extra&#39;: 0, &#39;start&#39;: 0, &#39;end&#39;: 0, &#39;cache&#39;: &#39;&#39;,
                 &#39;token&#39;: r&#39;[a-zA-Z_\x7f-\xff][a-zA-Z0-9_\x7f-\xff]*&#39;},
                {&#39;name&#39;: &#39;comment&#39;, &#39;next&#39;: &#39;commentline&#39;, &#39;extra&#39;: 1,
                 &#39;token&#39;: r&#39;//&#39;, &#39;start&#39;: 0, &#39;end&#39;: 0, &#39;cache&#39;: &#39;&#39;},
                {&#39;name&#39;: &#39;comment&#39;, &#39;next&#39;: &#39;commentline&#39;, &#39;extra&#39;: 1,
                 &#39;token&#39;: r&#39;#&#39;, &#39;start&#39;: 0, &#39;end&#39;: 0, &#39;cache&#39;: &#39;&#39;},
                {&#39;name&#39;: &#39;comment&#39;, &#39;next&#39;: &#39;comment&#39;, &#39;extra&#39;: 1,
                 &#39;token&#39;: r&#39;/\*&#39;, &#39;start&#39;: 0, &#39;end&#39;: 0, &#39;cache&#39;: &#39;&#39;},
                {&#39;name&#39;: &#39;string&#39;, &#39;next&#39;: &#39;string1&#39;, &#39;extra&#39;: 1,
                 &#39;token&#39;: r&#39;\&#39;&#39;, &#39;start&#39;: 0, &#39;end&#39;: 0, &#39;cache&#39;: &#39;&#39;},
                {&#39;name&#39;: &#39;string&#39;, &#39;next&#39;: &#39;string2&#39;, &#39;extra&#39;: 1,
                 &#39;token&#39;: r&#39;"&#39;, &#39;start&#39;: 0, &#39;end&#39;: 0, &#39;cache&#39;: &#39;&#39;},
                {&#39;name&#39;: &#39;symbol&#39;, &#39;next&#39;: &#39;&#39;, &#39;extra&#39;: 0, &#39;start&#39;: 0, &#39;end&#39;: 0, &#39;cache&#39;: &#39;&#39;,
                 &#39;token&#39;: r&#39;[\\\{\};:,\.\[\]\(\)\|\^&\+-/\*=%!~$<>\?@]&#39;}],
            &#39;string1&#39;: [
                {&#39;name&#39;: &#39;string&#39;, &#39;next&#39;: &#39;php&#39;, &#39;extra&#39;: 0,
                 &#39;token&#39;: r&#39;\&#39;&#39;, &#39;start&#39;: 0, &#39;end&#39;: 0, &#39;cache&#39;: &#39;&#39;},
                {&#39;name&#39;: &#39;string&#39;, &#39;next&#39;: &#39;escape1&#39;, &#39;extra&#39;: 1,
                 &#39;token&#39;: r&#39;\\&#39;, &#39;start&#39;: 0, &#39;end&#39;: 0, &#39;cache&#39;: &#39;&#39;},
                {&#39;name&#39;: &#39;string&#39;, &#39;next&#39;: &#39;&#39;, &#39;extra&#39;: 1,
                 &#39;token&#39;: r&#39;&#39;, &#39;start&#39;: 0, &#39;end&#39;: 0, &#39;cache&#39;: &#39;&#39;}],
            &#39;escape1&#39;: [
                {&#39;name&#39;: &#39;string&#39;, &#39;next&#39;: &#39;string1&#39;, &#39;extra&#39;: 1,
                 &#39;token&#39;: r&#39;.&#39;, &#39;start&#39;: 0, &#39;end&#39;: 0, &#39;cache&#39;: &#39;&#39;}],
            &#39;string2&#39;: [
                {&#39;name&#39;: &#39;string&#39;, &#39;next&#39;: &#39;php&#39;, &#39;extra&#39;: 0,
                 &#39;token&#39;: r&#39;\&#39;&#39;, &#39;start&#39;: 0, &#39;end&#39;: 0, &#39;cache&#39;: &#39;&#39;},
                {&#39;name&#39;: &#39;string&#39;, &#39;next&#39;: &#39;escape2&#39;, &#39;extra&#39;: 1,
                 &#39;token&#39;: r&#39;\\&#39;, &#39;start&#39;: 0, &#39;end&#39;: 0, &#39;cache&#39;: &#39;&#39;},
                {&#39;name&#39;: &#39;string&#39;, &#39;next&#39;: &#39;&#39;, &#39;extra&#39;: 1,
                 &#39;token&#39;: r&#39;&#39;, &#39;start&#39;: 0, &#39;end&#39;: 0, &#39;cache&#39;: &#39;&#39;}],
            &#39;escape2&#39;: [
                {&#39;name&#39;: &#39;string&#39;, &#39;next&#39;: &#39;string2&#39;, &#39;extra&#39;: 1,
                 &#39;token&#39;: r&#39;.&#39;, &#39;start&#39;: 0, &#39;end&#39;: 0, &#39;cache&#39;: &#39;&#39;}],
            &#39;commentline&#39;: [
                {&#39;name&#39;: &#39;comment&#39;, &#39;next&#39;: &#39;php&#39;, &#39;extra&#39;: 0,
                 &#39;token&#39;: r&#39;(\r|\n|\r\n)&#39;, &#39;start&#39;: 0, &#39;end&#39;: 0, &#39;cache&#39;: &#39;&#39;},
                {&#39;name&#39;: &#39;comment&#39;, &#39;next&#39;: &#39;php&#39;, &#39;extra&#39;: 0,
                 &#39;token&#39;: r&#39;&#39;, &#39;start&#39;: 0, &#39;end&#39;: 0, &#39;cache&#39;: &#39;&#39;}],
            &#39;comment&#39;: [
                {&#39;name&#39;: &#39;comment&#39;, &#39;next&#39;: &#39;php&#39;, &#39;extra&#39;: 0,
                 &#39;token&#39;: r&#39;\*/&#39;, &#39;start&#39;: 0, &#39;end&#39;: 0, &#39;cache&#39;: &#39;&#39;},
                {&#39;name&#39;: &#39;comment&#39;, &#39;next&#39;: &#39;&#39;, &#39;extra&#39;: 1,
                 &#39;token&#39;: r&#39;&#39;, &#39;start&#39;: 0, &#39;end&#39;: 0, &#39;cache&#39;: &#39;&#39;}]}

Copy after login

The above is the detailed content of Parsing Converter 3: Lexical part of handwritten PHP to Python compiler. For more information, please follow other related articles on the PHP Chinese website!

Statement of this Website

The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn

Hot AI Tools

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress images for free

Clothoff.io

AI clothes remover

Video Face Swap

Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Article

What's New in Windows 11 KB5054979 & How to Fix Update Issues

4 weeks ago By DDD

How to fix KB5055523 fails to install in Windows 11?

3 weeks ago By DDD

InZoi: How To Apply To School And University

1 months ago By DDD

How to fix KB5055518 fails to install in Windows 10?

3 weeks ago By DDD

Where to find the Site Office Key in Atomfall

4 weeks ago By DDD

Hot Tools

Notepad++7.3.1

Easy-to-use and free code editor

SublimeText3 Chinese version

Chinese version, very easy to use

Zend Studio 13.0.1

Powerful PHP integrated development environment

Dreamweaver CS6

Visual web development tools

SublimeText3 Mac version

God-level code editing software (SublimeText3)

Hot Topics

Where is the login entrance for gmail email?

7903

Java Tutorial

1652

CakePHP Tutorial

1411

Laravel Tutorial

1303

PHP Tutorial

1248

Related knowledge

How to avoid being detected by the browser when using Fiddler Everywhere for man-in-the-middle reading? Apr 02, 2025 am 07:15 AM

How to avoid being detected when using FiddlerEverywhere for man-in-the-middle readings When you use FiddlerEverywhere...

How to handle comma-separated list query parameters in FastAPI? Apr 02, 2025 am 06:51 AM

Fastapi ...

How to solve permission issues when using python --version command in Linux terminal? Apr 02, 2025 am 06:36 AM

Using python in Linux terminal...

How to teach computer novice programming basics in project and problem-driven methods within 10 hours? Apr 02, 2025 am 07:18 AM

How to teach computer novice programming basics within 10 hours? If you only have 10 hours to teach computer novice some programming knowledge, what would you choose to teach...

Python asyncio Telnet connection is disconnected immediately: How to solve server-side blocking problem? Apr 02, 2025 am 06:30 AM

About Pythonasyncio...

How to get news data bypassing Investing.com's anti-crawler mechanism? Apr 02, 2025 am 07:03 AM

Understanding the anti-crawling strategy of Investing.com Many people often try to crawl news data from Investing.com (https://cn.investing.com/news/latest-news)...

Python 3.6 loading pickle file error ModuleNotFoundError: What should I do if I load pickle file '__builtin__'? Apr 02, 2025 am 06:27 AM

Loading pickle file in Python 3.6 environment error: ModuleNotFoundError:Nomodulenamed...

What is the reason why pipeline files cannot be written when using Scapy crawler? Apr 02, 2025 am 06:45 AM

Discussion on the reasons why pipeline files cannot be written when using Scapy crawlers When learning and using Scapy crawlers for persistent data storage, you may encounter pipeline files...

See all articles