Home php教程 php手册 php中持久化存储模块开发总结

php中持久化存储模块开发总结

Jun 13, 2016 am 10:46 AM
php load exist storage develop Summarize Endurance module need need project

 

在项目中发现经常有这种需求,需要加载一些大的固定的格式化数据,比如对战中的一些技能数据,物品等。这些数据都是只读数据,并且可能会比较大,目前来看大约有上万条复杂数据,如果serialize的话,纯文本有20M左右。尝试过直接放一个array在php文件里,结果发现require这个文件很耗时,可能会花费几十ms的时间,并且这个时候io很重,因为需要加载几十m数据到内存;另外去调研了一下sqlite,这个东西还算比较靠谱,但问题在于,比如写操作函数,使用起来很不爽;于是就产生了自己写一个扩展的想法。于是折腾之旅就此展开。

 

 

 

一开始想的是,直接在MINIT里调用zend_execute_script方法来加载一个php文件,返回一个zval来存储到全局变量里。结果后来仔细一琢磨发现根本就是妄想。原因在于MINIT的时候php的vm还没初始化完,不可能让你调用zend_execute_script方法,并且这个方法也不会返回一个zval,要想拿到zval必须从EG中去拿,很麻烦。

 

 

 

于是转换思路,尝试用unserialize/serialize,结果发现,php_var_unserialize在MINIT阶段果然是可以调用的。于是开搞,调用这个方法得到一个zval,然后存在全局变量里,在get方法里返回这个zval。写完之后,在测试的时候杯具的发现,只要调用就会core呀。于是查文档,自己思考,最终发现PHP_RSHUTDOWN_FUNCTION函数里会将所有非pealloc分配的的变量给清除。因此在MINIT阶段还正常的数据,到了Request阶段已经被free了。

 

 

 

于是再查文档,发现php里提供了pealloc这类函数来提供persistent的数据分配。于是再转换思路,将全局变量里的hashtable用pealloc来分配,并且将hastable设置成persistent的(谢天谢地php的hashtable还要存代码和vm,因此有这个功能)。但是杯具的是php_unserialize只会返回一个zval,你根本无法控制它是否是persistent的。没办法,只能调用zend_hash_copy来做了。写完之后再测试,发现还是core,这就不明白了,为啥呢?中午吃饭的时候,突然想到,可能是浅拷贝的问题,zend_hash_copy提供了一个copy函数而我没有设置它。再加上深拷贝函数之后再测试,发现果然可以用了,使用起来很清爽。 www.2cto.com

 

 

 

接下来进行测试发现,内存使用率不能忍,一个20m的数据文件加载到内存,需要大约100m左右的内存,如果有100个php-cgi进程,那就多要10G内存,这根本不能忍。于是设想,可以用共享内存来解决这个问题,反正这部分数据只要能读就行了。php-cgi的主进程负责MINIT操作,子进程只要读这部分数据就行了。但是很麻烦的是,php里没提供任何让用户维护内存的接口,于是只能一个函数一个函数的扒了。

 

 

 

仔细看了一下php的hashtable实现,发现比较复杂,而且关键用到了realloc函数,这个太让人无语了,总不能我也写一个内存管理吧。目前只用共享内存实现了一个简单的线程分配内存的函数,从共享内存上依次往后分配空间。但是还好,这部分功能是resize功能根本不需要。因为我的目标是将php_var_unserialize里得到的zval拷贝到共享内存而已,而大小我明显已经知道了。并且也不需要updatea功能,因为是全新的copy。最终弄完之后,发现可以使用了,内存使用率果然降了。

 

 

 

接下来进行压力测试,突然发现又开始core了,这根本不能忍呀,为啥呀?根据core文件,发现是里面的hashtable的refcount降到0了。于是各种测试,发现单线程情况下是ok的,只有多线程大压力情况下会挂。于是想到refcount是会被修改的,而且多线程修改的话,必须可能被改乱。那怎么办呢?总不能加锁吧。

 

 

 

后来仔细想了一下,突然想到只要我每次在返回这个zval里将顶层的zval的refcount修改为大于php-cgi进程数的值,那即使会被改乱也没啥问题,因为根本不会改到0。于是修改了之后再测试,发现果然靠谱了。

 

 

 

到此,整个问题基本解决。但是还有另外一个问题,在重启Php-cgi时还是会core,原因是,当时把正在使用的一些变量给强制写成0了。其实共享内存的正确用法是,一个进程来写,另外的进程来读,但是我这个应用里将共享内存当做绝对地址在使用,因此不可能在一个地方写,在其他地方读,除非shmat里的第二个参数修改为固定值,但是这个就需要对进程的地址分配有充分的了解,知道哪些内存根本不可能被使用。不过这个应该还好,因为Php-cgi进程有内存上限,所以应该可以找到一块内存在php-cgi运行过程中无法被用到的。不过具体的情况得接下来具体研究一下。

作者 无心云

Statement of this Website
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

Video Face Swap

Video Face Swap

Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Tools

Notepad++7.3.1

Notepad++7.3.1

Easy-to-use and free code editor

SublimeText3 Chinese version

SublimeText3 Chinese version

Chinese version, very easy to use

Zend Studio 13.0.1

Zend Studio 13.0.1

Powerful PHP integrated development environment

Dreamweaver CS6

Dreamweaver CS6

Visual web development tools

SublimeText3 Mac version

SublimeText3 Mac version

God-level code editing software (SublimeText3)

PHP 8.4 Installation and Upgrade guide for Ubuntu and Debian PHP 8.4 Installation and Upgrade guide for Ubuntu and Debian Dec 24, 2024 pm 04:42 PM

PHP 8.4 brings several new features, security improvements, and performance improvements with healthy amounts of feature deprecations and removals. This guide explains how to install PHP 8.4 or upgrade to PHP 8.4 on Ubuntu, Debian, or their derivati

7 PHP Functions I Regret I Didn't Know Before 7 PHP Functions I Regret I Didn't Know Before Nov 13, 2024 am 09:42 AM

If you are an experienced PHP developer, you might have the feeling that you’ve been there and done that already.You have developed a significant number of applications, debugged millions of lines of code, and tweaked a bunch of scripts to achieve op

How To Set Up Visual Studio Code (VS Code) for PHP Development How To Set Up Visual Studio Code (VS Code) for PHP Development Dec 20, 2024 am 11:31 AM

Visual Studio Code, also known as VS Code, is a free source code editor — or integrated development environment (IDE) — available for all major operating systems. With a large collection of extensions for many programming languages, VS Code can be c

Explain JSON Web Tokens (JWT) and their use case in PHP APIs. Explain JSON Web Tokens (JWT) and their use case in PHP APIs. Apr 05, 2025 am 12:04 AM

JWT is an open standard based on JSON, used to securely transmit information between parties, mainly for identity authentication and information exchange. 1. JWT consists of three parts: Header, Payload and Signature. 2. The working principle of JWT includes three steps: generating JWT, verifying JWT and parsing Payload. 3. When using JWT for authentication in PHP, JWT can be generated and verified, and user role and permission information can be included in advanced usage. 4. Common errors include signature verification failure, token expiration, and payload oversized. Debugging skills include using debugging tools and logging. 5. Performance optimization and best practices include using appropriate signature algorithms, setting validity periods reasonably,

PHP Program to Count Vowels in a String PHP Program to Count Vowels in a String Feb 07, 2025 pm 12:12 PM

A string is a sequence of characters, including letters, numbers, and symbols. This tutorial will learn how to calculate the number of vowels in a given string in PHP using different methods. The vowels in English are a, e, i, o, u, and they can be uppercase or lowercase. What is a vowel? Vowels are alphabetic characters that represent a specific pronunciation. There are five vowels in English, including uppercase and lowercase: a, e, i, o, u Example 1 Input: String = "Tutorialspoint" Output: 6 explain The vowels in the string "Tutorialspoint" are u, o, i, a, o, i. There are 6 yuan in total

How do you parse and process HTML/XML in PHP? How do you parse and process HTML/XML in PHP? Feb 07, 2025 am 11:57 AM

This tutorial demonstrates how to efficiently process XML documents using PHP. XML (eXtensible Markup Language) is a versatile text-based markup language designed for both human readability and machine parsing. It's commonly used for data storage an

Explain late static binding in PHP (static::). Explain late static binding in PHP (static::). Apr 03, 2025 am 12:04 AM

Static binding (static::) implements late static binding (LSB) in PHP, allowing calling classes to be referenced in static contexts rather than defining classes. 1) The parsing process is performed at runtime, 2) Look up the call class in the inheritance relationship, 3) It may bring performance overhead.

What are PHP magic methods (__construct, __destruct, __call, __get, __set, etc.) and provide use cases? What are PHP magic methods (__construct, __destruct, __call, __get, __set, etc.) and provide use cases? Apr 03, 2025 am 12:03 AM

What are the magic methods of PHP? PHP's magic methods include: 1.\_\_construct, used to initialize objects; 2.\_\_destruct, used to clean up resources; 3.\_\_call, handle non-existent method calls; 4.\_\_get, implement dynamic attribute access; 5.\_\_set, implement dynamic attribute settings. These methods are automatically called in certain situations, improving code flexibility and efficiency.

See all articles