Web服务器日志统计分析完全解决方案 (2)
4.2 使用apache自带的rotatelogs实现日志轮循 apache提供了将不把日志直接写入文件,而是通过管道发送给另外 一个程序的能力,这样就大大的加强了对日志进行处理的能力,这个通过管道得到的程序可以是任何程序:如日志分析,压缩日志等。要实现将日志写到管
4.2 使用apache自带的rotatelogs实现日志轮循
apache提供了将不把日志直接写入文件,而是通过管道发送给另外 一个程序的能力,这样就大大的加强了对日志进行处理的能力,这个通过管道得到的程序可以是任何程序:如日志分析,压缩日志等。要实现将日志写到管道只需要将配置中日志文件部分的内容替换为“|程序名“即可,例如:
# compressed logs CustomLog "|/usr/bin/gzip -c >> /var/log/access_log.gz" common |
这样就可以实用apache自带的轮循工具:rotatelogs来对日志文件进行轮循。rotatelogs基本是用来按时间或按大小控制日志的。
CustomLog "|/www/bin/rotatelogs /www/logs/secfocus/access_log 86400" common |
上 面的示例中apache访问日志被发送给程序rotatelogs,rotatelogs将日志写入/www/logs/secfocus /access_log,并每隔86400秒(一天)对日志进行一次轮循。轮循以后的文件名为/www/logs/secfocus /access_log.nnnn,这里nnn是开始记录日志的时间。因此为了将日志按天对齐就需要在凌晨00:00启动服务,使得每天轮循得到的日志刚 好是完整一天的日志,以提供给访问统计分析程序进行处理。如果是00:00开始生成新的日志,那么轮循得到的日志就是access_log.0000。
4.3 使用cronolog实现日志轮循
首先需要下载和安装cronolog,可以到http://www.cronolog.org下载最新版本的cronolog。下载完毕以后,解压安装即可,方法如下所示:
[root@mail root]# tar xvfz cronolog-1.6.2.tar.gz [root@mail root]# cd cronolog-1.6.2 [root@mail cronolog-1.6.2]# ./configure [root@mail cronolog-1.6.2]# make [root@mail cronolog-1.6.2]# make check [root@mail cronolog-1.6.2]# make install |
这就完成了cronolog的配置和安装,默认情况下cronolog是安装在/usr/local/sbin下。
修改apache日志配置命令如下所示:
CustomLog "|/usr/local/sbin/cronolog /www/logs/secfocus/%w/access_log" combined |
这里%w表示按照日期星期几在不同的目录下保存日志,这种方式会保存一周的日志。为了进行日志分析,需要每天将该日志文件拷贝(或移动,如果不希望保存一周的日志)到一个固定的位置以方便日志分析统计文件进行处理,实用crontab –e,如下添加定时任务:
5 0 * * * /bin/mv /www/logs/secfocus/`date -v-1d +\% w`/access_log /www/logs/secfocus/access_log_yesterday |
这样再使用日志统计分析程序的对文件access_log_yesterday进行处理。
对 于使用负载均衡技术的大型站点,就存在多个服务器的访问日志的合并处理问题.对于这种情况,各个服务器定义或移动日志文件时就不能使用 access_log_yesterday了,就应该带上服务器编号了,例如服务器IP地址等信息以区分。然后在各个服务器上运行网站镜像和备份服务rsyncd(参考文章” 用rsync实现网站镜像和备份”,ttp://www.linuxaid.com.cn/engineer/ideal/article /rsync.htm),然后将每个服务器每天的安装配置文件通过rsync下载到专门进行访问统计分析的服务器上进行合并。
合并多个服务器的日志文件,例如:log1 log2 log3并输出到log_all中的方法是:
sort -m -t " " -k 4 -o log_all log1 log2 log3
-m: 使用 merge优化算法,-k 4表示根据时间进行排序,-o表示将排序结果存放到指定的文件中 |
五、日志统计分析程序webalizer的安装和配置
webalizer是一个高效的、免费的web服务器日志分析程序。其分析结果是HTML文件格式,从而可以很方便的通过web服务器进行浏览。Internet上的很多站点都使用webalizer进行web服务器日志分析。Webalizer具有以下一些特性:
1、是用C写的程序,所以其具有很高的运行效率。在主频为200Mhz的机器上,webalizer每秒钟可以分析10000条记录,所以分析一个40M大小的日志文件只需要15秒。
2、webalizer支持标准的一般日志文件格式(Common Logfile Format);除此之外,也支持几种组合日志格式(CombinedLogfile Format)的变种,从而可以统计客户情况以及客户操作系统类型。并且现在webalizer已经可以支持wu-ftpd xferlog日志格式以及squid日志文件格式了。
3、支持命令行配置以及配置文件。
4、可以支持多种语言,也可以自己进行本地化工作。
5、支持多种平台,比如UNIX、linux、NT, OS/2 和MacOS等。
上图是webalizer生成的访问统计分析报表第一页的内容,这里包含每个月的平均访问量的表格和条形图统计分析情况。点击每个月分,可以得到这个月每天的详细统计信息。
5.1 安装
在安装以前首先需要确保系统已经安装有gd库,可以使用:
[root@mail root]# rpm -qa|grep gdgd-devel-1.8.4-4gdbm-devel-1.8.0-14gdbm-1.8.0-14sysklogd-1.4.1-8gd-1.8.4-4 |
来确认系统已经安装有gd-deve和gd两个rpm包。
安装webalizer有两种方式,一种是下载源代码来安装,一种是直接使用rpm包来安装。
使用rpm包方式安装非常简单,从rpmfind.net找到webalizer包,下载以后:
rpm –ivh webalizer-2.01_10-1.i386.rpm |
即可实现安装。
对于源代码方式首先需要从http://www.mrunix.net/webalizer/下载,然后安装,首先解开源代码包:
tar xvzf webalizer-2.01-10-src.tgz |
在生成的目录中有个lang目录,该目录中保存了各种语言文件,但是只有繁体中文版本,可以自己转换成简体,或者自己重新翻译一下。然后进入生成的目录:
cd webalizer-2.01-10./configuremake --with-language=Chinesemake install |
编译成功后,会在/usr/local/bin/目录下安装一个webalizer可执行文件。
5.2 配置和运行
对webalizer运行的控制可以通过配置文件或者在命令行指定参数的两种方式进行。而使用配置文件方式是比较简单和灵活的,适用于自动web服务器日志统计分析的应用环境。
webalizer的默认配置文件为/etc/webalizer.conf,当启动 Webalizer时没有使用“-f“选项时,Webalizer就会寻找文件/etc/webalizer.conf,也可以使用“-f”来指定配置文 件(当服务器有虚拟主机时,就需要配置多份不同的webalizer配置文件,不同的虚拟主机的webalizer使用不同的配置文件。 Webalizer.conf配置文件中需要修改的配置选项如下:
LogFile /www/logs/secfocus/access_log |
用来指示配置文件的路径信息,webalizer会将该日志文件作为输入进行统计分析;
OutputDir /www/htdocs/secfocus/usage |
用来指示生成的统计报表的保存目录,在前面我们使用alias,使得用户可以使用http://www.secfocus.com/usage/来访问统计报表。
HostName www.secfocus.com |
用来指示主机名,统计报表中会引用该主机名。
其他选项就无需修改,配置文件修改完毕以后,就需要在定时webalizer,每天生成当日的统计分析。
以root身份运行:crontab –e 进入定时运行任务编辑状态,加入如下任务:
5 0 * * * /usr/local/bin/webalizer –f /etc/secfocus.webalizer.conf15 0 * * * /usr/local/bin/webalizer –f /etc/tomorrowtel.webalizer.conf |
我们这里假设系统运行有两个虚拟主机,并分别定义了日志分析配置文件 secfocus.webalizer.conf和tomorrowtel.webalizer.conf。这样我们定义在凌晨00:05对 secfocus的日志进行统计分析;在凌晨00:15对tomorrowtel的日志进行统计分析。
然后第二天分别使用http://www.secfocus.com/usage/和http://www.tomorrowtel.com/usage来察看各自的日志分析报表。
六、保护日志统计分析报告不被未授权用户访问
我们肯定不会希望自己网站访问统计信息随意被别人浏览,因此需要将usage目录保护起来,只允许合法用户访问。这里可以采用apache自带的基本的认证机制,配置以后再连接这个地址就会需要用户提供密码才能访问该页面:
1、条件
在配置文件中对目录"/"应该设置为:
DocumentRoot /www/htdocs/secfocus/AccessFileName .htaccessAllowOverride All |
2、需求
需求:限制对http://www.secfocus.com/usage/的访问,要求用户认证才能访问。这里设置用户为"admin",口令为"12345678"。
3、使用htpasswd建立用户文件
htpasswd -c /www/.htpasswd admin
此程序会询问用户"admin"的口令,你输入"12345678",两次生效。
4、建立.htaccess文件
用vi在/www/logs/secfocus/usage/目录下建立一个文件.htaccess,写入以下几行:
AuthName admin-onlyAuthType BasicAuthUserFile /www/.htpasswdrequire user admin |
5、测试
这时候通过浏览器访问http://www.secfocus.com/usage就会弹出框请求输入用户名和口令,这时候输入admin、12345678就可以才可以访问访问日志统计分析报表。

Hot AI Tools

Undresser.AI Undress
AI-powered app for creating realistic nude photos

AI Clothes Remover
Online AI tool for removing clothes from photos.

Undress AI Tool
Undress images for free

Clothoff.io
AI clothes remover

Video Face Swap
Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Article

Hot Tools

Notepad++7.3.1
Easy-to-use and free code editor

SublimeText3 Chinese version
Chinese version, very easy to use

Zend Studio 13.0.1
Powerful PHP integrated development environment

Dreamweaver CS6
Visual web development tools

SublimeText3 Mac version
God-level code editing software (SublimeText3)

Hot Topics

CrystalDiskMark is a small HDD benchmark tool for hard drives that quickly measures sequential and random read/write speeds. Next, let the editor introduce CrystalDiskMark to you and how to use crystaldiskmark~ 1. Introduction to CrystalDiskMark CrystalDiskMark is a widely used disk performance testing tool used to evaluate the read and write speed and performance of mechanical hard drives and solid-state drives (SSD). Random I/O performance. It is a free Windows application and provides a user-friendly interface and various test modes to evaluate different aspects of hard drive performance and is widely used in hardware reviews

foobar2000 is a software that can listen to music resources at any time. It brings you all kinds of music with lossless sound quality. The enhanced version of the music player allows you to get a more comprehensive and comfortable music experience. Its design concept is to play the advanced audio on the computer The device is transplanted to mobile phones to provide a more convenient and efficient music playback experience. The interface design is simple, clear and easy to use. It adopts a minimalist design style without too many decorations and cumbersome operations to get started quickly. It also supports a variety of skins and Theme, personalize settings according to your own preferences, and create an exclusive music player that supports the playback of multiple audio formats. It also supports the audio gain function to adjust the volume according to your own hearing conditions to avoid hearing damage caused by excessive volume. Next, let me help you

Cloud storage has become an indispensable part of our daily life and work nowadays. As one of the leading cloud storage services in China, Baidu Netdisk has won the favor of a large number of users with its powerful storage functions, efficient transmission speed and convenient operation experience. And whether you want to back up important files, share information, watch videos online, or listen to music, Baidu Cloud Disk can meet your needs. However, many users may not understand the specific use method of Baidu Netdisk app, so this tutorial will introduce in detail how to use Baidu Netdisk app. Users who are still confused can follow this article to learn more. ! How to use Baidu Cloud Network Disk: 1. Installation First, when downloading and installing Baidu Cloud software, please select the custom installation option.

NetEase Mailbox, as an email address widely used by Chinese netizens, has always won the trust of users with its stable and efficient services. NetEase Mailbox Master is an email software specially created for mobile phone users. It greatly simplifies the process of sending and receiving emails and makes our email processing more convenient. So how to use NetEase Mailbox Master, and what specific functions it has. Below, the editor of this site will give you a detailed introduction, hoping to help you! First, you can search and download the NetEase Mailbox Master app in the mobile app store. Search for "NetEase Mailbox Master" in App Store or Baidu Mobile Assistant, and then follow the prompts to install it. After the download and installation is completed, we open the NetEase email account and log in. The login interface is as shown below

The role of a DHCP relay is to forward received DHCP packets to another DHCP server on the network, even if the two servers are on different subnets. By using a DHCP relay, you can deploy a centralized DHCP server in the network center and use it to dynamically assign IP addresses to all network subnets/VLANs. Dnsmasq is a commonly used DNS and DHCP protocol server that can be configured as a DHCP relay server to help manage dynamic host configurations in the network. In this article, we will show you how to configure dnsmasq as a DHCP relay server. Content Topics: Network Topology Configuring Static IP Addresses on a DHCP Relay D on a Centralized DHCP Server

MetaMask (also called Little Fox Wallet in Chinese) is a free and well-received encryption wallet software. Currently, BTCC supports binding to the MetaMask wallet. After binding, you can use the MetaMask wallet to quickly log in, store value, buy coins, etc., and you can also get 20 USDT trial bonus for the first time binding. In the BTCCMetaMask wallet tutorial, we will introduce in detail how to register and use MetaMask, and how to bind and use the Little Fox wallet in BTCC. What is MetaMask wallet? With over 30 million users, MetaMask Little Fox Wallet is one of the most popular cryptocurrency wallets today. It is free to use and can be installed on the network as an extension

Common challenges faced by machine learning algorithms in C++ include memory management, multi-threading, performance optimization, and maintainability. Solutions include using smart pointers, modern threading libraries, SIMD instructions and third-party libraries, as well as following coding style guidelines and using automation tools. Practical cases show how to use the Eigen library to implement linear regression algorithms, effectively manage memory and use high-performance matrix operations.

Xiaomi car software provides remote car control functions, allowing users to remotely control the vehicle through mobile phones or computers, such as opening and closing the vehicle's doors and windows, starting the engine, controlling the vehicle's air conditioner and audio, etc. The following is the use and content of this software, let's learn about it together . Comprehensive list of Xiaomi Auto app functions and usage methods 1. The Xiaomi Auto app was launched on the Apple AppStore on March 25, and can now be downloaded from the app store on Android phones; Car purchase: Learn about the core highlights and technical parameters of Xiaomi Auto, and make an appointment for a test drive. Configure and order your Xiaomi car, and support online processing of car pickup to-do items. 3. Community: Understand Xiaomi Auto brand information, exchange car experience, and share wonderful car life; 4. Car control: The mobile phone is the remote control, remote control, real-time security, easy
