Uninote
Uninote
用户根目录
每日点滴

<!DOCTYPE html> <html><head><meta http-equiv='Content-Type' content='text/html; charset=utf8'><title>log parser/ana</title> <script> var g_resRoot = '../../__res/'; var g_json_outline = '{"children":[{"children":[],"level":1,"lvNumber":"1","text":"排错"},{"children":[],"level":1,"lvNumber":"2","text":"注意 note"},{"children":[],"level":1,"lvNumber":"3","text":"config"},{"children":[],"level":1,"lvNumber":"4","text":"log ana 日志"},{"children":[],"level":1,"lvNumber":"5","text":"check list"},{"children":[],"level":1,"lvNumber":"6","text":"部署步骤"},{"children":[],"level":1,"lvNumber":"7","text":"commits"},{"children":[],"level":1,"lvNumber":"8","text":"ana.php 一次性分析处理所有日志"},{"children":[],"level":1,"lvNumber":"9","text":"ana 优化 fgets"},{"children":[{"children":[],"level":2,"lvNumber":"10.1","text":"将上次的读取位置保存到mysql数据库中:"},{"children":[],"level":2,"lvNumber":"10.2","text":"mynote_log 加入logger字段"},{"children":[],"level":2,"lvNumber":"10.3","text":"logger时间精确到毫秒"}],"level":1,"lvNumber":"10","text":"mynote seaslog 日志解析"},{"children":[],"level":1,"lvNumber":"11","text":"实时分析 位置缓存"},{"children":[{"children":[],"level":2,"lvNumber":"12.1","text":"关联sessionid和用户"}],"level":1,"lvNumber":"12","text":"重复问题"},{"children":[],"level":1,"lvNumber":"13","text":"测试点"},{"children":[],"level":1,"lvNumber":"14","text":"log ana todo"},{"children":[],"level":1,"lvNumber":"15","text":"dev tools"},{"children":[],"level":1,"lvNumber":"16","text":"test cases"},{"children":[],"level":1,"lvNumber":"17","text":"mynote request log"},{"children":[],"level":1,"lvNumber":"18","text":"mynote log 混合编码"}],"level":0,"lvNumber":"","text":"log parser/ana"}'; </script> <link rel='stylesheet' type='text/css' href='../../__res/common.css'><style>.s0{color:#000000;} .s1{color:#0000ff;} .s2{color:#000000;} .s3{color:#7f01f5;} .s4{color:#000000;} .s5{color:#0000ff;} .s6{color:#000000;} .s7{color:#000000;} .s8{color:#000000;} .s9{color:#000000;} .s10{color:#000000;} .s11{color:#000000;} .s12{color:#0000ff;} .s13{color:#660000;} .s14{color:#660000;} .s15{color:#008000;} .s16{color:#000080;} .s17{color:#7f01f5;} .s18{color:#0000ff;} .s19{color:#000000;} .s20{color:#000000;} </style> </head><body> <script src='../../__res/util.js'></script><div id='outline'> <li><a href='#ol_0' style='font-weight:bold; font-size:22px'>log parser/ana</a></li> <li> <a href='#ol_1' >排错</a></li> <li> <a href='#ol_2' >注意 note</a></li> <li> <a href='#ol_3' >config</a></li> <li> <a href='#ol_4' >log ana 日志</a></li> <li> <a href='#ol_5' >check list</a></li> <li> <a href='#ol_6' >部署步骤</a></li> <li> <a href='#ol_7' >commits</a></li> <li> <a href='#ol_8' >ana.php 一次性分析处理所有日志</a></li> <li> <a href='#ol_9' >ana 优化 fgets</a></li> <li> <a href='#ol_10' >mynote seaslog 日志解析</a></li> <li> <a href='#ol_11' >将上次的读取位置保存到mysql数据库中:</a></li> <li> <a href='#ol_12' >mynote_log 加入logger字段</a></li> <li> <a href='#ol_13' >logger时间精确到毫秒</a></li> <li> <a href='#ol_14' >实时分析 位置缓存</a></li> <li> <a href='#ol_15' >重复问题</a></li> <li> <a href='#ol_16' >关联sessionid和用户</a></li> <li> <a href='#ol_17' >测试点</a></li> <li> <a href='#ol_18' >log ana todo</a></li> <li> <a href='#ol_19' >dev tools</a></li> <li> <a href='#ol_20' >test cases</a></li> <li> <a href='#ol_21' >mynote request log</a></li> <li> <a href='#ol_22' >mynote log 混合编码</a></li> </div>

<t id='5346e0d5-0cd1-4cfa-9bf5-4f2fd02ce03a'> <pre class='tnd_head'>lv1= lv2= type=</pre> <pre id='ol_0' class='title'>log parser/ana</pre> <pre><d class='s0'><a href='log_query.html'>log query 日志查询工具 </a></d></pre> <pre><d class='s1'><a href='http://a.add.my/log_storage'>http://a.add.my/log_storage</a></d></pre> <pre><d class='s1'></d></pre> <pre><d class='s2'>〖E:\uninote\additional_item\log_storage\bin\Analysis〗</d></pre> <pre><d class='s3'>obsolete:</d><d class='s2'></d></pre> <pre><d class='s2'><a href='http://a.myres.my/log.viewer/'>http://a.myres.my/log.viewer/</a></d></pre> <pre><d class='s2'>〖E:\uninote\mynote_res\logviewer〗</d></pre> <pre><d class='s4'></d></pre> <pre><d class='s4'>〖E:\uninote\mynote_res\log_analysis〗</d><d class='s5'></d></pre> <pre><d class='s2'>exception分析工具,相同的堆栈,md5作为标示</d></pre> <pre><d class='s2'>nginx log分析 耗时记录</d></pre> <pre><d class='s2'></d></pre> <pre><d class='s2'>定时器分析:??</d></pre> <pre><d class='s2'>E:\uninote\mynote\art_workman\start.php</d></pre> <pre><d class='s2'>E:\uninote\mynote\art_workman\Application\Mlog.php</d></pre> <pre><d class='s2'></d></pre> <h1 id='ol_1'>排错</h1> <pre><d class='s2'>mynote_indexfile 清理,重新分析</d></pre> <pre><d class='s0'></d></pre> <h1 id='ol_2'>注意 note</h1> <pre><d class='s0'>注意:编码、linefeed</d></pre> <pre><d class='s6'>目前只有mynote_log的分析有多行的问题,其他的日志都是单行记录。</d><d class='s0'></d></pre> <pre><d class='s0'></d></pre> <t id='fee932bb-f3a0-44b0-9ba8-08b2abb5a982'> <pre class='tnd_head'>lv1= lv2= type=</pre> <h1 id='ol_3'>config</h1> <pre><d class='s0'>同 reader config</d></pre> <pre><d class='s0'><a href='log_query.html#8d97b55a-901a-48c0-b882-f2420e2529f6'>log reader config </a></d></pre> </t> <pre><d class='s7'></d></pre> <t id='7274a260-f570-4f31-a63a-575a15ff6744'> <pre class='tnd_head'>lv1= lv2= type=</pre> <h1 id='ol_4'>log ana 日志</h1> <pre><d class='s0'>E:\uninote\additional_item\log_storage\bin\Analysis\Analysis.php</d></pre> <pre><d class='s0'>E:\uninote\additional_item\log_storage\logs\log.txt</d></pre> </t> <pre><d class='s0'></d></pre> <h1 id='ol_5'>check list</h1> <pre><d class='s8'>SHA-1: 4312417640264ac9086b466fccd2571eb85c5ca9</d></pre> <pre><d class='s8'>* bug fix</d><d class='s9'>:添加</d><d class='s10'>common.php </d><d class='s9'>的引用</d></pre> <pre><d class='s8'></d></pre> <pre><d class='s0'></d></pre> <h1 id='ol_6'>部署步骤</h1> <pre><d class='s2'>- 在php.ini中配置seaslog的输出模板:</d></pre> <pre><d class='s2'>seaslog.default_template = "%t -||- %L -||- %R -||- %C -||- %F -||- %Q -||- %M"</d></pre> <pre><d class='s2'></d></pre> <pre><d class='s2'>- 在mynote数据库中执行所有子目录下的create.sql,生成对应的日志表。</d></pre> <pre><d class='s2'><img src='..\pic\debcd9a08ba084425ae50f6c2d515224.jpg' /></d><d class='s2'></d></pre> <pre><d class='s2'></d></pre> <pre><d class='s2'>- 执行一次登录,并随意浏览网站</d></pre> <pre><d class='s2'>- 运行<a href='http://a.add.my/log_storage/ana.php'>http://a.add.my/log_storage/ana.php</a>,手动将生成的日志解析并存储到数据库中</d></pre> <pre><d class='s2'></d></pre> <pre><d class='s2'>或者直接访问 <a href='http://a.add.my/log_storage/'>http://a.add.my/log_storage/</a>,会自动分析</d></pre> <pre><d class='s0'></d></pre> <h1 id='ol_7'>commits</h1> <pre><d class='s11'>tbname & logpath 可以从url中覆盖</d></pre> <pre><d class='s8'>ed83711eca2a13c16db4d3a38a26983a397c7747</d></pre> <pre><d class='s11'></d></pre> <pre><d class='s11'>eg:</d></pre> <pre><d class='s11'><a href='http://a.add.my/log_analysis/mynote_request/process.php?logpath=logs.txt&tbname=abc'>http://a.add.my/log_analysis/mynote_request/process.php?logpath=logs.txt&tbname=abc</a></d></pre> <pre><d class='s11'>注意,必要时参数需要urlEncode</d></pre> <pre><d class='s12'></d></pre> <pre><d class='s2'>SHA-1: d04f40f687b150a79162145a5b6b196eaa31c5e3</d></pre> <pre><d class='s2'>* 【更新】- 日志采集功能更新,更改文件记录末尾行数存储mysql避免行号不对的错误</d><d class='s9'></d></pre> <pre><d class='s2'></d></pre> <t id='7354bcbf-d875-4392-b90d-85785577553e'> <pre class='tnd_head'>lv1= lv2= type=</pre> <h1 id='ol_8'>ana.php 一次性分析处理所有日志</h1> <pre><d class='s4'>直接 http 访问:</d><d class='s5'></d></pre> <pre><d class='s11'><a href='http://a.add.my/log_storage/ana.php'>http://a.add.my/log_storage/ana.php</a></d><d class='s12'></d></pre> <pre><d class='s11'>结果见日志</d></pre> <pre><d class='s0'><a href='log_ana.html#fee932bb-f3a0-44b0-9ba8-08b2abb5a982'>config </a></d></pre> <pre><d class='s0'></d></pre> <pre><d class='s0'>也可以在访问 GUI 时触发</d></pre> </t> <pre><d class='s12'></d></pre> <t id='581b79f5-4e9e-448a-9368-851d66f2e324'> <pre class='tnd_head'>lv1= lv2= type=</pre> <h1 id='ol_9'>ana 优化 fgets</h1> <pre><d class='s13'>这个有性能问题:</d><d class='s14'></d></pre> <pre><d class='s14'>$lines </d><d class='s7'>= explode(</d><d class='s15'>"</d><d class='s16'>\n</d><d class='s15'>"</d><d class='s7'>, </d><d class='s14'>$content</d><d class='s7'>);</d></pre> <pre><d class='s14'>$countLines </d><d class='s7'>= count(</d><d class='s14'>$lines</d><d class='s7'>);</d></pre> </t> <pre><d class='s11'></d></pre> <t id='d32ad704-63d1-4dec-a283-b248cd5e5003'> <pre class='tnd_head'>lv1= lv2= type=</pre> <h1 id='ol_10'>mynote seaslog 日志解析</h1> <pre><d class='s11'>文件夹:mynote_log</d></pre> <pre><d class='s12'></d></pre> <pre><d class='s0'>数据库表结构:</d></pre> <pre><d class='s0'>datetime level </d><d class='s17'>uri</d><d class='s0'> class filename reqid message</d></pre> <pre><d class='s0'>其中,需要将level字段翻译为level数字,参见<a href='..\mynote/seaslog.html#8e58dcac-7192-4da2-bf90-cbe60c638eee'>SeasLog 共将日志分成8个级别 </a></d></pre> <pre><d class='s0'></d></pre> <pre><d class='s0'>seaslog配置:</d></pre> <pre><d class='s0'>seaslog.default_template = "%t  %L  %R  %C  %F  %Q  %M"</d></pre> <pre><d class='s0'>seaslog.default_logger=default</d></pre> <pre><d class='s0'>seaslog.disting_folder = 1</d></pre> <pre><d class='s0'>seaslog.disting_type=0</d></pre> <pre><d class='s0'>seaslog.disting_by_hour=0</d></pre> <pre><d class='s0'>seaslog.use_buffer=1</d></pre> <pre><d class='s0'>seaslog.buffer_size=100</d></pre> <pre><d class='s0'>seaslog.level=8</d></pre> <pre><d class='s0'>seaslog.trace_error=1</d></pre> <pre><d class='s0'>seaslog.trace_exception=0</d></pre> <pre><d class='s0'></d></pre> <pre><d class='s0'>优化:</d></pre> <pre><d class='s0'>放到开头,作为一条log的标记。</d></pre> <pre><d class='s0'>fgets,不要一次性读取所有文件。</d></pre> <pre><d class='s0'></d></pre> <pre><d class='s0'>%R 不用了</d></pre> </t> <t id='789cc797-1c88-440a-9763-e72fcc74f109'> <pre class='tnd_head'>lv1=mynote lv2= type=done</pre> <pre><d class='s2'>不要存成一行,显示时需要换行</d></pre> <pre><d class='s2'> [cmd]: git commit -m 'update' 2>&1[update_0137822208 63a2a1e] 'update' 2 files changed, 6 insertions(+), 4 deletions(-) create mode 100644 1535965471</d></pre> <pre><d class='s2'></d></pre> <pre><d class='s2'>white-space: pre-line;</d></pre> </t> <pre><d class='s12'></d></pre> <t id='d6eaca46-50e7-40a2-bf88-64446e6cca5e'> <pre class='tnd_head'>lv1=mynote lv2= type=done</pre> <h2 id='ol_11'>将上次的读取位置保存到mysql数据库中:</h2> <pre><d class='s12'>log_analysis/common.php</d></pre> <pre><d class='s14'>$indexfile_path </d><d class='s7'>= </d><d class='s15'>"./last_end_index.txt"</d><d class='s7'>;</d></pre> <pre><d class='s16'>if </d><d class='s7'>(!file_exists(</d><d class='s14'>$indexfile_path</d><d class='s7'>)) {</d></pre> <pre><d class='s7'> file_put_contents(</d><d class='s14'>$indexfile_path</d><d class='s7'>, </d><d class='s18'>0</d><d class='s7'>);</d></pre> <pre><d class='s7'>}</d></pre> </t> <pre><d class='s12'></d></pre> <t id='d7ca66a6-0e76-4691-84e3-c6ba5f9cfe20'> <pre class='tnd_head'>lv1=mynote lv2= type=done</pre> <h2 id='ol_12'>mynote_log 加入logger字段</h2> <pre><d class='s11'>用于保存logger</d><d class='s12'></d></pre> </t> <pre><d class='s12'></d></pre> <t id='af24a816-d93b-4618-9d01-2794625523f2'> <pre class='tnd_head'>lv1=mynote lv2= type=done</pre> <h2 id='ol_13'>logger时间精确到毫秒</h2> <pre><d class='s2'>时间使用%t,修改解析处:</d></pre> <pre><d class='s2'>"%t -||- %L -||- %C -||- %F -||- %Q -||- %M"</d></pre> </t> <pre><d class='s12'></d></pre> <h1 id='ol_14'>实时分析 位置缓存</h1> <pre><d class='s2'>为了支持实时分析,在mynote_indexfile表中记录上次分析到的位置(文件偏移量)</d></pre> <pre><d class='s2'>注意大小写问题</d></pre> <pre><d class='s2'></d></pre> <h1 id='ol_15'>重复问题</h1> <t id='7d6e2466-e913-43f6-8846-0acc87f839f5'> <pre class='tnd_head'>lv1=mynote lv2= type=done</pre> <pre><d class='s2'>reqnum用get_time?重复太多</d></pre> </t> <pre><d class='s0'></d></pre> <pre><d class='s0'>log sessionid重复查询:</d></pre> <pre><d class='s0'>select count() as cnt, login.sessionid </d></pre> <pre><d class='s0'>from mynote_login login</d></pre> <pre><d class='s0'>GROUP BY login.sessionid</d></pre> <pre><d class='s0'>HAVING cnt >1</d></pre> <pre><d class='s0'></d></pre> <pre><d class='s0'>特定id查看:</d></pre> <pre><d class='s0'>SELECT * FROM mynote_login WHERE sessionid = 'k18v46c4k11m7tjs5vhndc0ktv'</d></pre> <pre><d class='s0'></d></pre> <pre><d class='s0'>mynote_request 记录请求时间</d></pre> <pre><d class='s0'></d></pre> <pre><d class='s0'>reqnum 重复查询:</d></pre> <pre><d class='s0'>SELECT count() as cnt, r.reqnum, r.sessionid</d></pre> <pre><d class='s0'>from mynote_request r</d></pre> <pre><d class='s0'>GROUP BY r.reqnum</d><d class='s2'></d></pre> <pre><d class='s2'>having cnt > 1</d></pre> <pre><d class='s2'></d></pre> <pre><d class='s2'>select from mynote_request where reqnum = 6951333</d></pre> <pre><d class='s2'></d></pre> <h2 id='ol_16'>关联sessionid和用户</h2> <pre><d class='s11'>在用户登录时记录到logger login:E:\uninote\mynote\basic\runtime\seaslog\login</d></pre> <pre><d class='s11'></d></pre> <pre><d class='s11'>但用户退出登录后,下次再登录,sessionid并不会改变。此时一条log就会对应两个登录记录(用户),查询时就是一条log显示两次。</d></pre> <pre><d class='s11'></d></pre> <pre><d class='s11'>解决办法:注销后前端清除sessionid</d></pre> <pre><d class='s8'>SHA-1: a8f4e3f1e8e03c484bf605ff8c7b6d965b58570f</d></pre> <pre><d class='s8'> </d><d class='s9'>注销后删除</d><d class='s10'>cookie</d><d class='s9'>,方便日志分析</d></pre> <pre><d class='s11'></d></pre> <pre><d class='s19'>但这种方法并不一定有效,比如:服务器上session过期,此时重新登录,还是会存在同一个sessionid两次登录的情况</d><d class='s11'></d></pre> <pre><d class='s11'></d></pre> <pre><d class='s2'>终极解决办法:<a href='log_query.html#acaa91cf-802c-4ed7-8800-5c8d1b25b53e'>每次变换reqnum,自动插入分隔符号 </a></d><d class='s0'></d></pre> <pre><d class='s2'></d></pre> <h1 id='ol_17'>测试点</h1> <pre><d class='s0'>图片接口</d></pre> <pre><d class='s0'></d></pre> <t id='3adb0b3d-30a8-488e-ad35-f56a692d63d7'> <pre class='tnd_head'>lv1= lv2= type=todo</pre> <h1 id='ol_18'>log ana todo</h1> <pre><d class='s2'></d></pre> <pre><d class='s2'>清理指定日期的数据 重新分析</d></pre> <pre><d class='s4'></d></pre> <pre><d class='s0'><a href='..\php/php.html#0a3bb354-b83e-4550-8168-8aca5fc9da01'>CT_log 写文件有并发问题 </a></d></pre> <pre><d class='s0'><a href='..\php/php.html#037a6d0e-2544-426d-9380-933712699c09'>logx.txt 按天分隔 </a></d></pre> <pre><d class='s0'></d></pre> <pre><d class='s0'>log user agent</d></pre> </t> <pre><d class='s12'></d></pre> <h1 id='ol_19'>dev tools</h1> <pre><d class='s0'></d></pre> <pre><d class='s6'>TRUNCATE table mynote_log;</d></pre> <pre><d class='s6'>truncate table mynote_request;</d></pre> <pre><d class='s6'>update mynote_indexfile set last_end_index = 0 where indexfile_path =</d><d class='s0'> 'E:/uninote/mynote/basic/runtime/seaslog/default/20190930.log';</d></pre> <pre><d class='s0'></d></pre> <pre><d class='s0'>select * from mynote_indexfile where indexfile_path = 'E:/mynote/basic/runtime/seaslog/default/20190422.log';</d></pre> <pre><d class='s0'></d></pre> <pre><d class='s0'>SELECT * from mynote_log;</d></pre> <pre><d class='s0'></d></pre> <h1 id='ol_20'>test cases</h1> <pre><d class='s10'>empty</d></pre> <pre><d class='s10'>one line</d></pre> <pre><d class='s10'>one line + oneline</d></pre> <pre><d class='s10'>mline + mline</d></pre> <pre><d class='s10'>oneline + mline</d></pre> <pre><d class='s10'>mline + online</d></pre> <pre><d class='s0'></d></pre> <t id='c286ec55-f0ff-48dc-a233-b242a49818ba'> <pre class='tnd_head'>lv1= lv2= type=</pre> <h1 id='ol_21'>mynote request log</h1> <pre><d class='s2'>SHA-1: f69244347bd8a3cbad51a5cd5a988ccec973c11d</d></pre> <pre><d class='s20'>* 用 finally 保证每个请求的 log 完整(REQUEST_END 的记录)</d><d class='s2'></d></pre> <pre><d class='s2'></d></pre> <pre><d class='s0'><a href='..\php/php.html#329bb4d5-dce3-448c-a67c-8b7bd301e38a'>php error reporting/handling exception 异常、错误处理 </a></d></pre> <pre><d class='s5'></d></pre> <pre><d class='s2'>finally {</d></pre> <pre><d class='s2'> /* CT log end */</d></pre> <pre><d class='s2'> CT_log("--REQUEST_END:" . $g_REQUEST_URI . "|| PHPSESSID: " . $g_PHPSESSID . "|| rnum: " . $request_num .</d></pre> <pre><d class='s2'> "|| time: " . (get_time() - $g_begintime) . "s");</d></pre> <pre><d class='s2'> CT_flush(true);</d><d class='s7'></d></pre> <pre><d class='s7'>}</d></pre> <pre><d class='s5'></d></pre> </t> <pre><d class='s2'></d></pre> <t id='5986db4c-2979-43c0-98fa-0483ea962b7a'> <pre class='tnd_head'>lv1= lv2= type=</pre> <h1 id='ol_22'>mynote log 混合编码</h1> <pre><d class='s20'>windows 下 yii redis 抛出的错是 gbk 编码的,其他都是utf8:</d><d class='s2'></d></pre> <pre><d class='s2'>[error][yii\redis\Connection] Failed to open redis DB connection (localhost:6379, database=0): 10061 - 由于目标计算机积极拒绝,无法连接。</d></pre> </t> </t> </body></html>

liaocheng_todo

log_query

点赞(0) 阅读(51) 举报
目录
标题