Turn plain text into business value
TYPO3 Conference Asia 2012
Christian Trabold
Logs == Plain Text
Log == Protocol
Every process creates logs
Aug 21 02:47:46 machine http://host/: - cms:
Locking [simple::...]: Released lock
Aug 21 02:47:48 machine http://host/: - Core:
File "/var/www/favicon.ico" was not found.
192.168.156.1 - - [20/Aug/2012:15:24:23 +0000]
"GET /typo3/ajax.php
HTTP/1.1" 200 7385 "http://host/index.php"
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:14.0)
Gecko/20100101 Firefox/14.0.1"
Just boring text?
NO!
Interesing!
192.168.156.1 <= Who?
[20/Aug/2012:15:24:23 +0000] <= When?
"GET /index.php <= What?
HTTP/1.1" 200 <= How?
"Mozilla/5.0 (Macintosh…) <= Who?
Gecko/20100101 Firefox/14.0.1" <= Who?
Alarming!
192.168.156.1 <= Who?
[20/Aug/2012:15:24:23 +0000] <= When?
"GET /index.php?id=1 or 1=1 -- <= What?
HTTP/1.1" 200 <= How?
"Mozilla/5.0 (Macintosh…) <= Who?
Gecko/20100101 Firefox/14.0.1" <= Who?
♥
Heartbeat of your application
And the best…
You can write own logs!
TYPO3 API
t3lib_div::sysLog($msg, $extKey, $severity=0)API change in TYPO3 6.0 https://review.typo3.org/#/c/13604/
This should be implemented around the source code, including the Core and both frontend and backend, logging serious errors.
… and for every action you find interesting!
To implement the sysLog in your applications:
t3lib_div::sysLog(
'[write message in English here]',
'extension key'
);Severities
0 info (default)
1 notice
2 warning
3 error
4 fatalNext:
Collect TYPO3 logs in system log
Send TYPO3 logs to syslog:
$TYPO3_CONF_VARS['SYS']['systemLog'] = 'syslog,USER';USER namespace avoids conflicts with other logs
Log every message
$TYPO3_CONF_VARS['SYS']['systemLogLevel'] = '0';No need for logging in database anymore
$TYPO3_CONF_VARS['SYS']['belogErrorReporting'] = 0;== More DB performance! :)
localconf.php
$TYPO3_CONF_VARS['SYS']['systemLog'] = 'syslog,USER';
$TYPO3_CONF_VARS['SYS']['systemLogLevel'] = '0';
$TYPO3_CONF_VARS['SYS']['belogErrorReporting'] = 0;Also make shure you enable PHP error logging in php.ini
log_errors = On
Example use cases
- Newsletter subscribe / unsubscribe
- User log in / out
- Failed Backend logins (Attacks?)
- Contact form submitted
Q: Why do I need this when I have Google Analytics? A: Because you can manage your data more flexible
Important
- Write Logs!
- Whereever you feel the need
- Logs provide useful values
- Values can be collected for metrics
- Metrics show business patterns
- Beware of cached elements
Measure to improve
You can’t improve what you
don’t measure.
http://www.littlethingsmatter.com/blog/2010/08/23/you-cant-improve-what-you-dont-measure/
Advanced metrics
Best practices
Normalize the logs
Convert different log formats into structured JSON objects
Because the log formats differe A LOT, we need to normalize and enrich them
Proven Architecture
fetch convert send
TYPO3 ---> SYSLOG <----- LOGSTASH ---> WEB INTERFACE
Web Interfaces like Kibana, Graylog2
Logstash
Normalize logs & creates structured JSON objects from plain text
Kibana
Focused on searching logs and statistics
Graylog2
Focused on reporting and streamlining logs
Searching is a sub-feature
If in doubt
log it!
Careful with user data
only store anonymous IDs
Showtime!
