블로그 운영 중에 종종 OOM(Out Of Memory)이 발생을 해서 httpd 또는 mariaDB 프로세스가 다운되어 블로그 운영이 중단된다.
VM Guest, CentOS7 x86_64, Memory 4G로 운영 중인데 hacking 공격으로 의심되는 공격이 발생되면 OOM이 발생되곤 한다.
그래서 로그와 대처 방법등을 확인해 본다.
1. messages 로그 확인
messages 로그를 확인해 보면 2018.05.01 12:50~12:51 경에 프로세스가 다운되었다.
[root@mws ~]# cat /var/log/messages* | grep -i 'out of' May 1 12:50:52 mws kernel: Out of memory: Kill process 1152 (mysqld) score 108 or sacrifice child May 1 12:50:59 mws kernel: Out of memory: Kill process 999 (httpd) score 16 or sacrifice child May 1 12:51:04 mws kernel: Out of memory: Kill process 25658 (httpd) score 14 or sacrifice child May 1 12:51:04 mws kernel: Out of memory: Kill process 25658 (httpd) score 14 or sacrifice child May 1 12:51:07 mws kernel: Out of memory: Kill process 813 (httpd) score 13 or sacrifice child
2. httpd error 로그 확인
OOM이 발생한 시각의 http error 로그를 보면 아래와 같다.
[root@mws ~]# cat /var/log/httpd/www.mapoo.net-error_log* | grep require
[Tue May 15 14:45:39.695540 2018] [:error] [pid 28363] [client 66.249.79.52:62631] require('wp-blog-header.php'), require_once('wp-includes/template-loader.php'), include('/themes/spacious/single.php'), get_footer, locate_template, load_template, require_once('/themes/spacious/footer.php'), get_sidebar, locate_template, load_template, require_once('/themes/spacious/sidebar-footer.php'), dynamic_sidebar, call_user_func_array, WP_Widget->display_callback, CountPerDay_Widget->widget, call_user_func, CountPerDay->getUserAll, CountPerDayCore->mysqlQuery\xec\x9d\xb4(\xea\xb0\x80) \xeb\xa7\x8c\xeb\x93\xa0 \xec\x9b\x8c\xeb\x93\x9c\xed\x94\x84\xeb\xa0\x88\xec\x8a\xa4 \xeb\x8d\xb0\xec\x9d\xb4\xed\x84\xb0\xeb\xb2\xa0\xec\x9d\xb4\xec\x8a\xa4 \xec\x98\xa4\xeb\xa5\x98 Server shutdown in progress(\xec\xbf\xbc\xeb\xa6\xac SELECT COUNT(*) FROM (SELECT 1 FROM wp_cpd_counter GROUP BY date, ip) t)
[Tue May 01 12:50:52.688661 2018] [:error] [pid 1289] [client 95.181.179.218:51431] require('wp-blog-header.php'), require_once('wp-includes/template-loader.php'), do_action('template_redirect'), WP_Hook->do_action, WP_Hook->apply_filters, call_user_func_array, wordfence::templateRedir, wordfence::doEarlyAccessLogging, wfLog->logLeechAndBlock, wfDB->queryWrite\xec\x9d\xb4(\xea\xb0\x80) \xeb\xa7\x8c\xeb\x93\xa0 \xec\x9b\x8c\xeb\x93\x9c\xed\x94\x84\xeb\xa0\x88\xec\x8a\xa4 \xeb\x8d\xb0\xec\x9d\xb4\xed\x84\xb0\xeb\xb2\xa0\xec\x9d\xb4\xec\x8a\xa4 \xec\x98\xa4\xeb\xa5\x98 Lost connection to MySQL server during query(\xec\xbf\xbc\xeb\xa6\xac insert IGNORE into wp_wfVulnScanners (IP, ctime, hits) values ('\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\xff\xff_\xb5\xb3\xda', unix_timestamp(), 1) ON DUPLICATE KEY UPDATE ctime = unix_timestamp(), hits = hits + 1)
[Tue May 01 12:50:52.690071 2018] [:error] [pid 1296] [client 95.181.179.218:51444] require('wp-blog-header.php'), require_once('wp-includes/template-loader.php'), do_action('template_redirect'), WP_Hook->do_action, WP_Hook->apply_filters, call_user_func_array, redirect_canonical, redirect_guess_404_permalink\xec\x9d\xb4(\xea\xb0\x80) \xeb\xa7\x8c\xeb\x93\xa0 \xec\x9b\x8c\xeb\x93\x9c\xed\x94\x84\xeb\xa0\x88\xec\x8a\xa4 \xeb\x8d\xb0\xec\x9d\xb4\xed\x84\xb0\xeb\xb2\xa0\xec\x9d\xb4\xec\x8a\xa4 \xec\x98\xa4\xeb\xa5\x98 Lost connection to MySQL server during query(\xec\xbf\xbc\xeb\xa6\xac SELECT ID FROM wp_posts WHERE post_name LIKE 'thumb-php%' AND post_type IN ('post', 'page', 'attachment') AND post_status = 'publish')
[Tue May 01 12:50:52.691638 2018] [:error] [pid 1285] [client 95.181.179.218:51461] require('wp-blog-header.php'), require_once('wp-includes/template-loader.php'), do_action('template_redirect'), WP_Hook->do_action, WP_Hook->apply_filters, call_user_func_array, wordfence::templateRedir, wordfence::doEarlyAccessLogging, wfLog->logLeechAndBlock, wfDB->queryWrite\xec\x9d\xb4(\xea\xb0\x80) \xeb\xa7\x8c\xeb\x93\xa0 \xec\x9b\x8c\xeb\x93\x9c\xed\x94\x84\xeb\xa0\x88\xec\x8a\xa4 \xeb\x8d\xb0\xec\x9d\xb4\xed\x84\xb0\xeb\xb2\xa0\xec\x9d\xb4\xec\x8a\xa4 \xec\x98\xa4\xeb\xa5\x98 Lost connection to MySQL server during query(\xec\xbf\xbc\xeb\xa6\xac insert IGNORE into wp_wfVulnScanners (IP, ctime, hits) values ('\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\xff\xff_\xb5\xb3\xda', unix_timestamp(), 1) ON DUPLICATE KEY UPDATE ctime = unix_timestamp(), hits = hits + 1)
[Tue May 01 12:50:52.691924 2018] [:error] [pid 1262] [client 95.181.179.218:51438] require('wp-blog-header.php'), require_once('wp-includes/template-loader.php'), include('/themes/spacious/404.php'), get_header, locate_template, load_template, require_once('/themes/spacious/header.php'), wp_head, do_action('wp_head'), WP_Hook->do_action, WP_Hook->apply_filters, call_user_func_array, wp_print_head_scripts, print_head_scripts, script_concat_settings, get_site_option, get_network_option, get_option\xec\x9d\xb4(\xea\xb0\x80) \xeb\xa7\x8c\xeb\x93\xa0 \xec\x9b\x8c\xeb\x93\x9c\xed\x94\x84\xeb\xa0\x88\xec\x8a\xa4 \xeb\x8d\xb0\xec\x9d\xb4\xed\x84\xb0\xeb\xb2\xa0\xec\x9d\xb4\xec\x8a\xa4 \xec\x98\xa4\xeb\xa5\x98 Lost connection to MySQL server during query(\xec\xbf\xbc\xeb\xa6\xac SELECT option_value FROM wp_options WHERE option_name = 'can_compress_scripts' LIMIT 1)
[Tue May 01 12:50:52.693969 2018] [:error] [pid 1283] [client 95.181.179.218:51450] require('wp-blog-header.php'), require_once('wp-includes/template-loader.php'), do_action('template_redirect'), WP_Hook->do_action, WP_Hook->apply_filters, call_user_func_array, redirect_canonical, redirect_guess_404_permalink\xec\x9d\xb4(\xea\xb0\x80) \xeb\xa7\x8c\xeb\x93\xa0 \xec\x9b\x8c\xeb\x93\x9c\xed\x94\x84\xeb\xa0\x88\xec\x8a\xa4 \xeb\x8d\xb0\xec\x9d\xb4\xed\x84\xb0\xeb\xb2\xa0\xec\x9d\xb4\xec\x8a\xa4 \xec\x98\xa4\xeb\xa5\x98 Lost connection to MySQL server during query(\xec\xbf\xbc\xeb\xa6\xac SELECT ID FROM wp_posts WHERE post_name LIKE 'thumb-php%' AND post_type IN ('post', 'page', 'attachment') AND post_status = 'publish')
[Tue May 01 12:50:52.697560 2018] [:error] [pid 1228] [client 95.181.179.218:52859] require('wp-blog-header.php'), require_once('wp-includes/template-loader.php'), include('/themes/spacious/404.php'), get_header, locate_template, load_template, require_once('/themes/spacious/header.php'), wp_head, do_action('wp_head'), WP_Hook->do_action, WP_Hook->apply_filters, call_user_func_array, wp_print_head_scripts, print_head_scripts, script_concat_settings, get_site_option, get_network_option, get_option\xec\x9d\xb4(\xea\xb0\x80) \xeb\xa7\x8c\xeb\x93\xa0 \xec\x9b\x8c\xeb\x93\x9c\xed\x94\x84\xeb\xa0\x88\xec\x8a\xa4 \xeb\x8d\xb0\xec\x9d\xb4\xed\x84\xb0\xeb\xb2\xa0\xec\x9d\xb4\xec\x8a\xa4 \xec\x98\xa4\xeb\xa5\x98 Lost connection to MySQL server during query(\xec\xbf\xbc\xeb\xa6\xac SELECT option_value FROM wp_options WHERE option_name = 'can_compress_scripts' LIMIT 1)
[Tue May 01 12:50:52.699475 2018] [:error] [pid 1249] [client 95.181.179.218:52842] require('wp-blog-header.php'), require_once('wp-load.php'), require_once('wp-config.php'), require_once('wp-settings.php'), do_action('init'), WP_Hook->do_action, WP_Hook->apply_filters, call_user_func_array, user_functions_for_captcha_booster, blocking_visitors_captcha_booster, get_ip_address_for_captcha_booster\xec\x9d\xb4(\xea\xb0\x80) \xeb\xa7\x8c\xeb\x93\xa0 \xec\x9b\x8c\xeb\x93\x9c\xed\x94\x84\xeb\xa0\x88\xec\x8a\xa4 \xeb\x8d\xb0\xec\x9d\xb4\xed\x84\xb0\xeb\xb2\xa0\xec\x9d\xb4\xec\x8a\xa4 \xec\x98\xa4\xeb\xa5\x98 Lost connection to MySQL server during query(\xec\xbf\xbc\xeb\xa6\xac SELECT meta_value FROM wp_captcha_booster_meta WHERE meta_key='other_settings')
[Tue May 01 12:50:53.019894 2018] [:error] [pid 1265] [client 95.181.179.218:52962] require('wp-blog-header.php'), require_once('wp-load.php'), require_once('wp-config.php'), require_once('wp-settings.php'), include_once('/plugins/wp-captcha-booster/wp-captcha-booster.php'), call_captcha_booster, include_once('/plugins/wp-captcha-booster/includes/logical-captcha.php')\xec\x9d\xb4(\xea\xb0\x80) \xeb\xa7\x8c\xeb\x93\xa0 \xec\x9b\x8c\xeb\x93\x9c\xed\x94\x84\xeb\xa0\x88\xec\x8a\xa4 \xeb\x8d\xb0\xec\x9d\xb4\xed\x84\xb0\xeb\xb2\xa0\xec\x9d\xb4\xec\x8a\xa4 \xec\x98\xa4\xeb\xa5\x98 Lost connection to MySQL server during query(\xec\xbf\xbc\xeb\xa6\xac SELECT meta_value FROM wp_captcha_booster_meta WHERE meta_key = 'display_settings')
[Tue May 01 12:50:53.088018 2018] [:error] [pid 1234] [client 95.181.179.218:52951] require('wp-blog-header.php'), require_once('wp-load.php'), require_once('wp-config.php'), require_once('wp-settings.php'), include_once('/plugins/wp-captcha-booster/wp-captcha-booster.php'), call_captcha_booster, include_once('/plugins/wp-captcha-booster/includes/logical-captcha.php')\xec\x9d\xb4(\xea\xb0\x80) \xeb\xa7\x8c\xeb\x93\xa0 \xec\x9b\x8c\xeb\x93\x9c\xed\x94\x84\xeb\xa0\x88\xec\x8a\xa4 \xeb\x8d\xb0\xec\x9d\xb4\xed\x84\xb0\xeb\xb2\xa0\xec\x9d\xb4\xec\x8a\xa4 \xec\x98\xa4\xeb\xa5\x98 Lost connection to MySQL server during query(\xec\xbf\xbc\xeb\xa6\xac SELECT meta_value FROM wp_captcha_booster_meta WHERE meta_key = 'error_message')
- OOM이 처음 발생하기 시작한 5월 1일 12:50:52에 95.181.179.218 IP(러시아)에서 php 공격을 시도한 것으로 보인다.
- 5월 12의 접근 IP 66.249.79.52은 google의 IP 이다.
- 러시아 IP와 구글 IP의 차이점은 WP_Hook->do_action, WP_Hook->apply_filters 구문을 쓰고 안쓰고 차이인거 같다.
아래는 hook 문구만으로 검색해 본 결과
[root@mws ~]# cat /var/log/httpd/www.mapoo.net-error_log* | grep require | grep -i hook
[Tue May 01 12:50:52.688661 2018] [:error] [pid 1289] [client 95.181.179.218:51431] require('wp-blog-header.php'), require_once('wp-includes/template-loader.php'), do_action('template_redirect'), WP_Hook->do_action, WP_Hook->apply_filters, call_user_func_array, wordfence::templateRedir, wordfence::doEarlyAccessLogging, wfLog->logLeechAndBlock, wfDB->queryWrite\xec\x9d\xb4(\xea\xb0\x80) \xeb\xa7\x8c\xeb\x93\xa0 \xec\x9b\x8c\xeb\x93\x9c\xed\x94\x84\xeb\xa0\x88\xec\x8a\xa4 \xeb\x8d\xb0\xec\x9d\xb4\xed\x84\xb0\xeb\xb2\xa0\xec\x9d\xb4\xec\x8a\xa4 \xec\x98\xa4\xeb\xa5\x98 Lost connection to MySQL server during query(\xec\xbf\xbc\xeb\xa6\xac insert IGNORE into wp_wfVulnScanners (IP, ctime, hits) values ('\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\xff\xff_\xb5\xb3\xda', unix_timestamp(), 1) ON DUPLICATE KEY UPDATE ctime = unix_timestamp(), hits = hits + 1)
[Tue May 01 12:50:52.690071 2018] [:error] [pid 1296] [client 95.181.179.218:51444] require('wp-blog-header.php'), require_once('wp-includes/template-loader.php'), do_action('template_redirect'), WP_Hook->do_action, WP_Hook->apply_filters, call_user_func_array, redirect_canonical, redirect_guess_404_permalink\xec\x9d\xb4(\xea\xb0\x80) \xeb\xa7\x8c\xeb\x93\xa0 \xec\x9b\x8c\xeb\x93\x9c\xed\x94\x84\xeb\xa0\x88\xec\x8a\xa4 \xeb\x8d\xb0\xec\x9d\xb4\xed\x84\xb0\xeb\xb2\xa0\xec\x9d\xb4\xec\x8a\xa4 \xec\x98\xa4\xeb\xa5\x98 Lost connection to MySQL server during query(\xec\xbf\xbc\xeb\xa6\xac SELECT ID FROM wp_posts WHERE post_name LIKE 'thumb-php%' AND post_type IN ('post', 'page', 'attachment') AND post_status = 'publish')
[Tue May 01 12:50:52.691638 2018] [:error] [pid 1285] [client 95.181.179.218:51461] require('wp-blog-header.php'), require_once('wp-includes/template-loader.php'), do_action('template_redirect'), WP_Hook->do_action, WP_Hook->apply_filters, call_user_func_array, wordfence::templateRedir, wordfence::doEarlyAccessLogging, wfLog->logLeechAndBlock, wfDB->queryWrite\xec\x9d\xb4(\xea\xb0\x80) \xeb\xa7\x8c\xeb\x93\xa0 \xec\x9b\x8c\xeb\x93\x9c\xed\x94\x84\xeb\xa0\x88\xec\x8a\xa4 \xeb\x8d\xb0\xec\x9d\xb4\xed\x84\xb0\xeb\xb2\xa0\xec\x9d\xb4\xec\x8a\xa4 \xec\x98\xa4\xeb\xa5\x98 Lost connection to MySQL server during query(\xec\xbf\xbc\xeb\xa6\xac insert IGNORE into wp_wfVulnScanners (IP, ctime, hits) values ('\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\xff\xff_\xb5\xb3\xda', unix_timestamp(), 1) ON DUPLICATE KEY UPDATE ctime = unix_timestamp(), hits = hits + 1)
[Tue May 01 12:50:52.691924 2018] [:error] [pid 1262] [client 95.181.179.218:51438] require('wp-blog-header.php'), require_once('wp-includes/template-loader.php'), include('/themes/spacious/404.php'), get_header, locate_template, load_template, require_once('/themes/spacious/header.php'), wp_head, do_action('wp_head'), WP_Hook->do_action, WP_Hook->apply_filters, call_user_func_array, wp_print_head_scripts, print_head_scripts, script_concat_settings, get_site_option, get_network_option, get_option\xec\x9d\xb4(\xea\xb0\x80) \xeb\xa7\x8c\xeb\x93\xa0 \xec\x9b\x8c\xeb\x93\x9c\xed\x94\x84\xeb\xa0\x88\xec\x8a\xa4 \xeb\x8d\xb0\xec\x9d\xb4\xed\x84\xb0\xeb\xb2\xa0\xec\x9d\xb4\xec\x8a\xa4 \xec\x98\xa4\xeb\xa5\x98 Lost connection to MySQL server during query(\xec\xbf\xbc\xeb\xa6\xac SELECT option_value FROM wp_options WHERE option_name = 'can_compress_scripts' LIMIT 1)
[Tue May 01 12:50:52.693969 2018] [:error] [pid 1283] [client 95.181.179.218:51450] require('wp-blog-header.php'), require_once('wp-includes/template-loader.php'), do_action('template_redirect'), WP_Hook->do_action, WP_Hook->apply_filters, call_user_func_array, redirect_canonical, redirect_guess_404_permalink\xec\x9d\xb4(\xea\xb0\x80) \xeb\xa7\x8c\xeb\x93\xa0 \xec\x9b\x8c\xeb\x93\x9c\xed\x94\x84\xeb\xa0\x88\xec\x8a\xa4 \xeb\x8d\xb0\xec\x9d\xb4\xed\x84\xb0\xeb\xb2\xa0\xec\x9d\xb4\xec\x8a\xa4 \xec\x98\xa4\xeb\xa5\x98 Lost connection to MySQL server during query(\xec\xbf\xbc\xeb\xa6\xac SELECT ID FROM wp_posts WHERE post_name LIKE 'thumb-php%' AND post_type IN ('post', 'page', 'attachment') AND post_status = 'publish')
[Tue May 01 12:50:52.697560 2018] [:error] [pid 1228] [client 95.181.179.218:52859] require('wp-blog-header.php'), require_once('wp-includes/template-loader.php'), include('/themes/spacious/404.php'), get_header, locate_template, load_template, require_once('/themes/spacious/header.php'), wp_head, do_action('wp_head'), WP_Hook->do_action, WP_Hook->apply_filters, call_user_func_array, wp_print_head_scripts, print_head_scripts, script_concat_settings, get_site_option, get_network_option, get_option\xec\x9d\xb4(\xea\xb0\x80) \xeb\xa7\x8c\xeb\x93\xa0 \xec\x9b\x8c\xeb\x93\x9c\xed\x94\x84\xeb\xa0\x88\xec\x8a\xa4 \xeb\x8d\xb0\xec\x9d\xb4\xed\x84\xb0\xeb\xb2\xa0\xec\x9d\xb4\xec\x8a\xa4 \xec\x98\xa4\xeb\xa5\x98 Lost connection to MySQL server during query(\xec\xbf\xbc\xeb\xa6\xac SELECT option_value FROM wp_options WHERE option_name = 'can_compress_scripts' LIMIT 1)
[Tue May 01 12:50:52.699475 2018] [:error] [pid 1249] [client 95.181.179.218:52842] require('wp-blog-header.php'), require_once('wp-load.php'), require_once('wp-config.php'), require_once('wp-settings.php'), do_action('init'), WP_Hook->do_action, WP_Hook->apply_filters, call_user_func_array, user_functions_for_captcha_booster, blocking_visitors_captcha_booster, get_ip_address_for_captcha_booster\xec\x9d\xb4(\xea\xb0\x80) \xeb\xa7\x8c\xeb\x93\xa0 \xec\x9b\x8c\xeb\x93\x9c\xed\x94\x84\xeb\xa0\x88\xec\x8a\xa4 \xeb\x8d\xb0\xec\x9d\xb4\xed\x84\xb0\xeb\xb2\xa0\xec\x9d\xb4\xec\x8a\xa4 \xec\x98\xa4\xeb\xa5\x98 Lost connection to MySQL server during query(\xec\xbf\xbc\xeb\xa6\xac SELECT meta_value FROM wp_captcha_booster_meta WHERE meta_key='other_settings')
3. 관련 사이트
아래 사이트에서 확인해보니 /etc/php.ini 설정파일에서 'memory_limit = 256M'로 변경하라고 가이드하는 듯 해서…
https://teamtreehouse.com/community/wphead-is-causing-memory-error
이 현상과 직접 연관은 없지만 우선 php.ini 값을 변경해보고 spacious 워드프레스 테마를 업데이트한 상태다. (2018.05.17 14:30)
default-constants.php 파일의 Memory limit 설정
[root@mws ~]# cat /home/mapoo-blog/wp-includes/default-constants.php | grep -i LIMIT
$current_limit = @ini_get( 'memory_limit' );
$current_limit_int = wp_convert_hr_to_bytes( $current_limit );
// Define memory limits.
if ( ! defined( 'WP_MEMORY_LIMIT' ) ) {
if ( false === wp_is_ini_value_changeable( 'memory_limit' ) ) {
define( 'WP_MEMORY_LIMIT', $current_limit );
define( 'WP_MEMORY_LIMIT', '64M' );
define( 'WP_MEMORY_LIMIT', '40M' );
if ( ! defined( 'WP_MAX_MEMORY_LIMIT' ) ) {
if ( false === wp_is_ini_value_changeable( 'memory_limit' ) ) {
define( 'WP_MAX_MEMORY_LIMIT', $current_limit );
} elseif ( -1 === $current_limit_int || $current_limit_int > 268435456 /* = 256M */ ) {
define( 'WP_MAX_MEMORY_LIMIT', $current_limit );
define( 'WP_MAX_MEMORY_LIMIT', '256M' );
// Set memory limits.
$wp_limit_int = wp_convert_hr_to_bytes( WP_MEMORY_LIMIT );
if ( -1 !== $current_limit_int && ( -1 === $wp_limit_int || $wp_limit_int > $current_limit_int ) ) {
@ini_set( 'memory_limit', WP_MEMORY_LIMIT );
더 지켜봐야 하겠다.
wordpress OOM(Out Of Memory)