Slave memory leak and trigger oom-killer_MySQL

Bug?Description  We?have?this?problem:  we?have?set?innodb_buffer_pool?=80GB?on?both?master?and?slave?,?master?offer?usually?workload,  but?slave?with?nothing?workload?except?these?slave?threads,?But?with?the?memory?consumption?is?increasing?,?  after?a?few?days?,?it?uses?about?120GB?of?memory(RES)?and?sometime?the?machine?starts?swapping?out.?  and?in?the?/var/log/message(loged?:Out?of?memory:?Kill?process?43817?(mysqld)?score?847?or?sacrifice?  childKilled?process?43817,?UID?500,?(mysqld)?total-vm:126723628kB,?anon-rss:125497616kB,?file-rss:3388kB)  We?have?the?exact?version?of?PerconaServer?on?the?master?(with?production?workload)?and?it?doesn't?  have?this?problem  This?problem?is?the?same?for?a?few?recent?versions?of?Percona?server,?not?just?the?last?one.  we?do?not?user?MyISAM?tables?and?just?use?Innodb?Tables;TokuDB?is?not?installed.  Other?Information:  this?one?just?OOM?last?night?T-T:  xiean@(none)?11:02>?s--------------mysql?Ver?14.14?Distrib?5.6.23-72.1,?for?Linux?(x86_64)?  using?EditLine?wrapper  Connection?id:?252482Current?database:Current?user:?xiean@localhostSSL:?Not?in?useCurrent?pager:?  stdoutUsing?outfile:?''Using?delimiter:?;Server?version:?5.6.23-72.1-log?Source?distributionProtocol?  version:?10Connection:?Localhost?via?UNIX?socketServer?characterset:?utf8Db?characterset:?  utf8Client?characterset:?utf8Conn.?characterset:?utf8UNIX?socket:?/data/mysql/3307/var/run/mysql.sockUptime:  ?10?hours?32?min?9?sec  Threads:?6?Questions:?5888709?Slow?queries:?312?Opens:?200?Flush?tables:?1?Open?tables:?193?Queries?per?  second?avg:?155.256--------------  CentOS?release?6.5?(Final)Linux?xxxxxxx?2.6.32-431.el6.x86_64?#1?SMP?Fri?Nov?22?03:15:09?  UTC?2013?x86_64?x86_64?x86_64?GNU/Linux  my.cnf-----------------------------------------------------[mysqld]  skip-external-lockingskip-name-resolvedefault-storage-engine?=?InnoDBcharacter-set-client-handshake?=?  FALSEcharacter-set-server?=?utf8collation-server?=?utf8_unicode_ciinit_connect='set?names?utf8'  connect_timeout?=?20back_log?=?600log_bin_trust_function_creators?=?1  binlog_format?=?rowmax_binlog_size?=?512Mbinlog_cache_size?=?2Mexpire-logs-days?=?7  slave-net-timeout?=?20log_slave_updates?=?1relay_log_recovery?=?1sync_master_info?=?1sync_relay_log_info?=?  1master_info_repository?=?TABLErelay_log_info_repository?=?TABLE  slow_query_log?=?1long_query_time?=?1  max_connections?=?2000max_user_connections?=?1000max_connect_errors?=?100000key_buffer_size?=?  64Mmax_allowed_packet?=?512Mtable_open_cache?=?6144table_definition_cache?=?4096sort_buffer_size?=?  512Kread_buffer_size?=?512Kread_rnd_buffer_size?=?512kjoin_buffer_size?=?512Ktmp_table_size?=?  64Mmax_heap_table_size?=?128Mquery_cache_type?=?0query_cache_size?=?0bulk_insert_buffer_size?=?  32Mthread_cache_size?=?64thread_stack?=?256K  innodb_data_file_path?=?ibdata1:1024M:autoextendinnodb_buffer_pool_size?=?80Ginnodb_buffer_pool_instances?=?  8innodb_additional_mem_pool_size?=?16Minnodb_use_sys_malloc?=?0innodb_log_file_size?=?  8196Minnodb_log_buffer_size?=?64Minnodb_log_files_in_group?=?3innodb_lock_wait_timeout?=?  30innodb_spin_wait_delay?=?96innodb_thread_concurrency?=?0innodb_max_dirty_pages_pct?=?  75innodb_support_xa?=?0innodb_flush_log_at_trx_commit?=?2innodb_flush_method?=?O_DIRECTinnodb_read_io_threads?=?  16innodb_write_io_threads?=?16innodb_io_capacity?=?2000innodb_flush_neighbors?=?0innodb_page_size?=?  4Kinnodb_log_block_size?=?4096innodb_file_per_table?=?1innodb_file_format?=?Barracudainnodb_file_format_max?=?  Barracudainnodb_strict_mode?=?1innodb_purge_threads?=?2innodb_purge_batch_size?=?32innodb_old_blocks_pct?=?  37innodb_old_blocks_time?=?1000innodb_change_buffering?=?allinnodb_stats_on_metadata?=?off  sync_binlog?=?0    innodb_buffer_pool_dump_at_shutdown?=?1innodb_buffer_pool_load_at_startup?=?1  show?engine?innodb?statusG  =====================================2016-03-22?11:08:07?7f8496af9700?INNODB?MONITOR?OUTPUT====================  =================Per?second?averages?calculated?from?the?last?3?seconds-----------------BACKGROUND?THREAD------  -----------srv_master_thread?loops:?35735?srv_active,?0?srv_shutdown,?2389?srv_idlesrv_master_thread?log?flush  ?and?writes:?38124----------SEMAPHORES----------OS?WAIT?ARRAY?INFO:?reservation?count?27319OS?WAIT?ARRAY?INFO:?  ?signal?count?694283Mutex?spin?waits?13779381,?rounds?15836207,?OS?waits?12843RW-shared?spins?3653866,?rounds  ??16984184,?OS?waits?11004RW-excl?spins?562597,?rounds?1648229,?OS?waits?2980Spin?rounds?per?wait:?1.15?mutex,  ??4.65?RW-shared,?2.93?RW-excl  ....................................................  ----------------------BUFFER?POOL?AND?MEMORY----------------------  Total?memory?allocated?107356939316;?in?additional?pool?allocated?8441088Total?memory?allocated?by?  read?views?496Internal?hash?tables?(constant?factor?+?variable?factor)????  Adaptive?hash?index?2118609544?(1359985288?+?758624256)????  Page?hash?42500456?(buffer?pool?0?only)????  Dictionary?cache?340576340?(339997712?+?578628)????  File?system?867904?(812272?+?55632)????  Lock?system?849995224?(849992296?+?2928)????  Recovery?system?0?(0?+?0)Dictionary?memory?allocated?578628Buffer?pool?size?20971512Buffer?pool?size,?  bytes?85899313152Free?buffers?793287Database?pages?19993014Old?database?pages?  7380256Modified?db?pages?857786Pending?reads?0Pending?writes:?LRU?0,?flush?list?0,?  single?page?0Pages?made?young?1500483,?not?young?7919383.00?youngs/s,?3.67?non-youngs/sPages?read?19777369,?  created?365977,?written?17055406.33?reads/s,?9.00?creates/s,?255.25?writes/sBuffer?pool?hit?rate?1000?/?1000,?  young-making?rate?0?/?1000?not?0?/?1000Pages?read?ahead?0.00/s,?evicted?without?access?0.00/s,?  Random?read?ahead?0.00/sLRU?len:?19993014,?unzip_LRU?len:?0I/O?sum[36104]:cur[1320],?unzip?sum[0]:cur[0]  ----------------------INDIVIDUAL?BUFFER?POOL?INFO-------------------------  BUFFER?POOL?0Buffer?pool?size?2621439Buffer?pool?size,?bytes?10737414144Free?buffers?1024Database  ?pages?2597173Old?database?pages?958702Modified?db?pages?110539Pending?reads?0Pending?writes:?LRU?0,?  ?flush?list?0,?single?page?0Pages?made?young?497074,?not?young?3051540.67?youngs/s,?3.67?non-youngs/sPages?  ?read?2603703,?created?46719,?written?2711371.00?reads/s,?0.33?creates/s,?17.33?writes/sBuffer?pool?hit?  ?rate?1000?/?1000,?young-making?rate?0?/?1000?not?0?/?1000Pages?read?ahead?0.00/s,?evicted?without  ??access?0.00/s,?Random?read?ahead?0.00/sLRU?len:?2597173,?unzip_LRU?len:?0I/O?sum[4513]:cur[165],?  ??unzip?sum[0]:cur[0]---BUFFER?POOL?1Buffer?pool?size?2621439Buffer?pool?size,?bytes?10737414144Free?  ??buffers?1024Database?pages?2597316Old?database?pages?958754Modified?db?pages?115355Pending?reads?  ??0Pending?writes:?LRU?0,?flush?list?0,?single?page?0Pages?made?young?496041,?not?young?2381371.00?youngs/s,?  ??0.00?non-youngs/sPages?read?2593888,?created?50425,?written?2643540.33?reads/s,?3.00?creates/s,?29.66  ???writes/sBuffer?pool?hit?rate?1000?/?1000,?young-making?rate?0?/?1000?not?0?/?1000Pages?read?ahead?0.00/s,?  ???evicted?without?access?0.00/s,?Random?read?ahead?0.00/sLRU?len:?2597316,?unzip_LRU?len:?0I/O?  ???sum[4513]:cur[165],?unzip?sum[0]:cur[0]---BUFFER?POOL?2Buffer?pool?size?2621439Buffer?pool?size,?  ???bytes?10737414144Free?buffers?1024Database?pages?2597259Old?database?pages?958733Modified?db?pages?  ???115748Pending?reads?0Pending?writes:?LRU?0,?flush?list?0,?single?page?0Pages?made?young?496770,?  ???not?young?2486471.00?youngs/s,?0.00?non-youngs/sPages?read?2595651,?created?51694,?  ???written?2406831.00?reads/s,?0.00?creates/s,?35.99?writes/sBuffer?pool?hit?rate?1000?/?1000,?  ???young-making?rate?0?/?1000?not?0?/?1000Pages?read?ahead?0.00/s,?evicted?without?access?0.00/s,?  ???Random?read?ahead?0.00/sLRU?len:?2597259,?unzip_LRU?len:?0I/O?sum[4513]:cur[165],?  ???unzip?sum[0]:cur[0]---BUFFER?POOL?3Buffer?pool?size?2621439Buffer?pool?size,?bytes?  ???10737414144Free?buffers?117082Database?pages?2481220Old?database?pages?915938Modified?db?pages?  ???112864Pending?reads?0Pending?writes:?LRU?0,?flush?list?0,?single?page?0Pages?made?young?2305,?  ???not?young?00.00?youngs/s,?0.00?non-youngs/sPages?read?2429753,?created?51467,?  ???written?2489670.00?reads/s,?0.00?creates/s,?40.99?writes/sBuffer?pool?hit?rate?1000?/?1000,?  ???young-making?rate?0?/?1000?not?0?/?1000Pages?read?ahead?0.00/s,?evicted?without?access?0.00/s,?  ???Random?read?ahead?0.00/sLRU?len:?2481220,?unzip_LRU?len:?0I/O?sum[4513]:cur[165],?  ???unzip?sum[0]:cur[0]---BUFFER?POOL?4Buffer?pool?size?2621439Buffer?pool?size,?  ???bytes?10737414144Free?buffers?207591Database?pages?2390638Old?database?pages?  ???882501Modified?db?pages?103991Pending?reads?0Pending?writes:?LRU?0,?flush?list?0,?single?page?  ???0Pages?made?young?2163,?not?young?00.33?youngs/s,?0.00?non-youngs/sPages?read?2345049,?created?45589,?  ???written?2144341.00?reads/s,?1.00?creates/s,?32.99?writes/sBuffer?pool?hit?rate?1000?/?1000,?young-making?  ???rate?0?/?1000?not?0?/?1000Pages?read?ahead?0.00/s,?evicted?without?access?0.00/s,?Random?read?ahead?0.00/sLRU  ????len:?2390638,?unzip_LRU?len:?0I/O?sum[4513]:cur[165],?unzip?sum[0]:cur[0]---BUFFER?POOL?5Buffer?pool?size?  ????2621439Buffer?pool?size,?bytes?10737414144Free?buffers?202652Database?pages?2395634Old?database?pages?  ????884332Modified?db?pages?99353Pending?reads?0Pending?writes:?LRU?0,?flush?list?0,?single?page?0Pages?made?  ????young?2061,?not?young?00.00?youngs/s,?0.00?non-youngs/sPages?read?2354114,?created?41520,?  ????written?1473470.67?reads/s,?3.67?creates/s,?30.99?writes/sBuffer?pool?hit?rate?1000?/?1000,?young-making?  rate?0?/?1000?not?0?/?1000Pages?read?ahead?0.00/s,?evicted?without?access?0.00/s,?Random?read?ahead?0.00/sLRU?  len:?2395634,?unzip_LRU?len:?0I/O?sum[4513]:cur[165],?unzip?sum[0]:cur[0]---BUFFER?POOL?6Buffer?pool?size?  2621439Buffer?pool?size,?bytes?10737414144Free?buffers?209285Database?pages?2389036Old?database?pages?  881910Modified?db?pages?96899Pending?reads?0Pending?writes:?LRU?0,?flush?list?0,?single?page?0Pages?  made?young?1913,?not?young?00.00?youngs/s,?0.00?non-youngs/sPages?read?2349930,?created?39106,?written?  1501741.33?reads/s,?0.67?creates/s,?38.65?writes/sBuffer?pool?hit?rate?1000?/?1000,?young-making?rate?0?/?  1000?not?0?/?1000Pages?read?ahead?0.00/s,?evicted?without?access?0.00/s,?Random?read?ahead?0.00/sLRU?  len:?2389036,?unzip_LRU?len:?0I/O?sum[4513]:cur[165],?unzip?sum[0]:cur[0]---BUFFER?POOL?7Buffer?pool?  size?2621439Buffer?pool?size,?bytes?10737414144Free?buffers?53605Database?pages?2544738Old?database?pages?  939386Modified?db?pages?103037Pending?reads?0Pending?writes:?LRU?0,?flush?list?0,?single?page?0Pages?made?  young?2156,?not?young?00.00?youngs/s,?0.00?non-youngs/sPages?read?2505281,?created?39457,?written?1684441.00?  reads/s,?0.33?creates/s,?28.66?writes/sBuffer?pool?hit?rate?1000?/?1000,?young-making?rate?0?/?1000?not?0?/?  1000Pages?read?ahead?0.00/s,?evicted?without?access?0.00/s,?Random?read?ahead?0.00/sLRU?len:?2544738,?unzip_LRU?  len:?0I/O?sum[4513]:cur[165],?unzip?sum[0]:cur[0]  --------------ROW?OPERATIONS--------------  0?queries?inside?InnoDB,?  0?queries?in?queue1?read?views?open?inside?InnoDB0?RW?transactions?  active?inside?InnoDB0?RO?transactions?active?inside?InnoDB0?out?of?1000?descriptors?  used---OLDEST?VIEW---Normal?read?viewRead?view?low?limit?trx?n:o?4402678824Read?view?  up?limit?trx?id?4402678824Read?view?low?limit?trx?id?4402678824Read?view?inpidually?stored?trx?ids:  -----------------Main?thread?process?no.?70096,?id?140207583508224,?  state:?sleepingNumber?of?rows?inserted?1347486,?updated?6484099,?deleted?505223,?  read?4665086631106.63?inserts/s,?1017.66?updates/s,?0.00?deletes/s,?1682192.60?reads/s  ----------------------------END?OF?INNODB?MONITOR?OUTPUT============================