June 23, 2010

How read_buffer_size Impacts Write Buffering and Write Performance

Even though the name read_buffer_size implies that the variable controls only read buffering, but it actually does dual purpose by providing sequential IO buffering for both reads and writes.

In case of write buffering, it groups the sequential writes until read_buffer_size(it is min(read_buffer_size, 8K)); and then actually does the physical write once the buffer is full. In most cases; this value is the initial value of read_buffer_size when server actually started first time; as this is a dynamic global variable; even if you change the value dynamically at run time; it will not affect write buffering size (and in some cases of read buffering as well) as this is stored one-time in my_default_record_cache_size (might be a bug ?); and that variable is used in initializing IO cache buffers.

Here is some use cases where read_buffer_size is actually used for buffering writes:

  • SELECT INTO … OUTFILE ‘fileName
    • When writing to the OUTFILE, the writes are buffered before writing to OUTFILE
  • When filesort is used, during merge buffers and when merged results are written to a temporary file, then writes are buffered

Normally you will see performance boost due to buffering on slower write disks or when you have IO saturation; but it does not matter when you have descent disks or raid controller with write_cache with BBU enabled.

Here is some stats on how many physical writes are actually posted for a simple SELECT … INTO OUTFILE (file size 384936838 bytes) for variable read_buffer_size values (server needs to be restarted in-order to get the new value):

read_buffer_size physical writes exe time in secs
=0 (defaults to 8200) 23495 28.39
=131072 (default) 2937 27.44
=16777216 23 26.71
=33554432 12 26.00
=536870912 1 26.72

Total writes are calculated using simple patch that I wrote around mysys/my_write.c to get the real physical writes posted as a global status counter.

mysql> show global status like 'Write_count';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| Write_count   | 23496 |
+---------------+-------+
1 row in set (0.00 sec)

As you can see, increase in read_buffer_size might save total physical writes and might help if you have lot of OUTFILEs or heavy file sorting to some extent; but again this will actually affect overall performance due to one of the known bug and the buffer is also allocated per query based; so be careful as allocation and initialization of big buffers are much costlier than real IO cost.

In either case; may be worth if the sequential read buffering is actually controlled by read_buffer_size and introduce new write_buffer_size that controls the write buffering instead of using the same for both.

June 12, 2010

Autocommit, Implicit Commit and Open Transactions

If you have any open transaction(s); and if there is any statement that causes implicit commit, especially DDL statements; then the current active transaction will be committed and transaction will be closed automatically. Here is the list of statements that causes implicit commit in MySQL.

But other day; we had an issue in production as all of a sudden everything started failing with the following error on RENAME TABLE statements:

ERROR 1192 (HY000): Can't execute the given command because you have active locked tables or an active transaction

This happened because I changed one of the table type to InnoDB from MyISAM; and as that table is only used for read-only purpose and most part of ETL operations failed with the above error even though no table is explicitly locked or running in any open transactions; and after debugging it further; found that its implicit commit that is not properly closing any open transactions when auto commit is set to ON mode.

Here is the simple repro in MySQL (5.0 and 5.1):

mysql> create table outer_tab(id int)Engine=InnoDB;
Query OK, 0 rows affected (0.12 sec)
 
mysql> insert into outer_tab values(10);
Query OK, 1 row affected (0.00 sec)
 
mysql> commit;
Query OK, 0 rows affected (0.00 sec)
 
mysql> 
mysql> set autocommit=0;
Query OK, 0 rows affected (0.00 sec)
 
mysql> select * from outer_tab;
+------+
| id   |
+------+
|   10 |
+------+
1 row in set (0.00 sec)
 
mysql> set autocommit=1;
Query OK, 0 rows affected (0.00 sec)
 
mysql> create table inner_tab(id int)Engine=InnoDB;
Query OK, 0 rows affected (0.18 sec)
 
mysql> rename table inner_tab to tmp_x;
ERROR 1192 (HY000): Can't execute the given command because you have active locked tables or an active transaction
mysql>

SET AUTOCOMMIT=1 should perform implicit commit and close any open transactions as per the documentation; but it only commits any pending transactional statements; but does not mark transaction as closed; and here is the code that does that in set_var.cc::set_option_autocommit().

 if ((org_options & OPTION_NOT_AUTOCOMMIT))
    {
      /* We changed to auto_commit mode */
      thd->options&= ~(ulonglong) (OPTION_BEGIN | OPTION_KEEP_LOG);
      thd->transaction.all.modified_non_trans_table= FALSE;
      thd->server_status|= SERVER_STATUS_AUTOCOMMIT;
      if (ha_commit(thd))
    return 1;
    }

This is handled by MySQL engine and is in-dependant of storage engines. The fix here is to reset SERVER_STATUS_IN_TRANS in thd->server_status after the successful ha_commit() or by calling end_active_trans() function instead of ha_commit() in the above set_option_autocommit() function; and this holds good for any other implicit commit statements as well, so that it marks transaction as closed. I also filed a bug report

In a side note; here is the same example against Oracle 10G:

SQL>
SQL> create table outer_tab(id int);
Table created.
 
SQL> insert into outer_tab values(10);
1 row created.
 
SQL> commit;
Commit complete.
 
SQL>
SQL> set autocommit off;
SQL> select * from outer_tab;
        ID
----------
        10
 
SQL> set autocommit on;
SQL>
SQL> create table inner_tab(id int);
Table created.
 
SQL>
SQL> rename inner_tab to tmp_x;
Table renamed.

		

June 7, 2010

InnoDB In a Complete Locked Mode

Today morning, we had a weird issue in one of the staging OLAP server (ETL); where all the InnoDB threads were locked and waiting on a signal condition for about 3-4 hours without performing any work related to InnoDB until we noticed it. Processlist indicates every thread is in ACTIVE state mode like Sending Data, update, Copying to tmp table etc; even queries that are acting on single row table are in Sending Data state for more than 3 hours; and slave thread is no exception as it was performing an update on InnoDB table and it got stuck in that query for 3+ hours.

Quick look at the stack trace as shown below indicates that, out of total 92 threads, none of them actually doing anything. Other than sleep, main and monitor threads, every other thread is waiting on a signal condition. The server was actually running with the following concurrency settings:

innodb_thread_concurrency=0
innodb_thread_sleep_delay=0

But looks like by accidentally thread_concurrency and concurrency_tickets count was toggled together; that forced currently executing threads to enter into this signal wait state (pthread_cond_wait@@GLIBC_2.3.2,os_event_wait_low). Resetting concurrency back to 0 or increasing tickets did not help; but that allowed me to execute new queries without any issues except that it couldn’t signal to wake rest of the threads.

One way to signal the event is by making a new thread to enter srv_conc_enter_innodb with no more concurrency tickets left; that means reset innodb_concurrency_tickets to a lower number along with setting innodb_thread_concurrency > 0 and by executing an InnoDB related query(create table, bunch of inserts, updates, selects and then drop); and by doing so, actually re-signaled the condition and all threads started working back.

This happened with 5.0.77 build with community patches and looks like a race condition that causes this in combination of innodb_thread_concurrency and innodb_concurrency_tickets when toggling back and forth on a long running active threads. I could not reproduce this again as its hard to simulate the load; but will try again on next weekend as week days we let production systems do their job

Stack trace:

     92 
     12 pthread_cond_wait@@GLIBC_2.3.2,os_event_wait_low,os_aio_simulated_handle,fil_aio_wait,io_handler_thread,start_thread,clone
      8 pthread_cond_wait@@GLIBC_2.3.2,end_thread,handle_one_connection,start_thread,clone
      1 select,os_thread_sleep,srv_lock_timeout_and_monitor_thread,start_thread,clone
      1 select,os_thread_sleep,srv_error_monitor_thread,start_thread,clone
      1 select,handle_connections_sockets,main
      1 read,read,buf=0x2ab822169530,my_real_read,my_net_read,handle_one_connection,start_thread,clone
      1 read,read,buf=0x2ab7e0000030,my_real_read,my_net_read,handle_one_connection,start_thread,clone
      1 read,read,buf=0x2ab7dfda85c0,my_real_read,my_net_read,handle_one_connection,start_thread,clone
      1 read,read,buf=0x2ab7dfda45b0,my_real_read,my_net_read,handle_one_connection,start_thread,clone
      1 read,read,buf=0x2ab7d4b035e0,my_real_read,my_net_read,handle_one_connection,start_thread,clone
      1 read,read,buf=0x2ab7a3d747e0,my_real_read,my_net_read,handle_one_connection,start_thread,clone
      1 read,read,buf=0x2ab7a0a1a970,my_real_read,my_net_read,handle_one_connection,start_thread,clone
      1 read,read,buf=0x2ab78c5ab320,my_real_read,my_net_read,handle_one_connection,start_thread,clone
      1 read,read,buf=0x2ab71686c1d0,my_real_read,my_net_read,handle_one_connection,start_thread,clone
      1 read,read,buf=0x2ab6d649ad10,my_real_read,my_net_read,handle_one_connection,start_thread,clone
      1 read,read,buf=0x2ab6cca043e0,my_real_read,my_net_read,handle_one_connection,start_thread,clone
      1 read,read,buf=0x2ab6bfd3bed0,my_real_read,my_net_read,handle_one_connection,start_thread,clone
      1 read,read,buf=0x2ab6bbd3e620,my_real_read,my_net_read,handle_one_connection,start_thread,clone
      1 read,read,buf=0x2ab6af60dbe0,my_real_read,my_net_read,handle_one_connection,start_thread,clone
      1 read,read,buf=0x2ab6903fefd0,my_real_read,my_net_read,handle_one_connection,start_thread,clone
      1 read,read,buf=0x2ab690000010,my_real_read,my_net_read,handle_one_connection,start_thread,clone
      1 read,read,buf=0x2ab68ddd4270,my_real_read,my_net_read,handle_one_connection,start_thread,clone
      1 read,read,buf=0x2ab682eea0c0,my_real_read,my_net_read,handle_one_connection,start_thread,clone
      1 read,read,buf=0x2ab681068d50,my_real_read,my_net_read,handle_one_connection,start_thread,clone
      1 read,read,buf=0x2aaaac0010a0,vio_read_buff,my_real_read,my_net_read,cli_safe_read,read_event,at,start_thread,clone
      1 pthread_cond_wait@@GLIBC_2.3.2,os_event_wait_low,srv_master_thread,start_thread,clone
      1 pthread_cond_wait@@GLIBC_2.3.2,os_event_wait_low,srv_conc_enter_innodb,innodb_srv_conc_enter_innodb,record=0x2ab74d2a9f00,write_record,mysql_insert,mysql_execute_command,Prepared_statement::execute,mysql_stmt_execute,dispatch_command,handle_one_connection,start_thread,clone
      1 pthread_cond_wait@@GLIBC_2.3.2,os_event_wait_low,srv_conc_enter_innodb,innodb_srv_conc_enter_innodb,record=0x2ab68a89a5e0,write_record,mysql_insert,mysql_execute_command,Prepared_statement::execute,mysql_stmt_execute,dispatch_command,handle_one_connection,start_thread,clone
      1 pthread_cond_wait@@GLIBC_2.3.2,os_event_wait_low,srv_conc_enter_innodb,innodb_srv_conc_enter_innodb,buf=0x2ab79032a2a0,join_read_key,sub_select,evaluate_join_record,sub_select,do_select,JOIN::exec,mysql_select,handle_select,mysql_execute_command,Prepared_statement::execute,mysql_stmt_execute,dispatch_command,handle_one_connection,start_thread,clone
      1 pthread_cond_wait@@GLIBC_2.3.2,os_event_wait_low,srv_conc_enter_innodb,innodb_srv_conc_enter_innodb,buf=0x2ab78c4ac270,join_read_always_key,sub_select,do_select,JOIN::exec,mysql_select,handle_select,mysql_execute_command,Prepared_statement::execute,mysql_stmt_execute,dispatch_command,handle_one_connection,start_thread,clone
      1 pthread_cond_wait@@GLIBC_2.3.2,os_event_wait_low,srv_conc_enter_innodb,innodb_srv_conc_enter_innodb,buf=0x2ab76e8e1ea0,ha_innobase::index_first,ha_innobase::rnd_next,rr_sequential,sub_select,do_select,JOIN::exec,mysql_select,handle_select,mysql_execute_command,mysql_parse,dispatch_command,handle_one_connection,start_thread,clone
      1 pthread_cond_wait@@GLIBC_2.3.2,os_event_wait_low,srv_conc_enter_innodb,innodb_srv_conc_enter_innodb,buf=0x2ab752517498,write_record,read_sep_field,ex=<value ,mysql_execute_command,mysql_parse,dispatch_command,handle_one_connection,start_thread,clone
      1 pthread_cond_wait@@GLIBC_2.3.2,os_event_wait_low,srv_conc_enter_innodb,innodb_srv_conc_enter_innodb,buf=0x2ab74bc72430,ha_innobase::index_first,ha_innobase::rnd_next,rr_sequential,sub_select,do_select,JOIN::exec,mysql_select,handle_select,mysql_execute_command,mysql_parse,dispatch_command,handle_one_connection,start_thread,clone
      1 pthread_cond_wait@@GLIBC_2.3.2,os_event_wait_low,srv_conc_enter_innodb,innodb_srv_conc_enter_innodb,buf=0x2ab6e87bffb0,rr_sequential,sub_select,do_select,JOIN::exec,mysql_select,handle_select,mysql_execute_command,Prepared_statement::execute,mysql_stmt_execute,dispatch_command,handle_one_connection,start_thread,clone
      1 pthread_cond_wait@@GLIBC_2.3.2,os_event_wait_low,srv_conc_enter_innodb,innodb_srv_conc_enter_innodb,buf=0x2ab6e3a77660,ha_innobase::index_first,ha_innobase::rnd_next,rr_sequential,sub_select,do_select,JOIN::exec,mysql_select,handle_select,mysql_execute_command,mysql_parse,dispatch_command,handle_one_connection,start_thread,clone
      1 pthread_cond_wait@@GLIBC_2.3.2,os_event_wait_low,srv_conc_enter_innodb,innodb_srv_conc_enter_innodb,buf=0x2ab6be153290,join_read_next_same,sub_select,evaluate_join_record,sub_select,do_select,JOIN::exec,mysql_select,handle_select,mysql_execute_command,Prepared_statement::execute,mysql_stmt_execute,dispatch_command,handle_one_connection,start_thread,clone
      1 pthread_cond_wait@@GLIBC_2.3.2,os_event_wait_low,srv_conc_enter_innodb,innodb_srv_conc_enter_innodb,buf=0x2ab6b7f893f0,write_record,select_insert::send_data,end_send,evaluate_join_record,sub_select,do_select,JOIN::exec,mysql_select,handle_select,mysql_execute_command,Prepared_statement::execute,mysql_stmt_execute,dispatch_command,handle_one_connection,start_thread,clone
      1 pthread_cond_wait@@GLIBC_2.3.2,os_event_wait_low,srv_conc_enter_innodb,innodb_srv_conc_enter_innodb,buf=0x2ab6b6b103c0,write_record,select_insert::send_data,end_send,evaluate_join_record,sub_select,do_select,JOIN::exec,mysql_select,handle_select,mysql_execute_command,Prepared_statement::execute,mysql_stmt_execute,dispatch_command,handle_one_connection,start_thread,clone
      1 pthread_cond_wait@@GLIBC_2.3.2,os_event_wait_low,srv_conc_enter_innodb,innodb_srv_conc_enter_innodb,buf=0x2ab6b4c1f570,write_record,select_insert::send_data,end_send,evaluate_join_record,sub_select,do_select,JOIN::exec,mysql_select,handle_select,mysql_execute_command,Prepared_statement::execute,mysql_stmt_execute,dispatch_command,handle_one_connection,start_thread,clone
      1 pthread_cond_wait@@GLIBC_2.3.2,os_event_wait_low,srv_conc_enter_innodb,innodb_srv_conc_enter_innodb,buf=0x2ab6af738490,write_record,select_insert::send_data,end_send,evaluate_join_record,sub_select,do_select,JOIN::exec,mysql_select,handle_select,mysql_execute_command,Prepared_statement::execute,mysql_stmt_execute,dispatch_command,handle_one_connection,start_thread,clone
      1 pthread_cond_wait@@GLIBC_2.3.2,os_event_wait_low,srv_conc_enter_innodb,innodb_srv_conc_enter_innodb,buf=0x2ab6a1268040,write_record,select_insert::send_data,end_send,evaluate_join_record,sub_select,do_select,JOIN::exec,mysql_select,handle_select,mysql_execute_command,Prepared_statement::execute,mysql_stmt_execute,dispatch_command,handle_one_connection,start_thread,clone
      1 pthread_cond_wait@@GLIBC_2.3.2,os_event_wait_low,srv_conc_enter_innodb,innodb_srv_conc_enter_innodb,buf=0x2ab68746fcb0,ha_innobase::index_first,ha_innobase::rnd_next,rr_sequential,sub_select,do_select,JOIN::exec,mysql_select,handle_select,mysql_execute_command,Prepared_statement::execute,mysql_stmt_execute,dispatch_command,handle_one_connection,start_thread,clone
      1 pthread_cond_wait@@GLIBC_2.3.2,os_event_wait_low,srv_conc_enter_innodb,innodb_srv_conc_enter_innodb,buf=0x2ab686f64c40,write_record,select_insert::send_data,end_send,evaluate_join_record,sub_select,do_select,JOIN::exec,mysql_select,handle_select,mysql_execute_command,Prepared_statement::execute,mysql_stmt_execute,dispatch_command,handle_one_connection,start_thread,clone
      1 pthread_cond_wait@@GLIBC_2.3.2,os_event_wait_low,srv_conc_enter_innodb,innodb_srv_conc_enter_innodb,buf=0x2ab685a3c340,write_record,select_insert::send_data,end_send,evaluate_join_record,sub_select,do_select,JOIN::exec,mysql_select,handle_select,mysql_execute_command,Prepared_statement::execute,mysql_stmt_execute,dispatch_command,handle_one_connection,start_thread,clone
      1 pthread_cond_wait@@GLIBC_2.3.2,os_event_wait_low,srv_conc_enter_innodb,innodb_srv_conc_enter_innodb,buf=0x2ab683cef620,handler::read_range_first,handler::read_multi_range_first,QUICK_RANGE_SELECT::get_next,rr_quick,mysql_delete,mysql_execute_command,Prepared_statement::execute,mysql_stmt_execute,dispatch_command,handle_one_connection,start_thread,clone
      1 pthread_cond_wait@@GLIBC_2.3.2,os_event_wait_low,srv_conc_enter_innodb,innodb_srv_conc_enter_innodb,buf=0x2ab681b97eb0,write_record,select_insert::send_data,end_send,evaluate_join_record,sub_select,do_select,JOIN::exec,mysql_select,handle_select,mysql_execute_command,Prepared_statement::execute,mysql_stmt_execute,dispatch_command,handle_one_connection,start_thread,clone
      1 pthread_cond_wait@@GLIBC_2.3.2,os_event_wait_low,srv_conc_enter_innodb,innodb_srv_conc_enter_innodb,buf=0x2ab67af5eac0,write_record,select_insert::send_data,end_send,evaluate_join_record,sub_select,do_select,JOIN::exec,mysql_select,handle_select,mysql_execute_command,Prepared_statement::execute,mysql_stmt_execute,dispatch_command,handle_one_connection,start_thread,clone
      1 pthread_cond_wait@@GLIBC_2.3.2,os_event_wait_low,srv_conc_enter_innodb,innodb_srv_conc_enter_innodb,buf=0x2ab672f44fc0,write_record,select_insert::send_data,end_send,evaluate_join_record,sub_select,do_select,JOIN::exec,mysql_select,handle_select,mysql_execute_command,Prepared_statement::execute,mysql_stmt_execute,dispatch_command,handle_one_connection,start_thread,clone
      1 pthread_cond_wait@@GLIBC_2.3.2,os_event_wait_low,srv_conc_enter_innodb,innodb_srv_conc_enter_innodb,buf=0x2ab66c88a778,handler::read_range_first,handler::read_multi_range_first,QUICK_RANGE_SELECT::get_next,rr_quick,sub_select,do_select,JOIN::exec,mysql_select,handle_select,mysql_execute_command,Prepared_statement::execute,mysql_stmt_execute,dispatch_command,handle_one_connection,start_thread,clone
      1 pthread_cond_wait@@GLIBC_2.3.2,os_event_wait_low,srv_conc_enter_innodb,innodb_srv_conc_enter_innodb,buf=0x2ab66bc2f120,ha_innobase::index_first,ha_innobase::rnd_next,rr_sequential,sub_select,do_select,JOIN::exec,mysql_select,handle_select,mysql_execute_command,mysql_parse,dispatch_command,handle_one_connection,start_thread,clone
      1 pthread_cond_wait@@GLIBC_2.3.2,os_event_wait_low,srv_conc_enter_innodb,innodb_srv_conc_enter_innodb,buf=0x2ab66bbcefe0,write_record,select_insert::send_data,end_send,evaluate_join_record,sub_select,do_select,JOIN::exec,mysql_select,handle_select,mysql_execute_command,Prepared_statement::execute,mysql_stmt_execute,dispatch_command,handle_one_connection,start_thread,clone
      1 pthread_cond_wait@@GLIBC_2.3.2,os_event_wait_low,srv_conc_enter_innodb,innodb_srv_conc_enter_innodb,buf=0x2ab66b054ef0,write_record,select_insert::send_data,end_send,evaluate_join_record,sub_select,do_select,JOIN::exec,mysql_select,handle_select,mysql_execute_command,Prepared_statement::execute,mysql_stmt_execute,dispatch_command,handle_one_connection,start_thread,clone
      1 pthread_cond_wait@@GLIBC_2.3.2,os_event_wait_low,srv_conc_enter_innodb,innodb_srv_conc_enter_innodb,buf=0x2ab66960f7a0,write_record,select_insert::send_data,end_send,evaluate_join_record,sub_select,do_select,JOIN::exec,mysql_select,handle_select,mysql_execute_command,Prepared_statement::execute,mysql_stmt_execute,dispatch_command,handle_one_connection,start_thread,clone
      1 pthread_cond_wait@@GLIBC_2.3.2,os_event_wait_low,srv_conc_enter_innodb,innodb_srv_conc_enter_innodb,buf=0x2ab6694a0620,write_record,select_insert::send_data,end_send,evaluate_join_record,sub_select,do_select,JOIN::exec,mysql_select,handle_select,mysql_execute_command,Prepared_statement::execute,mysql_stmt_execute,dispatch_command,handle_one_connection,start_thread,clone
      1 pthread_cond_wait@@GLIBC_2.3.2,os_event_wait_low,srv_conc_enter_innodb,innodb_srv_conc_enter_innodb,buf=0x2ab668cd2230,write_record,select_insert::send_data,end_send,evaluate_join_record,sub_select,do_select,JOIN::exec,mysql_select,handle_select,mysql_execute_command,Prepared_statement::execute,mysql_stmt_execute,dispatch_command,handle_one_connection,start_thread,clone
      1 pthread_cond_wait@@GLIBC_2.3.2,os_event_wait_low,srv_conc_enter_innodb,innodb_srv_conc_enter_innodb,buf=0x2ab667882de0,write_record,select_insert::send_data,end_send,evaluate_join_record,sub_select,do_select,JOIN::exec,mysql_select,handle_select,mysql_execute_command,Prepared_statement::execute,mysql_stmt_execute,dispatch_command,handle_one_connection,start_thread,clone
      1 pthread_cond_wait@@GLIBC_2.3.2,os_event_wait_low,srv_conc_enter_innodb,innodb_srv_conc_enter_innodb,buf=0x2ab660d71870,rr_sequential,sub_select,do_select,JOIN::exec,mysql_select,mysql_derived_filling,mysql_handle_derived,open_and_lock_tables,mysql_execute_command,Prepared_statement::execute,mysql_stmt_execute,dispatch_command,handle_one_connection,start_thread,clone
      1 pthread_cond_wait@@GLIBC_2.3.2,os_event_wait_low,srv_conc_enter_innodb,innodb_srv_conc_enter_innodb,buf=0x2ab66073a070,write_record,select_insert::send_data,end_send,evaluate_join_record,sub_select,do_select,JOIN::exec,mysql_select,handle_select,mysql_execute_command,Prepared_statement::execute,mysql_stmt_execute,dispatch_command,handle_one_connection,start_thread,clone
      1 pthread_cond_wait@@GLIBC_2.3.2,os_event_wait_low,srv_conc_enter_innodb,innodb_srv_conc_enter_innodb,buf=0x2ab6603a8ab0,rr_sequential,sub_select,do_select,JOIN::exec,mysql_select,handle_select,mysql_execute_command,Prepared_statement::execute,mysql_stmt_execute,dispatch_command,handle_one_connection,start_thread,clone
      1 pthread_cond_wait@@GLIBC_2.3.2,os_event_wait_low,srv_conc_enter_innodb,innodb_srv_conc_enter_innodb,buf=0x2ab6600964a0,write_record,select_insert::send_data,end_send,evaluate_join_record,sub_select,do_select,JOIN::exec,mysql_select,handle_select,mysql_execute_command,Prepared_statement::execute,mysql_stmt_execute,dispatch_command,handle_one_connection,start_thread,clone
      1 pthread_cond_wait@@GLIBC_2.3.2,os_event_wait_low,srv_conc_enter_innodb,innodb_srv_conc_enter_innodb,buf=0x2ab65fe180a0,write_record,select_insert::send_data,end_send,evaluate_join_record,sub_select,do_select,JOIN::exec,mysql_select,handle_select,mysql_execute_command,Prepared_statement::execute,mysql_stmt_execute,dispatch_command,handle_one_connection,start_thread,clone
      1 pthread_cond_wait@@GLIBC_2.3.2,os_event_wait_low,srv_conc_enter_innodb,innodb_srv_conc_enter_innodb,buf=0x2ab65f18eed0,rr_sequential,sub_select,do_select,JOIN::exec,mysql_select,handle_select,mysql_execute_command,Prepared_statement::execute,mysql_stmt_execute,dispatch_command,handle_one_connection,start_thread,clone
      1 pthread_cond_wait@@GLIBC_2.3.2,os_event_wait_low,srv_conc_enter_innodb,innodb_srv_conc_enter_innodb,buf=0x2ab65f1703d0,rr_sequential,sub_select,do_select,JOIN::exec,mysql_select,mysql_derived_filling,mysql_handle_derived,open_and_lock_tables,mysql_execute_command,Prepared_statement::execute,mysql_stmt_execute,dispatch_command,handle_one_connection,start_thread,clone
      1 pthread_cond_wait@@GLIBC_2.3.2,os_event_wait_low,srv_conc_enter_innodb,innodb_srv_conc_enter_innodb,buf=0x2ab65f145540,join_read_next_same,sub_select,evaluate_join_record,sub_select,do_select,JOIN::exec,mysql_select,handle_select,mysql_execute_command,Prepared_statement::execute,mysql_stmt_execute,dispatch_command,handle_one_connection,start_thread,clone
      1 pthread_cond_wait@@GLIBC_2.3.2,os_event_wait_low,srv_conc_enter_innodb,innodb_srv_conc_enter_innodb,buf=0x2ab65f1268e0,join_read_next_same,sub_select,evaluate_join_record,sub_select,do_select,JOIN::exec,mysql_select,handle_select,mysql_execute_command,Prepared_statement::execute,mysql_stmt_execute,dispatch_command,handle_one_connection,start_thread,clone
      1 pthread_cond_wait@@GLIBC_2.3.2,os_event_wait_low,srv_conc_enter_innodb,innodb_srv_conc_enter_innodb,buf=0x2ab65da6d390,join_read_next_same,sub_select,evaluate_join_record,sub_select,do_select,JOIN::exec,mysql_select,handle_select,mysql_execute_command,Prepared_statement::execute,mysql_stmt_execute,dispatch_command,handle_one_connection,start_thread,clone
      1 pthread_cond_wait@@GLIBC_2.3.2,os_event_wait_low,srv_conc_enter_innodb,innodb_srv_conc_enter_innodb,buf=0x2ab658caabe0,ha_innobase::index_first,ha_innobase::rnd_next,rr_sequential,sub_select,do_select,JOIN::exec,mysql_select,handle_select,mysql_execute_command,mysql_parse,dispatch_command,handle_one_connection,start_thread,clone
      1 pthread_cond_wait@@GLIBC_2.3.2,os_event_wait_low,srv_conc_enter_innodb,innodb_srv_conc_enter_innodb,buf=0x2ab6574905b0,join_read_always_key,sub_select,evaluate_join_record,sub_select,do_select,JOIN::exec,mysql_select,handle_select,mysql_execute_command,Prepared_statement::execute,mysql_stmt_execute,dispatch_command,handle_one_connection,start_thread,clone
      1 pthread_cond_wait@@GLIBC_2.3.2,os_event_wait_low,srv_conc_enter_innodb,innodb_srv_conc_enter_innodb,buf=0x2aaaae28e998,handler::read_range_first,handler::read_multi_range_first,QUICK_RANGE_SELECT::get_next,rr_quick,mysql_update,mysql_execute_command,mysql_parse,Query_log_event::exec_event,exec_relay_log_event,optimized,start_thread,clone
      1 pthread_cond_wait@@GLIBC_2.3.2,os_event_wait_low,srv_conc_enter_innodb,innodb_srv_conc_enter_innodb,buf=0x2aaaac4fc010,rr_sequential,sub_select,do_select,JOIN::exec,mysql_select,mysql_derived_filling,mysql_handle_derived,open_and_lock_tables,mysql_execute_command,Prepared_statement::execute,mysql_stmt_execute,dispatch_command,handle_one_connection,start_thread,clone
      1 pthread_cond_wait@@GLIBC_2.3.2,os_event_wait_low,srv_conc_enter_innodb,innodb_srv_conc_enter_innodb,buf=0x2aaaac03f000,write_record,select_insert::send_data,end_send,evaluate_join_record,sub_select,do_select,JOIN::exec,mysql_select,handle_select,mysql_execute_command,Prepared_statement::execute,mysql_stmt_execute,dispatch_command,handle_one_connection,start_thread,clone
      1 pthread_cond_wait@@GLIBC_2.3.2,os_event_wait_low,srv_conc_enter_innodb,innodb_srv_conc_enter_innodb,buf=0x2aaaab42ab00,write_record,select_insert::send_data,end_send,evaluate_join_record,sub_select,do_select,JOIN::exec,mysql_select,handle_select,mysql_execute_command,Prepared_statement::execute,mysql_stmt_execute,dispatch_command,handle_one_connection,start_thread,clone
      1 do_sigwait,sigwait,signal_hand,start_thread,clone