Advanced MySQL Administration
Volume Number: 25
Issue Number: 11
Column Tag: MySQL
Advanced MySQL Administration
How to use some more advanced
MySQL features
by Mihalis Tsoukalos
Introduction
MySQL is a very popular database. When first introduced, MySQL was considered to be a simplistic database but now things have changed! A lot of advanced features have been added to MySQL, making it everything but a simplistic database.
This article is going to present some advanced MySQL features along with their relevant commands.
This article assumes that you already have basic MySQL knowledge. Nevertheless, the commands presented here can be used by every MySQL DBA or programmer as they are practical and easy to follow.
Advanced Administration commands
The first part of this article is going to present some advanced commands for MySQL administration.
Check if the MySQL server process is already running
The simplest way to check if MySQL is already running is to use the following UNIX command from Terminal.app (or a similar application):
$ ps -axww | grep -i mysqld
I tried the above command on a PowerMac running Mac OS X 10.4 and MySQL 5.0.16. If you get an output that looks like the following, then MySQL is already running:
6040 p1 S 0:00.03 /bin/sh /sw/bin/mysqld_safe -datadir=/sw/var/mysql -pid-file=/sw/var/mysql/monastery.local.pid
6060 p1 S 0:02.15 /sw/sbin/mysqld -basedir=/sw -datadir=/sw/var/mysql -user=mysql -pid-file=/sw/var/mysql/monastery.local.pid
On a Leopard machine running MySQL 5.0.67, I got the following output:
92942 ttys001 0:00.02 /bin/sh /opt/local/lib/mysql5/bin/mysqld_safe -datadir=/opt/local/var/db/mysql5 -pid-file=/opt/local/var/db/mysql5/Monastery.local.pid
92970 ttys001
0:52.26 /opt/local/libexec/mysqld -basedir=/opt/local -datadir=/opt/local/var/db/mysql5 -user=mysql
-pid-file=/opt/local/var/db/mysql5/Monastery.local.pid -port=3306 -socket=/opt/local/var/run/mysql5/mysqld.sock
As you may understand, the biggest difference between the two outputs is that the first is from a MySQL database that was installed using fink, whereas the second was installed using macports.
Launching MySQL at startup
It would be very nice if MySQL could automatically run at startup J Well, macports (my preferred MySQL installation–yours may vary a little) has an easy way of doing this, which only requires running the following command once:
sudo launchctl load -w /Library/LaunchDaemons/org.macports.mysql5.plist
Now, every time your reboot your Mac OS X, MySQL will automatically start!
Where MySQL saves its actual table data
Again, on my default macports installation, the actual table data is stored inside /opt/local/var/db/mysql5.
Where MySQL saves its logs
As it happens with every respectable database, MySQL has log files. Log files are the perfect place to find information about MySQL startup, shutdown, troubleshooting and error messages. The default place for finding the default MySQL log file is inside your data directory. The log file is called <server_name>.err and you can change its location using the --log-error=<filename> command line option during MySQL startup.
Using the --log command line option, you can make MySQL create an additional log file (again inside the data directory) that saves data about both every connection made and every statement received. For example, giving the "use SA" command as the root MySQL user adds the following lines in the general query log file:
070505 20:55:26 1 Connect root@localhost on
070505 20:55:28 1 Query SELECT DATABASE()
1 Init DB sa
1 Query show databases
1 Query show tables
1 Field List NetData
1 Field List NetData_backup
1 Field List NetData_identical
1 Field List Populate
1 Field List deadlock
1 Field List delete_me
1 Field List t1
1 Field List t2
1 Field List test
The default filename for the general query log file is <hostname>.log.
There is another log file called "Slow Query Log" that will be created when using the -log-slow-query command line option when starting up MySQL. Its default filename is <hostname>-slow.log, its default directory is the data directory, and it saves information about queries that are slower than a given threshold. The threshold can be set using the -long-query-time=<seconds> command line option.
MySQL Deadlocks
First of all, let me introduce you to deadlocks. A deadlock happens when two or more transactions want to change the same resource at the same time, ending up waiting for each other to commit so that they can complete their own change. Deadlocks are pretty common to busy database servers.
The following MySQL commands (using two users, that is creating two transactions) create a deadlock:
USER 1 (MySQL admin)
> LOCK TABLE NetData WRITE;
(this is just an example - usually deadlocks do not happen that way)
USER 2 (starts typing after USER 1 finishes typing her command)
mysql> UPDATE NetData
-> SET sourcePort = "80"
-> WHERE sourcePort = "http";
The last command will not return. If you close the connection of USER 1, you will get an output that looks similar to:
Query OK, 2472 rows affected (36 min 11.25 sec)
Rows matched: 2472 Changed: 2472 Warnings: 0
Generating sample data
In order to be able to follow this section of the article you need to install the Super Smack package (http://vegan.net/tony/supersmack/). After installing it, you will be able to run the gen-data executable that is used for generating sample data.
Sample data can be useful for the following purposes:
Testing your database
Debugging your database
Evaluating the performance of your database
Testing the import process of "real" data
Now that you will also agree that sample data is useful, imagine that you have the following table (called Populate) that you want to populate with sample data:
mysql> desc Populate;
+-----------+--------------+-------+------+----------+-------+
| Field | Type | Null | Key | Default | Extra |
+-----------+--------------+-------+------+----------+-------+
| Name | varchar(50) | YES | | NULL | |
| SurName | varchar(50) | NO | | | |
| Telephone | varchar(20) | YES | | NULL | |
| Id | int(11) | YES | | NULL | |
+-----------+--------------+-------+------+----------+-------+
4 rows in set (0.03 sec)
mysql>
The following UNIX shell command will create 1000 rows of sample data for it:
$ gen-data -n 1000 -f %20-12s,%15-25s,%5-s-%d,%n | head
yxifwtdxjpprrmdcepmdtjhkiyd,kwbmrkieyevrwslkudqth,bdits-357571490,1
vfdwqyybdcidixgknponlatp,ebrufebxnxfhcqqmntb,wvdmi-1723153177,2
tejguakxrysmkhtiusnfgmrph,rvumvbpgnhqppekgw,xutex-272112289,3
fbxgnuntutqjbtnntpaj,wkrwowkvkxutskpajr,lectu-2119878818,4
rpqitwdwakmliclaksubroyg,bwktjfohpstfffgncbiyg,tniwb-897054849,5
ggotajfexekmuraagowxeaa,mewegxpdyjediwhxwfbdy,fqatu-684570285,6
xfihbdqhldotmiqumiwg,oxceiaekpwfffvhnabkhnos,keshk-1163313729,7
ncrstrwtudxdxkijgfyhxdwuagf,auiidbajfehyyryvke,iupxa-222902971,8
svyfvbytivvokxbpdbbffsyyn,aminriayuhlpqks,dpkrd-911254352,9
qkaycvmoaqrxrpyuxuloefe,oqgekxruqfrcoxblqq,vdvlu-102303120,10
Note: You have to write a small script in your favorite scripting language to insert the sample data in your MySQL database.
As you can easily understand, the information found in this section can be used not only in MySQL but everywhere you may need "realistic" data for testing purposes.
SHOW TABLE STATUS
The SHOW TABLE STATUS command is very informative because it displays useful data per table. A part of its output is the following:
mysql> show table status \G
*************************** 1. row *************************
Name: NetData
Engine: MyISAM
Version: 10
Row_format: Dynamic
Rows: 4912
Avg_row_length: 28
Data_length: 138520
Max_data_length: 281474976710655
Index_length: 1024
Data_free: 0
Auto_increment: NULL
Create_time: 2007-03-14 10:02:39
Update_time: 2007-04-12 10:06:14
Check_time: NULL
Collation: latin1_swedish_ci
Checksum: NULL
Create_options:
Comment:
*************************** 2. row ************************
Name: NetData_backup
Use this command when you want to have a general overview of the tables found in a MySQL database.
SHOW FULL PROCESSLIST
The SHOW FULL PROCESSLIST command is very useful for finding out what is running in your database server. The following output is from the deadlock example:
mysql> SHOW FULL PROCESSLIST \G
*************************** 1. row *************************
Id: 7
User: root
Host: localhost
db: sa
Command: Query
Time: 97
State: Locked
Info: UPDATE NetData SET sourcePort = "http" WHERE sourcePort = "80"
*************************** 2. row *************************
Id: 8
User: root
Host: localhost
db: sa
Command: Query
Time: 0
State: NULL
Info: show full processlist
2 rows in set (0.00 sec)
mysql>
Finding the privileges of a user
If you want to find out the privileges of a user ("mtsouk" in this example), you can use the SHOW GRANTS command:
mysql> mysql> SHOW GRANTS for 'mtsouk'@'localhost' \G
*************************** 1. row *************************
Grants for mtsouk@localhost: GRANT USAGE ON *.* TO 'mtsouk'@'localhost' IDENTIFIED BY PASSWORD '663c5dd53dae4ed0'
*************************** 2. row *************************
Grants for mtsouk@localhost: GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, REFERENCES, INDEX, ALTER, CREATE TEMPORARY TABLES, LOCK TABLES ON `widget`.* TO 'mtsouk'@'localhost'
*************************** 3. row *************************
Grants for mtsouk@localhost: GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, REFERENCES, INDEX, ALTER, CREATE TEMPORARY TABLES, LOCK TABLES ON `sa`.* TO 'mtsouk'@'localhost'
3 rows in set (0.01 sec)
Notice the \G string (two characters) at the end of the command. When you have long rows, \G improves the output making it easier to read and understand. Try the same command without the \G and you will understand what I am talking about.
The EXPLAIN command
Those of you who are interested in or familiar with database theory and/or performance issues will find the EXPLAIN command particularly interesting and handy. First, let me show you an example:
mysql> EXPLAIN select count(*) from NetData \G
************************** 1. row **************************
id: 1
select_type: SIMPLE
table: NULL
type: NULL
possible_keys: NULL
key: NULL
key_len: NULL
ref: NULL
rows: NULL
Extra: Select tables optimized away
1 row in set (0.00 sec)
mysql>
The output of the EXPLAIN command provides extremely useful information such as how MySQL thinks it is best to execute the given SQL statement, the PRIMARY and SECONDARY keys used, the indices used, etc. Put simply, a database has many ways of executing a query; the more complicated the query is, the more (equivalent) ways there are to execute it. The EXPLAIN command helps you determine if MySQL has chosen an optimal method for accessing its data.
If you think that your query is slow, try the EXPLAIN command as it may help you explain some of the reasons of its slow execution.
Creating an empty table that is identical to an existing table but without the table data
The following command will create a new table called NEW_TABLE that looks like the original OLD_TABLE table without its actual data:
CREATE TABLE NEW_TABLE
as
select * from OLD_TABLE
where 0 = 1;
Creating a backup copy of an existing table
The following output illustrates how to create a backup copy of the NetData table:
mysql> desc NetData;
+------+-------+---+---+-----+------+-----+---------+------+
| Field | Type | Null | Key | Default | Extra|
+------+-------+---+---+-----+------+-----+---------+------+
| Date | datetime | NO | | | |
| dt | float | YES | | NULL | |
| sourcePort | varchar(12) | YES | | NULL | |
| destPort | varchar(12) | YES | | NULL | |
+------+-------+---+---+-----+------+-----+---------+------+
4 rows in set (0.04 sec)
mysql> create table NetData_backup as select * from NetData;
Query OK, 4912 rows affected (0.02 sec)
Records: 4912 Duplicates: 0 Warnings: 0
Returning random rows
The MySQL rand() function can be used to return rows randomly. The following command returns 5 random rows:
select * from NetData
order by rand()
limit 3;
Two possible outputs are:
+----------------------+----------+------------+-----------+
| Date | dt | sourcePort | destPort |
+----------------------+----------+------------+-----------+
| 2007-01-27 15:12:11 | 0.000369 | 3144 | http |
| 2007-01-27 15:15:02 | 0.000323 | http | 4223 |
| 2007-01-27 15:09:04 | 0.000371 | 2610 | http |
+----------------------+----------+------------+-----------+
3 rows in set (0.14 sec)
+----------------------+----------+-------------+-----------+
| Date | dt | sourcePort | destPort |
+----------------------+----------+------------+-----------+
| 2007-01-27 15:04:19 | 0.000513 | smtp | 1108 |
| 2007-01-27 15:06:08 | 0.003117 | 2178 | http |
| 2007-01-27 15:07:13 | 0.000183 | http | 2590 |
+----------------------+----------+------------+-----------+
3 rows in set (0.01 sec)
Running multiple MySQL server processes on a single machine
A lesser-known feature of MySQL is the possibility of running multiple server processes on a single machine. It is very handy when you want to run an experimental MySQL installation or you do not have a second machine to try things. The difficult part is that you must manually specify certain parameters on the command line.
The following example (using a fink MySQL installation) starts a second instance of MySQL that uses port 3305 (the default port of MySQL server is 3306–you can verify it by typing "telnet localhost 3306"), stores its data inside the /tmp/mysql directory and is a console application:
sudo /sw/sbin/mysqld -port 3305 -datadir="/tmp/mysql" -console -user=mysql
Using this method, you can start as many MySQL server processes as you want. Keep in mind that the datadir directory must be both readable and writable by the MySQL user.
In order to connect to the above MySQL instance you have to run the MySQL client using the -port 3305 parameter.
Getting the MySQL source code
MySQL has a great advantage over databases such as IBM DB2 or Oracle; you can get, view or change its source code according to your needs, provided that you know what you are doing.
If you have OS X Server, MySQL binaries are installed as part of the OS. OS X needs to have MySQL installed, however. Even on OS X Server, you may want to run a different version than that supplied or be able to tweak the code yourself. There are three main ways of getting the MySQL source code:
a. Using the BitKeeper revision control application (the modern way)
b. Downloading the source code of a specific version release (the traditional way)
c. By telling fink or macports to get the code for you (the Mac way!)
In the Web Links and Bibliography section you will find more information about getting the MySQL source code.
Notice: it is truly useful to try to understand the MySQL source code by reading it; especially if you are familiar with the C programming language. People that have research interests in the database field will also benefit from reading the MySQL source code in order to learn more about database internals, execution plans, optimization techniques, cost estimation, etc.
Backup and Recovery
Two essential tasks of every database are backup and recovery.
Backing up a MySQL database
Let us say that you want to backup all the tables in the SA database. The following UNIX shell command will do what you want:
$ mysqldump SA -u root -p > sa_backup.sql
Enter password: xxxxx
$ ll sa_backup.sql
-rw-r-r- 1 mtsouk mtsouk 455563 Mar 14 09:54 sa_backup.sql
$ head -n 5 sa_backup.sql
- MySQL dump 10.10
-
- Host: localhost Database: SA
- ---------------------------
- Server version 5.0.16
SA is the name of the database that we want to backup, whereas sa_backup_sql is a text file that holds the necessary SQL commands that can restore the SA database.
Restoring a MySQL database
In order to restore a MySQL database, you can use the following way (again, from a UNIX shell):
$ mysql SA -u root -p < sa_backup.sql
It is highly advisable to check your backup strategy from time to time in order to make sure that everything works as expected. If you do not have a reliable backup strategy, you may get into trouble if you ever need to make a restore of you database.
Replication in MySQL
Replication is the creation of an identical database (as the one you have), in an alternative place.
Replication, a feature used to be found in advanced databases, is also available in MySQL. There are two types of replication: synchronous and asynchronous. MySQL implements asynchronous replication: data is transferred to the replicated systems with a small delay (lag).
Binary logs are utilized for the replication process to work. Everything that is changing the data in the master database is being written in a binary log. A process in the master database reads this binary log and transfers queries to a special log file in the slave database, called relay log. Another process in the slave database reads the relay log and executes the queries found.
The MySQL Data Dictionary
The tables that hold data about the database itself compose the Data Dictionary of the database. This part of the article will present some of them.
The mysql.user table
This table holds important information about the users of the database. It is a very critical MySQL table that DBAs are using all the time. To understand the contents of this table, use the desc (describe) command:
mysql> use mysql
mysql> desc mysql.user;
The INFORMATION_SCHEMA.TABLES table
The INFORMATION_SCHEMA.TABLES table holds metadata about the database tables on the server.
mysql> describe INFORMATION_SCHEMA.TABLES;
The INFORMATION_SCHEMA.COLUMNS table
The INFORMATION_SCHEMA.COLUMNS contains information about the columns found in the tables of the MySQL server.
mysql> desc INFORMATION_SCHEMA.COLUMNS;
Every database contains lots of tables and views in its data dictionary. This part of the article presented a tiny part of the MySQL data dictionary in order to give you an idea of how those tables or views look.
Benchmarking MySQL
Benchmarking is the process of creating performance results for some given testing scenarios. The results are then used for the following purposes:
a. To compare your system configuration with other systems
b. To help you understand the performance impact of changes you made to the database
c. To find the limits of your system
d. To find the problematic parts of your system
MySQL offers a benchmarking suite. Super Smack can also help you benchmark your MySQL database (See below).
The following is the output of the MySQL benchmarking suite using a normal MySQL user:
root# ./run-all-tests -pass="xyzw" -user="mtsouk"
Benchmark DBD suite: 2.15
Date of test: 2007-05-07 22:58:41
Running tests on: Darwin 8.9.0 Power Macintosh
Arguments:
Comments:
Limits from:
Server version: MySQL 5.0.16/
Optimization: None
Hardware:
alter-table: Total time: 12 wallclock secs ( 0.08 usr 0.03 sys + 0.00 cusr 0.00 csys = 0.11 CPU)
ATIS: Total time: 31 wallclock secs ( 8.65 usr 6.06 sys + 0.00 cusr 0.00 csys = 14.71 CPU)
big-tables: Total time: 34 wallclock secs ( 7.09 usr 13.08 sys + 0.00 cusr 0.00 csys = 20.17 CPU)
connect: Total time: 185 wallclock secs (46.70 usr 44.82 sys + 0.00 cusr 0.00 csys = 91.52 CPU)
create: Total time: 190 wallclock secs ( 9.58 usr 3.06 sys + 0.00 cusr 0.00 csys = 12.64 CPU)
insert: Total time: 1668 wallclock secs (467.41 usr 262.61 sys + 0.00 cusr 0.00 csys = 730.02 CPU)
select: Total time: 634 wallclock secs (46.81 usr 23.97 sys + 0.00 cusr 0.00 csys = 70.78 CPU)
transactions: Test skipped because the database doesn't support transactions
wisconsin: Total time: 13 wallclock secs ( 3.11 usr 1.91 sys + 0.00 cusr 0.00 csys = 5.02 CPU)
All 9 test executed successfully
Totals per operation:
Operation seconds usr sys cpu tests
alter_table_add 5.00 0.03 0.01 0.04 100
alter_table_drop 5.00 0.02 0.00 0.02 91
connect 13.00 6.13 1.73 7.86 10000
[many benchmarks omitted]
wisc_benchmark 4.00 1.87 1.11 2.98 114
TOTALS 2755.00 579.87 349.13 929.00 3425950
The output may look complicated at first sight, but should become more meaningful to you if you run the tests in different platforms, using various operating systems or if you change the amount of memory in your system.
Note: On my PowerMac G5 machine using a fink MySQL installation, the location of the run-all-tests script is the /sw/share/mysql/sql-bench directory. Yours may vary.
Summary
You can surely understand that this article could not cover everything about advanced MySQL administration. Its main purpose is to give you information about the capabilities of MySQL that you should further search and experiment with. Despite that, you learned many advanced MySQL commands in this article that you can use in every day tasks and make your DBA and programming life easier.
Web Links and Bibliography
http://www.mysql.org
http://www.mysql.com
Super Smack package: http://vegan.net/tony/supersmack
BitKeeper: www.bitkeeper.com
Available MySQL source trees: http://mysql.bkbits.net
MySQL source code: http://dev.mysql.com/downloads
Charles A. Bell, Expert MySQL, APress, 2007
Michael Kruckenberd and Jay Pipes, Pro MySQL, Apress, 2005
Russell Dyer, MySQL in a Nutshell, O'Reilly, 2005.
MySQL Reference Manual: http://dev.mysql.com/doc
Chad Russell and Jon Stephens. Beginning MySQL Database Design and Optimization: From Novice to Professional. Apress, 2004.
MyBench: http://jeremy.zawodny.com/mysql/mybench
Control MySQL server via launchd (manually creating the required files): http://www.macosxhints.com/article.php?story=20080128103022907&mode=print
Mihalis Tsoukalos enjoys digital photography, writing articles and programming iPhone apps. He is the author of the "Programming Dashboard Widgets" eBook. You can reach him at tsoukalos@sch.gr.
