Mastering MAMP's Metadata-Munching MySQL

Since this post ran a bit long I've included a TL;DR section.

The Problem

MAMP's MysQL uses a shared InnoDB file "ibdata1" which grows over time and never shrinks. Not even when dropping databases. All the metadata and indices for all databases stay in there forever.

The Solution

Tell MySQL to store medata and indices in separate files per database.

  1. Dump all databases in files and drop the databases.
  2. Delete ibdata1, ib_logfile0 and ib_logfile1.
  3. Add innodb_file_per_table to the [mysqld] section of /etc/my.cnf
  4. Restart MySQL and reimport the dumped databases.

The Article

Recently I had to import a MySQL dump of 9GB resulting in a database of 21GB. I didn't have enough space on my hard drive and after continuously deleting applications and music as the import went on I realised I wouldn't make it. I aborted the process and dropped the partial database. Funnily enough I noticed I didn't get all my space back. After a little scooting around on the file system I understood something I had been suspecting for a while already. MAMP's default MySQL settings suck.

jeroen@liver:/Applications/MAMP/db/mysql$ ls -hl | grep ibdata1
-rw-rw---- 1 jeroen admin 14G Nov 4 16:25 ibdata1

I've always been slightly opposed to using MAMP for serious work but the company I work for uses it and their setup scripts expect folder layouts the way MAMP sets them up. Now, after a year of using MAMP on a 128GB MacBook Air I saw the shared InnoDB file MySQL uses had grown to 14GB. That is a lot of wasted space.

Why is it wasted space you might ask? Surely MySQL needs it for something. Yes, MySQL needs it. It stores the _current_ database-indices in there as well as the metadata for them, you can even restore whole databases, after having dropped them, from this InnoDB data file. But, it also stores all that information for databases you have already dropped. See, MySQL's ibdata1 file never shrinks, it only grows.

As a web developer working with Drupal for about a year now, one of the things I do a lot is importing databases of ongoing projects, dropping them when I screw up and reimporting a backup to try again. Therefore I bet not too many web-developers know about MAMP's hungry MySQL or I would've heard about it earlier.

There is a solution and it is called innodb_file_per_table. When you put this in the MySQL configuration file it tells MySQL to store the metadata and indices for each database in a separate file within that database's folder. Then, when you drop a database and MySQL deletes the folder it will also remove the metadata files.

If you're wondering where you can find the MySQL config file your best guess is /etc/my.cnf unless of course you're using MAMP because it just starts MySQL with any meaningful parameters, neither does it install a my.cnf anywhere MySQL looks for one. (More info here: http://bensch.be/mysql-config-with-mamp) You can copy one of MAMP's default MySQL config files to /etc/my.cnf. Look for them under /Applications/MAMP/Library/support-files/

None of MAMP's MySQL config files however include the innodb_file_per_table setting so make sure to add this to the [mysqld] section of the file you chose to copy to the /etc/my.cnf:

innodb_file_per_table

When you restart your MySQL server it will start putting the medatadata in the database folders that will be created from then on, unfortunately this does not get rid of the huge ibdata1 file. All current databases must be dumped and dropped first, then the ibdata1, ib_logfile0 and ib_logfile1 fils can be deleted. MySQL can be restarted with the innodb_file_per_table setting in place and the dumped databases can be reimported.

You'll be saving gigabytes of disk space in your day-to-day development.

While I was writing this post I came across this Stack Overflow answer: http://stackoverflow.com/a/4056261 It tells the same story but with a bit more detail about what's in the ibdata1 file.