PHP is ideal for a task such as this: parsing directories and subdirectories for images is easy; resampling using the built-in library (GD) is a breeze; specifying the destination as a subdirectory is simple. The one minus was processor usage. Performing image manipulation eats up the CPU in a big way.

Luckily linux systems have a built-in utility for addressing a situation like this: nice. nice will “run a program with modified scheduling priority.” I’m running the image manipulation script using the following command:

nice --adjustment=19 php script.php

If nothing else is going on the script will use whatever resources are available. When anything with a higher priority executes, that program will take precedence over the script with regard to system resources. The script should thus not affect the responsiveness of the web server. This is the reason I was searching for this kind of functionality.

I created a series of SQL statements to run in MySQL that will convert the data from the packet_item_records and miscon_packet_refs tables to the packet_students, packet_data, and miscon_packetdata_refs tables. After the conversion I updated any pages that referenced the old data structure. So far in testing the data seems to have converted perfectly.

Minus one issue. Multiple selections for the answer choice questions (A, B, C, D) from 2006 were recorded as a generic Multiple rather than Y+NS, N+NS, etc. This value is not represented in the updated data format or on the data entry forms or summary tables. Rather then spend too much time addressing this issue I’m going to leave these values empty for now. I don’t expect this to be a problem since the researchers are focused on data for the current pilot and field tests. Also, I’m keeping the old version of the piloting data and scripts that interact with it online in case it’s needed.

Installation was not too difficult, though I did have to ensure that a few dependencies were installed before PEAR would load the package. Documentation for the module is not that great, but there are a number of samples and a fairly active discussion forum.

I did not find the graphing object very intuitive. It seems to me that a lot of the properties and functions are abstracted from the objects they affect. I suspect this was a design decision to allow for greater flexibility, but it may also be due to an older coding style. The current branch appears to have been developed in earnest starting in 2005, and it’s been a few years since a new release. Even so, my initial exposure leads me to believe the package may be stable enough for even a production-level project so long as sufficient QA has been performed.

It took me only a couple of days to hack together the demonstration. Though the requirements of the script were few, I decided to push the limits of my object-oriented PHP and create the graphing component as a class. Overall it was a good first-try at producing something of this nature. You can find the demonstrator at http://flora.p2061.org/weather_data/.

Updating the interface

First up is the form organization. Previously the forms were designed, I believe, with the researchers in mind. The most important data entry sections were placed at the top of the page with everything else following. Once we started using temps to enter the data, however, this format proved to be a hindrance to efficiency. So to improve efficiency I regrouped the form into logical sections: student responses, student demographics, and researcher notes. Plus, to allow for easier scanning I organized the student responses section so that it resembles the paper form.

In addition to organizing the form more logically, I wanted to make portions of the form collapsible. Since a lot of the data entry is performed by temps there is no reason for them to have to wade through the research notes, which are at the top of the page by request of the researchers. Brian W. has commented on the utility of Dreamweaver’s Spry widgets so I thought I’d give them a try. I must admit that they work quite nicely and are easy to implement, and I found just what I needed in the Collapsible Panel widget. The benefit of using the panels is that I could position the researcher notes at the top of the page without significantly hindering the ability of the temps to access the student responses by having this panel closed by default.

Which brings me to a minor change I made to the functionality of the panels: having the appropriate panels open when the data entry form is loaded (e.g. student responses always open; researcher notes closed unless otherwise requested by the user; demographics only open if no data has been entered). I used a combination of PHP and JavaScript to set the initial state. Unfortunately this seems to caused some problems for Dreamweaver. When you open the packet data entry form in Dreamweaver the program appears to freeze. It appears Dreamweaver is attempting to process the spry widgets and choking on the PHP-derived JavaScript variables. Eventually Dreamweaver informs you that “A script in file … has been running for a long time. Do you want to continue?” Answering no to this will return Dreamweaver to normal operation. I could get around this problem by setting the state of the panels after they have been initialized, but this produces a disorienting collapse of panels after the page has loaded.

The final interface modification was a small script that makes it possible to deselect radio buttons. The main purpose of developing this feature was to streamline the form which, in previous revisions, had an extra option to clear the selected response.

Improving the foundation

As I mentioned, while the form is still hard-coded (each question coded by hand) I did lay out the foundation for a more dynamic approach. First I redeveloped the table that stores the data related to a packet. I reused the packets and packet_items tables, but created a new table called packet_data that stores only the question number, answer, and any comments. The benefit here is that forms with varying numbers of questions can be stored in this one table. I also moved the student meta data into a separate table, a design decision more in line with database normalization practice.

Another modification I made was to use a structured array to define the questions in the packet. While this has not been fully developed as of yet, I have used it to significantly simplify the script that saves the data. Rather than specify each question and corresponding HTML form element and writing the query to insert that information in the database I designed the PHP to loop through the array and generate the SQL code on the fly.

Finally, I used another structured array to hold data related to the packet, packet items, and students while generating the packet data entry page. I do not utilize any methods to automate the form generation at present, though there is little to prevent this from being developed. One reason I decided to code the form by hand is to allow for a level of customization that is difficult to accommodate when automating the visual side of things. Even so, utilizing structured arrays allows for more consistency between PHP pages.

Moving forward

Data from years prior has not yet been ported into the new structure. I do not expect this task to require too much work, but I am not sure when I’ll be able to perform the task. At the very least this will be part of the redevelopment taking place around subversion if not sooner.

To fully accommodate the data in the utility I’ll need to also update the summary tables, the school report generator, and packet data export. These updates will need to be completed fairly soon in order to support the item review that will begin taking place fairly soon.

Because this is more of a long-term project it is being performed along side any current development of the items utility. As a result any modifications to the items utility should be brought to my attention so that they can be ported to the SVN working copy if necessary.

Workflow

Since we’re not using an IDE that has integrated subversion capability, and because I wanted to keep the process as simple as possible, I decided to try a combined approach to development. A working copy of project files are checked out from subversion to the development server. Dreamweaver uses the working copy for its remote. Code development will continue as usual in Dreamweaver. When a development milestone is reached the changes are committed to subversion. When the project is ready for deployment, we use tagging to mark a stable copy in the repository and check it out to the production server.

The main problem I’ve found with this setup right now is that there is no way to hide the subversion structure from Dreamweaver. Subversion creates a directory to house control files for each directory in the working copy. The subversion control directories can be cloaked, allowing Dreamweaver to function as if these directories don’t exist, but this is not as feasible for a project with lots of directories.

I found an extension that can automatically cloak any subversion directories, but it only works for the local copy. So if you’re willing to go through the process of downloading the entire site you can use the extension to cloak all subversion directories automatically. Read about the extension on Adobe Exchange; I’ve saved a copy of the extension on the techlog.

I have set up subversion to ignore the typical Dreamweaver development files (lock files, notes, backups). This allows the benefits of using the Dreamwever environment to its fullest without mucking up the subversion repository with unnecessary data.

We should also implement configuration options in a single file that is not part of the project files. We would still want to provide a sample configuration file, but the actual configuration file should be created during initial deployment by copying the sample and modifying the parameters. Using a sample file instead of actual configuration in the repository will allow us to upgrade the production server without having to worry about overwriting local settings. Overwriting the configuration is actually not much of a concern so long as the configuration file is not updated in the repository. But implementing this kind of setup does give us a little more security in that regard.

Subversion setup

First step in the process is to create a subversion repository. We’ll create this on the production server because we can see the production server from the development server, but not vice versa. To set up the subversion repository in /inet/svn, run the following command on the production server:

svnadmin create /inet/svn

Next we’ll populate the repository with a project. Since we’re just getting started with subversion we’ll utilize a well-known repository layout: project, project/branches, project/tags, project/trunk. Create the directory structure on the development server and place the current development files in project/trunk. There are a number of ways to access a subversion server, but for now we’ll use file-based access over ssh. If I create the project directory in /tmp, I would run the following command to import the project:

svn import /tmp/benchmarks svn+ssh://production.server/inet/svn

Working with project files

Now that we have the data in the repository we need to set up a working copy for Dreamweaver. First change to the directory where you want to store the working copy. Then run the following command to check out a working copy:

svn checkout svn+ssh://production.server/inet/svn/benchmarks/trunk .

Set up this directory as your remote in Dreamweaver and edit as usual.

At some point in the future you’ll want to record the changes in the subversion repository. This can be at any time in the development process, but at the very least it’ll need to be done when a project has reached a stable state and is ready for deployment. Run the following command from the working copy directory:

svn commit

Once the project has reached a stable state we need to have a way of maintaining that stable copy in the face of future modifications. This is where the branches/tags/trunk structure really comes in handy. The in-development files are always in /trunk. To create a stable release we merely need to create a copy in the /tags directory (where ver.no is the version number, such as 1.0):

svn copy svn+ssh://production.server/inet/svn/benchmarks/trunk svn+ssh://production.server/inet/svn/benchmarks/tags/ver.no

Deploy the project

Whether we’re talking about a home-brewed application or one from an external organization, the steps for deployment are the same. Check out a working copy or export a clean copy to the appropriate directory on the production server. I prefer to check out a working copy because it enables greater flexibility in deployment. If we need to modify a file in the production environment, a working copy enables non-destructive upgrades in the future. Only the files that have been modified in the repository will be updated during a version upgrade. Any conflicting updates will be noted by the subversion client so that they can be addressed. (Of course, this is only really a benefit if significant modifications have not been made to the project between versions.)

Updating a working copy to a new version is as simple as the following command (executed from the working copy directory):

svn switch repository_address .

Belt and suspenders

To avoid loss of our subversion repository we should consider creating a copy on our development server. This would only be used to restore the subversion repository in case of loss or corruption. This can be done using the svnsync command and should be done automatically (via cron). This may not be a necessity as much as a convenience, however, since the repository should be included in the nightly backup.

General Reading

What I attempted to provide here was the core commands needed to get started. There’s a lot more to working with subversion than what I covered and I would recommend .

• Version Control with Subversion (all you need to know)
  
• Tracking PeopleAggregator releases with Subversion vendor branches
• Subversion Vendor Branches Howto
• Vendor Branches in Subversion

I found the vendor branch concept a bit confusing; so a few extra links for that.

Purpose and Need

As I mentioned, we’re using the full text search capability of MySQL as the core of our search functionality. By default, MySQL performs a natural language full text search. This mode has a few restrictions that can affect the results that are returned, but most notable for our purposes is that a word provided by the user must match exactly a word in the record. What this means is that records with a variation of the word (such as a plural when a singular was provided) will not be returned. For example, if you search for “atom,” MySQL will not return records that have “atoms” or “atomic” but not “atom.”

I’ve found the easiest way to address this limitation is by performing the search using the keywords as typed, but then adding on results that have matches to the stem of the keywords. By using these two full text search methods in conjunction we can get the best results for our user.

Finding the Stem

The hardest part of this whole process was determining a way to find the stem for a word. Initially I developed a simple function that stripped plurality from a word. Is was simplistic, but provided a good sample of how effective stemming could be. A little more research led me to a PHP extension that uses an established algorithm to determine a word stem. The extension is available from the PECL PHP library and is called “stem.” Installation is simple. On *nix you first install the extension:

sudo pecl install stem

Next, update the php.ini file to enable the extension by adding the following line:

extension=stem.so

Finally, restart the web server for the new setting to take affect:

sudo /etc/init.d/apachectl graceful

On Windows you first download the dll from pecl4win.php.net (PECL installation actually involves compiling the extension binary which is not quite as straightforward on Windows). Place the dll in your php/ext directory. Next, update the php.ini file to enable the extension by adding the following line:

extension=stem.dll

Finally, restart IIS for the settings to take affect.

How to Use

Performing a search with stem expansion using MySQL’s full text search functionality is actually quite easy. First we need to break the search terms up so that we can stem them. To do this we need to isolate each word in the search phrase. We can do this by splitting the search phrase into an array. It’s impossible to know exactly what the user will type, so we can be greedy and use something like:

$araKeys = preg_split('/\W/', $strSearch);

Then we need to modify each keyword so that we have its stem:

$araKeys = array_map('stem', $araKeys );

Recombine the keywords into a new boolean search phrase:

$strSearchBool = join('* ', $keysArray) . '*'

Finally, we combine the original natural language search with a boolean mode search:

MATCH(fulltext_index_columns) AGAINST ('$strSearch') OR MATCH(fulltext_index_columns) AGAINST ('$strSearchBool' IN BOOLEAN MODE)

Order the results by the sum of the relevance values for the full text searches and you’re done:

ORDER BY (MATCH(fulltext_index_columns) AGAINST ('$strSearch')) + (MATCH(fulltext_index_columns) AGAINST ('$strSearchBool' IN BOOLEAN MODE)) DESC


The two combined provides a good sense of how well a result matches. Records that match the entered search terms exactly will bubble to the top of the list since they match on both the natural language and boolean searches. Records that match only on the stem-based boolean search will come next, and the more words they match the higher in this secondary list they will be. These stem-based matches would not be returned by the natural language search since the exact search terms were not present in the records.

Future Development

MySQL has stated a desire to enhance full text searching with stem and proximity information. The implementation of these features will negate the need for this to be done via the hack described above. At that point this extra code should be removed, though I doubt it will cause problems with the search results if left in place.

• Added a save feature so that packet/item selections for a report can be retrieved at a later date.
• Added the ability to set thresholds that must be met before the script will include statistics in the report.
• Updated tables to reflect changes in displayed information and formatting.

In regards to the above changes, the save feature is of particular note for it inscrutibility. While the technique used to save/retrieve the report setting is adequate, the code to enable this functionality is not well-implemented. Unfortunately, time constraints required a fast, rather than best, implementation.

As modifications are requested this script is getting to be a little harder to work with. The code was created over the course of a week or two back in January. I was able to make quick work of it by using some concepts I initially worked out for summary table generation (and based on discussions with Brian W). While the data is stored in a psuedo object-oriented format, the script itself is fairly linear in design. If more modifications are requested the script may need some rewrites to enable a bit more flexibility.

MyISAM Recovery

MyISAM is a nice format for speed and has support for functionality not available in other MySQL storage engines (such as full text search). Unfortunately MyISAM is not nearly as robust as InnoDB. Since the data files are dealt with directly sans transactions a system crash can cause table corruption and loss of data (particularly is an INSERT/UPDATE operation were in progress). To ensure that the tables have not been corrupted at any time we can set myisam-recover=BACKUP,FORCE. This will tell MySQL to check a MyISAM table when it is opened, repair it if necessary, and make a backup of the table.

There are some drawbacks with this setting. First, if a row is corrupted the data from that row could be lost. That’s why we use the BACKUP option. Also, there can be a performance hit due to recovery operations, particularly if a large number of tables have to be repaired simultaneously. Not to mention that the recovery check is done every time a table is opened.

Another method of checking the MyISAM tables we should consider is a cron job that checks the tables outside of MySQL. This would give us the benefit of automated repair (or at least notification) while mitigating possible performance bottlenecks.

References:

• MySQL Reference Manual: 5.1.2. Command Options
• MySQL Reference Manual: 5.1.3. System Variables
• MySQL Performance Blog: Using MyISAM in production
• MySQL Performance Blog: MySQL Crash Recovery

Thread Cache

MySQL assigns a thread to each connection made by a client. Thread creation/destruction can be a relatively expensive process, but MySQL gets around this by implementing thread caching. Thread caching allows MySQL to reuse a thread once a connection is finished with it. A thread is not destroyed unless the number of active threads exceeds the thread cache limit. The thread cache does not have to be large enough to handle all simultaneous connections, particularly since maintaining a thread uses up system resources. The cache should be large enough, however, such that the number of threads created is small. Check the Threads_created server status variable.

We currently do not have thread caching enabled. Our threads created is at 60,000, which is extremely high (~460 per day). Unless you set up persistent connections to MySQL using the pconnect() function PHP will open/close a connection each time a web page is loaded. I’m going to go with a value of 20. Assuming maximum concurrent connection could reach 25 this provides plenty of cached connections for average usage. If the number of threads created does not budge past the cache limit we should consider lowering the value somewhat to free up resources.

For an hint of how important thread caching is, see MySQL, Linux, and Thread Caching and Optimize MySQL: The Thread Cache.

References:

• MySQL Reference Manual: 7.5.7. How MySQL Uses Threads for Client Connections
• MySQL Performance Blog: What to tune in MySQL Server after installation

Table Cache

Opening a table can be a slow process as data descriptors are created and file headers are modified. To mitigate this MySQL uses a file-based table cache that maintains a table in an open state for future connections. A unique entry in the table cache is required for each concurrent access to a table (multiple users or multiple accesses by a single user in a query). Every time a table is opened an unused table cache entry for that table is sought. If none is found a new one is created. Once the table cache reaches the limit specified by the table_cache variable MySQL has to close old cached connections before opening new ones, adding even more time to table access. MySQL recommends sizing the table cache so that it can handle the largest number of concurrent connections multiplied by the largest number of tables accessed by a single query. This is at the high end. You can start lower and watch opened_tables to see if the table cache is constantly swapping out tables. The faster opened_tables rises the more urgently the table cache needs to be increased.

One caveat to consider when setting the table cache is the per-process file pointer limit. Each cache entry is associated with MySQL. If the number of files held open by MySQL exceeds the limit allowed by the operating system no further files can be opened. MySQL does not fail gracefully in this situation and may, according to the documentation, “refuse connections, fail to perform queries, and be very unreliable.” You can find the file usage limit by issuing the following command cat /proc/sys/fs/file-max. It’s very unlikely we’ll have a problem; the current value indicated by this command is 50569. But the results of going over this limit appear to be fairly severe for MySQL, so it’s a good idea to check.

The current table cache of is set to 160 and is full, but the value of opened tables is rising slowly. Still, our maximum concurrent connections has already hit ten, so I believe we could easily see the table cache get overwhelmed once the database is exposed to a larger traffic base. If we assume maximum concurrent connections of 25 and a crazy join of 10 tables then we’re looking at a table cache of around 250. I’ll start with this number and watch the opened tables stat to see if the cache needs to go higher or can be lowered.

References:

• MySQL Reference Manual: 7.4.8. How MySQL Opens and Closes Tables
• MySQL Performance Blog: What to tune in MySQL Server after installation
• Database Journal: Optimizing the mysqld variables

Temporary Tables

Temporary tables may be used by MySQL when performing queries. By default these tables are created in memory. However, there are two situations in which a temporary table will be written to disk, resulting in a bit of a performance hit: when a temporary table grows beyond the maximum allowed; and when a condition exists that prevents the use of a temporary table. The former situation is one determined by the tmp_table_size/max_heap_table_size parameters. The latter is determined by table and query structure. tmp_table_size is specific to temporary tables while max_heap_table_size applies to all memory tables, so make sure that max_heap_table_size is at least as large as tmp_table_size.

The allowable size of temporary tables should be large enough to avoid writing to disk where possible, but small enough that memory is not eaten up. There is no provision to limit the number of temporary tables stored in memory. If there are many simultaneous connections and each connection is working with a large temporary table memory could be filled rather quickly.

You can determine whether or not your temporary tables are being created in memory by looking at the number of temporary tables that had to be written to disk (Created_tmp_disk_tables). Ours is hovering around 50% of the total number of temporary tables (Created_tmp_tables), but this isn’t enough information to make a decision about the optimal setting for tmp_table_size. What we don’t know is the reason a table is written to disk. That’s something that can only be determined using the EXPLAIN statement.

Since the number of disk-based temporary tables is relatively high I’m going to increase the maximum size allowed for memory-based tables and see if that improves things.

References:

• MySQL Reference Manual: 5.1.3. System Variables
• MySQL Reference Manual: 5.1.5. Status Variables
• MySQL Reference Manual: 7.5.9. How MySQL Uses Internal Temporary Tables
• Database Journal: Optimizing the mysqld variables

Query Cache

For SELECT queries, the speed of the response can be affected by factors such as query structure and which columns are indexed. MySQL is able to provide fast results for often-run queries by storing the results in the query cache. The first time a query is run the query and its result set are stored in the query cache. Subsequent runs of the exact same query will pull the results from the cache (so long as the cached entry is available). There are some qualifications for the query cache to be used successful. First, a query must match in a binary manner in order for MySQL to use a cached query. This means a query must match character for character in a case-sensitive way with the previous run. Second, the query results must not exceed a defined size (1MB by default) or it will not be cached.

To enable the query cache set query-cache-type = 1 and give query_cache_size a value. The best size of the query cache is a guessing game. I’m starting with a value of 32M. Though we’re likely to see little performance improvement on the web-based applications due to constant table updates (which invalidates any cached results for that table) we should see decent performance improvement for more static tables such as those used to serve dbBOL. As a result, judging the efficiency of the cache based on the usage data provided by MySQL will be somewhat difficult. Still, I would expect to see a relatively high value for Qcache_hits when compared to Qcache_inserts and a low value for Qcache_lowmem_prunes. It’s also important to keep an eye on Qcache_free_blocks in order to ensure the cache memory is not fragmented. This number should remain low and can be improved temporarily by issuing a FLUSH QUERY CACHE statement, which will “defragment” the query cache.

One thing to keep in mind is that any cached query will be dropped if the table it references is updated. Tables that are updated often should include the SQL_NO_CACHE attribute in any SELECT queries to prevent caching. This will help prevent the extra overhead of storing and dropping queries from the cache when those caches can rarely be reused.

References:

• MySQL Reference Manual: 7.5.4. The MySQL Query Cache
• MySQL: A Practical Look at the MySQL Query Cache
• Database Journal: MySQL’s Query Cache
• Whatever….: mysql query cache

Key Buffer

MySQL says this is one of the most important performance tuning variables and recommends allocating as much memory as possible. However, since the key buffer is stored in RAM a setting should be used that (in consideration with other settings) won’t cause the server to page memory. MySQL also recommends that the key_reads/key_read_requests be less than 0.01.

The MySQL Performance Blog recommends up to 40% of your system memory, taking into account the size of the MyISAM table indexes and available memory.

Currently the setting for this parameter is 16M and our key_reads/key_read_requests are very low (0.0002). So right now we seem to be doing great in regard to this value, but since MySQL recommends a high value I’m going to increase this to 24M or about 4.7% of system memory. Though this value is below the roughly 58M that the table indexes add up to, the maximum portion of the key buffer that has been used at any one time so far is only about 70% (based on a block size of 1M and Key_blocks_used showing 11K, meaning roughly 11M of 16M in the current setup).

Due to how MySQL uses the indexes, we should be O.K. with a key buffer size smaller than the sum of the indexes. As I understand it, MySQL divides indexes into blocks which allows MySQL to only access the part of an index it needs. Only used portions of a particular index need to be stored in the key buffer. Since some of the tables in the database are rarely (if ever) used their indexes won’t add to the overall key buffer usage.

Special note: some tables have rather large indexes. These tables should be reviewed to determine if any optimizations to the table structure can be made.

References:

• MySQL Reference Manual: 5.1.3. System Variables
• MySQL Reference Manual: 7.4.6. The MyISAM Key Cache
• MySQL Reference Manual: 7.5.2. Tuning Server Parameters
• MySQL Performance Blog: What to tune in MySQL Server after installation
• Database Journal: Optimizing the mysqld variables

Sort Buffer

The sort buffer is used whenever ORDER BY or GROUP BY is used in a query. If the amount of memory required to perform an operation of this type exceeds the value of sort_buffer_size MySQL has to sort the current working set, write the data out to disc, and start another working set. So not only do you have the problem of sorting parts of the result set separately, but then those groupings have to be combined on disc and sorted. Conceivably, then, a larger value would be beneficial. You can see how often a sort buffer has to be written out to disc and combined with additional buffered data by looking at the Sort_merge_passes status variable.

The sort buffer is a per-connection setting, meaning that each connection can allocate the amount specified by this value. As a result, care should be taken when setting this value in order to avoid eating up too much memory. Also, the MySQL Performance Blog has benchmarks showing a larger sort buffer actually hurting performance. With all this in mind it may be wise to use a relatively low value in the server settings and specify a larger value when necessary for a specific connection. Also, it may be advisable to perform some testing in various scenarios to see if there is an optimal minimum to cover most situations.

Still, MySQL recommends using a larger value to help improve sorting. Our current value was 512K. Our merge passes appear to be rising steadily (though not severely). So for now I’m doubling the value.

References:

• MySQL Reference Manual: 5.1.3. System Variables
• MySQL Reference Manual: 5.1.5. Status Variables
• MySQL Reference Manual: 7.2.11. ORDER BY Optimization
• MySQL Performance Blog: How fast can you sort data with MySQL?
• MySQL Performance Blog: Are larger buffers always better?
• Database Journal: Optimizing the mysqld variables

Read Buffers

MySQL uses read buffers when accessing table data. There are two settings to pay attention to here: read_buffer_size and read_rnd_buffer_size. MySQL allocates read_buffer_size when a sequential scan is performed. A sequential scan is when every row in a table is read and typically would be done when an index can’t be used to satisfy a query. MySQL allocates read_rnd_buffer_size when tables rows are read based on a key sort.

As with the sort buffer, the read buffers are a per-connection setting. So setting this value with consideration of our available memory is important. Plus, the MySQL Performance Blog has found some performance issues with larger values for these settings, similar to the issues with the sort buffer. Once again these are variables that may be best increased as needed. And again, testing should be done to find an optimal minimum.

Since these settings can show something of a benefit we want to increase the value if possible. Until further testing can be done on our system, however, I’ll keep these values below the theorized threshold of decreasing returns, 256K (at least until further testing and optimization can be done with our own system).

References:

• MySQL Reference Manual: 5.1.3. System Variables
• MySQL Reference Manual: 7.5.2. Tuning Server Parameters
• MySQL Performance Blog: MySQL: what read_buffer_size value is optimal?
• MySQL Performance Blog: Are larger buffers always better?
• MySQL Performance Blog: What exactly is read_rnd_buffer_size?
• MySQL Performance Blog: Read Buffers, mmap, malloc and MySQL Performance
• MySQL-HA: Read Buffer performance hit
• Database Journal: Optimizing the mysqld variables

InnoDB

The MySQL Performance Blog recommends a number of settings for enhancement of InnoDB performance. After much consideration, however, I believe it may be best if we forgo usage of the InnoDB storage engine where appropriate. I have a few reasons for this opinion

1. InnoDB tables are more resource intensive due to their use of transactions.
2. Fulltext indexing, a feature of MySQL that we have come to depend on for a number of applications, is not currently available for InnoDB tables.
3. Our current applications do not generally require the added safety of a transactional database.

If we do decide in the future to implement InnoDB for a future application (such as an online ordering system) we should revisit the optimization settings.

To ensure that tables are MyISAM by default I have set default-storage-engine = MyISAM. This can be overrode by specifying the table engine when creating a new table or by altering a table.

Fin

Once settings changes have been made to /etc/my.cnf restart the server and check the server variables (SHOW GLOBAL STATUS or use PHPMyAdmin) to ensure all settings have been implemented correctly.

Also, see my earlier post on optimization.

http://floradev.p2061.org/benchmarks/index.php?bmtabver=1993

To disable the switch change the value from 1993 to 2007 (or delete the “bmtabver” cookie from your browser).

This switch is not publicly noted.