Let’s review one of those situations. Say we have the following (simplified) models:

class Item extends AppModel {
  var $hasMany = array('Answer');
  var $belongsTo = array('Packet');
}
 
class Student extends AppModel {
  var $hasMany = array('Answer');
}
 
class Answer extends AppModel {
  var $belongsTo = array('Student','Item','Packet');
}
 
class Packet extends AppModel {
  var $hasMany = array('Answer','Student');
  var $hasAndBelongsToMany = array('Item');
}

Let me explain the model relationships. We have test items that are assigned to packets. Packets are sent out for testing to a group of student. So each answer is unique based on the packet, item, and student. The (somewhat circular) relationships between these models provides for easy access to specific data. For example, let’s say we want to know all the answers from a particular state for a couple of our items. Here’s an example query that present some optimization issues:

$queryOpts = array(
  'fields' => array('Item.*');
  'conditions' => array('Item.id' => array(1995, 1726, 1971, 1707, 1972));
  'contain' => array(
    'Answer' => array(
      'fields' => array('Answer.*');
      'conditions' => array(
        'Answer.packet_id IN (SELECT id FROM packets WHERE state = "CA")'
      )
    )
  );
);
$items = $this->Item->find('all',$queryOpts);

You’ll note that I’m not using CakePHP syntax for the Answer model conditional. We had been using this conditional in our web application prior to moving to CakePHP. Since there is not, so far as I can tell, an easy way to work with SQL subqueries we decided to start by forcing a query similar to the original.

Running the above creates the following SQL statement when querying the answers table:

SELECT `Answer`.* FROM `answers` AS `Answer` WHERE `Answer`.`packet_id` IN (SELECT id FROM packets WHERE packet_type = "F") AND `Answer`.`item_id` IN (1995, 1726, 1971, 1707, 1972)

We have a fairly large data set so the above query takes ~32 seconds to run on my system. But that query is not optimized; reordering the conditions in the WHERE clause produces the following query:

SELECT `Answer`.* FROM `answers` AS `Answer` WHERE `Answer`.`item_id` IN (1995, 1726, 1971, 1707, 1972) AND `Answer`.`packet_id` IN (SELECT id FROM packets WHERE packet_type = "F")

The optimized query performs significantly better, <1 second.

Since for this example we know which items we want we can update the query to manually force the desired ordering:

$queryOpts = array(
  'fields' => array('Item.*');
  'conditions' => array('Item.id' => array(1995, 1726, 1971, 1707, 1972));
  'contain' => array(
    'Answer' => array(
      'fields' => array('Answer.*');
      'conditions' => array(
        'Item.id' => array(1995, 1726, 1971, 1707, 1972)
        'Answer.packet_id IN (SELECT id FROM packets WHERE state = "CA")'
      )
    )
  );
);
$items = $this->Item->find('all',$queryOpts);

But what if we don’t know what items we want. Say we’re pulling items (and answers) as part of another containable query. We have no way to specify the items in the Answer model conditionals. But we can improve the query so that we’re not using a subquery each time (i.e. we’re doing it the “CakePHP” way):

$fields = array('Packet.id');
$conditions = array('Packet.packet_type' => 'F');
$packet_list = $this->Item->Packet->find('list',array('fields'=>$fields,'conditions'=>$conditions));
 
$queryOpts = array(
  'fields' => array('Item.*');
  'conditions' => array('Item.id' => array(1995, 1726, 1971, 1707, 1972));
  'contain' => array(
    'Answer' => array(
      'fields' => array('Answer.*');
      'conditions' => array(
        'Answer.packet_id' => $packet_list
      )
    )
  );
);
$items = $this->Item->find('all',$queryOpts);

Which produces the following two queries:

SELECT `Packet`.`id` FROM `packets` AS `Packet` WHERE `Packet`.`packet_type` = 'F';
SELECT `Answer`.*, `Answer`.`item_id` FROM `answers` AS `Answer` WHERE `Answer`.`packet_id` IN (1278, 1277, 1276, 1274,...,1726, 1971, 1972, 1995);

Again we’re seeing the kind of performance we want, <1 second for both queries. As good as when we knew all our conditions in advance. The main drawback to doing it this way is the size of the query string sent to your database. If your first query is pulling a large number of results then your second query will be quite large. And most databases have limits on how large a query can be.

This isn’t the only way to get to the end point either. As with any programming task there are a variety of ways to arrive at the desired result. For example, you could break up the query into multiple queries that build off of the previous query. Sticking with our items/answers query you could first get the items in one query. Then foreach through the results and run a query to get the answers, integrating the results manually. It’s more work, and likely would decrease performance since you’re increasing the number of database queries, but if you can’t seem to optimize your all-in-one queries it’s worthwhile to try other options to see if you can get better performance.

Aside: performance problems due to poorly optimized queries can be somewhat mitigated by query caching. Even so, you’re better of with an optimized query to begin with since it saves you from a performance hit every time the relevant cached result is flushed.

One of the great things about CakePHP is that if it doesn’t have some core functionality you want/need there are easy ways to add it. More and more I’m taking advantage of this ability. This all comes about because I wanted to have calculated fields available inline with the model data. By calculated fields I mean results that are not data columns (e.g. SELECT *, CURDATE() AS current_date FROM users … yes, that’s a fairly contrived example).

By default Cake places results from calculated fields outside the model data, like this:

Array
(
    [0] => Array
        (
            [User] => Array
                (
                    [id] => 1
                    [username] => aaas
                )
 
            [0] => Array
                (
                    [current_date] => '2011-05-13'
                )
 
        )
)

What we want is to place the “current_date” calculated field inside the User model, so it’s more naturally accessed with $users[0]['User']['current_date'] instead of $users[0]['User'][0]['current_date']. Easy enough to do through a model’s afterFind() callback method (to make it widely available place the function in app_model.php).

function afterFind($results, $primary=false) {
	if($primary == true) {
		if(Set::check($results, '0.0') && Set::check($results, "0." . $this->alias)) {
			$fields = array_keys( $results[0][0] );
			foreach($results as $key=>$value) {
				foreach( $fields as $fieldName ) {
					$results[$key][$this->alias][$fieldName] = $value[0][$fieldName];
				}
				unset($results[$key][0]);
			}
		}
	}
	return $results;
}

And yet, even as I add such enhanced functionality to my web apps I’m finding limits. The above logic is complicated a little because you don’t know what type of results you’re getting. The results you get from calling, for example, find('all') versus find('count') are not the same. But Cake doesn’t give any hints to the afterFind() callback, and as a result additional logic is included to try and guess at our data structure. The above adds a quick but incomplete hack by a) checking that the results are for the primary model queried (e.g. not from contained model), b) checking for the presence of nested numerical keys, and c) checking that there is model data to integrate with.

The take away is that while the code produces the desired data structure, in its current form it does so only for specific results.

I recently noticed the following alert in the system log:

ALERT – configured POST variable limit exceeded – dropped variable ‘included_packet_ids[]‘ (attacker USER_IP_ADDRESS, file REPORT_FILTERING_PAGE)

One of the reasons I use POST variables on this page is to avoid the relatively small data size limit of GET. Suhosin adds additional limits, including in the number of times you can reference an individual variable.  Our limit was set at 1000, meaning there were over 1000 packets selected. This points to a need to adjust how the filter “selects all” … but for now I’ve adjusted the suhosin limit upward by modifying the suhosin.post.max_vars setting.

References:

• ALERT – configured POST variable limit exceeded ::Suhosin configuration blocks script | Tudor Barbu’s professional blog

ALERT – script tried to increase memory_limit to 268435456 bytes which is above the allowed value (attacker SERVER_IP_ADDRESS, file WP_MAIN_ADMIN_PAGE, line 96)

The source of the issue is WordPress. The application is trying to raise the memory limit and suhosin won’t let it. Apparently WordPress will try to set a 256MB memory limit before executing certain functions. The necessity of adjusting this setting seems questionable to me, but I also understand that it’s often better to play it safe when developing software for public consumption.

I don’t particularly like applications attempting to specify their own resource usage in a web environment. In my mind applications should specify a required/recommended memory limit in the system requirements and stay away from adjusting this setting behind-the-scenes. Tell me during setup if the current setting may result in non-optimal performance or even a halt in script execution. That’s not how it’s done here, but really no harm is done in the long run beyond the annoyance of suhosin throwing errors at the system logs.

There are two easy fixes to the problem:

1. Set the PHP memory limit to 256MB
2. Modify the suhosin.memory_limit parameter to 256MB

In our particular situation it’s just as easy to set the PHP memory limit. There’s always a risk of overloading the physical resources, but this site receives little enough traffic that I’m not concerned about the right confluence of request occurring to cause a crash.

References:

• Mysteries about the WordPress Memory Limit | hakre on wordpress
• Hardened-PHP Project – PHP Security – Configuration

From what I can tell, when CakePHP specifies the values for all columns when it creates a new record. A column with a defined default that is not specifically set by the user is manually set by CakePHP (rather than let MySQL handle defaults upon record insertion). But CakePHP doesn’t understand the CURRENT_TIMESTAMP keyword and so treats it as a string and wraps it in quotes. This breaks the resulting INSERT statement and you receive an error:

Incorrect datetime value: ‘CURRENT_TIMESTAMP’

Interestingly, columns that are named “created” and “modified” receive special handling by CakePHP. These columns are treated like auto-update columns by CakePHP and it sets them as expected. With the special handling of these columns in mind it is possible to get around the CURRENT_TIMESTAMP bug by following the recommended settings for created/modified columns, i.e. specifying the field as DATETIME with a default of NULL. CakePHP will automatically update the columns when inserting/updating records.

References:

• MySQL Reference Manual: 10.3.1.1. TIMESTAMP Properties
• CakePHP Cookbook: 3.7.2.3 created and modified

In my case, I was taking a form submission and redirecting it to a new page, appending the submitted data as URL parameters (e.g. of the form /controller/action/name:value). CakePHP handles ampersand in POST-submitted forms just fine, but during the redirect I noticed that the value of the parameter was cut off at the ampersand. For example, I’m creating a search form for a list of items (located at /items/index). When you submit the form with a value of “ball & chain” the controller sees it just fine. The form is submitted to the search action (/items/search) which takes the values, constructs a new URL, and redirects back to the item list (/items/index/Search.keywords:ball+%26+chain). However, the index action only sees “ball ” … the rest of the value is assumed to be additional key/value pairs in the querystring.

This problem seems like a pretty major issue to me. A bug report has already been filed, but I don’t believe it will be addressed anytime soon. And for good reason, the bug is actually in Apache’s mod_rewrite module rather than in CakePHP. When mod_rewrite processes a URL it unescapes any escaped characters (newer releases unescape to two levels). What this means for CakePHP is that any escaped characters in the URL become normal characters in the querystring (%26 becomes &).

The best overview of the problem has a good work-around, but it requires modification of the core CakePHP scripts. I’ll leave that to the experts. On our install I’ve found that a triple-escaped value will only be unescaped twice, resulting in the proper escaping after being parsed by mod_rewrite.

References

• The Apache mod_rewrite Problem
• mod_rewrite & ampersand
• Escape problem in mod_rewrite [P] action…
• Make mod_rewrite escaping optional / expose internal map
• Handle ampersands in search queries and other URLs when clean URLs are on
• Short URL/Ampersand solution

Data Collection Hassles

For the past few years the Assessment team has sent out a large batch of items in what we call pilot tests. These tests are used to gauge the effectiveness of assessment items. The format consists of an item followed by a set of questions about the item (Was there anything confusing?, Did the picture help?, Is answer A correct?, etc.). The questions have set answers (Yes, No, Not Sure) and/or a space for the student to provide a short written response.

In previous years we have used temporary workers to input the student responses into the Items Utility. This proved to be extremely slow and relatively expensive.

Last year we attempted a new method of data collection using an automated system to capture the Yes/No/Not Sure responses. This system used optical mark recognition (OMR) and required the packets to be formatted in a consistent manner on each page. Formatted in this way we could create a master file indicating how to judge answer selection. This proved fairly successful, though we did have to outsource the scanning to a third party. The results of the scan came back as Excel documents. A little PHP coding and I easily imported the scanned data into the Items Utility. The main drawback was that we still had to hire temps to enter the written responses.

This year we embarked on a new automated scanning system for the Yes/No/Not Sure responses that proved even better than in previous years. This system uses an in-house scanner and design software. A lot of prep work had to be performed by the researchers to set up their test packets. And we did have to use specific printers to produce the packets. But I believe in the end we saved money, particularly since the majority of the leg work had already been done in figuring out how to import the data into the utility. The only snag is that we again had no way of capturing the written responses electronically.

However, one of the benefits to the new system is that in addition to creating a data file for the Yes/No/Not Sure values it can produce images of the scanned regions. At first I thought maybe we could find some way of performing character recognition on the scanned images. Unfortunately, there’s just no economical way to do that on hand-written text. So our only real option was, again, transcription by a human being. Past experience with the time and money required made the option unpalatable and we were likely going to just move ahead without the hand-written data.

Enter the Turk

Amazon recently released a web-based service, Mechanical Turk (MTurk), that provides access to a distributed workforce. The service provides a means of creating what Amazon calls “Human Intelligence Tasks” (HITs), tasks that are currently too complex for a computer but ideally suited for your average person. We decided to try the service out for transcription of the written responses.

Since MTurk is web-based, the HITs are set up as web pages. An online editor makes setting up HIT templates fairly simple, though it helps to know a bit of HTML. Each template consists of instructions and a form where the worker enters the requested information. It’s up to the requester to determine the amount paid for each HIT completed (starting as low as one cent). Amazon tacks on an overhead fee for each HIT that is completed by a worker and accepted by the requester.

The template uses variables to fill in each HIT with unique information. Once the template is set up you provide a data file that contains the variable data. MTurk then publishes your HITs on a public interface where anyone in the world with an internet connection can perform the work.

Using MTurk

As I said, the program we used to scan the packets is able to save scanned regions as images. These images were organized such that we could easily determine the packet/student/item combination based solely on the file name and directory in which the image was located. This information is all we needed to insert data into the Items Utility, and thus provided an easy identifier to use with MTurk.

To minimize the amount of time a worker spends on each HIT it’s good to try and have all the data the user needs on the screen. Since a MTurk HIT is a web page it can reference images located on external servers. I used URLs for the image scan as values for the template variables. The HIT displayed the image side by side with a text area where the workers could input the transcribed text.

Initially I thought to just link to the original images. A quick test, however, and I realized that the size of the images would be a problem. The images were larger than needed and any bandwidth constraints on the worker end would significantly increase the time per HIT. Smaller files were needed, and to save space I thought to dynamically resize the images using PHP. A sample proved the folly of this idea. Our system resources are limited and the processing power and memory required to enable this for more than a few simultaneous workers caused the HITs to load slower than if we hadn’t resized to begin with. In the end we decided to resize all images, a scenario we had avoided due to space constraints. The smaller image would be referenced on each HIT with a link to the full size in cases where a zoomed view might be benefitial.

Since Amazon tacks on a handling fee for each HIT I attempted to organize our HITs in a way that would work out best for both us and the workers. We needed HITs that could be completed quickly but that also collected sufficient data so that the cost-per-HIT would balance out against the amount of work required. Plus we were concerned about providing equitable pay for the work being done. Based on our testing we were able to make a determination of how many images per HIT and at what rate would keep our fees acceptable while also resulting in an equitable wage.

Our due dilligence paid off. We had 48,351 HITs comprised of 193,401 images. The work was completed within a couple of days, which was beyond my wildest estimates. The amount paid and the time taken to complete the work was significantly less than what we would have paid to hire traditional temporary workesrs to do the same.

Once the work was complete MTurk provided a data file that included the original data along with the worker input, all the information we needed to import the data into the utility.

Wrapping Up

While this was a bit of a learning process, using MTurk to transcribe this data proved successful. I fully expect to use this system in the future. This has been more of a review of the process, but there were a few technical aspects involved. Rather than go into detail on these I’m providing links to the files involved which should provide sufficient information in that regard (except for perhaps the import script).

• PHP Script to Generate a List of Files
• PHP Script to Resize Images
• Data Set Sent to MTurk
• MTurk HIT Template
• Result Data Set Provided by MTurk
• Simplified Result Data Set
• PHP Script to Import Data

PHP is ideal for a task such as this: parsing directories and subdirectories for images is easy; resampling using the built-in library (GD) is a breeze; specifying the destination as a subdirectory is simple. The one minus was processor usage. Performing image manipulation eats up the CPU in a big way.

Luckily linux systems have a built-in utility for addressing a situation like this: nice. nice will “run a program with modified scheduling priority.” I’m running the image manipulation script using the following command:

nice --adjustment=19 php script.php

If nothing else is going on the script will use whatever resources are available. When anything with a higher priority executes, that program will take precedence over the script with regard to system resources. The script should thus not affect the responsiveness of the web server. This is the reason I was searching for this kind of functionality.

Installation was not too difficult, though I did have to ensure that a few dependencies were installed before PEAR would load the package. Documentation for the module is not that great, but there are a number of samples and a fairly active discussion forum. (There used to be a decent community, but the site appears to have gone under.)

I did not find the graphing object very intuitive. It seems to me that a lot of the properties and functions are abstracted from the objects they affect. I suspect this was a design decision to allow for greater flexibility, but it may also be due to an older coding style. The current branch appears to have been developed in earnest starting in 2005, and it’s been a few years since a new release. Even so, my initial exposure leads me to believe the package may be stable enough for even a production-level project so long as sufficient QA has been performed.

It took me only a couple of days to hack together the demonstration. Though the requirements of the script were few, I decided to push the limits of my object-oriented PHP and create the graphing component as a class. Overall it was a good first-try at producing something of this nature. You can find the demonstrator at http://flora.p2061.org/weather_data/.

Updating the interface

First up is the form organization. Previously the forms were designed, I believe, with the researchers in mind. The most important data entry sections were placed at the top of the page with everything else following. Once we started using temps to enter the data, however, this format proved to be a hindrance to efficiency. So to improve efficiency I regrouped the form into logical sections: student responses, student demographics, and researcher notes. Plus, to allow for easier scanning I organized the student responses section so that it resembles the paper form.

In addition to organizing the form more logically, I wanted to make portions of the form collapsible. Since a lot of the data entry is performed by temps there is no reason for them to have to wade through the research notes, which are at the top of the page by request of the researchers. Brian W. has commented on the utility of Dreamweaver’s Spry widgets so I thought I’d give them a try. I must admit that they work quite nicely and are easy to implement, and I found just what I needed in the Collapsible Panel widget. The benefit of using the panels is that I could position the researcher notes at the top of the page without significantly hindering the ability of the temps to access the student responses by having this panel closed by default.

Which brings me to a minor change I made to the functionality of the panels: having the appropriate panels open when the data entry form is loaded (e.g. student responses always open; researcher notes closed unless otherwise requested by the user; demographics only open if no data has been entered). I used a combination of PHP and JavaScript to set the initial state. Unfortunately this seems to caused some problems for Dreamweaver. When you open the packet data entry form in Dreamweaver the program appears to freeze. It appears Dreamweaver is attempting to process the spry widgets and choking on the PHP-derived JavaScript variables. Eventually Dreamweaver informs you that “A script in file … has been running for a long time. Do you want to continue?” Answering no to this will return Dreamweaver to normal operation. I could get around this problem by setting the state of the panels after they have been initialized, but this produces a disorienting collapse of panels after the page has loaded.

The final interface modification was a small script that makes it possible to deselect radio buttons. The main purpose of developing this feature was to streamline the form which, in previous revisions, had an extra option to clear the selected response.

Improving the foundation

As I mentioned, while the form is still hard-coded (each question coded by hand) I did lay out the foundation for a more dynamic approach. First I redeveloped the table that stores the data related to a packet. I reused the packets and packet_items tables, but created a new table called packet_data that stores only the question number, answer, and any comments. The benefit here is that forms with varying numbers of questions can be stored in this one table. I also moved the student meta data into a separate table, a design decision more in line with database normalization practice.

Another modification I made was to use a structured array to define the questions in the packet. While this has not been fully developed as of yet, I have used it to significantly simplify the script that saves the data. Rather than specify each question and corresponding HTML form element and writing the query to insert that information in the database I designed the PHP to loop through the array and generate the SQL code on the fly.

Finally, I used another structured array to hold data related to the packet, packet items, and students while generating the packet data entry page. I do not utilize any methods to automate the form generation at present, though there is little to prevent this from being developed. One reason I decided to code the form by hand is to allow for a level of customization that is difficult to accommodate when automating the visual side of things. Even so, utilizing structured arrays allows for more consistency between PHP pages.

Moving forward

Data from years prior has not yet been ported into the new structure. I do not expect this task to require too much work, but I am not sure when I’ll be able to perform the task. At the very least this will be part of the redevelopment taking place around subversion if not sooner.

To fully accommodate the data in the utility I’ll need to also update the summary tables, the school report generator, and packet data export. These updates will need to be completed fairly soon in order to support the item review that will begin taking place fairly soon.