Comments for Project 2061 Techlog

Comment on Items Utility: Piloting Data Entry Updates by BrianS

BrianS — Thu, 19 Jun 2008 19:03:58 +0000

A quick look at the database after a few days of data entry and I was beginning to think may have to reconsider the table structure. Data for 1,030 test items had been entered (packet+student+item). That results in 19,570 rows in the table weighing in at 61.2MB.

For comparison, the previous table structure contains 60,864 rows and weighs in at 13.8 MB. Since a row in the old table correlates with all the data for a test item it was significantly more efficient in terms of storage. The new table, in order to accommodate the same number of items and growing at the current rate, would take up 60 times as much space.

After some research I found that that the trouble has to do with the format of the row. The current format is static (fixed) length, but for variable-length data a dynamic row is better suited for space considerations. Most of our tables use the dynamic format. Updating this table to dynamic cuts the size of the table down by a factor of 100.

There are some drawbacks to dynamic rows, however. I’ll be adding some documentation regarding this setting to the upcoming post “Optimizing MySQL Table Structure and Queries.” In the meantime you can find out more information about this setting from the MySQL Reference Manual. See 13.1.3. MyISAM Table Storage Formats.

Comment on Optimizing MySQL Server Runtime Parameters by BrianS

BrianS — Tue, 27 May 2008 21:06:39 +0000

11-day report:
The settings I’ve implemented definitely seem to be helping with database performance.

My assumption of the max number of concurrent connections appears to be fairly close. So far we’ve hit a concurrent connection max of 16. I’m not sure how MySQL determines when to cache a thread, but our thread cache is only at 8. Probably related to thread age, but something that should be researched.

I’m going to lower the maximum connections to 50 (from the default of 100). We’re not coming anywhere near 100 and since the purpose of this setting is to prevent resource depletion it’s better to play safe than sorry. At some point in the future we should calculate the maximum per-connection resource usage and adjust this setting based on that information.

The query cache seems to be performing well with what appears to be a high hit/insert ratio (1,179K/31K). It has plenty of free memory after 11 days of usage and no low memory prunes. It does show some modest fragmentation as indicated by qcache_free_blocks having a value of 47.

The key buffer setting also appears to be acceptable. Our key_reads/key_read_requests is still significantly below the .01 ceiling recommended by MySQL.

For the number of temporary tables created (148K) we’re not looking too bad on the number that had to be written to disk (1,840).

The table cache appears to be doing well, though still not being used fully. Again something that might relate to pruning by MySQL based on age, but I’ll need to research this more as well.

The one number that is disconcerting in the stats is handler_read_rnd_next with a value of 538M. The documentation states that a high value here probably indicates that tables indexes are not optimal for the queries being run. This is also bourne out by the values of select_full_join and select_scan. We should spend some time analyzing our queries to determine where optimizations can be made.

Next update at around one month.

Comment on Optimizing MySQL by Project 2061 Techlog » Optimizing MySQL Server Runtime Parameters

Project 2061 Techlog » Optimizing MySQL Server Runtime Parameters — Wed, 07 May 2008 18:56:36 +0000

[...] Also, see my earlier post on optimization. [...]

Comment on BOL Text utility updates by bwalker

bwalker — Thu, 27 Mar 2008 15:34:55 +0000

After releasing yesterday’s update I realized the utility did not support content changes for Chapters 13-16 although this content is stored in the same table. I have remedied this by making all of the menus dynamic and asynchronously updated. In the process I have added 2 new tables bol_chapters and bol_sections whose purpose is self-explanatory. The band menu options are collected from the bol_text table based on selected chapter and section.

Comment on Windows Media Player Plugin for Mozilla by bwalker

bwalker — Fri, 21 Mar 2008 14:21:29 +0000

While it seemed this plug-in resolved the error mentioned above, the error reappeared yesterday. I tried reinstalling the plug-in, but it did not have any effect. Additional testing determined the error occurred whenever I tried to over-write the player object html. It’s possible whatever mechanism the player uses to close/unload the object has a bug or is conflicting with something. I still don’t know the root cause of the problem, but a work-around is to clear the contents of the parent container using innerHTML or equivalent loading any new scripts.

Comment on Working with OAI by BrianS

BrianS — Wed, 05 Mar 2008 20:29:00 +0000

The OAI code has been updated to correct some problems with the previous version.

Comment on Updates to field test registrant_list_demo by BrianS

BrianS — Tue, 04 Mar 2008 17:44:40 +0000

The NCES data was a one-time download last year. The data in the demographics database is the entire set. Though registrations are related to the demographic information by ZIP code, the NCES data does not indicate which ZIP codes are contained in each school district. The only ZIP codes available to search on relate to the address of the main office for the district.

In order to relate a ZIP code to a school district I used a kludge. When doing the matching I search for two possibilities. Either an exact match to the ZIP code listed, or a partial match to the first three digits of the ZIP code when the city also matches.

This does not always work, as you have noted. For the registrations that do not have a match I went online to the NCES school district search and tried to find which school district served a particular ZIP code. I would then add this data to the nces_school_zips table.

The main problem here is that the matches aren’t always perfect (though they should be adequate for our purposes). Plus, it requires a lot of manual intervention to match ZIP codes manually. Unfortunately, it was difficult finding demographic information that we could use last year. It may be worthwhile seeing if additional data sets are available this year.

Comment on image upload problem (New Items) by BrianS

BrianS — Tue, 04 Mar 2008 16:59:53 +0000

I did make a change to the way FCKE is set up. I decided to move the configuration options out of the default fckconfig.js file and into an external one, which can now be found in items/includes/fckeditor. In addition to the config file there is a template definitions file and editor content stylesheet.

It’s possible I missed some configuration options when I set up the new configuration file. I’ll have to check out the setup and see what’s going on.

Comment on floradev not accepting connections by bwalker

bwalker — Mon, 03 Mar 2008 18:57:56 +0000

Connecting via Citrix over the weekend was not a problem, and this morning the server seemed to be connecting. However, since then I’ve been interacting with the server regularly and the server has dropped my connection and stopped accepting connections 3 times in the span of a couple hours!!

I think this rules out a connection to the idle/power-save features.
In addition whatever is happening seems to be getting worse day by day.
No atypical messages were found in the messages log for these failures, which leads me to believe this is some sort of network failure (vs. a server failure).

Dave mentioned a possible IP conflict, but I’m not really sure how, or if, I can test for such a thing.