Are you running an Intel Linux system with at least one (E)IDE hard drive?

Wouldn’t it be neat if there were a magical command to instantly double the I/O performance of your disks? Or, in some cases, show 6 to 10 times your existing throughput?

Did you ever just wonder how to tell what kind of performance you’re getting on your “tricked-out” Linux box?

Don’t overlook hdparm(8). If you’ve never heard of it, don’t worry. Most people I’ve talked to haven’t either. But if you’re running an IDE/Linux system (as many folks are,) you’ll wonder how you ever got this far without it. I know I did.

What’s the big deal?

So, you’ve got your brand-new UltraATA/66 EIDE drive with a screaming brand-new controller chipset that supports multiple PIO modes and DMA and the leather seat option and extra chrome… But is your system actually taking advantage of these snazzy features? The hdparm(8) command will not only tell you how your drives are performing, but will let you tweak them out to your heart’s content.

Now before you get too excited, it is worth pointing out that under some circumstances, these commands CAN CAUSE UNEXPECTED DATA CORRUPTION! Use them at your own risk! At the very least, back up your box and bring it down to single-user mode before proceeding.

With the usual disclaimer out of the way, I’d like to point out that if you are using current hardware (i.e. your drive AND controller AND motherboard were manufactured in the last two or three years), you are at considerably lower risk. I’ve used these commands on several boxes with various hardware configurations, and the worst I’ve seen happen is the occasional hang, with no data problems on reboot. And no matter how much you might whine at me and the world in general for your personal misfortune, we all know who is ultimately responsible for the well-being of YOUR box: YOU ARE. Caveat Fair Reader.

Now, then. If I haven’t scared you away yet, try this (as root, preferably in single-user mode):

hdparm -Tt /dev/hda

You’ll see something like:

/dev/hda:
 Timing buffer-cache reads:   128 MB in  1.34 seconds =95.52 MB/sec
 Timing buffered disk reads:  64 MB in 17.86 seconds = 3.58 MB/sec

What does this tell us? The -T means to test the cache system (i.e., the memory, CPU, and buffer cache). The -t means to report stats on the disk in question, reading data not in the cache. The two together, run a couple of times in a row in single-user mode, will give you an idea of the performance of your disk I/O system. (These are actual numbers from a PII/350 / 128M Ram / newish EIDE HD; your numbers will vary.)

But even with varying numbers, 3.58 MB/sec is PATHETIC for the above hardware. I thought the ad for the HD said something about 66MB per second!!?!? What gives?

Well, let’s find out more about how Linux is addressing your drive:

hdparm /dev/hda

/dev/hda:
 multcount    =  0 (off)
 I/O support  =  0 (default 16-bit)
 unmaskirq    =  0 (off)
 using_dma    =  0 (off)
 keepsettings =  0 (off)
 nowerr       =  0 (off)
 readonly     =  0 (off)
 readahead    =  8 (on)
 geometry     = 1870/255/63, sectors = 30043440, start = 0

These are the defaults. Nice, safe, but not necessarily optimal. What’s all this about 16-bit mode? I thought that went out with the 386! And why are most of the other options turned off?

Well, it’s generally considered a good idea for any self-respecting distribution to install itself in the kewlest, slickest, but SAFEST way it possibly can. The above settings are virtually guaranteed to work on any hardware you might throw at it. But since we know we’re throwing something more than a dusty, 8-year-old, 16-bit multi-IO card at it, let’s talk about the interesting options:

• multcount: Short for multiple sector count. This controls how many sectors are fetched from the disk in a single I/O interrupt. Almost all modern IDE drives support this. The man page claims:
  
  
  

  When this feature is enabled, it typically reduces operating system overhead for disk I/O by 30-50%. On many systems, it also provides increased data throughput of anywhere from 5% to 50%.

  
  
• I/O support: This is a big one. This flag controls how data is passed from the PCI bus to the controller. Almost all modern controller chipsets support mode 3, or 32-bit mode w/sync. Some even support 32-bit async. Turning this on will almost certainly double your throughput (see below.)
  
  
  
• unmaskirq: Turning this on will allow Linux to unmask other interrupts while processing a disk interrupt. What does that mean? It lets Linux attend to other interrupt-related tasks (i.e., network traffic) while waiting for your disk to return with the data it asked for. It should improve overall system response time, but be warned: Not all hardware configurations will be able to handle it. See the manpage.
  
  
  
• using_dma: DMA can be a tricky business. If you can get your controller and drive using a DMA mode, do it. But I have seen more than one machine hang while playing with this option. Again, see the manpage (and the example on the next page)!

Turbocharged

So, since we have our system in single-user mode like a good little admin, let’s try out some turbo settings:

hdparm -c3 -m16 /dev/hda

/dev/hda:
 setting 32-bit I/O support flag to 3
 setting multcount to 16
 multcount    =  16 (on)
 I/O support  =  3 (32-bit w/sync)

Great! 32-bit sounds nice. And some multi-reads might work. Let’s re-run the benchmark:

hdparm -tT /dev/hda


/dev/hda:
 Timing buffer-cache reads:   128 MB in  1.41 seconds =90.78 MB/sec
 Timing buffered disk reads:  64 MB in  9.84 seconds = 6.50 MB/sec

WOW! Almost double the disk throughput without really trying! Incredible.

But wait, there’s more: We’re still not unmasking interrupts, using DMA, or even a using decent PIO mode! Of course, enabling these gets riskier. (Why is it always a trade-off between freedom and security?) The man page mentions trying Multiword DMA mode2, so:

hdparm -X34 -d1 -u1 /dev/hda

…Unfortunately this seems to be unsupported on this particular box (it hung like an NT box running a Java app.) So, after rebooting it (again in single-user mode), I went with this:

hdparm -X66 -d1 -u1 -m16 -c3 /dev/hda

/dev/hda:
 setting 32-bit I/O support flag to 3
 setting multcount to 16
 setting unmaskirq to 1 (on)
 setting using_dma to 1 (on)
 setting xfermode to 66 (UltraDMA mode2)
 multcount    = 16 (on)
 I/O support  =  3 (32-bit w/sync)
 unmaskirq    =  1 (on)
 using_dma    =  1 (on)

And then checked:

hdparm -tT /dev/hda

/dev/hda:
 Timing buffer-cache reads:   128 MB in  1.43 seconds =89.51 MB/sec
 Timing buffered disk reads:  64 MB in  3.18 seconds =20.13 MB/sec

20.13 MB/sec. A far cry from the miniscule 3.58 we started with…

By the way, notice how we specified the -m16 and -c3 switch again? That’s because it doesn’t remember your hdparm settings between reboots. Be sure to add the above line (not the test line with -tT flags!) to your /etc/rc.d/* scripts once you’re sure the system is stable (and preferably after your fsck runs; having an extensive fs check run with your controller in a flaky mode may be a good way to generate vast quantities of entropy, but it’s no way to administer a system. At least not with a straight face…)

Now, after running the benchmark a few more times, reboot in multi-user mode and fire up X. Load Netscape. And try not to fall out of your chair.

In conclusion

This is one of those interesting little tidbits that escapes many “seasoned” Linux veterans, especially since one never sees any indication that the system isn’t using the most optimal settings. (Gee, all my kernel messages have looked fine….) And using hdparm isn’t completely without risk, but is well worth investigating.

And it doesn’t stop at performance: hdparm lets you adjust various power saving modes as well. See the hdparm(8) for the final word.

Many thanks to Mark Lord for putting together this nifty utility. If your particular distribution doesn’t include hdparm (usually in /sbin or /usr/sbin), get it from the source at http://metalab.unc.edu/pub/Linux/system/hardware/

Happy hacking!

The Preferences API was first covered here shortly after it was introduced with the 1.4 version of the standard platform: the July 15, 2003 article, the Preferences API.

That article described how to get and set user specific preferences. There is more to the Preferences API than just getting and setting user specific settings. There are system preferences, import and export preferences, and event notifications associated with preferences. There is even a way to provide your own custom location for storage of preferences. The first three options mentioned will be described here. Creating a custom preferences factory will be left to a later tip.

System Preferences

The Preferences API provides for two separate sets of preferences. The first set is for the individual user, allows multiple users on the same machine to have different settings defined. These are called user preferences. Each user who shares the same machine can have his or her own unique set of values associated with a group of preferences. Something like this could be like a user password or starting directory. You don’t want every person on the same machine to have the same password and home directory. Well, I would hope you don’t want that.

The other form of preferences is the system type. All users of a machine share the same set of system preferences. For instance, the location of an installed printer would typically be a system preference. You wouldn’t necessarily have a different set of printers installed for different users. Everyone running on one machine would know about all printers known by that machine.

Another example of a system preference would be the high score of a game. There should only be one overall high score. That’s what a system preference would be used for. In the previous tip you saw how userNodeForPackge() — and subsequently userRoot() — was used to acquire the user’s preference node, the following example shows how to get the appropriate part of the system preferences tree with systemNodeForPackage() — or systemRoot() for the root. Other than the method call to get the right preference node, the API usage is identical.

The example is a simple game, using the game term loosely here. It picks a random number from 0 to 99. If the number is higher than the previously saved number, it updates the “high score.” The example also shows the current high score. The Preferences API usage is rather simple. The example just gets the saved value with getSavedHighScore() , providing a default of -1 if no high score had been saved yet, and updateHighScore(int value) to store the new high score. The HIGH_SCORE key is a constant shared by the new Preferences API accesses.

And, here’s what the screen looks like after a few runs. The 61 score is not apt to be your high score, but it certainly could be.

You can try running the application as different users to see that they all share the same high score.

Import and Export

In the event that you wish to transfer preferences from one user to another or from one system to another, you can export the preferences from that one user/system, and then import them to the other side. When preferences are exported, they are exported into an XML formatted document whose DTD is specified by http://java.sun.com/dtd/preferences.dtd , though you don’t really need to know that. You can export either a whole subtree with the exportSubtree() method or just a single node with the exportNode() method. Both methods accept an OutputStream argument to specify where to store things. The XML document will be UTF-8 character encoded. Importing of the data then happens via the importPreferences() method, which takes an InputStream argument. From an API perspective, there is no difference in importing a system node/tree or a user node.

Adding a few lines of code to the previous example will export the newly updated high score to the file high.xml. Much of the added code is responsible for launching a new thread to save the file and for handling exceptions. There are only three lines to export the single node:

Notice the root element has a type attribute that says “system “. This states the type of node it is. The node also has a name attribute valued at “<unnamed> “. Since the High class was not placed in a package, you get to work in the unnamed system node area. The entry attribute provide the current high score value, 95 in the example here, though your value could differ.

While we won’t include any import code in the example here, the way to import is just a static method call on Preferences, passing in the appropriate input stream:

Since the XML file includes information about whether the preferences are system or user type, the import call doesn’t have to explicitly include this bit of information. Besides the typical IOExceptions that can happen, the import call will throw an InvalidPreferencesFormatException if the file format is invalid. Exporting can also throw a BackingStoreException if the data to export can’t be read correctly from the backing store.

Event Notifications

The original version of the High game updated the high score preference, then explicitly made a call to update the label on the screen. A better way to perform this action would be to add a listener to the preferences node, then a value change can automatically trigger the label to update its value. That way, if the high score is ever updated from multiple places, you won’t need to remember to add code to update the label after saving the updated value.

The two lines:

There is a PreferenceChangeListener and its associated PreferenceChangeEvent for just such a task. The listener will be notified for all changes to the associated node, so you need to check for which key-value pair was modified, as shown here.

The PreferenceChangeEvent has three important properties: the key, new new value, and the node itself. The new value doesn’t have all the convenience methods of Preferences though. For example, you can’t retrieve the value as an int. Instead you must manually convert the value yourself. Here’s what the modified High class looks like:

In addition to the PreferenceChangeListener/Event class pair, there is a NodeChangeListener and NodeChangeEvent combo for notification of preference changes. However, these are for notification nodes additions and removals, not changing values of specific nodes. Of course, if you are writing something like a Preferences viewer, clearly you’d want to know if/when nodes appear and disappear so these classes may be of interest, too.

The whole Preferences API can be quite handy to store data beyond the life of your application without having to rely on a database system. For more information on the API, see the article Sir, What is Your Preference?

The enhanced for-loop is a popular feature introduced with the Java SE platform in version 5.0. Its simple structure allows one to simplify code by presenting for-loops that visit each element of an array/collection without explicitly expressing how one goes from element to element.

Because the old style of coding didn’t become invalid with the new for-loop syntax, you don’t have to use an enhanced for-loop when visiting each element of an array/collection. However, with the new style, one’s code would typically change from something like the following:

for (int i=0; i<array.length; i++) {
    System.out.println(“Element: ” + array[i]);
}

to the newer form:

for (String element : array) {
    System.out.println(“Element: ” + element);
}

Assuming “array” is defined to be an array of String objects, each element is assigned to the element variable as it loops through the array. These basics of the enhanced for-loop were covered in an earlier Tech Tip: The Enhanced For Loop, from May 5, 2005.

If you have a class called Colony which contains a group of Penguin objects, without doing anything extra to get the enhanced for-loop to work, one way you would loop through each penguin element would be to return an Iterator and iterate through the colony. Unfortunately, the enhanced for-loop does not work with Iterator , so the following won’t even compile:

You cannot just pass an Iterator into the enhanced for-loop. The 2nd line of the following will generate a compilation error:

In order to be able to use your class with an enhanced for-loop, it does need an Iterator , but that Iterator must be provided via the Iterable interface:

Actually, to be more correct, you can use a generic T , allowing the enhanced for-loop to avoid casting, returning the designated generic type, instead of just a plain old Object .

It is this Iterable object which is then provided to the enhanced for-loop. By making the Colony class implement Iterable , and having its new iterator() method return the Iterator that getPenguins() provides, you’ll be able to loop through the penguins in the colony via an enhanced for-loop.

By adding the proper implements clause:

Keep in mind that the individual penguins are internally kept in a Set type collection so the returned order doesn’t necessarily match the insertion order, which in this case it doesn’t.

Remember to genericize the implements clause for the class “implements Iterable<T> ” and not just say “implements Iterable “. With the latter, the enhanced for-loop will only return an Object for each element.

For more information on the enhanced for-loop, please see the Java Programming Language guide from JDK 1.5.

Iterating over a collection is uglier than it needs to be. Consider the following method, which takes a collection of timer tasks and cancels them:

    void cancelAll(Collection<TimerTask> c) {
        for (Iterator<TimerTask> i = c.iterator(); i.hasNext(); )
            i.next().cancel();
    }

The iterator is just clutter. Furthermore, it is an opportunity for error. The iterator variable occurs three times in each loop: that is two chances to get it wrong. The for-each construct gets rid of the clutter and the opportunity for error. Here is how the example looks with the for-each construct:

    void cancelAll(Collection<TimerTask> c) {
        for (TimerTask t : c)
            t.cancel();
    }

When you see the colon (:) read it as “in.” The loop above reads as “for each TimerTask t in c.” As you can see, the for-each construct combines beautifully with generics. It preserves all of the type safety, while removing the remaining clutter. Because you don’t have to declare the iterator, you don’t have to provide a generic declaration for it. (The compiler does this for you behind your back, but you need not concern yourself with it.)

Here is a common mistake people make when they are trying to do nested iteration over two collections:

    List suits = …;
    List ranks = …;
    List sortedDeck = new ArrayList();

    // BROKEN – throws NoSuchElementException!
    for (Iterator i = suits.iterator(); i.hasNext(); )
        for (Iterator j = ranks.iterator(); j.hasNext(); )
            sortedDeck.add(new Card(i.next(), j.next()));

Can you spot the bug? Don’t feel bad if you can’t. Many expert programmers have made this mistake at one time or another. The problem is that the next method is being called too many times on the “outer” collection (suits). It is being called in the inner loop for both the outer and inner collections, which is wrong. In order to fix it, you have to add a variable in the scope of the outer loop to hold the suit:

    // Fixed, though a bit ugly
    for (Iterator i = suits.iterator(); i.hasNext(); ) {
        Suit suit = (Suit) i.next();
        for (Iterator j = ranks.iterator(); j.hasNext(); )
            sortedDeck.add(new Card(suit, j.next()));
    }

So what does all this have to do with the for-each construct? It is tailor-made for nested iteration! Feast your eyes:

    for (Suit suit : suits)
        for (Rank rank : ranks)
            sortedDeck.add(new Card(suit, rank));

The for-each construct is also applicable to arrays, where it hides the index variable rather than the iterator. The following method returns the sum of the values in an int array:

    // Returns the sum of the elements of a
    int sum(int[] a) {
        int result = 0;
        for (int i : a)
            result += i;
        return result;
    }

So when should you use the for-each loop? Any time you can. It really beautifies your code. Unfortunately, you cannot use it everywhere. Consider, for example, the expurgate method. The program needs access to the iterator in order to remove the current element. The for-each loop hides the iterator, so you cannot call remove. Therefore, the for-each loop is not usable for filtering. Similarly it is not usable for loops where you need to replace elements in a list or array as you traverse it. Finally, it is not usable for loops that must iterate over multiple collections in parallel. These shortcomings were known by the designers, who made a conscious decision to go with a clean, simple construct that would cover the great majority of cases.

Before enabling HTTP file upload on your server, one important thing that you must consider is security, as improper design and configuration will make your server vulnerable to attacks.

For example, the PHP file upload script and JSP file upload script that were covered earlier are not secure. One problem is that we have not checked what the user entered in the optional filename text box. This gives malicious users the chance to modify the server’s files (e.g. system files or password files). For example, if a malicious user enters a path such as “../password/password.dat” in the optional filename text box, our PHP and JSP script will save the uploaded file to the destination “/file_uploads/../password/password.dat”, which is actually the path “/password/password.dat”.

Here are a few security tips that may be useful to you. We will only provide some brief descriptions here. For more details, please refer to other sources.

• 
  

  Check all information provided by the client to ensure that it is safe. For example:

  
  
  
  
  • 
    

    The HTTP request received includes a MIME type that describes what the uploaded file contains. A malicious user can provide a wrong value to trick you to think that the uploaded file is of another type. Hence, you should not rely on the MIME type included in the HTTP request but should perform a check by your own at the server-side. For instance, the photo album example covered earlier does not perform any checks to ensure the uploaded files are really image files. To enhance security, we can include a check on the uploaded files using the PHP function getimagesize() at the server-side. If getimagesize() returns false, that means the uploaded file is not a valid image file and it should be rejected.

    
    
  • 
    

    The HTTP request received includes the uploaded file’s original file name at the client-side. A malicious user can provide an unsafe value to trick you to modify system or password files. This problem is similar to the one described in the second paragraph of this section, so we will not describe it once more.
    In addition, you should prepare for the situation that the file name contains special characters that are not allowed to appear in file names or non-English characters. Make sure your WAP/web application will not crash or be left in an erroneous state when such situations occur.

  
  
• 
  

  Set a file size limit so that the user cannot upload files that are too large or too small.

  
  
• 
  

  Do not run web servers or application servers with the administrator account. Create and configure an account that is specifically for their use. Limit the file access permissions of the account so that even if your WAP/web application has security holes, the OS will not allow it to work with system files or files of other users.

  
  
• 
  

  Make sure your WAP/web application does not reveal too much information to the user when an error occurs. The information revealed can help a malicious user find ways to attack your system.

  
  
• 
  

  Log down the details (such as the time, the client’s IP address and the user name) of file uploads and other related events. Although the logs only tell you what has happened, they can help you check what types of attacks have been made against your server and whether there were any successful attacks.

Below shows a JSP file upload script that is used to print out the name-value pair received from the earlier XHTML MP document and save the uploaded file to a certain location on the WAP server. Remember to change the action attribute of the <form> element in the earlier XHTML MP document from file_upload.php to file_upload.jsp for this example to work.

Below shows a PHP file upload script that is written to handle the form data submitted from the earlier XHTML MP document. The PHP script prints out the information obtained from the HTTP request and saves the uploaded file to the “/file_uploads” directory of the WAP server.