Perishable Press

HTTP Headers for ZIP File Downloads

Wednesday, 17 November 2010 16:26

You know when you you’re working on a project and get stuck on something, so you scour the Web for solutions only to find that everyone else seems to be experiencing the exact same thing. Then, after many hours trying everything possible, you finally stumble onto something that seems to work. This time, the project was setting up a secure downloads area for Digging into WordPress. And when I finally discovered a solution, I told myself that it was definitely something I had to share here at Perishable Press.

Apparently, there is much to be desired when it comes to sending proper HTTP headers for file downloads. Different browsers (and not just IE) require different headers, and will error if not present exactly the way they expected. Confounding that equation is the fact that different file types also require specific headers. Then there are issues with sending an accurate (or should I say “acceptable”?) Content-Length headers when file compression is involved. Needless to say, finding a set of headers that works for all file types in all browsers is next to impossible. And I won’t even get into the issues involved with readfile() and large-download file-sizes.

Download Headers that actually work

After trying hundreds of different headers and combinations, I hit upon a set that works great for ZIP downloads (and other file types as well) in all tested browsers. Here’s what they look like using PHP:

<?php // HTTP Headers for ZIP File Downloads
// http://perishablepress.com/press/2010/11/17/http-headers-file-downloads/

// set example variables
$filename = "Inferno.zip";
$filepath = "/var/www/domain/httpdocs/download/path/";

// http headers for zip downloads
header("Pragma: public");
header("Expires: 0");
header("Cache-Control: must-revalidate, post-check=0, pre-check=0");
header("Cache-Control: public");
header("Content-Description: File Transfer");
header("Content-type: application/octet-stream");
header("Content-Disposition: attachment; filename=\"".$filename."\"");
header("Content-Transfer-Encoding: binary");
header("Content-Length: ".filesize($filepath.$filename));
ob_end_flush();
@readfile($filepath.$filename);
?>

This PHP script is known to work under the following conditions:

Operating System: Linux
Server: Apache/2.2.3 (CentOS)
MYSQL Version: 5.0.77-log
PHP Version: 5.2.6
PHP Safe Mode: Off
PHP Allow URL fopen: On
PHP Memory Limit: 256M
PHP Max Upload Size: 2M
PHP Max Post Size: 8M
PHP Max Script Execute Time: 30s

With this code, the downloads work in the following tested browsers:

Firefox 3.0, 3.5 (Mac & PC)
Opera 8, 9, 10 (Mac & PC)
Internet Explorer 7, 8
Chrome 7.0.517
Camino 2
Safari 5 (PC)
Safari 3 (Mac)

The downloads work for the following types of files (including small and large file types):

.zip
.txt
.pdf
.jpg

Obviously, I didn’t test every file type in every browser, but the positive results from those listed here suggest a much wider range of files and browsers that will work. For the file sizes, I tested small files only a few bytes in length, and also large files up to around 20MB or so. Also would like to give a shout to the Live HTTP Headers extension for Firefox. It proved indispensable throughout the troubleshooting/testing/pulling-my-hair-out process.

As always, if you can contribute to the content of this post with further information about sending proper HTTP Headers for file downloads, you may just save a life ;)

Source: Perishable Press

Take your WordPress skills to the next level with Digging into WordPress!

Latest Blacklist Entries

Tuesday, 09 November 2010 11:22

from-deny-the-user-agent-with

Recently cleared several megabytes of log files, detecting patterns, recording anomalies, and blacklisting gross offenders. Gonna break it down into three sections:

User Agents

User-agents come and go, and are easily spoofed, but it’s worth a few lines of htaccess to block the more persistent bots that repeatedly scan your site with malicious requests.

# Nov 2010 User Agents
SetEnvIfNoCase User-Agent "MaMa " keep_out
SetEnvIfNoCase User-Agent "choppy" keep_out
SetEnvIfNoCase User-Agent "heritrix" keep_out
SetEnvIfNoCase User-Agent "Purebot" keep_out
SetEnvIfNoCase User-Agent "PostRank" keep_out
SetEnvIfNoCase User-Agent "archive.org_bot" keep_out
SetEnvIfNoCase User-Agent "msnbot.htm)._" keep_out

<Limit GET POST PUT>
 Order Allow,Deny
 Allow from all
 Deny from env=keep_out
</Limit>

The first line blocks any user-agent containing “MaMa ”. If that scares you, then replace that line with these two:

SetEnvIfNoCase User-Agent "MaMa CyBer" keep_out
SetEnvIfNoCase User-Agent "MaMa Xirio" keep_out

The other lines block the latest batch of “loser-agents,” which may completely disappear overnight. My current strategy is to block for a few months and then start fresh. Stuff like heritrix, Purebot, and PostRank have made the list numerous times.

Character Strings

There must be some exciting new vulnerability, because suddenly I’m seeing TONS of requests for the following resources in just about every virtual directory imaginable:

fpw.php
xmlpc.php
pingserver.php
test00.comze.com

What’s the best way to deal with endless requests for non-existent resources? I prefer to respond with 403 Forbidden and call it done:

# Nov 2010 Char Strings
<IfModule mod_alias.c>
 RedirectMatch 403 fpw.php
 RedirectMatch 403 xmlpc.php
 RedirectMatch 403 pingserver.php
 RedirectMatch 403 test00.comze.com
</IfModule>

Of course, make sure you aren’t actually using any of these files anywhere on your site before using this code.

IP Addresses

Last but not least, here’s the latest batch of nefarious IP addresses. There’s no reason to block random botnet IPs, so only the most rogue static addresses make the list:

# Nov 2010 IPs
<Limit GET POST PUT>
 Order Allow,Deny
 Allow from all
 Deny from 65.55.3.211
 Deny from 72.229.57.27
 Deny from 77.93.2.81
 Deny from 77.221.130.18 
 Deny from 91.205.96.13
 Deny from 94.75.229.132
 Deny from 95.108.157.252
 Deny from 99.22.93.95
 Deny from 173.193.219.168
 Deny from 174.133.177.66
 Deny from 178.234.154.230
 Deny from 178.33.3.23
 Deny from 190.174.198.86
 Deny from 203.89.212.187
 Deny from 207.241.228.166
 Deny from 213.55.76.224
 Deny from 216.171.98.77
</Limit>

As with the user-agents, I like to block IPs for a month or so at a time. Implement (or not) as you see fit.

Bonus IPs! – Looking for more bad IPs to block? Check out Vladimir’s post in the comments.

Just one fix..

Don’t take my word for it. Check your own logs and see what shouldn’t be there. “Know thy enemy,” as they say ;)

For more help on blacklisting, check out Eight Ways to Blacklist with Apache’s mod_rewrite.

Source: Perishable Press

Take your WordPress skills to the next level with Digging into WordPress!

Importing WordPress Users via CSV Files

Thursday, 04 November 2010 17:15

the-and-data-with-from

I recently did some time in Microsoft Excel, preparing large CSV files for import into WordPress. Each of these CSV files contained data for 1000 WordPress users. Here is a screenshot showing the structure of the file:

[ Screenshot: User Data in CSV Format ]

Conceptually, the idea is simple: import the data to create actual users for a WordPress-powered site. The trick is to clean the data as much as possible to ensure valid username and password information. Once the data is good, importing is easy using a plugin.

Here is a step-by-step tutorial that combines WordPress and Excel techniques to register users en masse from a CSV file.

Please remember to backup your database and CSV file(s) before making any changes.

Step 1: Prepare the Data

This project begins with a CSV file containing only first & last names and email addresses. To create the usernames, we combine the first two cells and remove any whitespace:

=SUBSTITUTE(A2&B2," ","")

Place that in cell C2 and then drag the plus sign down the column to apply the formula and create usernames for all users. Then convert the formula data into actual values:

Copy the entire username column
Go to Edit > Paste Special
Then Paste as Values only

Now we have everything but the passwords. One way to create a column of random passwords in Excel is by creating a user-defined custom function:

Public Function RndPass(Length As Integer, Optional Lower As Boolean) As String

Dim Max As Integer
Dim Min As Integer

Dim RndPassLoop As String

Max = 126
Min = 48

Randomize Timer

If Length < 8 Then
Length = 8
End If

For i = 1 To Length
RndPassLoop = RndPassLoop & Chr(Int((Max - Min + 1) * Rnd + Min))
Next i

If Lower = False Then
RndPass = RndPassLoop
Else
RndPass = StrConv(RndPassLoop, vbLowerCase)
End If

End Function

To use this function, follow these steps:

Switch to VBA mode by pressing Alt+F11
Insert a new module by going to Insert > Module
Paste in the RndPass function
Save the new module by pressing Ctrl+S
Quit VBA mode by pressing Alt+Q

Then apply the function by pressing Shift+F3 and selecting the RndPass from the “User Defined” dropdown menu. The syntax is RndPass(password-length, lowercase). For example, we would use this to generate 16-character passwords that include both upper- and lower-case characters:

RndPass(16,false)

After generating the passwords, replace the formula data with actual values as we did with the usernames. Four columns of clean data is the goal here, so next step is to clean it up and prepare for import into WordPress.

Step 2: Cleaning the Data

At this point, the CSV file contains all of the required data:

[ Screenshot: User Data in CSV Format ]

Now we want to make sure everything is squeaky clean:

Remove all special characters from firstname, lastname, and username
Remove all invalid characters from the auto-generated passwords
Quick check for email validity

Cleaning up the data is really the key to a successful import. If present, certain characters will cause errors, resulting in unusable user data and a big waste of time. By taking the time to prepare good data, we ensure that the plugin can do its job and that all imported user accounts will be valid and ready to use.

Remove all special characters from firstname, lastname, and username

For firstname, lastname, and username data, we want to remove anything that isn’t upper/lowercase alphanumeric. We could do this manually using Find & Replace for stuff like:

| % . + ; , - ! _ ' ? * ~
Characters such as é and ó
Whitespace

Another option is filter WordPress’ sanitize_user function to allow such characters, but why modify WordPress when you can just clean the data instead? Here is a simple user-function for removing all non-alphanumeric characters from any cell data:

Function GETALPHANUMERIC(text)

str_all = "abcdefghijklmnopqrstuvwxyz1234567890"
For lenstr = 1 To Len(text)
    If InStr(str_all, LCase(Mid(text, lenstr, 1))) Then
        GETALPHANUMERIC = GETALPHANUMERIC & Mid(text, lenstr, 1)
    End If
Next

End Function

Just insert into your VBA module from Step 1 and apply the function to the entire firstname, lastname, and username columns. Viola! Squeaky clean user names! But the function isn’t perfect, and may return a zero (0) if a calculation fails. So just to be safe, Find any zero values and replace them with valid data.

Remove all invalid characters from the auto-generated passwords

From the intel I gathered, WordPress passwords may contain the following characters:

Any lowercase/uppercase alphanumeric characters
These symbols: . ? / ' " ; : | ] } [ { = + - _ ) ( * & ^ % $ # @ ! ~
Whitespace

So more flexible than username data, but we still need to remove the following characters:

The backslash: \
Left-angled brackets: <
Right-angled brackets: >
The dreaded comma: ,

To remove these items from the passwords, we could modify the GETALPHANUMERIC function from above, or just perform the following Find-&-Replace steps:

Replace #NAME? with a valid, random password
Replace RndPass with a valid, random password
Replace = with nothing
Replace @ with any valid character (or nothing)
Replace \ with any valid character (or nothing)
Replace < with any valid character (or nothing)
Replace > with any valid character (or nothing)
Replace , with any valid character (or nothing)

The first two replacements fix any errors from the RndPass function from Step 1. The next two replacements are needed to prevent Excel from treating the passwords as formulas. And the final four steps remove invalid characters for WordPress.

Tip: Replace any formula data with actual values by copying the column and then Pasting Special as Values.

Once the names and passwords have been prepped and cleaned, spend a few moments to examine your CSV data and make sure everything looks good. If so, your file should be ready for import into the WordPress database.

Step 3: Importing the Data

Now for the fun stuff! To import our freshly pimped CSV data into WordPress, we use the CSV User Import plugin to do the job. From the Plugin Directory:

[CSV User Import] allows you to import a list of users taken from an uploaded CSV file. It will add users with basic information, including firstname, lastname, username, password and email address. Each user who is added will be a ‘subscriber’ by default, and be able to login to your site.

Exactly what we want. And there’s only one screen in the Admin, so everything is self-explanatory:

[ Screenshot: CSV User Import Plugin ]

Find this screen under Users > Import. Before you select your CSV file and import your new users, a few things to keep in mind:

WordPress stores user data in two tables, users and usermeta – backup these two tables individually before importing any data
The imported CSV data will populate only the users table
Depending on your server settings, you may need to break up the CSV file into smaller files before import
If you’ve changed the default table prefix, you’ll need to edit a few lines in the plugin

Once everything is all set, import your data and check out your new users in the Admin. Make sure they have the correct permissions, and scan the columns for any missing usernames or email info. If you see any errors, you can either fix them manually or restore your backup and try again.

Again, this is the technique I used recently to import and register several thousand users into WordPress. I’ll try to help with questions, but Excel is certainly not my specialty ;)

Source: Perishable Press

Take your WordPress skills to the next level with Digging into WordPress!

Target iPhone and iPad with CSS3 Media Queries

Wednesday, 20 October 2010 14:59

the-and-ipad-margin-10px

When designing a website, it’s always a good idea to test on as many different platforms, devices, and browsers as possible. These days, pimping your websites for the iPhone and iPad is an important step in the design process. Especially on the iPad, sites tend to look about 20% cooler than on desktop browsers, so you definitely want to take the time to fine-tune the details. And when dealing with iDevices, it’s often necessary to deliver some custom CSS to make everything just right.

Want to apply CSS styles to the iPad and iPhone? Here is the plug-n-play, copy-&-paste code that actually works.

As you may have heard, I’ve been super-busy behind the scenes building an Angry-Birds fan site, of all things. The site is looking great so far, but needed some tweaking to appear slick on the iPad and iPhone. After testing a number of different solutions, here is what I found that actually works..

Target iPad & iPhone with separate CSS files

If you want to apply styles to the iPad and iPhone using external stylesheets, put this code in your <head>:

<!--[if !IE]>-->
<link type="text/css" rel="stylesheet" media="only screen and (max-device-width: 480px)" href="http://example.com/iPhone.css" />
<link type="text/css" rel="stylesheet" media="only screen and (min-device-width: 768px) and (max-device-width: 1024px)" href="http://example.com/iPad.css" />
<!--<![endif]-->

For this to work, you need to edit the path to each of the stylesheets. Then add some styles exclusively for the iPhone (first <link>) or the iPad (second <link>).

How does it work? Well, we first exclude IE by wrapping the links in a simple conditional comment. Then we’re using some CSS3 media queries to target each device specifically. Safari on the iPhone responds to a max-device-width of 480px, and Safari on the iPad seems to respond best when both min-device-width and max-device-width are used in the query. Together, these media queries apply styles in either portrait or landscape orientation.

Target iPad & iPhone from within your existing stylesheet

Using external stylesheets to target iStuff works great, but the extra HTTP requests can reduce performance. To avoid this, we can apply iPad/iPhone-specific CSS using our existing stylesheet. You know, the same one that we’re using for Firefox, Opera, and other desktop browsers. After much experimentation and testing, here is what I found that works:

/* iPad [portrait + landscape] */
@media only screen and (min-device-width: 768px) and (max-device-width: 1024px) {
	.selector-01 { margin: 10px; }
	.selector-02 { margin: 10px; }
	.selector-03 { margin: 10px; }
}

/* iPhone [portrait + landscape] */
@media only screen and (max-device-width: 480px) {
	.selector-01 { margin: 10px; }
	.selector-02 { margin: 10px; }
	.selector-03 { margin: 10px; }
}

Just place that code into your stylesheet and change the declaration blocks to something more useful. Unfortunately, combining these two media queries won’t work:

/* == iPad/iPhone [portrait + landscape] == */
@media only screen and (min-device-width: 768px) and (max-device-width: 1024px), 
@media only screen and (max-device-width: 480px) {
	.selector-01 { margin: 10px; }
	.selector-02 { margin: 10px; }
	.selector-03 { margin: 10px; }
}

As always, comments, questions, and suggestions are welcome! :)

Source: Perishable Press

Take your WordPress skills to the next level with Digging into WordPress!

Print Version of Digging into WordPress 3.0

Friday, 01 October 2010 16:30

the-and-wordpress-into-with

Just a note to let everyone know about the new printed editions of Digging into WordPress 3.0. This is the latest version of DiW, featuring revamped core content, new graphics, popouts and asides, as well as a new chapter devoted entirely to WordPress 3.0. It’s certainly packed with WordPress goodness, with nearly 450 pages of practical, hands-on tips, tricks, and information. The PDF is awesome, but seeing it all printed up in full color on quality paper and heavy, glossy card-stock cover is enough to make you drool:

[ Digging into WordPress 3.0 ]

The book looks and feels good – the durable coil binding lets the book stay open flat for easy reading, and the quality of the printing is excellent, with crisp text and vivid colors throughout the book. And the new black cover looks amazing on the glossy 80-lb card stock. Even with so many pages of WordPress content, the book feels solid and durable. The copy in the photos above was used for proof-reading and has been flipped through by several people, but as you can see it still looks great. Here are some more shots of the book:

[ Digging into WordPress 3.0 ]

So here’s the deal: as with previous versions, we have a limited number of books to sell, and they are going quickly. So if you want one, now is the time. The books are priced at $70 + shipping/handling, which is around $10 within the US. This includes delivery confirmation.

YES we are shipping internationally! At least for now. Last time there were too many problems so we had to disable it. For now though, international shipping is available, but the shipping is quite a bit more expensive. International shipping includes tracking and possibly delivery confirmation (depending on location and availability).

What else? The printed book includes a FREE copy of the PDF, which includes our exclusive WordPress themes and free lifetime updates.

Sound good? Get your copy here!

Source: Perishable Press

Take your WordPress skills to the next level with Digging into WordPress!

Perishable Press

HTTP Headers for ZIP File Downloads

Download Headers that actually work

Related articles

Latest Blacklist Entries

User Agents

Character Strings

IP Addresses

Just one fix..

Related articles

Importing WordPress Users via CSV Files

Step 1: Prepare the Data

Step 2: Cleaning the Data

Remove all special characters from firstname, lastname, and username

Remove all invalid characters from the auto-generated passwords

Step 3: Importing the Data

Related articles

Target iPhone and iPad with CSS3 Media Queries

Target iPad & iPhone with separate CSS files

Target iPad & iPhone from within your existing stylesheet

Related articles

Print Version of Digging into WordPress 3.0

Related articles

More Articles...