Why It Happens
Drupal's setup includes a .htaccess that has the following effects:
Drupal sets up index.php as the only file that the Apache web server looks for in a folder should a filename not be specified by the visitor.
For example, if your visitor types "http://www.example.com/non-drupal-folder/" in the browser, the web server will look for a file named index.php in the "non-drupal-folder" directory. If it is not found, it will stop looking and load the 404 error document.
Drupal sets up the 404 error document to point to its own script. So if at any time the server cannot find a document, it will send the request for the file to Drupal. Drupal will then look into its database to see if the URL is one that corresponds to a post that you have made. If not, it will issue its 404 File Not Found error.
Another side effect is if you have set up a password-protected directory using .htaccess but have not set up the 401 error handler to point to a valid file. When you try to access the directory, the web server issues a 401 error. This error is normally handled by the web browser and causes it to issue a request for the password. However, if the 401 error handler does not point to a valid file, it will return a 404 File Not Found error instead. And as we have seen above, 404 errors are intercepted by Drupal, which means that you will never get the browser prompt for a password. Instead, you get a mysterious "File Not Found" error.
Fear not. All these problems can be easily solved.
How to Prevent Drupal from Intercepting Your Requests for Certain Folders
Fix the 401 Error Handler on Your Site
The first thing to do is to make sure that your 401 error handler points to a valid file on your website's root directory. If you are not trying to access a password protected folder on your site, you can skip this part, although it should not harm your site unless you do something wrong. You can also skip this section if your 401 error document handler already points to a valid file, as is the case in some web hosts.
Download the .htaccess file from your website's root directory
First, download the .htaccess file from the main directory of your website. This is typically in the "public_html" or "www" folder when you log in with your FTP program. If you cannot find a file named ".htaccess", you will need to configure your FTP program to show all files, including those that begin with a dot ("."). Find out from your FTP documentation how you can do this.
Add the following line to the top of the .htaccess file
Make a copy of the downloaded .htaccess file and name it something like "original-htaccess-do-not-delete". This is your backup copy. Do not modify it. Do not delete it. If you make a mistake so that your website no longer functions, you can quickly rename this copy back to .htaccess and re-upload it to your site while you figure out what you did wrong. This is an important step - it's very easy to disable your entire site by making some small typo in the .htaccess file.
Open the original .htaccess file (not the backup copy) with an ASCII text editor. If you are using Windows, an ASCII text editor named Notepad can be found in the Accessories folder of the Start menu. Add the following lines to the top of the file, before anything else:
ErrorDocument 401 /401.html
Then create a web page named "401.html" and write whatever you want inside. If you have no idea what to put in there, just type "You do not have permission to access this folder." into your favorite web editor. If you don't have a web editor, type the following text into a blank Notepad document:
401 Password Needed
401 Password Needed
You don't have permission to access this folder.
Save the file as "401.html". Include the quotes when saving the file if you are using Notepad, to avoid a Notepad problem.
Upload both the .htaccess and the 401.html files
Now upload both files to your website, to the same folder you got the .htaccess file originally.
Enable Support for other Directory Index Files
Now you have to enable the support for the default index files like "index.html" and "index.shtml" in the folder you want to access.
If the folder you want to access does not have an existing .htaccess file, create a new .htaccess file with the following content. Otherwise, add the following line to the existing .htaccess file. Note that this is not the same .htaccess file you modified earlier. This one resides in the folder you are trying to allow access. Remember: you must use an ASCII text editor like Notepad, not a word processing program like Word or WordPad.
DirectoryIndex index.php index.shtml index.html
If your setup relies on index.pl or index.cgi being executed when the folder is accessed, you should add those to the DirectoryIndex line as well. Put it immediately before or after "index.php".
Save the file as ".htaccess" (including the quotes, if you are using Notepad), and upload it into the directory that you are trying to enable access. Watch out where you are uploading so that you do not accidentally overwrite Drupal's .htaccess file.
That's all there is to it. Clear your browser's cache (also known as "Temporary Internet Files" in one browser) and try to access the folder using your web browser and you should be successful this time. If not, go back through the instructions and make sure that you have completed every step correctly. If things go desperately wrong, so that you can't even access your website, don't panic. All you have to do is to delete the .htaccess files you have created or modified, and re-upload the one you have backed up earlier. (You did back it up like I told you, right?) Once you have done that, you can fix your errors at leisure without worrying that a visitor might be blocked from your site in the meantime.