Can a user be given read permissions to a specific folder in a document library without being able to access the rest of the library?

We often have the following situation occur and still have not found a good solution for it:

A department would like a user outside of the department to be able to access a specific folder in their default "Shared Documents" library without that user also being able to see the rest of the contents of the document library outside of that specific folder.

So if I give the user "read" permissions on the specific folder, they still cannot access the folder contents because they don't have "read" permission on the document library (it seems "limited access" permission on the document library is not enough).  So I end up having to grant the user read access on the document library, and then on all the top-level folders, I need to break inheritance and make sure that the user doesn't have access to any of the other top-level folders.

But then the problem because if we want to add a new user/group to the document library, we have to go into all the top-level folders and specifically grant the same access since we broke the permission inheritance earlier.

I feel that I must be doing something wrong - it seems there should be an easier solution!