Understanding Authentication with Active Directory Groups

I am trying to understand the process with which SP authenticates when using AD groups for permissions. I have had a SP2010 with NTLM environment up and running for the past 4 years and have used AD groups as well as direct user permissions since the beginning. For the most part this worked without issue if I recall, but it seems it doesn't quite work too well now.

Currently, our SP environment (which is part of the primary domain) is primarily used for our local LAN users with some WAN access for specific personnel. In the past we allowed full domain access for adding new users from the SP "Address Book". However, our local policy changed a couple years ago and we restricted the ability to add new users to our local OU. and we implimented local OU AD groups and add users from the WAN to that group.

My question is... How does the authentication of users using AD Groups work? If I ad new WAN users using AD to our OU security groups, they are denied permissions though the AD group is assigned permissions in SP. Does the authentication happen at the time the user is accessing information? Does the restricted settings on the Address  Book in SP have some impact? Does somehow SP Search need to crawl this ad group to know who the user names are?

I know that looking at the back end data the group account exists but the user does not unless specifically added using the SP interface. Also, I do not use the SP/AD management feature and I do not have USer Profile service running. Doesn't work, never could get it too.

Thanks!