Index on lookup column hitting list view threshold

I have a list with over 150,000 records in it.  I have an index on one of the columns in that list which also happens to be of type "lookup" (with multiselect not on).  I am trying to use that indexed column in a view and it is hitting the list view threshold which I have set to 10,000.  During the large query window I have set I ran the view just to see how many records would be coming back and it is only bringing back 52 records.  I have indexes on other columns and I use those in view filters and they work just fine so I am not sure why those work and this one does not.  Are there any known issues with using indexes on lookup columns in SharePoint 2010?