SharePoint incoming email in load balanced web servers farm

I have two webservers in sharepoint farm with a VIP load balance and the web address we use ishttps://sharepoint.contoso.com , I have installed smtp on webserver1 and in the incoming email settings in SharePoint central admin I have given the server display address assharepoint.contoso.com same as my site name. there is also an alias created in webserver1 smtp called sharepoint but I wont recieve emails when I send it tolibrary1@sharepoint.contoso.com and will recieve emails when i send it tolibrary1@webserver1.contoso.com. How do i make the emails to be recieved when they are send tolibrary1@sharepoint.contoso.com . Please note the webservers2 which is loadbalanced doesnt even have smtp and I dont want to install smpt on it.