Scenario: Using Excel Web Access Webpart giving error "Unable to process the request. Wait a few minutes and try performing this operation again". This is when request goes through WFE.
Environment: SharePoint 2010, Windows Server 2008 R2
These are the errors getting logged:
01/02/2013 16:17:26.59 w3wp.exe (0x12B0) 0x0D28 Excel Services Application Web Front End
g0pd High ServerSession.GetHealthScoreCallback: Failed to get health score for server
http://prod003:32843/71a521ef812b4d48b1bf5d0fd87698ec/ExcelService*.asmx [id: b8c7e663-8225-4d87-9b56-1862c0161f9e] due to the connection timing out. Exception: System.TimeoutException: The request channel timed out while waiting for a reply after 00:00:20.
Increase the timeout value passed to the call to Request or increase the SendTimeout value on the Binding. The time allotted to this operation may have been a portion of a longer timeout. ---> System.TimeoutException: The HTTP request to 'http://prod003:32843/71a521ef812b4d48b1bf5d0fd87698ec/ExcelService.asmx'
has exceeded the allotted timeout of 00:00:20. The time allotted to this operation may have been a portion of a longer timeout. ---> System.Net.We...
01/02/2013 16:17:26.59* w3wp.exe (0x12B0) 0x0D28 Excel Services Application Web Front End
g0pd High ...bException: The operation has timed out at System.Net.HttpWebRequest.GetResponse() at System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan
timeout) --- End of inner exception stack trace --- at System.ServiceModel.Channels.HttpChannelUtilities.ProcessGetResponseWebException(WebException webException, HttpWebRequest request, HttpAbortReason abortReason)
at System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout) at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout)
--- End of inner exception stack trace --- Server stack trace: at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan ...
01/02/2013 16:17:26.59* w3wp.exe (0x12B0) 0x0D28 Excel Services Application Web Front End
g0pd High ...timeout) at System.ServiceModel.Channels.SecurityChannelFactory`1.SecurityRequestChannel.Request(Message message, TimeSpan timeout) at System.ServiceModel.Dispatcher.RequestChannelBinder.Request(Message
message, TimeSpan timeout) at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout) at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage
methodCall, ProxyOperationRuntime operation) at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message) Exception rethrown at [0]: at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage
reqMsg, IMessage retMsg) at System.Runtime.Remoting.Proxies.RealPro...
01/02/2013 16:17:26.59* w3wp.exe (0x12B0) 0x0D28 Excel Services Application Web Front End
g0pd High ...xy.PrivateInvoke(MessageData& msgData, Int32 type) at Microsoft.Office.Excel.Server.CalculationServer.Proxy.IExcelServiceSoap.GetHealthScore() at Microsoft.Office.Excel.Server.CalculationServer.Proxy.ServerSession.GetServerHealthScore()
at Microsoft.Office.Excel.Server.CalculationServer.Proxy.ServerSession.<>c__DisplayClass155.<ExecuteWithSecurityContext>b__154() at Microsoft.Office.Excel.Server.CalculationServer.Proxy.ServerSession.ExecuteWithSecurityContext(Boolean
forceRevert, CoreWebMethod coreWebMethod, CommandParameter parameter) at Microsoft.Office.Excel.Server.CalculationServer.Proxy.ServerSession.GetHealthScoreCallback(Object state)
01/02/2013 16:17:36.57 w3wp.exe (0x12B0) 0x1928 Excel Services Application Web Front End
144h Medium ServerSession.StampLastSentRequest: server
http://prod004:32843/71a521ef812b4d48b1bf5d0fd87698ec/ExcelService*.asmx [id: 2758ed0b-5f48-46a2-b297-ba303d61a96b] is on probation
01/02/2013 16:17:36.57 w3wp.exe (0x12B0) 0x1928 Excel Services Application Excel Calculation
Services d51n Medium MossHost.GetEndpointAddress: Server endpoint Uri:
http://prod004:32843/71a521ef812b4d48b1bf5d0fd87698ec/ExcelService.asmx.
01/02/2013 16:17:36.57 w3wp.exe (0x12B0) 0x1928 SharePoint Foundation
Topology e5mc Medium WcfSendRequest: RemoteAddress: 'http://prod004:32843/71a521ef812b4d48b1bf5d0fd87698ec/ExcelService.asmx'
Channel: 'Microsoft.Office.Excel.Server.CalculationServer.Proxy.IExcelServiceSoap' Action: 'http://schemas.microsoft.com/office/Excel/Server/WebServices/ExcelServerInternalService/ExcelServiceSoap/GetHealthScore' MessageId: 'urn:uuid:cb36928f-b4de-4b9c-896a-2739414875b7'
01/02/2013 16:17:36.57 w3wp.exe (0x12B0) 0x1E90 Excel Services Application Web Front End
144h Medium ServerSession.StampLastSentRequest: server
http://prod003:32843/71a521ef812b4d48b1bf5d0fd87698ec/ExcelService*.asmx [id: b8c7e663-8225-4d87-9b56-1862c0161f9e] is on probation
01/02/2013 16:17:36.57 w3wp.exe (0x12B0) 0x1E90 Excel Services Application Excel Calculation
Services d51n Medium MossHost.GetEndpointAddress: Server endpoint Uri:
http://prod003:32843/71a521ef812b4d48b1bf5d0fd87698ec/ExcelService.asmx.
01/02/2013 16:17:36.57 w3wp.exe (0x12B0) 0x1E90 SharePoint Foundation
Topology e5mc Medium WcfSendRequest: RemoteAddress: 'http://prod003:32843/71a521ef812b4d48b1bf5d0fd87698ec/ExcelService.asmx'
Channel: 'Microsoft.Office.Excel.Server.CalculationServer.Proxy.IExcelServiceSoap' Action: 'http://schemas.microsoft.com/office/Excel/Server/WebServices/ExcelServerInternalService/ExcelServiceSoap/GetHealthScore' MessageId: 'urn:uuid:7cfa82e7-d215-4d6b-9675-3bc193595873'
01/02/2013 16:17:36.57 w3wp.exe (0x12B0) 0x0D28 Excel Services Application Web Front End
144h Medium ServerSession.StampLastSentRequest: server
http://prod002:32843/71a521ef812b4d48b1bf5d0fd87698ec/ExcelService*.asmx [id: 765495b3-0ac4-4734-860e-b3e3c804d7ca] is on probation
01/02/2013 16:17:36.57 w3wp.exe (0x12B0) 0x0D28 Excel Services Application Excel Calculation
Services d51n Medium MossHost.GetEndpointAddress: Server endpoint Uri:
http://prod002:32843/71a521ef812b4d48b1bf5d0fd87698ec/ExcelService.asmx.
01/02/2013 16:17:36.57 w3wp.exe (0x12B0) 0x0D28 SharePoint Foundation
Topology e5mc Medium WcfSendRequest: RemoteAddress: 'http://prod002:32843/71a521ef812b4d48b1bf5d0fd87698ec/ExcelService.asmx'
Channel: 'Microsoft.Office.Excel.Server.CalculationServer.Proxy.IExcelServiceSoap' Action: 'http://schemas.microsoft.com/office/Excel/Server/WebServices/ExcelServerInternalService/ExcelServiceSoap/GetHealthScore' MessageId: 'urn:uuid:deede8d5-5dc7-4b9a-bfac-f75c4a548d4e'
01/02/2013 16:17:36.57 w3wp.exe (0x12B0) 0x1B5C Excel Services Application Web Front End
144h Medium ServerSession.StampLastSentRequest: server
http://prod001:32843/71a521ef812b4d48b1bf5d0fd87698ec/ExcelService*.asmx [id: 5c00a33f-a710-4ec1-880c-d7436d55aea4] is on probation
01/02/2013 16:17:36.57 w3wp.exe (0x12B0) 0x1B5C Excel Services Application Excel Calculation
Services d51n Medium MossHost.GetEndpointAddress: Server endpoint Uri:
http://prod001:32843/71a521ef812b4d48b1bf5d0fd87698ec/ExcelService.asmx.
01/02/2013 16:18:06.57 w3wp.exe (0x12B0) 0x2288 Excel Services Application Web Front
End 144h Medium ServerSession.StampLastSentRequest: server
http://prod004:32843/71a521ef812b4d48b1bf5d0fd87698ec/ExcelService*.asmx [id: 2758ed0b-5f48-46a2-b297-ba303d61a96b] is on probation
01/02/2013 16:18:06.57 w3wp.exe (0x12B0) 0x2288 Excel Services Application Excel Calculation
Services d51n Medium MossHost.GetEndpointAddress: Server endpoint Uri:
http://prod004:32843/71a521ef812b4d48b1bf5d0fd87698ec/ExcelService.asmx.
01/02/2013 16:18:06.57 w3wp.exe (0x12B0) 0x2288 SharePoint Foundation Topology e5mc Medium WcfSendRequest: RemoteAddress: 'http://prod004:32843/71a521ef812b4d48b1bf5d0fd87698ec/ExcelService.asmx'
Channel: 'Microsoft.Office.Excel.Server.CalculationServer.Proxy.IExcelServiceSoap' Action: 'http://schemas.microsoft.com/office/Excel/Server/WebServices/ExcelServerInternalService/ExcelServiceSoap/GetHealthScore' MessageId: 'urn:uuid:0e1fa739-395e-44b5-b9f0-dbd65ee6b808'
01/02/2013 16:18:06.57 w3wp.exe (0x12B0) 0x1E90 Excel Services Application Web Front
End 144h Medium ServerSession.StampLastSentRequest: server
http://prod003:32843/71a521ef812b4d48b1bf5d0fd87698ec/ExcelService*.asmx [id: b8c7e663-8225-4d87-9b56-1862c0161f9e] is on probation
01/02/2013 16:18:06.57 w3wp.exe (0x12B0) 0x1E90 Excel Services Application Excel Calculation
Services d51n Medium MossHost.GetEndpointAddress: Server endpoint Uri:
http://prod003:32843/71a521ef812b4d48b1bf5d0fd87698ec/ExcelService.asmx.
01/02/2013 16:18:06.57 w3wp.exe (0x12B0) 0x1E90 SharePoint Foundation Topology e5mc Medium WcfSendRequest: RemoteAddress: 'http://prod003:32843/71a521ef812b4d48b1bf5d0fd87698ec/ExcelService.asmx'
Channel: 'Microsoft.Office.Excel.Server.CalculationServer.Proxy.IExcelServiceSoap' Action: 'http://schemas.microsoft.com/office/Excel/Server/WebServices/ExcelServerInternalService/ExcelServiceSoap/GetHealthScore' MessageId: 'urn:uuid:7a464167-460a-42d4-8d94-69d5fbd73019'
01/02/2013 16:18:06.57 w3wp.exe (0x12B0) 0x0D28 Excel Services Application Web Front
End 144h Medium ServerSession.StampLastSentRequest: server
http://prod002:32843/71a521ef812b4d48b1bf5d0fd87698ec/ExcelService*.asmx [id: 765495b3-0ac4-4734-860e-b3e3c804d7ca] is on probation
01/02/2013 16:18:06.57 w3wp.exe (0x12B0) 0x0D28 Excel Services Application Excel Calculation
Services d51n Medium MossHost.GetEndpointAddress: Server endpoint Uri:
http://prod002:32843/71a521ef812b4d48b1bf5d0fd87698ec/ExcelService.asmx.
01/02/2013 16:18:06.57 w3wp.exe (0x12B0) 0x0D28 SharePoint Foundation Topology e5mc Medium WcfSendRequest: RemoteAddress: 'http://prod002:32843/71a521ef812b4d48b1bf5d0fd87698ec/ExcelService.asmx'
Channel: 'Microsoft.Office.Excel.Server.CalculationServer.Proxy.IExcelServiceSoap' Action: 'http://schemas.microsoft.com/office/Excel/Server/WebServices/ExcelServerInternalService/ExcelServiceSoap/GetHealthScore' MessageId: 'urn:uuid:426aef47-d20a-481c-9247-8e98bd487b8f'
01/02/2013 16:18:06.57 w3wp.exe (0x12B0) 0x1B5C Excel Services Application Web Front
End 144h Medium ServerSession.StampLastSentRequest: server
http://prod001:32843/71a521ef812b4d48b1bf5d0fd87698ec/ExcelService*.asmx [id: 5c00a33f-a710-4ec1-880c-d7436d55aea4] is on probation
01/02/2013 16:18:06.57 w3wp.exe (0x12B0) 0x1B5C Excel Services Application Excel Calculation
Services d51n Medium MossHost.GetEndpointAddress: Server endpoint Uri:
http://prod001:32843/71a521ef812b4d48b1bf5d0fd87698ec/ExcelService.asmx.
01/02/2013 16:22:10.64 w3wp.exe (0x12B0) 0x1E90 Excel Services Application Web Front
End 6k3v Medium LoadBalancer.GetNextServer: failed to get server using load balancing scheme WorkbookUrl, falling down to health base afe0ed53-6377-4e20-83fb-3d919d543faa
01/02/2013 16:22:10.64 w3wp.exe (0x12B0) 0x1E90 Excel Services Application Web Front
End abho High ServerSession.GetNextServerHealthBased: There are no healthy servers although the number of available server in
the farm is {4}; updated with no candidates available for the health based load balancing. The update duration was {0} millisecs afe0ed53-6377-4e20-83fb-3d919d543faa
01/02/2013 16:22:10.64 w3wp.exe (0x12B0) 0x1E90 Excel Services Application Web Front
End ablz High LoadBalancer.GetNextServerHealthBased: There are no healthy server in the farm afe0ed53-6377-4e20-83fb-3d919d543faa
01/02/2013 16:22:10.64 w3wp.exe (0x12B0) 0x1E90 Excel Services Application Web Front
End abgn Medium LoadBalancer.GetFailoverServer: Failed to get a failover server using health based load balancing scheme. afe0ed53-6377-4e20-83fb-3d919d543faa
01/02/2013 16:22:10.64 w3wp.exe (0x12B0) 0x1E90 Excel Services Application Browser bdke Medium Utils.ExecuteWebEventHandler: An exception occured Microsoft.Office.Excel.WebUI.ExcelWebRendererException: Unable to process the request. ---> Microsoft.Office.Excel.Server.CalculationServer.Proxy.ServerSessionException:
Unable to process the request. at Microsoft.Office.Excel.Server.CalculationServer.Proxy.LoadBalancer.GetFailoverServer(String excludeServerId) at Microsoft.Office.Excel.Server.CalculationServer.Proxy.LoadBalancer.GetNextServer(String
workbookUrl) at Microsoft.Office.Excel.Server.CalculationServer.Proxy.LoadBalancer.ExecuteNewSessionMethod(CreateServerSessionMethod createServerSessionMethod, CoreNewSessionMethod newSessionMethod, String workbookUrl)
at Microsoft.Office.Excel.Server.CalculationServer.Proxy.ExcelServerLoadBalancer.OpenWorkbook(String url, Open... afe0ed53-6377-4e20-83fb-3d919d543faa
01/02/2013 16:22:10.64* w3wp.exe (0x12B0) 0x1E90 Excel Services Application Browser bdke Medium ...WorkbookFlags flags, CommandParameter commandParameter, CultureCommandParameter cultureParameter, String subscriberId, OpenWorkbookOutFlags& outFlags, OpenWorkbookOutParameters& outParameters, ServerSession&
serverSession) at Microsoft.Office.Excel.Server.CalculationServer.Proxy.ExcelServerSession.OpenWorkbook(String url, OpenWorkbookFlags flags, CommandParameter commandParameter, CultureCommandParameter cultureParameter, OpenWorkbookOutFlags&
outFlags, OpenWorkbookOutParameters& outParameters, ServerSession& serverSession) at Microsoft.Office.Excel.WebUI.EwaState.OpenWorkbookAndUpdateState(String sessionId, CommandParameter commandParameter, OpenWorkbookFlags flags,
String plainTextWorkbookUrl) at Microsoft.Office.Excel.WebUI.EwaState.OpenWorkbook() --- End of inner except... afe0ed53-6377-4e20-83fb-3d919d543faa
01/02/2013 16:22:10.64* w3wp.exe (0x12B0) 0x1E90 Excel Services Application Browser bdke Medium ...ion stack trace --- at Microsoft.Office.Excel.WebUI.EwaState.OpenWorkbook() at Microsoft.Office.Excel.WebUI.ExcelWebRenderer.<InternalPreRender>b__c(Object param)
at Microsoft.Office.Excel.WebUI.Utils.ExecuteWebEventHandler(CoreWebEventHandler coreWebEventHandler, CoreWebEventHandler baseClassHandler, Object param, IMainControl mainControl) afe0ed53-6377-4e20-83fb-3d919d543faa
01/02/2013 16:22:10.67 w3wp.exe (0x12B0) 0x1E90 Excel Services Application Browser 3kdd Medium ExcelWebRenderer.InternalRenderWebPart: Finished rendering EWR afe0ed53-6377-4e20-83fb-3d919d543faa
01/02/2013 16:22:36.59 w3wp.exe (0x12B0) 0x1E90 Excel Services Application Web Front
End acbd Critical Unable to reach Excel Calculation Services
http://prod003:32843/71a521ef812b4d48b1bf5d0fd87698ec/ExcelService*.asmx. [Session: User: WGLCO_DOMAIN\Svc-sp-web1-int-prd]
01/02/2013 16:22:36.59 w3wp.exe (0x12B0) 0x1E90 Excel Services Application Web Front
End accf Medium ServerSession.ProcessWebException: A Web exception during ExecuteWebMethod has occurred for server:
http://prod003:32843/71a521ef812b4d48b1bf5d0fd87698ec/ExcelService*.asmx, method: GetHealthScore, ex: System.Net.WebException: The remote server returned an error: (503) Server Unavailable. at System.Net.HttpWebRequest.GetResponse()
at System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout), response: 'System.Net.HttpWebResponse', status ProtocolError, user name: WGLCO_DOMAIN\Svc-sp-web1-int-prd.
01/02/2013 16:22:36.59 w3wp.exe (0x12B0) 0x1E90 Excel Services Application Web Front
End acco Critical There was an error in communicating with Excel Calculation Services
http://prod003:32843/71a521ef812b4d48b1bf5d0fd87698ec/ExcelService*.asmx exception: The remote server returned an error: (503) Server Unavailable. [Session: User: WGLCO_DOMAIN\Svc-sp-web1-int-prd].
I already tried restarting app server, tried all load balanced scheme of excel services (workbook url, round robin, local), iisreset. But NO LUCK :(
Please help in resolving this issue?
Thanks,
KK