<< Go Back

Analyzing Httpwebrequest Connection Saturation In Net

- 24 Oct 2008

WinDbg is a powerful tool and can use the SOS extension (that ships with the .NET framework) to look into and analyze Managed Code. Another powerful tool is Debug Diagnostics. It uses the same Debugger Engine that WinDbg uses and can run analysis scripts. You can use Debug Diagnostics and a custom script to analyze hangs and performance issues with the HttpWebRequest class.

I will walk you through some techniques that will allow you to customize your own Debug Diagnostics script to analyze .NET HttpWebRequest connection saturation.

The technique I will use is to:

1. Walk the threads in a Dump and look for Managed threads that have System.Net.HttpWebRequest.GetResponse on the stack. To me that means we are waiting on a response and may indicate an issue.

2. For each of the threads found above, I will get the ServicePoint object and see how many connections it allows vs. how many it currently has.

3. Make a suggestion on how to fix it!

DebugDiag (Debug Diagnostics) has a great help file and some starter scripts. I wanted to use jscript so the starter scripts did not do much for me since they were written in VBScript. I put all the metadata for the script on one line so that when I get script errors I can easily map this to the script I am editing in Visual Studio 2008. Save the script (with a .asp extension) to the Scripts sub-directory of where DebugDiag is installed and it will show up when you start DebugDiag.

<?xml:namespace prefix = o ns = "urn:schemas-microsoft-com:office:office" />

Here is the starting script:

<%@ Language = javascript %><%@ Category = Crash/Hang Analyzers %><%@ Description = Template System.Net analysis script to build from…%><%@ TopLevelScript = True %><%

main();

// this kicks off everything

function main()

{

Manager.Write( “System.Net sample analysis script
”);

}

Select a dump file and run this script against it. You see it will just print some text.

Now let’s get to work. We need to iterate over all the dump files the DebugDiag is pointing to (in the event there is more than one) and we want to iterate over all of the threads.

Here is the Main function:

// this kicks off everything

function main()

{

var a_DataFile;

var i;

var a_NumberOfDumps;

g_GlobalBookMark=0;

Manager.Write( “System.Net sample analysis script
”);

// interesting that this does not work

//for (a_DataFile in Manager.DataFiles)

Manager.Progress.SetOverallRange(0,100);

Manager.Progress.OverallPosition = 10;

Manager.Progress.OverallStatus=“Initializing”;

a_NumberOfDumps = Manager.DataFiles.Count;

// for each data file specified…

for(i=0;i<a_NumberOfDumps;i++)

{

// get the dump

a_DataFile = Manager.DataFiles(i);

Manager.Progress.OverallStatus=“Starting analysis of: “ + a_DataFile;

// get an instance of a debugger

g_Debugger = Manager.GetDebugger(a_DataFile);

Manager.Write(“

Analysis of: “ + a_DataFile + “</h2>”);


 // try to load the managed debugging extension

 if(LoadSOS())

 {

 // initialize some object information for performance

 initGlobals();

 // walk all threads but report only specific managed ones

 walkManagedThreads();

 reportErrors();

 }





 // done with the debugger, so close it

 Manager.CloseDebugger(a_DataFile);

 Manager.Progress.OverallPosition = 100/a_NumberOfDumps * (i+1);

 }

}

Now all that remains is to fill in the functions: LoadSOS(), initGlobals(), walkManagedThreads() and reportErrors();

// Find where mscorwks is loaded and then load SOS from the same place

function LoadSOS()

{

 var abSuccess;

 var begPos;

 var endPos;

 var CmdOutput;

 abSuccess = false;

 // Find where mscorwks module came from…

 CmdOutput = g_Debugger.Execute(“lmv m mscorwks”);



 begPos = CmdOutput.indexOf(“Image path:”);



 if (begPos>0)

 {

 begPos=begPos+11; // skip over Image path:

 endPos=CmdOutput.indexOf(“mscorwks.dll”);



 Manager.Write (“Loading SOS from: ” + CmdOutput.substr(begPos,endPos-begPos) + “
”);

// load it!

 CmdOutput = g_Debugger.Execute(“!load “ + CmdOutput.substr(begPos,endPos-begPos) + “sos.dll”);



 if (CmdOutput.length == 0)

 {

 abSuccess=true;

 }

 else

 {

 Manager.Write( “Cannot load Managed Debugging extensions. Error: “ + CmdOutput + “
” );

 }

 }



 return abSuccess;

}

function initGlobals()

{

 g_HttpObjectOffsets = new ManagedFieldOffsets(new Array(“_ServicePoint”, “_Uri”, “_HttpRequestHeaders”, “_Proxy”),“system.dll!System.Net.HttpWebRequest”);

 g_SvcPointObjectOffsets = new ManagedFieldOffsets(new Array(“m_ConnectionLimit”, “m_CurrentConnections”),“system.dll!System.Net.ServicePoint”);

 g_UriObjectOffsets = new ManagedFieldOffsets(new Array(“m_String”), “system.dll!System.Uri”);

 g_StringObjectOffsets = new ManagedFieldOffsets(new Array( “m_firstChar”), “mscorlib.dll!System.String”);

 g_MaxConnectionErrorThreads = “”;

 g_MaxConnectionWarningThreads = “”;

}

The ManagedFieldOffsets function is pretty interesting. You could use several techniques in this script to get the information you want from a managed object. The SOS extension has a !do command that can dump the fields of an managed object and you can hunt through that for each object to get the value of a field you want, then !do that object etc… until you get to the value you want. However, I want this script to perform well, so I will use offsets from the object base address to get to the data I want. I could hard code these offsets, but if they change or I am using a different version of the CRL, I would have to change my script. So this handy function (actually used lick a class) is an expando based class that will store these offsets for me.

NOTE: If you want to learn about these commands, load a dump file with HttpWebRequest on the heap and play with each of the g_Debugger.Execute commands I am using!

/////////////////// Class – ManagedFieldOffsets

// this class uses expando properties and managed helpers to cache offsets

// in managed classes.

//

// usage: var obj = new ManagedFieldOffsets(new Array(“_ServicePoint”, “_Uri”, “_HttpRequestHeaders”, “_Proxy”),”system.dll!System.Net.HttpWebRequest”);

//

// aArray is an array of fields you are intersted in. It will create properties

// of the same name and look up the offset value for this field

//

// the second parameter aObject is the location and name of the managed object.

//

// you then can use the obj as defined above like this obj._Proxy to get the offset values

function ManagedFieldOffsets( aArray, aObject )

{

 var aStr;

 var i;

 var aCmdOutput;



 // get the Class information

 aCmdOutput = g_Debugger.Execute(“!DumpClass “ + getEEClass(aObject));



 for(i in aArray)

 {

 this[aArray[i]] = findOffset(aCmdOutput,aArray[i]);

 }

}

////////////////// Helper Function – getEEClass

// given a string, extract the EEClass address

// theClass name is direct input to !Name2EE must be MODULENAME!CLASSNAM

function getEEClass( theClassName )

{

 var retStr=“”;



 var aCmdOutput;

 var aFoundIdx;

 var aEOL;



 aCmdOutput = g_Debugger.Execute(“!Name2EE “ + theClassName);



 aFoundIdx = aCmdOutput.indexOf(“EEClass:”);

 if (aFoundIdx != -1)

 {

 aEOL = aCmdOutput.indexOf(“\n”,aFoundIdx);

 retStr=trim( aCmdOutput.substr( aFoundIdx + “EEClass:”.length , aEOL – (aFoundIdx + “EEClass:”.length)));

 }



 return retStr;

}

////////////////// Helper Function – findOffset

// given a string, extract the field offset specified

// this is really an excercise it string parsing

function findOffset(aCmdOutput, aField)

{

 var retStr=“”;



 var aFoundIdx;

 var aBOL;

 var sArray;



 // find the field

 aFoundIdx = aCmdOutput.indexOf(aField);

 if (aFoundIdx != -1)

 {

 // look backwards from there for the offset.

 // find the previous line

 aBOL = aCmdOutput.lastIndexOf(“\n”,aFoundIdx)+1;

 //need to strip all but one space from the string and make an array of the column values

 sArray = normalizeWhitespace(aCmdOutput.substr(aBOL, aFoundIdx-aBOL)).split(” “);

 // 7 columns total, offset is third column (0 based array)

 retStr=sArray[2];

 //Manager.Write(“
0:” + sArray[0] +”
1:” + sArray[1] +”
2:” + sArray[2] +”
3:” + sArray[3] + “
”);

 }



 return retStr;

}

Finally this function walks the threads and if it finds the thread is waiting on a request, dumps it out and inspects for connection saturation:

// this starts the analysis…

function walkManagedThreads()

{

 var aCmdOutput;

 var aStackOutput;

 var aThreadInfo;

 var aDebugThread;

 var aNumThreads;

 var aAddr;



 var i;

 var aFoundIdx;

 var bFoundFirstThread =false;

 aThreadInfo = g_Debugger.ThreadInfo;

 aNumThreads = aThreadInfo.Count;

 Manager.Progress.SetCurrentRange(0,aNumThreads);



 Manager.Progress.CurrentStatus=“Finding threads waiting on ‘System.Net.HttpWebRequest.GetResponse()'”;





 for (i=0;i<aNumThreads;i++)

 {



 Manager.Progress.CurrentPosition = i;

 aDebugThread = aThreadInfo.Item(i);

 if (aDebugThread)

 {

 aStackOutput = g_Debugger.Execute(“~” + aDebugThread.ThreadID + “e!CLRStack”);

 aFoundIdx = aStackOutput.indexOf(“System.Net.HttpWebRequest.GetResponse”);

 // highlight the call???

 //is the tread processing GetResponse?

 if (aFoundIdx != -1)

 {

 if (!bFoundFirstThread)

 {

 bFoundFirstThread=true;

 Manager.Write(“

Managed Threads Waiting on ‘System.Net.HttpWebRequest.GetResponse()'

”);

}

Manager.Write( “<A NAME=’ThreadAnchor” + aDebugThread.ThreadID + “‘>” );

// dump managed objects on the stack

aCmdOutput = g_Debugger.Execute(“~” + aDebugThread.ThreadID + “e!dso”);

// Collect the HttpWebRequest objects on the stack.

aFoundIdx = aCmdOutput.indexOf(“System.Net.HttpWebRequest”);

while (aFoundIdx != -1)

{

// if we found an object of interest on the stack,

// sav the address fo the object and the thread ID.

aAddr = aCmdOutput.substr(aFoundIdx-9, 8);

checkObj(aAddr,aDebugThread.ThreadID);

//build error/warning strings here instead.

//display total number of managed threads?

// look for another one…

aFoundIdx = -1;

}

// print the stack pretty by replacing line feeds with breaks Manager.Write( aStackOutput.replace(/\n/g, “
” ) + “
”);

}

I have included the completed script. It needs some error checking and polishing, but it will get the job done. It will report out any threads waiting on requests and dump out any HttpWebRequest objects that have ServicePoint objects with the CurrentConnections >= ConnectionLimit

Here is the complete code listing (save to the DebugDiag Scripts directory as WebRequestConnections.asp):

<%@ Language = javascript %><%@ Category = Crash/Hang Analyzers %><%@ Description = Template System.Net analysis script to build from…%><%@ TopLevelScript = True %><%

var g_Progress;

var g_Debugger;

var g_HttpObjectOffsets;

var g_SvcPointObjectOffsets;

var g_UriObjectOffsets;

var g_StringObjectOffsets;

var g_MaxConnectionErrorThreads; // array of thread id’s that have exceed max connection.

var g_MaxConnectionWarningThreads; // array of thread id’s that have 50% max connection.

main();

// this kicks off everything

function main()

{

var a_DataFile;

var i;

var a_NumberOfDumps;

Manager.Write( “System.Net sample analysis script
”);

// interesting that this does not work

//for (a_DataFile in Manager.DataFiles)

Manager.Progress.SetOverallRange(0,100);

Manager.Progress.OverallPosition = 10;

Manager.Progress.OverallStatus=“Initializing”;

a_NumberOfDumps = Manager.DataFiles.Count;

// for each data file specified…

for(i=0;i<a_NumberOfDumps;i++)

{

// get the dump

a_DataFile = Manager.DataFiles(i);

Manager.Progress.OverallStatus=“Starting analysis of: “ + a_DataFile;

// get an instance of a debugger

g_Debugger = Manager.GetDebugger(a_DataFile);

Manager.Write(“

Analysis of: “ + a_DataFile + “</h2>”);


 // try to load the managed debugging extension

 if(LoadSOS())

 {

 // initialize some object information for performance

 initGlobals();

 // walk all threads but report only specific managed ones

 walkManagedThreads();

 reportErrors();

 }





 // done with the debugger, so close it

 Manager.CloseDebugger(a_DataFile);

 Manager.Progress.OverallPosition = 100/a_NumberOfDumps * (i+1);

 }

}

function initGlobals()

{

 g_HttpObjectOffsets = new ManagedFieldOffsets(new Array(“_ServicePoint”, “_Uri”, “_HttpRequestHeaders”, “_Proxy”),“system.dll!System.Net.HttpWebRequest”);

 g_SvcPointObjectOffsets = new ManagedFieldOffsets(new Array(“m_ConnectionLimit”, “m_CurrentConnections”),“system.dll!System.Net.ServicePoint”);

 g_UriObjectOffsets = new ManagedFieldOffsets(new Array(“m_String”), “system.dll!System.Uri”);

 g_StringObjectOffsets = new ManagedFieldOffsets(new Array( “m_firstChar”), “mscorlib.dll!System.String”);

 g_MaxConnectionErrorThreads = “”;

 g_MaxConnectionWarningThreads = “”;

}

// this starts the analysis…

function walkManagedThreads()

{

 var aCmdOutput;

 var aStackOutput;

 var aThreadInfo;

 var aDebugThread;

 var aNumThreads;

 var aAddr;



 var i;

 var aFoundIdx;

 var bFoundFirstThread =false;

 aThreadInfo = g_Debugger.ThreadInfo;

 aNumThreads = aThreadInfo.Count;

 Manager.Progress.SetCurrentRange(0,aNumThreads);



 Manager.Progress.CurrentStatus=“Finding threads waiting on ‘System.Net.HttpWebRequest.GetResponse()'”;





 for (i=0;i<aNumThreads;i++)

 {



 Manager.Progress.CurrentPosition = i;

 aDebugThread = aThreadInfo.Item(i);

 if (aDebugThread)

 {

 aStackOutput = g_Debugger.Execute(“~” + aDebugThread.ThreadID + “e!CLRStack”);

 aFoundIdx = aStackOutput.indexOf(“System.Net.HttpWebRequest.GetResponse”);

 // highlight the call???

 //is the tread processing GetResponse?

 if (aFoundIdx != -1)

 {

 if (!bFoundFirstThread)

 {

 bFoundFirstThread=true;

 Manager.Write(“

Managed Threads Waiting on ‘System.Net.HttpWebRequest.GetResponse()'

”);

}

Manager.Write( “<A NAME=’ThreadAnchor” + aDebugThread.ThreadID + “‘>” );

// dump managed objects on the stack

aCmdOutput = g_Debugger.Execute(“~” + aDebugThread.ThreadID + “e!dso”);

// Collect the HttpWebRequest objects on the stack.

aFoundIdx = aCmdOutput.indexOf(“System.Net.HttpWebRequest”);

while (aFoundIdx != -1)

{

// if we found an object of interest on the stack,

// sav the address fo the object and the thread ID.

aAddr = aCmdOutput.substr(aFoundIdx-9, 8);

checkObj(aAddr,aDebugThread.ThreadID);

//build error/warning strings here instead.

//display total number of managed threads?

// look for another one…

aFoundIdx = -1; //aCmdOutput.indexOf(“System.Net.HttpWebRequest”, aFoundIdx + 25);

}

Manager.Write( aStackOutput.replace(/\n/g, “
” ) + “
”);

// Manager.Write( “Thread ID: ” + aDebugThread.ThreadID + ” Waiting
”);

}

function reportErrors()

{

var i;

var threadList;

if (g_MaxConnectionErrorThreads.length)

{

Manager.ReportError(“The number connections exceeds the number of available connections for these threads:” + g_MaxConnectionErrorThreads.replace(/, +$/g, “”) ,“Increase the number of connections available per this article: <a TARGET=_blank href=’http://msdn.microsoft.com/en-us/library/7af54za5.aspx’>Managing Connections</a>”);

}

/////////////////////// Managed code helpers ///////////////////////////////////

// Find where mscorwks is loaded and then load SOS from the same place

function LoadSOS()

{

var abSuccess;

var begPos;

var endPos;

var CmdOutput;

abSuccess = false;

CmdOutput = g_Debugger.Execute(“lmv m mscorwks”);

begPos = CmdOutput.indexOf(“Image path:”);

if (begPos>0)

{

begPos=begPos+11; // skip over Image path:

endPos=CmdOutput.indexOf(“mscorwks.dll”);

Manager.Write (“Loading SOS from: ” + CmdOutput.substr(begPos,endPos-begPos) + “
”);

CmdOutput = g_Debugger.Execute(“!load “ + CmdOutput.substr(begPos,endPos-begPos) + “sos.dll”);

if (CmdOutput.length == 0)

{

abSuccess=true;

}

else

{

Manager.Write( “Cannot load Managed Debugging extensions. Error: “ + CmdOutput + “
” );

}

return abSuccess;

}

/////////////////// Class – ManagedFieldOffsets

// this class uses expando properties and managed helpers to cache offsets

// in managed classes.

// usage: var obj = new ManagedFieldOffsets(new Array(“_ServicePoint”, “_Uri”, “_HttpRequestHeaders”, “_Proxy”),”system.dll!System.Net.HttpWebRequest”);

// aArray is an array of fields you are intersted in. It will create properties

// of the same name and look up the offset value for this field

// the second parameter aObject is the location and name of the managed object.

// you then can use the obj as defined above like this obj._Proxy to get the offset values

function ManagedFieldOffsets( aArray, aObject )

{

var aStr;

var i;

var aCmdOutput;

// get the Class information

aCmdOutput = g_Debugger.Execute(“!DumpClass “ + getEEClass(aObject));

for(i in aArray)

{

this[aArray[i]] = findOffset(aCmdOutput,aArray[i]);

}

function getManagedLongVal( aObjAddr, aOffset )

{

return parseInt(g_Debugger.Execute(“dt long “ + aObjAddr + “+” + aOffset));

}

////////////////// Helper Function – findOffset

// given a string, extract the field offset specified

// this is really an excercise it string parsing

function findOffset(aCmdOutput, aField)

{

var retStr=“”;

var aFoundIdx;

var aBOL;

var sArray;

// find the field

aFoundIdx = aCmdOutput.indexOf(aField);

if (aFoundIdx != -1)

{

// look backwards from there for the offset.

// find the previous line

aBOL = aCmdOutput.lastIndexOf(“\n”,aFoundIdx)+1;

//need to strip all but one space from the string and make an array of the column values

sArray = normalizeWhitespace(aCmdOutput.substr(aBOL, aFoundIdx-aBOL)).split(” “);

// 7 columns total, offset is third column (0 based array)

retStr=sArray[2];

//Manager.Write(“
0:” + sArray[0] +”
1:” + sArray[1] +”
2:” + sArray[2] +”
3:” + sArray[3] + “
”);

}

return retStr;

}

////////////////// Helper Function – getEEClass

// given a string, extract the EEClass address

// theClass name is direct input to !Name2EE must be MODULENAME!CLASSNAM

function getEEClass( theClassName )

{

var retStr=“”;

var aCmdOutput;

var aFoundIdx;

var aEOL;

aCmdOutput = g_Debugger.Execute(“!Name2EE “ + theClassName);

aFoundIdx = aCmdOutput.indexOf(“EEClass:”);

if (aFoundIdx != -1)

{

aEOL = aCmdOutput.indexOf(“\n”,aFoundIdx);

retStr=trim( aCmdOutput.substr( aFoundIdx + “EEClass:”.length , aEOL – (aFoundIdx + “EEClass:”.length)));

}

return retStr;

}

// utility functions

// utility function removing leading and trailing white space

function trim(stringToTrim) {

return stringToTrim.replace(/^\s+|\s+$/g,“”);

}

// utility function removing extra whitespace and replaceing it with a single space

// for example ” some text here” will be converted to ” some text here”

function normalizeWhitespace(stringToTrim) {

return stringToTrim.replace(/\s+/g,‘ ‘)

}

// calculate and save offsets for the Object fields we need

// this is how you would get the information in the debugger with the sos.dll extension

//0:000> !Name2EE system.dll!System.Net.HttpWebRequest

//Module: 7a726000 (System.dll)

//Token: 0x020003d0

//MethodTable: 7a7681d4

//EEClass: 7a768154 <<—– This is it!

//Name: System.Net.HttpWebRequest

//0:000> !DumpClass 7a768154

//Class Name: System.Net.HttpWebRequest

//mdToken: 020003d0 (C:\Windows\assembly\GAC_MSIL\System\2.0.0.0__b77a5c561934e089\System.dll)

//Parent Class: 7a768574

//Module: 7a726000

//Method Table: 7a7681d4

//Vtable Slots: 2c

//Total Method Slots: da

//Class Attributes: 102001

//NumInstanceFields: 45

//NumStaticFields: 7

// MT Field Offset Type VT Attr Value Name

//790fd0f0 400018a 4 System.Object 0 instance __identity

//7a7978b0 4001d3b 14 System.Int32 1 instance m_AuthenticationLevel

//…

//7a769fd8 4001f19 40 ….Net.ServicePoint 0 instance _ServicePoint

//…

//7a757bf0 4001f22 5c System.Uri 0 instance _Uri

//////////////////////////////// Non-generic managed object helpers….

// Given an address of an object get the service point object address

function getSvcPointFromReq(aHttpObjAddr)

{

var aRetStr = “”;

var aCmdOutput;

aCmdOutput = g_Debugger.Execute(“dd “ + aHttpObjAddr + “+” + g_HttpObjectOffsets._ServicePoint + ” L1″);

aRetStr = aCmdOutput.substr(aCmdOutput.length – 9,8);

return aRetStr;

}

function getUriString(aObjAddr)

{

//strip the first 9 chars off (the address)

return g_Debugger.Execute(“du poi(poi(“ + aObjAddr + “+” + g_HttpObjectOffsets._Uri + “)+” +g_UriObjectOffsets.m_String + “)+” + g_StringObjectOffsets.m_firstChar).substr(9);

}

// obj is an address we can do !do on to get information from

function checkObj(aObjAddr, threadID)

{

//get service point and see if it is OK.

var bSvcPointErrorOrWarning = false;

var maxConn;

var currConn;

var aCmdOutput;

var i;

var aFoundIdx;

aCmdOutput = getSvcPointFromReq(aObjAddr);

if( (aCmdOutput!=“00000000”) && (aCmdOutput!=“????????”))

{

maxConn = getManagedLongVal( aCmdOutput, g_SvcPointObjectOffsets.m_ConnectionLimit);

currConn = getManagedLongVal(aCmdOutput, g_SvcPointObjectOffsets.m_CurrentConnections);

if(currConn>=maxConn)

{

// report out info and error

g_MaxConnectionErrorThreads = g_MaxConnectionErrorThreads + “<a href=’#ThreadAnchor” + threadID + “‘>” + threadID +“</a>, “;

Manager.Write(“
 HttpRequest URI:” + getUriString(aObjAddr) + ” ServicePoint – ConnectionLimit:” + maxConn + ” CurrentConnections:” + currConn +“
”);

}

else

{

if(((currConn)*2)>=maxConn)

{

// report out info and error

g_MaxConnectionWarningThreads = g_MaxConnectionWarningThreads + “<a href=’#ThreadAnchor” + threadID + “‘>” + threadID +“</a>, “;

Manager.Write(“
 HttpRequest URI:” + getUriString(obj) + ” ServicePoint – ConnectionLimit:” + maxConn + ” CurrentConnections:” + currConn +“
”);

}

<< Go Back

Jeff Sanders Technical Blog

I am a Microsoft employee that has worked on all aspects of the Web Stack for a long time. I hope these blogs are useful to you! Use this information at your own risk.

Analyzing Httpwebrequest Connection Saturation In Net

Managed Threads Waiting on ‘System.Net.HttpWebRequest.GetResponse()'

Managed Threads Waiting on ‘System.Net.HttpWebRequest.GetResponse()'