[prev in list] [next in list] [prev in thread] [next in thread]
List: httpclient-commons-dev
Subject: RE: Code for "Problems under heavy load"
From: "Derek Sweet" <derek () arizonabay ! com>
Date: 2006-01-20 20:35:22
Message-ID: 20060120203522.7B32644CBC7 () relay2 ! r2 ! iad ! emailsrvr ! com
[Download RAW message or body]
I didn't expect anybody to debug the code for me, but just wanted to provide
as much information as I could. Thank you for the response. I will think
about what you've told me and see if there is any way I can make the process
of starting and killing processes more streamlined.
Derek
-----Original Message-----
From: Oleg Kalnichevski [mailto:olegk@apache.org]
Sent: Friday, January 20, 2006 1:16 PM
To: HttpClient Project
Subject: Re: Code for "Problems under heavy load"
Derek Sweet wrote:
> Apparantly it doesn't like attachments, which I should have expected.
> Here is the code:
Derek,
I seriously doubt someone will debug this amount of code for you. What I can
tell you is that I strongly suspect there is a race condition / thread
synchronization problem in your code. Some thread seems to try to execute a
method that has been just aborted from another one.
This explains the fact that the code works under moderate load in testing
and falls flat in production under heavier load. The nasty thing about race
conditions is that they tend to occur mostly in production, when
applications are running on multiprocessor boxes.
Oleg
>
> /*
> Filename: AsyncHTTPSingleton.java
> Author: Derek Sweet
> Date Created: Oct 27, 2005
>
> Description: This is the single .java file for the Asynchronous
> operation system. It hosts
> the 2-dimensional array of HttpClient employing
AsyncHTTP
> classes that will effectively serve the purposes of
> our ColdFusion system.
>
> Dependencies: This code requires that the jakarta commons HttpClient
> classes be included, as well as the jakarta commons codec.
> HttpClient:
> http://jakarta.apache.org/commons/httpclient/
> Commons Codec:
> http://jakarta.apache.org/commons/codec/
> */
>
> import org.apache.commons.httpclient.HttpClient;
> import
> org.apache.commons.httpclient.MultiThreadedHttpConnectionManager;
> import org.apache.commons.httpclient.HttpMethod;
> import org.apache.commons.httpclient.methods.GetMethod;
> import org.apache.commons.httpclient.methods.PostMethod;
> import org.apache.commons.httpclient.HttpException;
> import java.net.*;
> import java.io.*;
> import java.lang.reflect.*;
>
> public class AsyncHTTPSingleton{
> //This is the Multithreaded Connection Manager that is passed into
> any HttpClient constructor to ensure smooth asynchrnous operation
> private static MultiThreadedHttpConnectionManager connectionManager
=
> new MultiThreadedHttpConnectionManager();
> //This is the array of our custom AsyncHTTP classes that will each
> represent one instance of a HTTP GET or POST to a partner
> private volatile AsyncHTTP[][] asyncStruct;
>
> //This is the constructor for the Singleton class. It takes in a
size
> parameter to specify how large the main array should be.
> public AsyncHTTPSingleton(int Size){
> asyncStruct = new AsyncHTTP[Size][20];
> //null every member so that the array is primed for UUID
checking
> and array creation
> for (int i = 0; i < Size; i++){
> asyncStruct[i] = null;
> }
> }
>
> //This is a function that will create a new array for us to store
> AsyncHTTP classes in. We only make the 2nd dimension 20
> //because it is not likely that any given process will be pinging
> more than 10 people. 20 is overboard.
> public void CreateGroup(int UUID){
> asyncStruct[UUID] = new AsyncHTTP[20];
> }
>
> //This function will check to see if a given UUID is being used
> public boolean CheckUUID(int UUID){
> boolean tempBool = false;
> if (asyncStruct[UUID] != null){
> tempBool = true;
> }
> return tempBool;
> }
>
> //This is the function that will abort one individual process if it
> is taking too long to respond or we don't care about it anymore
> private void KillProcess(int UUID, int PPID){
> if (asyncStruct[UUID][PPID] != null){
> asyncStruct[UUID][PPID].abortMethod();
> }
> }
>
> //This function will kill an entire group of processes once we have
> found the response we're waiting for
> public void KillGroup(int UUID){
> for (int i = 0; i < Array.getLength(asyncStruct[UUID]);
> i++){
> KillProcess(UUID, i);
> }
> //Be sure to clean up our object by removing the array
element for
> future use
> asyncStruct[UUID] = null;
> }
>
> //This function will create one instance of an AsyncHTTP object that
> holds the specified properties. UUID and PPID are specified.
> public void CreateAsyncHTTP(String url, String names, String values,
> String method, int UUID, int PPID, String returnAddress){
> HttpClient tempHttpClient = new
> HttpClient(connectionManager);
> //The next two lines will parse the comma delimited
name/value pair
> strings into arrays of the individual values
> String[] nameArray = parseString(names,',');
> String[] valueArray = parseString(values,',');
> //If this is a GET method then pass the name/value pairs in
the URL
> and create a GetMethod object
> if (method == "GET"){
> for (int i = 0; i < Array.getLength(nameArray);
> i++){
> if (i == 0){
> url = url + "?";
> }
> else{
> url = url + "&";
> }
> url = url + nameArray[i] + "=" +
> valueArray[i];
> }
> GetMethod tempMethod = new GetMethod(url);
> asyncStruct[UUID][PPID] = new AsyncHTTP(UUID, PPID,
returnAddress,
> tempHttpClient, tempMethod);
> }
> //Else if this is a POST method then create the PostMethod
object
> and attach the name/value pairs as parameters
> else if (method == "POST"){
> PostMethod tempMethod = new PostMethod(url);
> for (int i = 0; i < Array.getLength(nameArray);
> i++){
> tempMethod.addParameter(nameArray[i],
> valueArray[i]);
> }
> asyncStruct[UUID][PPID] = new AsyncHTTP(UUID, PPID,
returnAddress,
> tempHttpClient, tempMethod);
> }
>
> //Start the asyncrhonous process as defined in the AsyncHTTP
class'
> run() method
> asyncStruct[UUID][PPID].start();
> }
>
> //This is a quick parsing function that will turn our comma delimted
> name/value pair strings into arrays
> private String[] parseString(String stringToSplit, char delimiter) {
> stringToSplit = stringToSplit.trim();
>
> int index = 0;
> int stringsFound = 0;
> String[] result = new String[stringsFound];
> while (index < stringToSplit.length()) {
> StringBuffer sb = new StringBuffer();
> char c;
> boolean inString = false;
> while (true) {
> c = stringToSplit.charAt(index);
> if (c == delimiter) break;
> else sb.append(c);
> index++;
> if (index == stringToSplit.length()) break;
> }
> String s = sb.toString().trim();
> stringsFound++;
> String[] tempArray = new String[stringsFound];
> System.arraycopy(result, 0, tempArray, 0,
stringsFound-1);
> tempArray[stringsFound-1] = s;
> result = tempArray;
> index++;
> }
> return result;
> }
>
> //Here is the AsyncHTTP class that allows asynchrnous operation.
> When the start() method is called this class will execute it's
> //run() method because it extends the Thread class.
> static class AsyncHTTP extends Thread{
> private HttpClient myHttpClient;
> private HttpMethod myMethod;
> private int UUID;
> private int PPID;
> private String returnAddress;
>
> //Basic constructor that simply sets all the local variables
> public AsyncHTTP(int UUIDToUse, int PPIDToUse, String
> returnAddressToUse, HttpClient httpClientToUse, HttpMethod methodToUse){
> this.UUID = UUIDToUse;
> this.PPID = PPIDToUse;
> this.returnAddress = returnAddressToUse;
> this.myHttpClient = httpClientToUse;
> this.myMethod = methodToUse;
> }
>
> //This is a function that will simply cause the HTTP process
> associated with this object to abort.
> public void abortMethod(){
> this.myMethod.abort();
> }
>
> //This is the main run() method that is called when the
instatiated
> object's start() method is called.
> public void run(){
> //This string will hold the response returned from
the HTTP
> GET/POST
> String StrResponse = new String("");
>
> final int SOCKET_TIMEOUT = 1000;
> myMethod.getParams().setSoTimeout(SOCKET_TIMEOUT);
>
> //Try to execute the method will the local
HttpClient and store the
> response in StrResponse
> try{
> myHttpClient.executeMethod(myMethod);
> StrResponse =
> myMethod.getResponseBodyAsString();
> } catch (HttpException he) {
> } catch (IOException ioe){
> } finally {
> myMethod.releaseConnection();
> }
>
> //Create a quick HttpClient to send the response
back to the
> ColdFusion system, along with the PPID and UUID
> HttpClient tempHttpClient = new
> HttpClient(connectionManager);
> PostMethod tempReturnMethod = new
> PostMethod(returnAddress);
> tempReturnMethod.addParameter("PPID",
> Integer.toString(PPID));
> tempReturnMethod.addParameter("UUID",
> Integer.toString(UUID));
> tempReturnMethod.addParameter("responseBody", new
> String(StrResponse));
> try{
>
> tempHttpClient.executeMethod(tempReturnMethod);
> } catch (HttpException he) {
> } catch (IOException ioe){
> } finally {
> tempReturnMethod.releaseConnection();
> }
> //Clean up our object by removing the temporary
HttpClient and
> PostMethod
> tempHttpClient = null;
> tempReturnMethod = null;
> }
> }
> }
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail:
> httpclient-dev-help@jakarta.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: httpclient-dev-help@jakarta.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: httpclient-dev-help@jakarta.apache.org
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic