[prev in list] [next in list] [prev in thread] [next in thread]
List: httpclient-users
Subject: AW: AW: HttpClient.execute blocks until EOF of InputStreamBody instead of sending chunks on the fly
From: Frenzel Stefan <S.Frenzel () cadenas ! de>
Date: 2013-02-21 6:16:39
Message-ID: 8995DF86E158F740A8EE0B883AC1BE4F84B2FDA2 () INMAIL01
[Download RAW message or body]
Hi Francois-Xavier,
Subclassing InputStream as you suggested worked very well, thank you so much for this \
idea.
Here is what I have done to produce CSV (SuperCSV library) from Hibernate's \
ScrollableResults and upload it to a servlet on the fly:
public class ScrollableResultsInputStreamProducer<E> extends InputStream {
private static final Logger logger = \
Logger.getLogger(ScrollableResultsInputStreamProducer.class);
private final Session session;
private final ScrollableResults cursor;
private final Transformer<E> transformer;
private ByteArrayInputStream in;
private boolean eof;
public ScrollableResultsInputStreamProducer(Session session, ScrollableResults \
cursor, Transformer<E> transformer) throws IOException {
this.session = session;
this.cursor = cursor;
this.transformer = transformer;
this.in = new ByteArrayInputStream(transformer.transformHeader());
}
@Override
public int read() throws IOException {
if (eof) {
return -1;
}
if (in != null) {
int data = in.read();
if (data > -1) {
return data;
}
else {
in.close();
in = null;
}
}
if (!cursor.next()) {
eof = true;
return -1;
}
in = new ByteArrayInputStream(transformer.transform((E) cursor.get(0)));
session.clear();
return in.read();
}
@Override
public void close() throws IOException {
IOUtils.closeQuietly(in);
try {
cursor.close();
}
catch (Exception ex) {
logger.warn(ex.getMessage(), ex);
}
}
}
public interface Transformer<E> {
byte[] transform(E entity) throws IOException;
byte[] transformHeader() throws IOException;
}
public abstract class CsvTransformer<E> implements Transformer<E> {
private CellProcessor[] processors;
public CsvTransformer() {
processors = new CellProcessor[getHeader().size()];
CellProcessor notNullProcessor = new ConvertNullTo("");
for (int i = 0; i < processors.length; i++) {
processors[i] = notNullProcessor;
}
}
public byte[] transform(E entity) throws IOException {
ByteArrayOutputStream out = new ByteArrayOutputStream();
Writer writer = new OutputStreamWriter(out, "UTF-8");
CsvListWriter csvWriter = new CsvListWriter(writer, \
CsvPreference.STANDARD_PREFERENCE); List<List<String>> rows = getRows(entity);
for (List<String> row : rows) {
csvWriter.write(row, processors);
}
csvWriter.close();
return out.toByteArray();
}
public byte[] transformHeader() throws IOException {
ByteArrayOutputStream out = new ByteArrayOutputStream();
out.write(ByteOrderMark.UTF_8.getBytes());
out.flush();
Writer writer = new OutputStreamWriter(out, "UTF-8");
CsvListWriter csvWriter = new CsvListWriter(writer, \
CsvPreference.STANDARD_PREFERENCE); csvWriter.writeHeader(getHeader().toArray(new \
String[getHeader().size()])); csvWriter.close();
return out.toByteArray();
}
protected abstract List<String> getHeader();
protected abstract List<List<String>> getRows(E entity);
}
public class Uploader {
private static final Logger logger = Logger.getLogger(Uploader.class);
private final HttpClient client;
private final HttpPost request;
private final InputStream in;
private HttpResponse response;
public Uploader(String fileName, Date expiry, InputStream in) {
client = new DefaultHttpClient();
request = new HttpPost("http://localhost/foo/Servlet?filename=" + fileName + \
"&expiry=" + expiry.getTime()); this.in = new BufferedInputStream(in);
request.setEntity(new InputStreamEntity(in, -1));
}
public String upload() throws IOException {
String token = null;
try {
response = client.execute(request);
token = EntityUtils.toString(response.getEntity());
}
finally {
close();
}
return token;
}
protected void close() {
try {
EntityUtils.consume(response.getEntity());
}
catch (Exception ex) {
logger.warn(ex.getMessage(), ex);
}
IOUtils.closeQuietly(in);
try {
client.getConnectionManager().shutdown();
}
catch (Exception ex) {
logger.warn(ex.getMessage(), ex);
}
}
}
public class UploadServlet extends BaseServlet {
protected void processRequest(HttpServletRequest httpRequest, HttpServletResponse \
httpResponse, Map<RequestParam, String> params) throws Exception {
/* Stream to database or whatever ... */
/* String token = dao.create(new \
Upload(httpRequest.getInputStream(),params.get(RequestParam.FILE_NAME),params.get(RequestParam.EXPIRY));
sendResponse(token, httpResponse); */
}
protected RequestParam[] getMandatoryParameters() {
return new RequestParam[] { RequestParam.FILE_NAME, RequestParam.EXPIRY };
}
}
Hope that might be useful for anyone.
Br, Stefan
-----Ursprüngliche Nachricht-----
Von: Francois-Xavier Bonnet [mailto:fx.bonnet@gmail.com] Im Auftrag von \
François-Xavier Bonnet
Gesendet: Freitag, 15. Februar 2013 12:19
An: HttpClient User Discussion
Cc: Frenzel Stefan
Betreff: Re: AW: HttpClient.execute blocks until EOF of InputStreamBody instead of \
sending chunks on the fly
If you don't want to spawn new threads, you cannot use HttpAsyncClient.
Then the most simple way is to produce your csv file in memory and then pass a \
ByteArrayInputStream to HttpClient. But if the data is too big to be buffered to \
memory, you can write your own InputStream subclass that produces the data as \
HttpClient reads it.
It would be something like this:
InputStream producer = new InputStream() {
private ByteArrayInputStream data;
private boolean eof = false;
private int count = 0;
@Override
public int read() throws IOException {
if (eof)
return -1;
if (data != null) {
int result = data.read();
if (result > -1)
return data.read();
else {
data.close();
data = null;
}
}
String newData = nextLine();
if (newData == null) {
eof = true;
return -1;
}
data = new ByteArrayInputStream(newData.getBytes("UTF-8"));
return data.read();
}
private String nextLine() {
if (count < 10) {
System.out.println("Writing line " + count);
String result = "new Line " + count + "\n";;
count++;
return result;
}
return null;
}
};
On 15/02/2013 11:15, Frenzel Stefan wrote:
> Hi Francois-Xavier,
>
> Thank you very much for your answer. Switching to HttpAsyncClient made it. It's \
> quite easy to use, good work! However, my client is within an EJB. According to EJB \
> 3.0 specification it's not allowed to spawn new threads which is exclusively the \
> container's job. Any suggestions or best practices?
>
> Br, Stefan
>
>
> -----Ursprüngliche Nachricht-----
> Von: Francois-Xavier Bonnet [mailto:fx.bonnet@gmail.com] Im Auftrag
> von François-Xavier Bonnet
> Gesendet: Freitag, 15. Februar 2013 09:33
> An: HttpClient User Discussion
> Cc: Frenzel Stefan
> Betreff: Re: HttpClient.execute blocks until EOF of InputStreamBody
> instead of sending chunks on the fly
>
> Hi Stefan,
>
> You cannot use piped streams to write and read in the same thread as each read / \
> write operation is blocking. Here client.execute(post) is blocking while trying to \
> read the InputStrean. You should either create separate threads for read and write \
> or consider using HttpAsyncClient.
> Francois-Xavier
>
>
> On 15/02/2013 09:05, Frenzel Stefan wrote:
> > Hey guys,
> >
> > I am pretty new to HttpComponents and just wondered if it is possible to stream \
> > data from input stream of unknown length to an upload servlet. I've already tried \
> > to get it working with ClientChunkEncodedPost example bundled with HttpComponents \
> > 4.2.3 for the client side as well as ServletFileUpload for the server side. \
> > However, HttpClient.execute waits until EOF which will never be reached at this \
> > stage.
> > This is what I have:
> >
> > 1. Client:
> > pout = new PipedOutputStream();
> > pin = new PipedInputStream(pout, 8192);
> > writer = new OutputStreamWriter(pout, "UTF-8");
> > csvWriter = new CsvListWriter(writer,
> > CsvPreference.EXCEL_PREFERENCE);
> >
> > csvWriter.writeHeader(header);
> > client = new DefaultHttpClient();
> > client.getParams().setParameter(CoreProtocolPNames.PROTOCOL_VERSION, \
> > HttpVersion.HTTP_1_1); post = new HttpPost("http://localhost/Foo/Servlet");
> >
> > MultipartEntity entity = new
> > MultipartEntity(HttpMultipartMode.BROWSER_COMPATIBLE);
> >
> > /* csvWriter is not written to at this stage, so is pin */
> > entity.addPart("file", new InputStreamBody(pin,
> > "text/csv", fileName));
> >
> > post.setEntity(entity);
> >
> > /* here it blocks until EOF */
> > HttpResponse response = client.execute(post);
> >
> >
> > 2. Server:
> >
> > protected void processRequest(HttpServletRequest httpRequest, HttpServletResponse \
> > httpResponse, Map<RequestParam, String> params) throws Exception {
> > InputStream in = null;
> > try {
> >
> > if(!ServletFileUpload.isMultipartContent(httpRequest)) {
> > throw new Exception("Not multipart");
> > }
> >
> > ServletFileUpload upload = new ServletFileUpload();
> > FileItemIterator it = upload.getItemIterator(httpRequest);
> > while(it.hasNext()) {
> > FileItemStream item = it.next();
> > System.out.println(item.getFieldName());
> > in = item.openStream();
> > if(item.isFormField()) {
> > /* ... */
> > } else {
> > /* Write upload to database */
> > }
> > }
> > finally {
> > IOUtils.closeQuietly(in);
> > }
> > }
> >
> > Any help would be appreciated. Thanks!
> >
> >
> > Br, Stefan
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic