'[PyQt] Looking for Efficient way to capture QWebView "frames" for input to ML'

[prev in list] [next in list] [prev in thread] [next in thread] 

List:       pykde
Subject:    [PyQt] Looking for Efficient way to capture QWebView "frames" for input to ML
From:       Chuck Hirstius <chuck () hirstius ! com>
Date:       2016-10-22 19:47:18
Message-ID: CAE90dUkrH1HMBfYoc+-rkm6_dLfDqb7kaNjkMNTLg4a_Jai37Q () mail ! gmail ! com
[Download RAW message or body]

[Attachment #2 (multipart/alternative)]

Hello all,

I'm working on a project where I am hoping to teach a general ML system,
how to play games visually. As most of the active/powerful ML systems seems
to be written in/for Python, I am trying to use Python as the basis for the
project. I'm fairly new to it, but I am very new to PyQt (Qt in general).

Most examples of this kind of learning are around Pong, or other Atari-type
games, and there are tools that help you run those games and feed the
frames in. I would like to move beyond that and train the system on modern,
web-based games. One example is Slither.io

I've been able to embed a WebView and get it working, but I've stumbled on
trying to capture the output of the WebView at a high sample rate (ideally
30-60FPS). Every method I've been able to make work so far, involves
"grabbing" the screen, and while these seemingly don't appear to be CPU
blocked, the best I've been able to get has been ~15-17FPS. Here's an
example:

https://www.dropbox.com/s/1gbb82aps8j2zw9/slither.gif?raw=1

Though, the framerate of the gif is limited itself, the webview is
perfectly smooth in practice, and the transformed, scaled image, is ok,
better than previous attempts that got 8-10fps, but not ideal.

Another limitation with the current method is that the source windows needs
to be visible and positioned exactly right, to be captured. I'd love to be
able to render off-screen to a framebuffer (I'm on Windows) and just
capture from there, and display if/when I need to only, but again, I don't
seem to be able to work out how to do that successfully with python and Qt
- despite C++ examples that seem to show it can be done.

I've also had no success with using render() from the webview, every time I
try, it just crashes the app immediately, with no errors logged, so I don't
really know what I'm doing wrong there.

I'm happy to post my (awful) code, but generally, does anyone have
experience doing anything like this who can offer some guidance or methods
for efficiently capturing the output of a webview so that it can be
scaled/transformed?

Thanks!

[Attachment #5 (text/html)]

<div dir="ltr">Hello all,<div><br></div><div>I&#39;m working on a project where I am \
hoping to teach a general ML system, how to play games visually. As most of the \
active/powerful ML systems seems to be written in/for Python, I am trying to use \
Python as the basis for the project. I&#39;m fairly new to it, but I am very new to \
PyQt (Qt in general).</div><div><br></div><div>Most examples of this kind of learning \
are around Pong, or other Atari-type games, and there are tools that help you run \
those games and feed the frames in. I would like to move beyond that and train the \
system on modern, web-based games. One example is \
Slither.io</div><div><br></div><div>I&#39;ve been able to embed a WebView and get it \
working, but I&#39;ve stumbled on trying to capture the output of the WebView at a \
high sample rate (ideally 30-60FPS). Every method I&#39;ve been able to make work so \
far, involves &quot;grabbing&quot; the screen, and while these seemingly don&#39;t \
appear to be CPU blocked, the best I&#39;ve been able to get has been ~15-17FPS. \
Here&#39;s an example:<br><br><a \
href="https://www.dropbox.com/s/1gbb82aps8j2zw9/slither.gif?raw=1">https://www.dropbox \
.com/s/1gbb82aps8j2zw9/slither.gif?raw=1</a><br></div><div><br></div><div>Though, the \
framerate of the gif is limited itself, the webview is perfectly smooth in practice, \
and the transformed, scaled image, is ok, better than previous attempts that got \
8-10fps, but not ideal.</div><div><br></div><div>Another limitation with the current \
method is that the source windows needs to be visible and positioned exactly right, \
to be captured. I&#39;d love to be able to render off-screen to a framebuffer \
(I&#39;m on Windows) and just capture from there, and display if/when I need to only, \
but again, I don&#39;t seem to be able to work out how to do that successfully with \
python and Qt - despite C++ examples that seem to show it can be \
done.</div><div><br></div><div>I&#39;ve also had no success with using render() from \
the webview, every time I try, it just crashes the app immediately, with no errors \
logged, so I don&#39;t really know what I&#39;m doing wrong \
there.</div><div><br></div><div>I&#39;m happy to post my (awful) code, but generally, \
does anyone have experience doing anything like this who can offer some guidance or \
methods for efficiently capturing the output of a webview so that it can be \
scaled/transformed?</div><div><br></div><div>Thanks!</div></div>

[Attachment #6 (text/plain)]

_______________________________________________
PyQt mailing list    PyQt@riverbankcomputing.com
https://www.riverbankcomputing.com/mailman/listinfo/pyqt

[prev in list] [next in list] [prev in thread] [next in thread] 