'Re: [Rd] Single-threaded aspect'

[prev in list] [next in list] [prev in thread] [next in thread] 

List:       r-devel
Subject:    Re: [Rd] Single-threaded aspect
From:       Dirk Eddelbuettel <edd () debian ! org>
Date:       2016-05-12 14:52:07
Message-ID: 22324.39063.896654.102426 () max ! nulle ! part
[Download RAW message or body]

On 12 May 2016 at 09:25, Charles Determan wrote:
| Thank you Simon for the detailed reply.=A0 That explains much more of wha=
t I was
| looking for from the R side.
| =

| Dirk, I'm sorry if I seem hung up on anything here but I am trying to
| understand the details.=A0 My reply about XPtr or XPtr on arma/Eigen was =
to
| confirm my understanding was correct, which it appears it was.=A0 I was n=
ot aware

I still do not think so.

Step back, have a cup of tea or two, and start with the simple and short
OpenMP examples in Rcpp itself.  They have been there for years and should
still work.  I would encourage you to work through these, maybe take notes
and possibly even submit the notes as a new short piece in the Rcpp Gallery.

| the RVector/RMatrix objects don't connect to R as I am just now familiari=
zing
| myself with the package, that explains more of my confusion.=A0 I will lo=
ok at
| doing work within the compiled code as you have suggested.

Sounds good.  OpenMP and Intel TBB (as in RcppParallel) will only become mo=
re
important as we move to more and more cores.  Working with them is not all
that obvious as you are finding out.  Let's try to work to make the
documentation better.

Dirk
 =

| Regards,
| Charles
| =

| On Thu, May 12, 2016 at 9:18 AM, Dirk Eddelbuettel <edd@debian.org> wrote:
| =

|    =

|     On 12 May 2016 at 13:11, Mark van der Loo wrote:
|     | Charles,
|     |
|     | 1. Perhaps this question is better directed at the R-help or
|     | R-pacakge-devel mailinglist.
|     |
|     | 2. It basically means that R itself can only evaluate one R express=
ion at
|     | the time.
|     |
|     | The parallel package circumvents this by starting multiple R-sessio=
ns and
|     | dividing workload.
|     |
|     | Compiled code called by R (such as C++ code through RCpp or C-code
|     through
|     | base R's interface) can execute multi-threaded code for internal
|     purposes,
|     | using e.g. openMP. A limitation is that compiled code cannot call R=
's C
|     API
|     | from multiple threads (in many cases). For example, it is not threa=
d-safe
|     | to create R-variables from multiple threads running in C. (R's vari=
able
|     | administration is such that the order of (un)making them from compi=
led
|     code
|     | matters).
| =

|     Well put.
|    =

|     Charles seems to hang himself up completely about a small detail, fai=
ling
|     to
|     see the forest for the trees.
| =

|     My answer to this problem is to only use non-R data structures. That =
is
|     what
|     RcpParallel does in the actual parallel code portions in all examples=
 --
|     types RVector and RMatrix do NOT connect back to R. There are several
|     working
|     examples.=A0 That is also what the OpenMP examples at the Rcpp Galler=
y do.
| =

|     Charles seems to be replying 'but I use XPtr' or 'I use XPtr on arma:=
:mat
|     or
|     Eigen::Matrixxd' and seems to forget that these are proxy objects to =
SEXPs.
|     XPtr just wrap the SEXP for external pointers; Arma's and Eigen's mat=
rices
|     are performant via RcppArmadillo and RcppEigen because we use R memor=
y via
|     proxies.=A0 All of that is 'too close to R' for comfort.
| =

|     So the short answer is:=A0 enter compiled code from R, set a mutex (e=
ither
|     conceptually or explicitly), _copy_ your data in to plain C++ data
|     structures
|     and go to town in parallel via OpenMP and other multithreaded approac=
hes.
|     Then collect the result, release the mutex and move back up.
| =

|     I hope this help.
| =

|     Dirk
| =

|     |
|     | Best,
|     | Mark
|     |
|     |
|     |
|     |
|     |
|     |
|     |
|     |
|     |
|     |
|     | Op do 12 mei 2016 om 14:46 schreef Charles Determan <
|     cdetermanjr@gmail.com>:
|     |
|     | > R Developers,
|     | >
|     | > Could someone help explain what it means that R is single threade=
d?=A0 I
|     am
|     | > trying to understand what is actually going on inside R when user=
s want
|     to
|     | > parallelize code.=A0 For example, using mclapply or foreach (with=
 some
|     | > backend) somehow allows users to benefit from multiple CPUs.
|     | >
|     | > Similarly there is the RcppParallel package for RMatrix/RVector
|     objects.
|     | > But none of these address the general XPtr objects in Rcpp.=A0 So=
me
|     readers
|     | > here may recognize my question on SO (
|     | >
|     | > http://stackoverflow.com/questions/37167479/
|     rcpp-parallelize-functions-that-return-xptr
|     | > )
|     | > where I was curious about parallel calls to C++/Rcpp functions th=
at
|     return
|     | > XPtr objects.=A0 I am being a little more persistent here as this
|     limitation
|     | > provides a very hard stop on the development on one of my package=
s that
|     | > heavily uses XPtr objects.=A0 It's not meant to be a criticism or
|     intended to
|     | > be rude, I just want to fully understand.
|     | >
|     | > I am willing to accept that it may be impossible currently but I =
want
|     to at
|     | > least understand why it is impossible so I can explain to future =
users
|     why
|     | > parallel functionality is not available.=A0 Which just echos my o=
riginal
|     | > question, what does it mean that R is single threaded?
|     | >
|     | > Kind Regards,
|     | > Charles
|     | >
|     | >=A0 =A0 =A0 =A0 =A0[[alternative HTML version deleted]]
|     | >
|     | > ______________________________________________
|     | > R-devel@r-project.org mailing list
|     | > https://stat.ethz.ch/mailman/listinfo/r-devel
|     | >
|     |
|     |=A0 =A0 =A0 =A0[[alternative HTML version deleted]]
|     |
|     | ______________________________________________
|     | R-devel@r-project.org mailing list
|     | https://stat.ethz.ch/mailman/listinfo/r-devel
| =

|     --
|     http://dirk.eddelbuettel.com | @eddelbuettel | edd@debian.org
| =

| =

-- =

http://dirk.eddelbuettel.com | @eddelbuettel | edd@debian.org

______________________________________________
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
[prev in list] [next in list] [prev in thread] [next in thread] 