[prev in list] [next in list] [prev in thread] [next in thread]
List: wekalist
Subject: [Wekalist] Re: Help: Nifti Loader
From: Leonel Rangel Jimenez <leonel.rangeljime () gmail ! com>
Date: 2020-05-19 14:59:16
Message-ID: CAAxw3pqWR6-tZcecFcBaY3Wc_NahypquurzCtwVdajRWyzux=g () mail ! gmail ! com
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
Hello again Eibe!
Thanks again for your correspondence, using the heap memory command I was
able to open the .nii.gz files well and started building the classifier
combination as described in the paper. I attached the class in this email,
which takes a Bayes Net, a Naive Bayes, and a Metacost/CostSensitive
Classifier (with a Bayes Net base) and combines them using a meta.vote
method to combine them.
So far everything seems to work fine in the preprocessing, but when I try
to run the classifier, I get the error that meta.vote cannot handle numeric
classes, so in the preprocessing tab, I selected the filter for
numerictonominal since voting accepts nominal classes, however, it seems to
be extremely expensive and exceeds the 8 GB heap memory I've implemented.
Might you have any suggestions on converting to nominal attributes that
doesn't crash the system?
Best,
Leonel Rangel Jimenez
On Thu, May 14, 2020 at 7:44 PM Eibe Frank <eibe@waikato.ac.nz> wrote:
> I have downloaded UKBiobank_BrainImaging_GroupMeanTemplates, which seems
> to be the data associated with that paper, and it has a number of .nii.gz
> files that can be loaded with the NIfTI loader. Unless you reduce the size
> of the data somehow when you create the CSV files, using CSV will probably
> not be faster (but this is just a guess).
>
> You may want to uninstall (or temporarily prevent loading) of any WEKA
> packages that you do not need. For example, the RPlugin can struggle with
> very high-dimensional data.
>
> The fMRI.nii.gz file in that collection of datasets took a few minutes to
> load with the NIfTIFileLoader on my iMac.
>
> If you run out of heap space (i.e., you get an out-of-memory error), you
> need to increase the heap space available to the Java virtual machine. One
> way to do this is to set the
>
> _JAVA_OPTIONS
>
> environment variable on your computer to provide the appropriate setting
> for Java. For example, give it the value
>
> -Xmx8g
>
> to give Java 8 GB of heap memory. Just make sure that you have a bit more
> than that as actual RAM on your computer.
>
> You will need to restart WEKA for this variable to be read. Also, on Macs,
> you will need to look up how to set environment variables globally for app
> applications.
>
> Cheers,
> Eibe
>
> > On 15/05/2020, at 11:03 AM, Leonel Rangel Jimenez <
> leonel.rangeljime@gmail.com> wrote:
> >
> > Thanks for the quick response and thorough explanation! I'm currently
> looking through all the source "s you provided and perhaps due to the
> inefficiency of the nifti loader it might be best for me to try the CSV
> method you mentioned. Would you happen to have a good source on converting
> a Nifti Directory to a CSV file?
> >
> > On Wed, May 13, 2020 at 7:52 PM Eibe Frank <eibe@waikato.ac.nz> wrote:
> > I had a quick look at that paper. Are you sure they used the NIfTI
> loaders for WEKA? Perhaps they processed the data into a CSV file and then
> loaded that into WEKA?
> >
> > The NIfTIDirectoryLoader assumes that you have several directories
> corresponding to different classes of observations. It works the same way
> as the TextDirectoryLoader in WEKA. So, if you wanted to classify brains
> into "diseased" and "healthy", you would make one sub-directory (i.e.,
> sub-folder) containing the data of healthy brains and another one
> containing the data of diseased brains. Each directory would contain a
> collection of .nii files ("observations" to be classified). The directory
> containing these two sub-directories would be what you would specify as the
> argument for NIfTIDirectoryLoader.
> >
> > However, this loader turns out to be quite inefficient. Alternatively,
> there is also a NIfTIFileLoader that loads a single .nii file with all the
> necessary information.
> >
> > The package directory for niftiLoader
> ($WEKA_HOME/wekafiles/packages/niftiLoader) includes an example script
> (both as a Groovy script and as a BASH script) that downloads example fMRI
> data and runs a cross-validation experiment on it. Using web access to the
> WEKA SVN server, you can also find these scripts here:
> >
> >
> https://svn.cms.waikato.ac.nz/svn/weka/trunk/packages/internal/niftiLoader/example-groovy-scripts/classifyPymvpaExampledata.groovy
>
> >
> https://svn.cms.waikato.ac.nz/svn/weka/trunk/packages/internal/niftiLoader/example-shell-scripts/classify-pymvpa-example-data.sh
>
> >
> > There is also a trivial plotting example based on data of a single class:
> >
> >
> https://svn.cms.waikato.ac.nz/svn/weka/trunk/packages/internal/niftiLoader/example-groovy-scripts/PlotFace.groovy
>
> >
> > The data for this plotting example is here:
> >
> >
> https://svn.cms.waikato.ac.nz/svn/weka/trunk/packages/internal/niftiLoader/example_data/
>
> >
> > The Javadoc for the package is here:
> >
> > https://weka.sourceforge.io/doc.packages/niftiLoader/
> >
> > The above scripts should be fairly readable, but let me know if you have
> any further questions. I'm happy to help with this if I can.
> >
> > Cheers,
> > Eibe
> >
> > > On 14/05/2020, at 9:50 AM, Leonel Rangel Jimenez <
> leonel.rangeljime@gmail.com> wrote:
> > >
> > > I am new to machine learning and am looking for some assistance in
> working with Weka and MRI T1 datasets. I found the Weka Nifti Loader
> package and installed it, hoping to be able to load several T1s at once and
> then combine three classifiers for the purpose of having a way to
> automatically run QC on these files as seen in the following paper:
> > >
> > > Alfaro-Almagro F, Jenkinson M, Bangerter NK, et al. Image processing
> and Quality Control for the first 10,000 brain imaging datasets from UK
> Biobank. NeuroImage. 2018;166:400-424. doi:10.1016/j.neuroimage.2017.10.034
> > >
> > > However, the paper has very little documentation on the use of Weka, so
> upon trying to load the directory (in which I have ~100 T1s), I got an
> error stating that 'no such file or directory exists' despite it being
> easily found in my Mac's Finder. So Far, I have only attempted this within
> the Weka UI since I have no experience in using the terminal commands for
> Weka. I also attempted to work with a single .nii.gz file but got the
> OutOfMemory error.
> > >
> > > Organization of the Nifti files is also a little confusing given the
> package description that 'The directory to be loaded must contain as many
> subdirectories as there are classes of MRI data.' Not entirely sure what is
> meant by this, so how should the files be organized?
> > >
> > > Any suggestions for how to load the files? Or are the
> number/organization of files and size the issue here?
> > >
> > > If anyone has an example directory to run trials with, I would greatly
> appreciate it. Please let me know if I can provide anymore information
> regarding this.
> > >
> > >
> > >
> > >
> > > _______________________________________________
> > > Wekalist mailing list -- wekalist@list.waikato.ac.nz
> > > Send posts to wekalist@list.waikato.ac.nz
> > > To unsubscribe send an email to wekalist-leave@list.waikato.ac.nz
> > > To subscribe, unsubscribe, etc., visit
> https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
> > > List etiquette:
> http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
> >
> > _______________________________________________
> > Wekalist mailing list -- wekalist@list.waikato.ac.nz
> > Send posts to wekalist@list.waikato.ac.nz
> > To unsubscribe send an email to wekalist-leave@list.waikato.ac.nz
> > To subscribe, unsubscribe, etc., visit
> https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
> > List etiquette:
> http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
> > --
> > Leonel Rangel Jimenez
> > _______________________________________________
> > Wekalist mailing list -- wekalist@list.waikato.ac.nz
> > Send posts to wekalist@list.waikato.ac.nz
> > To unsubscribe send an email to wekalist-leave@list.waikato.ac.nz
> > To subscribe, unsubscribe, etc., visit
> https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
> > List etiquette:
> http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
> _______________________________________________
> Wekalist mailing list -- wekalist@list.waikato.ac.nz
> Send posts to wekalist@list.waikato.ac.nz
> To unsubscribe send an email to wekalist-leave@list.waikato.ac.nz
> To subscribe, unsubscribe, etc., visit
> https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz
> List etiquette:
> http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
>
[Attachment #5 (text/html)]
<div dir="ltr">Hello again Eibe!<div><br></div><div>Thanks again for your \
correspondence, using the heap memory command I was able to open the .nii.gz files \
well and started building the classifier combination as described in the paper. I \
attached the class in this email, which takes a Bayes Net, a Naive Bayes, and a \
Metacost/CostSensitive Classifier (with a Bayes Net base) and combines them using a \
<a href="http://meta.vote">meta.vote</a> method to combine them. \
</div><div><br></div><div>So far everything seems to work fine in the preprocessing, \
but when I try to run the classifier, I get the error that <a \
href="http://meta.vote">meta.vote</a> cannot handle numeric classes, so in the \
preprocessing tab, I selected the filter for numerictonominal since voting accepts \
nominal classes, however, it seems to be extremely expensive and exceeds the 8 GB \
heap memory I've implemented. Might you have any suggestions on converting to \
nominal attributes that doesn't crash the \
system?</div><div><br></div><div>Best,</div><div><br clear="all"><div><div dir="ltr" \
class="gmail_signature" data-smartmail="gmail_signature">Leonel Rangel \
Jimenez<br><br><br><br></div></div><br></div></div><br><div class="gmail_quote"><div \
dir="ltr" class="gmail_attr">On Thu, May 14, 2020 at 7:44 PM Eibe Frank <<a \
href="mailto:eibe@waikato.ac.nz">eibe@waikato.ac.nz</a>> \
wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">I have downloaded \
UKBiobank_BrainImaging_GroupMeanTemplates, which seems to be the data associated with \
that paper, and it has a number of .nii.gz files that can be loaded with the NIfTI \
loader. Unless you reduce the size of the data somehow when you create the CSV files, \
using CSV will probably not be faster (but this is just a guess).<br> <br>
You may want to uninstall (or temporarily prevent loading) of any WEKA packages that \
you do not need. For example, the RPlugin can struggle with very high-dimensional \
data.<br> <br>
The fMRI.nii.gz file in that collection of datasets took a few minutes to load with \
the NIfTIFileLoader on my iMac.<br> <br>
If you run out of heap space (i.e., you get an out-of-memory error), you need to \
increase the heap space available to the Java virtual machine. One way to do this is \
to set the<br> <br>
_JAVA_OPTIONS<br>
<br>
environment variable on your computer to provide the appropriate setting for Java. \
For example, give it the value<br> <br>
-Xmx8g<br>
<br>
to give Java 8 GB of heap memory. Just make sure that you have a bit more than that \
as actual RAM on your computer. <br> <br>
You will need to restart WEKA for this variable to be read. Also, on Macs, you will \
need to look up how to set environment variables globally for app applications.<br> \
<br> Cheers,<br>
Eibe<br>
<br>
> On 15/05/2020, at 11:03 AM, Leonel Rangel Jimenez <<a \
href="mailto:leonel.rangeljime@gmail.com" \
target="_blank">leonel.rangeljime@gmail.com</a>> wrote:<br> > <br>
> Thanks for the quick response and thorough explanation! I'm currently looking \
through all the source "s you provided and perhaps due to the inefficiency of the \
nifti loader it might be best for me to try the CSV method you mentioned. Would you \
happen to have a good source on converting a Nifti Directory to a CSV file?<br> > \
<br> > On Wed, May 13, 2020 at 7:52 PM Eibe Frank <<a \
href="mailto:eibe@waikato.ac.nz" target="_blank">eibe@waikato.ac.nz</a>> \
wrote:<br> > I had a quick look at that paper. Are you sure they used the NIfTI \
loaders for WEKA? Perhaps they processed the data into a CSV file and then loaded \
that into WEKA?<br> > <br>
> The NIfTIDirectoryLoader assumes that you have several directories corresponding \
to different classes of observations. It works the same way as the \
TextDirectoryLoader in WEKA. So, if you wanted to classify brains into "diseased" and \
"healthy", you would make one sub-directory (i.e., sub-folder) containing the data of \
healthy brains and another one containing the data of diseased brains. Each directory \
would contain a collection of .nii files ("observations" to be classified). The \
directory containing these two sub-directories would be what you would specify as the \
argument for NIfTIDirectoryLoader.<br> > <br>
> However, this loader turns out to be quite inefficient. Alternatively, there is \
also a NIfTIFileLoader that loads a single .nii file with all the necessary \
information.<br> > <br>
> The package directory for niftiLoader \
($WEKA_HOME/wekafiles/packages/niftiLoader) includes an example script (both as a \
Groovy script and as a BASH script) that downloads example fMRI data and runs a \
cross-validation experiment on it. Using web access to the WEKA SVN server, you can \
also find these scripts here:<br> > <br>
> <a href="https://svn.cms.waikato.ac.nz/svn/weka/trunk/packages/internal/niftiLoader/example-groovy-scripts/classifyPymvpaExampledata.groovy" \
rel="noreferrer" target="_blank">https://svn.cms.waikato.ac.nz/svn/weka/trunk/packages \
/internal/niftiLoader/example-groovy-scripts/classifyPymvpaExampledata.groovy</a><br> \
> <a href="https://svn.cms.waikato.ac.nz/svn/weka/trunk/packages/internal/niftiLoader/example-shell-scripts/classify-pymvpa-example-data.sh" \
rel="noreferrer" target="_blank">https://svn.cms.waikato.ac.nz/svn/weka/trunk/packages \
/internal/niftiLoader/example-shell-scripts/classify-pymvpa-example-data.sh</a><br> \
> <br> > There is also a trivial plotting example based on data of a single \
class:<br> > <br>
> <a href="https://svn.cms.waikato.ac.nz/svn/weka/trunk/packages/internal/niftiLoader/example-groovy-scripts/PlotFace.groovy" \
rel="noreferrer" target="_blank">https://svn.cms.waikato.ac.nz/svn/weka/trunk/packages/internal/niftiLoader/example-groovy-scripts/PlotFace.groovy</a><br>
> <br>
> The data for this plotting example is here:<br>
> <br>
> <a href="https://svn.cms.waikato.ac.nz/svn/weka/trunk/packages/internal/niftiLoader/example_data/" \
rel="noreferrer" target="_blank">https://svn.cms.waikato.ac.nz/svn/weka/trunk/packages/internal/niftiLoader/example_data/</a><br>
> <br>
> The Javadoc for the package is here:<br>
> <br>
> <a href="https://weka.sourceforge.io/doc.packages/niftiLoader/" rel="noreferrer" \
target="_blank">https://weka.sourceforge.io/doc.packages/niftiLoader/</a><br> > \
<br> > The above scripts should be fairly readable, but let me know if you have \
any further questions. I'm happy to help with this if I can. <br> > <br>
> Cheers,<br>
> Eibe<br>
> <br>
>> On 14/05/2020, at 9:50 AM, Leonel Rangel Jimenez <<a \
href="mailto:leonel.rangeljime@gmail.com" \
target="_blank">leonel.rangeljime@gmail.com</a>> wrote:<br> >> <br>
>> I am new to machine learning and am looking for some assistance in working \
with Weka and MRI T1 datasets. I found the Weka Nifti Loader package and installed \
it, hoping to be able to load several T1s at once and then combine three classifiers \
for the purpose of having a way to automatically run QC on these files as seen in the \
following paper:<br> >> <br>
>> Alfaro-Almagro F, Jenkinson M, Bangerter NK, et al. Image processing and \
Quality Control for the first 10,000 brain imaging datasets from UK Biobank. \
NeuroImage. 2018;166:400-424. doi:10.1016/j.neuroimage.2017.10.034<br> >> <br>
>> However, the paper has very little documentation on the use of Weka, so upon \
trying to load the directory (in which I have ~100 T1s), I got an error stating that \
'no such file or directory exists' despite it being easily found in my \
Mac's Finder. So Far, I have only attempted this within the Weka UI since I have \
no experience in using the terminal commands for Weka. I also attempted to work with \
a single .nii.gz file but got the OutOfMemory error.<br> >> <br>
>> Organization of the Nifti files is also a little confusing given the package \
description that 'The directory to be loaded must contain as many subdirectories \
as there are classes of MRI data.' Not entirely sure what is meant by this, so \
how should the files be organized?<br> >> <br>
>> Any suggestions for how to load the files? Or are the number/organization of \
files and size the issue here?<br> >> <br>
>> If anyone has an example directory to run trials with, I would greatly \
appreciate it. Please let me know if I can provide anymore information regarding \
this.<br> >> <br>
>> <br>
>> <br>
>> <br>
>> _______________________________________________<br>
>> Wekalist mailing list -- <a href="mailto:wekalist@list.waikato.ac.nz" \
target="_blank">wekalist@list.waikato.ac.nz</a><br> >> Send posts to <a \
href="mailto:wekalist@list.waikato.ac.nz" \
target="_blank">wekalist@list.waikato.ac.nz</a><br> >> To unsubscribe send an \
email to <a href="mailto:wekalist-leave@list.waikato.ac.nz" \
target="_blank">wekalist-leave@list.waikato.ac.nz</a><br> >> To subscribe, \
unsubscribe, etc., visit <a \
href="https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz" \
rel="noreferrer" target="_blank">https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz</a><br>
>> List etiquette: <a \
href="http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html" \
rel="noreferrer" target="_blank">http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html</a><br>
> <br>
> _______________________________________________<br>
> Wekalist mailing list -- <a href="mailto:wekalist@list.waikato.ac.nz" \
target="_blank">wekalist@list.waikato.ac.nz</a><br> > Send posts to <a \
href="mailto:wekalist@list.waikato.ac.nz" \
target="_blank">wekalist@list.waikato.ac.nz</a><br> > To unsubscribe send an email \
to <a href="mailto:wekalist-leave@list.waikato.ac.nz" \
target="_blank">wekalist-leave@list.waikato.ac.nz</a><br> > To subscribe, \
unsubscribe, etc., visit <a \
href="https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz" \
rel="noreferrer" target="_blank">https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz</a><br>
> List etiquette: <a \
href="http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html" \
rel="noreferrer" target="_blank">http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html</a><br>
> -- <br>
> Leonel Rangel Jimenez<br>
> _______________________________________________<br>
> Wekalist mailing list -- <a href="mailto:wekalist@list.waikato.ac.nz" \
target="_blank">wekalist@list.waikato.ac.nz</a><br> > Send posts to <a \
href="mailto:wekalist@list.waikato.ac.nz" \
target="_blank">wekalist@list.waikato.ac.nz</a><br> > To unsubscribe send an email \
to <a href="mailto:wekalist-leave@list.waikato.ac.nz" \
target="_blank">wekalist-leave@list.waikato.ac.nz</a><br> > To subscribe, \
unsubscribe, etc., visit <a \
href="https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz" \
rel="noreferrer" target="_blank">https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz</a><br>
> List etiquette: <a \
href="http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html" \
rel="noreferrer" target="_blank">http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html</a><br>
_______________________________________________<br>
Wekalist mailing list -- <a href="mailto:wekalist@list.waikato.ac.nz" \
target="_blank">wekalist@list.waikato.ac.nz</a><br> Send posts to <a \
href="mailto:wekalist@list.waikato.ac.nz" \
target="_blank">wekalist@list.waikato.ac.nz</a><br> To unsubscribe send an email to \
<a href="mailto:wekalist-leave@list.waikato.ac.nz" \
target="_blank">wekalist-leave@list.waikato.ac.nz</a><br> To subscribe, unsubscribe, \
etc., visit <a href="https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz" \
rel="noreferrer" target="_blank">https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz</a><br>
List etiquette: <a href="http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html" \
rel="noreferrer" target="_blank">http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html</a><br>
</blockquote></div>
["classifiercombo" (application/octet-stream)]
_______________________________________________
Wekalist mailing list -- wekalist@list.waikato.ac.nz
Send posts to wekalist@list.waikato.ac.nz
To unsubscribe send an email to wekalist-leave@list.waikato.ac.nz
To subscribe, unsubscribe, etc., visit \
https://list.waikato.ac.nz/postorius/lists/wekalist.list.waikato.ac.nz List \
etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic