[prev in list] [next in list] [prev in thread] [next in thread]
List: python-distutils-sig
Subject: Re: [Distutils] Fwd: How to get pip to really, really, I mean it -- rebuild this damn package!
From: Chris Barker <chris.barker () noaa ! gov>
Date: 2016-01-29 23:56:01
Message-ID: CALGmxE+d8Ln_SQWbu13xUxhdYEAfJtLXc8QWOrtNjGLxh3GMCQ () mail ! gmail ! com
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
On Fri, Jan 29, 2016 at 12:25 PM, Robert Collins <robertc@robertcollins.net>
wrote:
> Please try pip 7.1? Latest before 8; we're not meant to be caching
> wheels of by-location things from my memory, but it may have
> regressed/changed with the cache changes made during the 8 development
> cycle.
>
indeed this is looking like a regression. I have done:
* create 2 fresh conda environments
* downgrade pip to 7.1.2 in both (conda seems to put the latest pip in a
new environment by default)
$ pip --version
pip 7.1.2 from
/Users/chris.barker/miniconda2/envs/test2/lib/python2.7/site-packages
(python 2.7)
* build and install the package in one of them:
$ pip install ./
Processing /Users/chris.barker/Temp/geojson-1.3.2
Requirement already satisfied (use --upgrade to upgrade): setuptools in
/Users/chris.barker/miniconda2/envs/test1/lib/python2.7/site-packages/setuptools-19.4-py2.7.egg
(from geojson==1.3.2)
Building wheels for collected packages: geojson
Running setup.py bdist_wheel for geojson
Stored in directory:
/Users/chris.barker/Library/Caches/pip/wheels/67/47/d8/01cf2332293b60900ec87ff03bbe9fff92bc85f194e0eb0e74
Successfully built geojson
Installing collected packages: geojson
Successfully installed geojson-1.3.2
You are using pip version 7.1.2, however version 8.0.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
It built and installed fine:
(test1)orrw-m-4179497:geojson-1.3.2 chris.barker$ python -c "import geojson"
(test1)orrw-m-4179497:geojson-1.3.2 chris.barker$
Now I go to install in the other environment:
first, check if geojson is there:
(test2)orrw-m-4179497:Temp chris.barker$ python -c "import geojson"
Traceback (most recent call last):
File "<string>", line 1, in <module>
ImportError: No module named geojson
nope.
so use pip to install from source
(test2)orrw-m-4179497:geojson-1.3.2 chris.barker$ pip install ./
Processing /Users/chris.barker/Temp/geojson-1.3.2
Requirement already satisfied (use --upgrade to upgrade): setuptools in
/Users/chris.barker/miniconda2/envs/test2/lib/python2.7/site-packages/setuptools-19.4-py2.7.egg
(from geojson==1.3.2)
Building wheels for collected packages: geojson
Running setup.py bdist_wheel for geojson
Stored in directory:
/Users/chris.barker/Library/Caches/pip/wheels/67/47/d8/01cf2332293b60900ec87ff03bbe9fff92bc85f194e0eb0e74
Successfully built geojson
Installing collected packages: geojson
Successfully installed geojson-1.3.2
You are using pip version 7.1.2, however version 8.0.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
All good. See if it works:
(test2)orrw-m-4179497:Temp chris.barker$ python -c "import geojson; print
geojson.__version__"
1.3.2
(test2)orrw-m-4179497:Temp chris.barker$
yup -- all good.
However -- AARRGG! I then repeated that same exercise with pip 8.0.1, which
was failing for me before (multiple times) but it jsut worked!
This is really, really frustrating...did the downgrade, then re-upgrade of
pip somehow fix this? really, really weird.
OK -- well, I'm done for now with pip for building conda packages....
-CHB
>
> -Rob
>
> On 30 January 2016 at 04:48, Chris Barker - NOAA Federal
> <chris.barker@noaa.gov> wrote:
> >>> Requirement already satisfied (use --upgrade to upgrade): gsw==3.0.3
> from
> >>> file:///Users/chris.barker/miniconda2/conda-bld/work/gsw-3.0.3 in
> >>> /Users/chris.barker/miniconda2/conda-bld/work/gsw-3.0.3
> >>
> >> I think this is saying that pip thinks it has found an
> >> already-installed version of gsw 3.0.3 in sys.path, and that the
> >> directory in your sys.path where it's already installed is
> >>
> >> /Users/chris.barker/miniconda2/conda-bld/work/gsw-3.0.3
> >
> > That is the temp dir conda sets up to unpack downloaded files, and do
> > its work in -- hence the name. I'll look and see what's there. I'm
> > pretty sure conda build starts out with an empty dir, however. And
> > that dir should not be on sys.path.
> >
> >> I think this means that that directory is (a) in sys.path, and (b)
> >> contains a .egg-info/.dist-info directory for gsw 3.0.3. Part (a)
> >> seems weird and broken.
> >
> > Indeed. And I get the same symptoms with a clean environment that I've
> > set up outside conda build. Though with the same source dir. But with
> > conda build, it's a fresh unpack of the tarball.
> >
> >> Do you have "." in your PYTHONPATH or anything like that?
> >
> > God no!
> >
> >> Don't know why it seems to be building a wheel for it, if it already
> >> thinks that it's installed... this is also odd.
> >
> > Yes it is. But it doesn't install it :-(
> >
> >>
> >> $PYTHON -m pip install --no-cache-dir --upgrade --force-reinstall ./
> >>
> >> ? Though I'd think that -I would have the same affect as
> --force-reinstall...
> >>
> > So did I, and I think I tried --force-reinstall already, but I will
> again.
> >
> >> (It doesn't look like the cache dir is your problem here, but you do
> >> probably want to use --no-cache-dir anyway just as good practice, just
> >> because you don't want to accidentally package up a stale version of
> >> the software that got pulled out of your cache instead of the version
> >> you thought you were packaging in the tree in front of you.
> >
> > Exactly. Doesn't seem to make a difference, though.
> >
> >> Also, I think it's a bug in pip that it caches builds of source trees
> >> -- PyPI can enforce the rule that each (package name, version number)
> >> sdist is unique, but for a work-in-progress VCS checkout it's just not
> >> true that (package name, version number) uniquely identifies a
> >> snapshot of the whole tree. So in something like 'pip install .', then
> >> requirement resolution code should treat this as a special requirement
> >> that it wants *this exact tree*, not just any package that has the
> >> same (package name, version number) as this tree; and the resulting
> >> wheel should not be cached.
> >
> > Absolutely! In fact, I'll bet that approach is the source of the
> > problem here. If not automagically, there should be a flag, at least.
> >
> > However, what seems to be happening is that pip is looking outside the
> > current Python environment somewhere to see if this package needs to
> > be installed. It may be something that works with virtualenv, but
> > doesn't with conda environments for some reason.
> >
> > I guess on some level pip simply isn't designed to build and install
> > from local source :-(
> >
> > In the end, I'm still confused: does pip install give me anything that:
> >
> > setup.py install single-version-externally-managed
> >
> > Doesn't? Other that support for non-setuptools installs, anyway.
> >
> > CHB
> >
> >
> >> I don't know if there are any bugs filed
> >> in pip on this...)
> >>
> >> -n
> >>
> >> --
> >> Nathaniel J. Smith -- https://vorpus.org
> > _______________________________________________
> > Distutils-SIG maillist - Distutils-SIG@python.org
> > https://mail.python.org/mailman/listinfo/distutils-sig
>
>
>
> --
> Robert Collins <rbtcollins@hpe.com>
> Distinguished Technologist
> HP Converged Cloud
>
>
> --
> Robert Collins <rbtcollins@hpe.com>
> Distinguished Technologist
> HP Converged Cloud
> _______________________________________________
> Distutils-SIG maillist - Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
--
Christopher Barker, Ph.D.
Oceanographer
Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception
Chris.Barker@noaa.gov
[Attachment #5 (text/html)]
<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Fri, Jan 29, 2016 \
at 12:25 PM, Robert Collins <span dir="ltr"><<a \
href="mailto:robertc@robertcollins.net" \
target="_blank">robertc@robertcollins.net</a>></span> wrote:<br><blockquote \
class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div \
class=""><div class="h5">Please try pip 7.1? Latest before 8; we're not meant to \
be caching<br> wheels of by-location things from my memory, but it may have<br>
regressed/changed with the cache changes made during the 8 development<br>
cycle.<br></div></div></blockquote><div><br></div><div>indeed this is looking like a \
regression. I have done:</div><div><br></div><div><br></div><div><br></div><div>* \
create 2 fresh conda environments</div><div><br></div><div>* downgrade pip to 7.1.2 \
in both (conda seems to put the latest pip in a new environment by \
default)</div><div>
<p class=""><span class="">$ pip --version</span></p>
<p class=""><span class="">pip 7.1.2 from \
/Users/chris.barker/miniconda2/envs/test2/lib/python2.7/site-packages (python \
2.7)</span></p><p class=""><span class=""></span></p><div>* build and install the \
package in one of them:</div></div><div><br></div><div>
<p class=""><span class=""><font face="monospace, monospace">$ pip install \
./</font></span></p> <p class=""><span class=""><font face="monospace, \
monospace">Processing /Users/chris.barker/Temp/geojson-1.3.2</font></span></p> <p \
class=""><span class=""><font face="monospace, monospace">Requirement already \
satisfied (use --upgrade to upgrade): setuptools in \
/Users/chris.barker/miniconda2/envs/test1/lib/python2.7/site-packages/setuptools-19.4-py2.7.egg \
(from geojson==1.3.2)</font></span></p> <p class=""><span class=""><font \
face="monospace, monospace">Building wheels for collected packages: \
geojson</font></span></p> <p class=""><span class=""><font face="monospace, \
monospace"> Running setup.py bdist_wheel for geojson</font></span></p> <p \
class=""><span class=""><font face="monospace, monospace"> Stored in directory: \
/Users/chris.barker/Library/Caches/pip/wheels/67/47/d8/01cf2332293b60900ec87ff03bbe9fff92bc85f194e0eb0e74</font></span></p>
<p class=""><span class=""><font face="monospace, monospace">Successfully built \
geojson</font></span></p> <p class=""><span class=""><font face="monospace, \
monospace">Installing collected packages: geojson</font></span></p> <p class=""><span \
class=""><font face="monospace, monospace">Successfully installed \
geojson-1.3.2</font></span></p> <p class=""><span class=""><font face="monospace, \
monospace">You are using pip version 7.1.2, however version 8.0.2 is \
available.</font></span></p> <p class=""><span class=""><font face="monospace, \
monospace">You should consider upgrading via the 'pip install --upgrade pip' \
command.</font></span></p></div><div>It built and installed fine:</div><div>
<p class=""><span class=""><font face="monospace, \
monospace">(test1)orrw-m-4179497:geojson-1.3.2 chris.barker$ python -c "import \
geojson"</font></span></p><p class="">
</p><p class=""><span class=""><font face="monospace, \
monospace">(test1)orrw-m-4179497:geojson-1.3.2 chris.barker$ \
</font></span></p></div><div>Now I go to install in the other \
environment:</div><div><br></div><div>first, check if geojson is there:</div><div>
<p class=""><span class=""><font face="monospace, \
monospace">(test2)orrw-m-4179497:Temp chris.barker$ python -c "import \
geojson"</font></span></p> <p class=""><span class=""><font face="monospace, \
monospace">Traceback (most recent call last):</font></span></p> <p class=""><span \
class=""><font face="monospace, monospace"> File "<string>", line \
1, in <module></font></span></p> <p class=""><span class=""><font \
face="monospace, monospace">ImportError: No module named geojson</font></span></p><p \
class=""><span class="">nope.</span></p><p class=""><span class="">so use pip to \
install from source</span></p><p class=""><span class=""><font face="monospace, \
monospace">(test2)orrw-m-4179497:geojson-1.3.2 chris.barker$ pip install \
./</font></span></p><p class=""><span class=""><font face="monospace, \
monospace">Processing /Users/chris.barker/Temp/geojson-1.3.2</font></span></p><p \
class=""><span class=""><font face="monospace, monospace">Requirement already \
satisfied (use --upgrade to upgrade): setuptools in \
/Users/chris.barker/miniconda2/envs/test2/lib/python2.7/site-packages/setuptools-19.4-py2.7.egg \
(from geojson==1.3.2)</font></span></p><p class=""><span class=""><font \
face="monospace, monospace">Building wheels for collected packages: \
geojson</font></span></p><p class=""><span class=""><font face="monospace, \
monospace"> Running setup.py bdist_wheel for geojson</font></span></p><p \
class=""><span class=""><font face="monospace, monospace"> Stored in directory: \
/Users/chris.barker/Library/Caches/pip/wheels/67/47/d8/01cf2332293b60900ec87ff03bbe9fff92bc85f194e0eb0e74</font></span></p><p \
class=""><span class=""><font face="monospace, monospace">Successfully built \
geojson</font></span></p><p class=""><span class=""><font face="monospace, \
monospace">Installing collected packages: geojson</font></span></p><p class=""><span \
class=""><font face="monospace, monospace">Successfully installed \
geojson-1.3.2</font></span></p><p class=""><span class=""><font face="monospace, \
monospace">You are using pip version 7.1.2, however version 8.0.2 is \
available.</font></span></p><p class=""><span class=""><font face="monospace, \
monospace">
</font></span></p><p class=""><span class=""><font face="monospace, monospace">You \
should consider upgrading via the 'pip install --upgrade pip' \
command.</font></span></p><p class=""><span class="">All good. See if it \
works:</span></p><p class=""><span class=""><font face="monospace, \
monospace">(test2)orrw-m-4179497:Temp chris.barker$ python -c "import geojson; \
print geojson.__version__"</font></span></p><p class=""><span class=""><font \
face="monospace, monospace">1.3.2</font></span></p><p class=""><span class=""><font \
face="monospace, monospace">
</font></span></p><p class=""><span class=""><font face="monospace, \
monospace">(test2)orrw-m-4179497:Temp chris.barker$ </font></span></p></div><div>yup \
-- all good.</div><div><br></div><div>However -- AARRGG! I then repeated that same \
exercise with pip 8.0.1, which was failing for me before (multiple times) but it jsut \
worked!</div><div><br></div><div>This is really, really frustrating...did the \
downgrade, then re-upgrade of pip somehow fix this? really, really \
weird.</div><div><br></div><div>OK -- well, I'm done for now with pip for \
building conda packages....</div><div><br></div><div>-CHB</div><div><br></div><div><br \
></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div> \
> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px \
> 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div \
> class=""><div class="h5">
<br>
-Rob<br>
<br>
On 30 January 2016 at 04:48, Chris Barker - NOAA Federal<br>
<<a href="mailto:chris.barker@noaa.gov">chris.barker@noaa.gov</a>> wrote:<br>
>>> Requirement already satisfied (use --upgrade to upgrade): gsw==3.0.3 \
from<br> >>> file:///Users/chris.barker/miniconda2/conda-bld/work/gsw-3.0.3 \
in<br> >>> /Users/chris.barker/miniconda2/conda-bld/work/gsw-3.0.3<br>
>><br>
>> I think this is saying that pip thinks it has found an<br>
>> already-installed version of gsw 3.0.3 in sys.path, and that the<br>
>> directory in your sys.path where it's already installed is<br>
>><br>
>> /Users/chris.barker/miniconda2/conda-bld/work/gsw-3.0.3<br>
><br>
> That is the temp dir conda sets up to unpack downloaded files, and do<br>
> its work in -- hence the name. I'll look and see what's there. \
I'm<br> > pretty sure conda build starts out with an empty dir, however. \
And<br> > that dir should not be on sys.path.<br>
><br>
>> I think this means that that directory is (a) in sys.path, and (b)<br>
>> contains a .egg-info/.dist-info directory for gsw 3.0.3. Part (a)<br>
>> seems weird and broken.<br>
><br>
> Indeed. And I get the same symptoms with a clean environment that I've<br>
> set up outside conda build. Though with the same source dir. But with<br>
> conda build, it's a fresh unpack of the tarball.<br>
><br>
>> Do you have "." in your PYTHONPATH or anything like that?<br>
><br>
> God no!<br>
><br>
>> Don't know why it seems to be building a wheel for it, if it already<br>
>> thinks that it's installed... this is also odd.<br>
><br>
> Yes it is. But it doesn't install it :-(<br>
><br>
>><br>
>> $PYTHON -m pip install --no-cache-dir --upgrade --force-reinstall ./<br>
>><br>
>> ? Though I'd think that -I would have the same affect as \
--force-reinstall...<br> >><br>
> So did I, and I think I tried --force-reinstall already, but I will again.<br>
><br>
>> (It doesn't look like the cache dir is your problem here, but you do<br>
>> probably want to use --no-cache-dir anyway just as good practice, just<br>
>> because you don't want to accidentally package up a stale version of<br>
>> the software that got pulled out of your cache instead of the version<br>
>> you thought you were packaging in the tree in front of you.<br>
><br>
> Exactly. Doesn't seem to make a difference, though.<br>
><br>
>> Also, I think it's a bug in pip that it caches builds of source \
trees<br> >> -- PyPI can enforce the rule that each (package name, version \
number)<br> >> sdist is unique, but for a work-in-progress VCS checkout \
it's just not<br> >> true that (package name, version number) uniquely \
identifies a<br> >> snapshot of the whole tree. So in something like 'pip \
install .', then<br> >> requirement resolution code should treat this as a \
special requirement<br> >> that it wants *this exact tree*, not just any \
package that has the<br> >> same (package name, version number) as this tree; \
and the resulting<br> >> wheel should not be cached.<br>
><br>
> Absolutely! In fact, I'll bet that approach is the source of the<br>
> problem here. If not automagically, there should be a flag, at least.<br>
><br>
> However, what seems to be happening is that pip is looking outside the<br>
> current Python environment somewhere to see if this package needs to<br>
> be installed. It may be something that works with virtualenv, but<br>
> doesn't with conda environments for some reason.<br>
><br>
> I guess on some level pip simply isn't designed to build and install<br>
> from local source :-(<br>
><br>
> In the end, I'm still confused: does pip install give me anything that:<br>
><br>
> setup.py install single-version-externally-managed<br>
><br>
> Doesn't? Other that support for non-setuptools installs, anyway.<br>
><br>
> CHB<br>
><br>
><br>
>> I don't know if there are any bugs filed<br>
>> in pip on this...)<br>
>><br>
>> -n<br>
>><br>
>> --<br>
>> Nathaniel J. Smith -- <a href="https://vorpus.org" rel="noreferrer" \
target="_blank">https://vorpus.org</a><br> > \
_______________________________________________<br> > Distutils-SIG maillist - \
<a href="mailto:Distutils-SIG@python.org">Distutils-SIG@python.org</a><br> > <a \
href="https://mail.python.org/mailman/listinfo/distutils-sig" rel="noreferrer" \
target="_blank">https://mail.python.org/mailman/listinfo/distutils-sig</a><br> <br>
<br>
<br>
--<br>
Robert Collins <<a href="mailto:rbtcollins@hpe.com">rbtcollins@hpe.com</a>><br>
Distinguished Technologist<br>
HP Converged Cloud<br>
<br>
<br>
--<br>
Robert Collins <<a href="mailto:rbtcollins@hpe.com">rbtcollins@hpe.com</a>><br>
Distinguished Technologist<br>
HP Converged Cloud<br>
_______________________________________________<br>
Distutils-SIG maillist - <a \
href="mailto:Distutils-SIG@python.org">Distutils-SIG@python.org</a><br> <a \
href="https://mail.python.org/mailman/listinfo/distutils-sig" rel="noreferrer" \
target="_blank">https://mail.python.org/mailman/listinfo/distutils-sig</a><br> \
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div \
class="gmail_signature"><br>Christopher Barker, \
Ph.D.<br>Oceanographer<br><br>Emergency Response Division<br>NOAA/NOS/OR&R \
(206) 526-6959 voice<br>7600 Sand Point Way NE (206) 526-6329 \
fax<br>Seattle, WA 98115 (206) 526-6317 main reception<br><br><a \
href="mailto:Chris.Barker@noaa.gov" target="_blank">Chris.Barker@noaa.gov</a></div> \
</div></div>
_______________________________________________
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic