[prev in list] [next in list] [prev in thread] [next in thread] 

List:       ivy-user
Subject:    Re: <ivy:configure> vs. <ivy:settings>
From:       David Weintraub <qazwart () gmail ! com>
Date:       2012-11-30 6:18:04
Message-ID: 1FF04785-C06F-43CA-8A02-4283A0616BC4 () gmail ! com
[Download RAW message or body]

On Nov 29, 2012, at 12:32 PM, Mitch Gitman <mgitman@gmail.com> wrote:
> As you describe, the immediate problem is that, with settings, the
> EXECUTOR_NUMBER environment variable is not being picked up. The following
> line is really just establishing a default:
> <property name="env.EXECUTOR_NUMBER" value="0" override="false"/>

That's correct. The environment variable EXECUTOR_NUMBER is set by Jenkins, If you're \
not running on Jenkins, then, EXECUTOR_NUMBER isn't set. I use that line to set it if \
it's not already set.

> For debugging's sake, I'd try leaving that out. The executor number should
> be required.
> 
> For debugging's sake, I'd also try outputting the value in the Ant build
> just prior to calling ivy:settings.
> 
> Also, I'd make sure that ivy:settings is always getting called before
> ivy:cleancache.

All three of these are true. In fact, when I first used my ivysettings.xml file, I \
set the executor number manually in my environment. When I did the build, the cache \
directory was being set correctly. At least when the <ivy:resolve> took place, and \
the correct cache was being created. I also made sure that the cache was set to \
$HOME/.ivy2/cache-o when there was no EXECUTOR_NUMBER environment variable. To me, \
this was the default way the user would run it. Everything looked great.

Suddenly builds on Jenkins started to fail and I figured that happened when parallel \
builds took place and were wiping out each other's cache. Printing a few debug \
statements before and after the <ivy:settings> task led me to realize that the cache \
wasn't being set. That's when I read through the <ivy:settings> task documentation \
and discovered that <ivy:configure> did more or less the same thing, but that the \
<ivy:configure> task did the configuration immediately instead of "when needed". \
Switching to <ivy:configure> fixed the issue.

> Also, I'd make sure that ivy:settings is always getting called before
> ivy:cleancache.

It is. My Ivy project is a subdirectory to the user's project. The developer puts in \
the lines "<property name="ivy.dir" value="${basedir}/ivy.dir"/> and <import \
file="${ivy.dir}/ivy.tasks.xml"/> into their build. The "ivy.task.xml" file calls the \
"<ivy:settings>" task for the user. It runs before any other task is called. The user \
calls <ivy:cleancache> this way:

<property name="ivy.cleancache" value="true"/>

<target name="clean">
    <if>
       <istrue value="${ivy.cleancache}"/>
       <then>
           <ivy:cleancache/>
       </then>
    </if>
    <delete>
        <fileset dir="${target.dir}"/>
    </fileset>
</target>

The user can override the value of ${ivy.cleancache} via a build.properties file, so \
the cache isn't cleaned every time the developer runs the "clean" target. Thus, \
<ivy:settings/> is called first before <ivy:cleancache/>.

> It appears that your resolution cache is not being defined on a
> per-executor basis. If each executor build has its own target directory,
> that's OK. But then, why not do that for both your resolution cache and
> your repository cache? So you'd specify resolutionCacheDir and
> repositoryCacheDir attributes (no defaultCacheDir) in a consistent way.

The resolution cache needs to be defined so that parallel builds don't interfere with \
each other. There can also be an issue of a developer using multiple working \
directories of the same project, and this is also an issue with Jenkins too. The \
resolution cache gets built pretty quickly, so if you delete it and it has to be \
built from scratch, it takes maybe another second to do. Thus, putting the resolution \
cache per checkout resolves the issue of multiple builds sharing the same resolution \
cache since each will have its own. When the developer does a clean, they delete the \
"target" directory and the resolution cache. No real problem since it can be quickly \
rebuilt.

The Ivy cache is a different issue. If this cache is deleted, all the jars must be \
redownloaded which takes time. Besides if project A downloads version 1.3 of foo.jar, \
and project B also needs that jar, project B doesn't have to download it. That's the \
whole purpose of the Ivy cache. Thus, the resolution cache is placed per project \
while the Ivy jar cache is shared between different projects.

> I forget if Jenkins checks out the source into a different location for
> each executor build (it should), but if that's the case and you're sending
> your output to a location relative to your source, then establishing a
> relative path to your caches should suffice, and there's no need to
> establish cache-${env.EXECUTOR_NUMBER} directories. And in fact, now that I
> think of it, specifying Ivy caches for a CI build that are based on
> ivy.default.ivy.user.dir, considering that its default is user.home/.ivy2/,
> is a bit of an antipattern.

No, each job has an established working directory and it uses the same working \
directory no matter which executor is running. This makes sense since the first time \
a job might run, it runs on executor #1 and the second  time on executor #0. If there \
were separate working directories per executor, you couldn't take advantage of a \
version control update which only has to copy a few files over instead of a whole new \
working directory.

The "cache" really doesn't exist on Jenkins because I wipe it out for each build. \
That's why each executor needs a separate build. I could just as easily put that \
under the "target" directory like I do the resolution cache. Unfortunately, the same \
build.xml is also used by the developer. Their machines are slower and on a slower \
network connection than the servers. Jenkins can download all of the jars in seconds. \
On the developer's machines, it can take minutes. Thus, the cache is very important \
to the developers.

This is the problem: Both developers and Jenkins use the same build process, and the \
same build files. But, both use the Ivy cache very differently. Developers want it to \
act like a cache: Saving them valuable time by not having to redownload files over \
and over. Jenkins wants a clean slate to make sure that there isn't a jar issue \
during its build process. Satisfying these two completely different requirements \
while using the same build process and files lead me to this design.

On Nov 29, 2012, at 12:32 PM, Mitch Gitman <mgitman@gmail.com> wrote:
> Let me take a stab at addressing your immediate problem, even though I
> don't have a direct answer. But then I'm going to suggest a different
> approach that avoids that problem.
> 
> Frankly, I've never used configure, only settings. The configure
> documentation explains the primary difference (basically settings is lazy):
> http://ant.apache.org/ivy/history/latest-milestone/use/configure.html
> 
> As you describe, the immediate problem is that, with settings, the
> EXECUTOR_NUMBER environment variable is not being picked up. The following
> line is really just establishing a default:
> <property name="env.EXECUTOR_NUMBER" value="0" override="false"/>
> 
> For debugging's sake, I'd try leaving that out. The executor number should
> be required.
> 
> For debugging's sake, I'd also try outputting the value in the Ant build
> just prior to calling ivy:settings.
> 
> Also, I'd make sure that ivy:settings is always getting called before
> ivy:cleancache.
> 
> And there's something I find odd about your caches specification:
> <caches
> defaultCacheDir="${ivy.default.ivy.user.dir}/cache-${env.EXECUTOR_NUMBER}"
> resolutionCacheDir="${ivy.dir}/../target/ivy.cache"/>
> 
> It appears that your resolution cache is not being defined on a
> per-executor basis. If each executor build has its own target directory,
> that's OK. But then, why not do that for both your resolution cache and
> your repository cache? So you'd specify resolutionCacheDir and
> repositoryCacheDir attributes (no defaultCacheDir) in a consistent way.
> 
> I forget if Jenkins checks out the source into a different location for
> each executor build (it should), but if that's the case and you're sending
> your output to a location relative to your source, then establishing a
> relative path to your caches should suffice, and there's no need to
> establish cache-${env.EXECUTOR_NUMBER} directories. And in fact, now that I
> think of it, specifying Ivy caches for a CI build that are based on
> ivy.default.ivy.user.dir, considering that its default is user.home/.ivy2/,
> is a bit of an antipattern.
> 
> On Thu, Nov 29, 2012 at 8:35 AM, David Weintraub <qazwart@gmail.com> wrote:
> 
> > When is it better to use <ivy:settings> vs. <ivy:configure>?
> > 
> > I ask because I was having a problem with my Ivy setup. We use Subversion
> > and have multiple projects setup that use Ant. In order to centralize our
> > jar dependencies, I've decided to implement Ivy into our process.
> > 
> > I did this by creating an Ivy project (https://github.com/qazwart/ivy.dir)
> > in Subversion. Users merely set `svn:externals` to include this project,
> > make some minor changes in their build.xml, and add in a "ivy.xml" file,
> > and everything is set. I control the ivysettings.xml file, so I point Ivy
> > to our repository. The Ivy jar is part of the Ivy project, so users don't
> > have to install it. The idea was to make Ivy integration as painless as
> > possible.
> > 
> > We use Jenkins as our build system, and I decided it would be good if we
> > could clean the Ivy cache before each build. However, out Jenkins system
> > has six executors, so as many as six builds could happen at once. Cleaning
> > the cache in one build while another one runs could cause problems, so I
> > modified my ivysettings.xml file to use a different cache depending upon
> > the executor:
> > 
> > <ivysettings>
> > <property name="env.EXECUTOR_NUMBER" value="0" override="false"/>
> > <caches
> > 
> > defaultCacheDir="${ivy.default.ivy.user.dir}/cache-${env.EXECUTOR_NUMBER}"
> > resolutionCacheDir="${ivy.dir}/../target/ivy.cache"/>
> > <settings defaultResolver="default"/>
> > <include file="${ivy.dir}/ivysettings-public.xml"/>
> > <include url="${ivy.default.settings.dir}/ivysettings-shared.xml"/>
> > <include url="${ivy.default.settings.dir}/ivysettings-local.xml"/>
> > <include url="${ivy.default.settings.dir}/ivysettings-main-chain.xml"/>
> > <include
> > url="${ivy.default.settings.dir}/ivysettings-default-chain.xml"/>
> > </ivysettings>
> > 
> > The problem is that if I did an <ivy:cleancache>, the cache ID was always
> > set to "cache-0". When I switched from "<ivy:settings>" to
> > "<ivy:configure>", the problem went away. (In the standard build process,
> > the <ivy:cleancache> task is executed before the <ivy:resolve> task).
> > 
> > I'm told that "<ivy:settings>" can do multiple configurations, but it looks
> > like <ivy:configure> also can do multiple settings too. I also read that
> > "<ivy:configure>" was deprecated in some mailings, but it doesn't state
> > that in the on line documentation.
> > 
> > So what is the difference between <ivy:settings> and <ivy:configure>, and
> > when should I use one over the other?
> > 
> > --
> > David Weintraub
> > qazwart@gmail.com
> > 


[prev in list] [next in list] [prev in thread] [next in thread]