'Re: [TCWG CI] 433.milc slowed down by 4% after llvm: Add missing header'

[prev in list] [next in list] [prev in thread] [next in thread] 

List:       linaro-toolchain
Subject:    Re: [TCWG CI] 433.milc slowed down by 4% after llvm: Add missing header
From:       David Blaikie <dblaikie () gmail ! com>
Date:       2021-12-07 18:13:16
Message-ID: CAENS6Et5aHTrM3KUtD8cMsj+Z-CxGBCMzmHaCo7az-+QKzJGqA () mail ! gmail ! com
[Download RAW message or body]

OK , thanks - appreciate all the hard work of making performance regression
analysis & continued improvement to reduce false positives!

On Tue, Dec 7, 2021 at 4:03 AM Maxim Kuvyrkov <maxim.kuvyrkov@linaro.org>
wrote:

> Hi David,
> 
> This is a false positive, sorry for the noise.  Our CI bisects performance
> regressions down to a single commit, and notifies patch authors about
> significant regressions.
> 
> Since this is benchmarking CI, some noise is expected.  Apparently,
> 433.milc started to show bi-modal performance, and bisection [mistakenly]
> converged on this patch.  We work to reduce benchmarking noise and
> otherwise improve benchmarking CI.
> 
> Regards!
> 
> --
> Maxim Kuvyrkov
> https://www.linaro.org
> 
> > On 7 Dec 2021, at 06:12, David Blaikie <dblaikie@gmail.com> wrote:
> > 
> > Seems... unlikely this change had a performance impact.
> > 
> > Also is this email meant to be sent to public contributors like myself,
> or
> > only intended for some Linaro folks?
> > 
> > On Sun, Dec 5, 2021 at 6:18 AM <ci_notify@linaro.org> wrote:
> > 
> > > After llvm commit bd4c6a476fd037fb07a1c484f75d93ee40713d3d
> > > Author: David Blaikie <dblaikie@gmail.com>
> > > 
> > > Add missing header
> > > 
> > > the following benchmarks slowed down by more than 2%:
> > > - 433.milc slowed down by 4% from 12427 to 12916 perf samples
> > > 
> > > Below reproducer instructions can be used to re-build both "first_bad"
> and
> > > "last_good" cross-toolchains used in this bisection.  Naturally, the
> > > scripts will fail when triggerring benchmarking jobs if you don't have
> > > access to Linaro TCWG CI.
> > > 
> > > For your convenience, we have uploaded tarballs with pre-processed
> source
> > > and assembly files at:
> > > - First_bad save-temps:
> > > 
> https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-aarch64-s \
> pec2k6-O2_LTO/34/artifact/artifacts/build-bd4c6a476fd037fb07a1c484f75d93ee40713d3d/save-temps/
> 
> > > - Last_good save-temps:
> > > 
> https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-aarch64-s \
> pec2k6-O2_LTO/34/artifact/artifacts/build-7d4da4e1ab7f79e51db0d5c2a0f5ef1711122dd7/save-temps/
> 
> > > - Baseline save-temps:
> > > 
> https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-aarch64-spec2k6-O2_LTO/34/artifact/artifacts/build-baseline/save-temps/
> 
> > > 
> > > Configuration:
> > > - Benchmark: SPEC CPU2006
> > > - Toolchain: Clang + Glibc + LLVM Linker
> > > - Version: all components were built from their tip of trunk
> > > - Target: aarch64-linux-gnu
> > > - Compiler flags: -O2 -flto
> > > - Hardware: NVidia TX1 4x Cortex-A57
> > > 
> > > This benchmarking CI is work-in-progress, and we welcome feedback and
> > > suggestions at linaro-toolchain@lists.linaro.org .  In our improvement
> > > plans is to add support for SPEC CPU2017 benchmarks and provide "perf
> > > report/annotate" data behind these reports.
> > > 
> > > THIS IS THE END OF INTERESTING STUFF.  BELOW ARE LINKS TO BUILDS,
> > > REPRODUCTION INSTRUCTIONS, AND THE RAW COMMIT.
> > > 
> > > This commit has regressed these CI configurations:
> > > - tcwg_bmk_llvm_tx1/llvm-master-aarch64-spec2k6-O2_LTO
> > > 
> > > First_bad build:
> > > 
> https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-aarch64-s \
> pec2k6-O2_LTO/34/artifact/artifacts/build-bd4c6a476fd037fb07a1c484f75d93ee40713d3d/
> > > Last_good build:
> > > 
> https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-aarch64-s \
> pec2k6-O2_LTO/34/artifact/artifacts/build-7d4da4e1ab7f79e51db0d5c2a0f5ef1711122dd7/
> > > Baseline build:
> > > 
> https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-aarch64-spec2k6-O2_LTO/34/artifact/artifacts/build-baseline/
> 
> > > Even more details:
> > > 
> https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-aarch64-spec2k6-O2_LTO/34/artifact/artifacts/
> 
> > > 
> > > Reproduce builds:
> > > <cut>
> > > mkdir investigate-llvm-bd4c6a476fd037fb07a1c484f75d93ee40713d3d
> > > cd investigate-llvm-bd4c6a476fd037fb07a1c484f75d93ee40713d3d
> > > 
> > > # Fetch scripts
> > > git clone https://git.linaro.org/toolchain/jenkins-scripts
> > > 
> > > # Fetch manifests and test.sh script
> > > mkdir -p artifacts/manifests
> > > curl -o artifacts/manifests/build-baseline.sh
> > > 
> https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-aarch64-spec2k6-O2_LTO/34/artifact/artifacts/manifests/build-baseline.sh
> 
> > > --fail
> > > curl -o artifacts/manifests/build-parameters.sh
> > > 
> https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-aarch64-spec2k6-O2_LTO/34/artifact/artifacts/manifests/build-parameters.sh
> 
> > > --fail
> > > curl -o artifacts/test.sh
> > > 
> https://ci.linaro.org/job/tcwg_bmk_ci_llvm-bisect-tcwg_bmk_tx1-llvm-master-aarch64-spec2k6-O2_LTO/34/artifact/artifacts/test.sh
> 
> > > --fail
> > > chmod +x artifacts/test.sh
> > > 
> > > # Reproduce the baseline build (build all pre-requisites)
> > > ./jenkins-scripts/tcwg_bmk-build.sh @@
> > > artifacts/manifests/build-baseline.sh
> > > 
> > > # Save baseline build state (which is then restored in
> artifacts/test.sh)
> > > mkdir -p ./bisect
> > > rsync -a --del --delete-excluded --exclude /bisect/ --exclude
> /artifacts/
> > > --exclude /llvm/ ./ ./bisect/baseline/
> > > 
> > > cd llvm
> > > 
> > > # Reproduce first_bad build
> > > git checkout --detach bd4c6a476fd037fb07a1c484f75d93ee40713d3d
> > > ../artifacts/test.sh
> > > 
> > > # Reproduce last_good build
> > > git checkout --detach 7d4da4e1ab7f79e51db0d5c2a0f5ef1711122dd7
> > > ../artifacts/test.sh
> > > 
> > > cd ..
> > > </cut>
> > > 
> > > Full commit (up to 1000 lines):
> > > <cut>
> > > commit bd4c6a476fd037fb07a1c484f75d93ee40713d3d
> > > Author: David Blaikie <dblaikie@gmail.com>
> > > Date:   Mon Nov 29 16:29:25 2021 -0800
> > > 
> > > Add missing header
> > > ---
> > > llvm/lib/Demangle/DLangDemangle.cpp | 1 +
> > > 1 file changed, 1 insertion(+)
> > > 
> > > diff --git a/llvm/lib/Demangle/DLangDemangle.cpp
> > > b/llvm/lib/Demangle/DLangDemangle.cpp
> > > index faf91b239490..f380aa90035e 100644
> > > --- a/llvm/lib/Demangle/DLangDemangle.cpp
> > > +++ b/llvm/lib/Demangle/DLangDemangle.cpp
> > > @@ -17,6 +17,7 @@
> > > #include "llvm/Demangle/StringView.h"
> > > #include "llvm/Demangle/Utility.h"
> > > 
> > > +#include <cctype>
> > > #include <cstring>
> > > #include <limits>
> > > 
> > > </cut>
> > _______________________________________________
> > linaro-toolchain mailing list
> > linaro-toolchain@lists.linaro.org
> > https://lists.linaro.org/mailman/listinfo/linaro-toolchain
> 
> 
_______________________________________________
linaro-toolchain mailing list
linaro-toolchain@lists.linaro.org
https://lists.linaro.org/mailman/listinfo/linaro-toolchain

[prev in list] [next in list] [prev in thread] [next in thread] 