[cfarm-users] Setting up GitLab CI for git.git on the farm

[Segher Boessenkool]

On Sun, Nov 25, 2018 at 02:57:09PM +0100, Ævar Arnfjörð Bjarmason wrote:
> On Sun, Nov 25 2018, Segher Boessenkool via cfarm-users wrote:
> > On Sun, Nov 25, 2018 at 10:23:41AM +0100, Baptiste Jonglez wrote:
> >> On 24-11-18, Segher Boessenkool via cfarm-users wrote:
> >> > On Sat, Nov 24, 2018 at 11:28:58PM +0100, Ævar Arnfjörð Bjarmason via cfarm-users wrote:
> >> > > I've tried to be conservative about resources. It's all nice -n 19'd,
> >> > > and with a conservative -j value relative to the number of cores:
> >> > > https://gitlab.com/git-vcs/git-gitlab-ci/blob/b8d4645891aa/ci/gitlab/run-on-gcc-farm.sh#L62-163
> >> >
> >> > At least for the Power machines, that isn't conservative at all.
> >> > -j1 is conservative.  -j24 is not conservative on a machine with 20 CPUs
> >> > (gcc112), or 32 CPUs (gcc135).  The AIX (gcc119) jobs seem to run for over
> >> > an hour on half the machine?  That's no good :-(
> >>
> >> According to ansible [https://cfarm.tetaneutral.net/machines/list/] gcc112
> >> has 160 cores, and gcc135 has 128 cores.  Is ansible getting this wrong?
> 
> Thanks all. I definitely don't want to misuse resources here. Will tweak
> it way down.

Thanks!

> I was going by e.g. 128 on gcc135 and $(getconf
> _NPROCESSORS_ONLN), but can see that lscpu gives the numbers discussed
> her.

(That is just the "nproc" command, fwiw).

Yes, you get the number of threads.  What Linux calls a "CPU" is an SMT
thread, a processor context.

> > gcc110 is a Power7 with 16 cores, 64 threads.
> > gcc112 is a Power8 with 20 cores, 160 threads.
> > gcc135 is a Power9 with 32 cores, 128 threads.
> >
> > (Running more than 4 threads per core on a Power8 does not really help,
> > so you can count gcc112 as 80 threads).
> 
> I was curious to see what this was like on Power8 so on gcc135 I ran
> this as a one-off:

(That's Power9).

>     for j in 8 16 32 64 96 128 160 192 224 256; do for i in {1..3}; do git clean -dxf; time perl -MTime::HiRes=time -wE 'my $n = shift; my $t0 = -time(); system "make -j$n CFLAGS=-O0 NO_CURL=Y"; $t0 += time(); open my $fh, ">>", "/tmp/perf.txt"; printf $fh "%.2f\t%d\n", $t0, $n' $j ; done; done
> 
> I.e. compile with -O0 3 times for 8, 16 etc. values of -j:

(edited for presentation:)

     2.87 2.98 3.04    256
     2.93 2.93 2.96    224
     2.90 2.92 2.95    192
     2.91 2.93 2.94    160
     2.86 2.93 2.96    128
     3.02 3.02 3.06    96
     4.05 4.13 4.16    64
     4.97 4.97 5.05    32
     6.27 6.28 6.33    16
     9.83 9.85 9.86    8

So it improves all the way up to 128, and 128 is significantly better
than 64.

As you can see for <32 threads, your build doesn't scale super well, but
you still get very significant improvements up to 96 threads, and a bit
all the way to 128 even.


Segher