Add tooling to remove version dependencies for a given list of python
modules found in a subset of files in a given directory. This is
useful for cases where packages in pypi have out of date dependencies
relative to the test of the ecoysystem.
Signed-off-by: William Douglas <william.douglas@intel.com>
Because miltiple binaries might share the same exact contents, include
the filepath as part of the hash.
Signed-off-by: William Douglas <william.douglas@intel.com>
If --path was not passed previously elf-move would crash, it also was
incorrectly handling the filepath instead of the virtpath (only care
about the path after the DESTDIR for --path, makes handling easier
too).
Finally make --path override the --skip behavior so they can work
together to append new content to a file.
The first 4096 bytes were not being accounted for, resulting in any ELF
content with size <= 4096 having the same sha256.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
Add a new utility that will moving elf binaries to an alternate
location (renamed to their sha256 hash) and create a mapping file of
the binaries moved.
Signed-off-by: William Douglas <william.douglas@intel.com>
In `objdump -d` output, lines with instructions may list function names
that have "xmm", "ymm", or "zmm" in their names.
Avoid detecting lines like this as sse/avx2/avx512 by searching the
instruction argument field for "%xmm", "%ymm", or "%zmm" instead.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
There is duplicate counting of AVX2&AVX512 or SSE&AVX2 with original
matching order:
...
sse_score = is_sse(ins, arg)
avx2_score = is_avx2(ins, arg)
avx512_score = is_avx512(ins, arg)
...
with -d command line applied, we could see following output:
python3 avxjudge.py -d /tmp/libopenblas_skylakexp-r0.3.3.so
...
duplicate count for sse & avx2 ? vfmadd213ss (%r12,%r15,4),%xmm2,%xmm0 102
duplicate count for sse & avx2 ? vfmadd213ss -0x4(%r12),%xmm1,%xmm0 103
duplicate count for sse & avx2 ? vinsertf128 $0x1,%xmm3,%ymm0,%ymm0 104
...
duplicate count for avx2 & avx512 ? vmovups 0x60(%rdx),%ymm16 5
duplicate count for sse & avx2 ? vinsertf128 $0x1,%xmm1,%ymm0,%ymm0 115
duplicate count for avx2 & avx512 ? vmovups 0x80(%rdx),%ymm17 6
duplicate count for sse & avx2 ? vinsertf128 $0x1,%xmm1,%ymm0,%ymm0 116
duplicate count for avx2 & avx512 ? vmovups 0xa0(%rdx),%ymm18 7
duplicate count for sse & avx2 ? vinsertf128 $0x1,%xmm1,%ymm0,%ymm0 117
duplicate count for avx2 & avx512 ? vmovups 0xc0(%rdx),%ymm19 8
...
File total (SSE): 542451 instructions with score 126060
File total (AVX2): 114279 instructions with score 112833
File total (AVX512): 80538 instructions with score 30653
File duplicate count of sse&avx2 66813 , duplicate count of avx2&avx512 11502
So we should do is_avx512() matching first and trust it's result,
only do is_avx2() when is_avx512() fails, then the matching is
re-ordered as:
avx512_score = is_avx512(ins, arg)
if avx512_score <= 0:
avx2_score = is_avx2(ins, arg)
if avx2_score <= 0 and avx512_score <= 0:
sse_score = is_sse(ins, arg)
Though is_avx512(), is_avx2() and is_sse() are not accurate 100%, but we
should trust their result and assume they could match accurately.
Then the duplicate counting issue is fixed in a simple way.
Signed-off-by: Ethan Zhao <haifeng1.zhao@intel.com>
Check if being run in delete mode (and quiet) and exit early when the
file being judged would be kept. This allows improved speed when
processing files for deletion rather than gathering statistics.
This is actually a large rewrite to avoid mutable globals cleverly
disguised as parsing objdump output as a stream rather than waiting
for all the content to finish loading in the python buffer. The
functionality should *not* have changed as this is just a speculative
performance patch.
The script is not really fast on big libraries, so parallelising the run
should sorten considerably the post-install procedures, especially on
build servers with a lot of cores.
Signed-off-by: Thiago Macieira <thiago.macieira@intel.com>
That's the usual case with libraries:
libfoo.so -> libfoo.so.1 -> libfoo.so.1.2.3
We prepended the symlinks so we did readelf on them first, before
moving. But there was a problem in case moved the middle symlink, such
that the first one became broken.
So instead of building a file list to scan, then scan and move in one
go, we scan while building the file list, doing all the moves at the
end.
Signed-off-by: Thiago Macieira <thiago.macieira@intel.com>
If a package is built with both AVX512 and AVX2, when the
%make_install_avx2 target is run, we'll already have AVX512 files in
/usr/bin/haswell, /usr/lib64/haswell and plugins named *.so.avx512. So
don't recurse into the libsubdir and don't rename anything called
*.so.avx*.
This patch is a little bigger than necessary so we don't recurse at all
for binary directories.
Note: this only works if AVX512 is run first, because it depends on the
AVX512 directory being a subdir of the AVX2 directory.
Signed-off-by: Thiago Macieira <thiago.macieira@intel.com>
Traceback (most recent call last):
File "/usr/bin/avxjudge.py", line 344, in <module>
main()
File "/usr/bin/avxjudge.py", line 315, in main
do_file(args.filename)
File "/usr/bin/avxjudge.py", line 231, in do_file
print(function,"\t",ratio(sse_count/instructions),"\t", ratio(avx2_count / instructions), "\t", ratio(avx512_count/instructions), "\t", avx2_score,"\t", avx512_score)
NameError: name 'ratio' is not defined
Signed-off-by: Thiago Macieira <thiago.macieira@intel.com>
The original shell script was mkdir -p. The perl function mkdir()
doesn't do that, though, so we have to use File::Path::make_path().
Signed-off-by: Thiago Macieira <thiago.macieira@intel.com>
VFMADD132SS is an FMA instruction, which was added to Haswell
architecture at the same time it got AVX2. This was correct prior to the
patch in PR #1.
Signed-off-by: Thiago Macieira <thiago.macieira@intel.com>
Use the `endswith()` from string objects to determine if any of the
register names ends with `mm16`..`mm31`.
Signed-off-by: Leandro Pereira <leandro.pereira@intel.com>
Python's `in` operator are translated to a hash table lookup in Python;
it's more efficient than using it with lists or tuples. Modify the
instruction tables to use sets instead.
Also changes how the instruction tables are initialized; initializing
them using this syntax is slighly more efficient.
Analyzing the libc library on my system went from 3.40s to 2.61s with
this change alone.
Signed-off-by: Leandro Pereira <leandro.pereira@intel.com>
avxjudge.py: used automatically by the RPM scripts to heuristically
determine whether a library or plugin created with AVX2 or AVX512
support is worth keeping.
clr-avx-move.pl: inserted into RPM builds by Autospec to move the
libraries and plugins to their correct locations.
clr-python-avx2 and clr-python-avx512: wrappers for avxjudge.py.
Signed-off-by: Thiago Macieira <thiago.macieira@intel.com>
William Douglas