aboutsummaryrefslogtreecommitdiffhomepage
path: root/Eigen/src/Core/arch/AVX
Commit message (Collapse)AuthorAge
...
* Adding lowlevel APIs for optimized RHS packet load in TensorFlowGravatar Anuj Rawat2019-04-20
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | SpatialConvolution Low-level APIs are added in order to optimized packet load in gemm_pack_rhs in TensorFlow SpatialConvolution. The optimization is for scenario when a packet is split across 2 adjacent columns. In this case we read it as two 'partial' packets and then merge these into 1. Currently this only works for Packet16f (AVX512) and Packet8f (AVX2). We plan to add this for other packet types (such as Packet8d) also. This optimization shows significant speedup in SpatialConvolution with certain parameters. Some examples are below. Benchmark parameters are specified as: Batch size, Input dim, Depth, Num of filters, Filter dim Speedup numbers are specified for number of threads 1, 2, 4, 8, 16. AVX512: Parameters | Speedup (Num of threads: 1, 2, 4, 8, 16) ----------------------------|------------------------------------------ 128, 24x24, 3, 64, 5x5 |2.18X, 2.13X, 1.73X, 1.64X, 1.66X 128, 24x24, 1, 64, 8x8 |2.00X, 1.98X, 1.93X, 1.91X, 1.91X 32, 24x24, 3, 64, 5x5 |2.26X, 2.14X, 2.17X, 2.22X, 2.33X 128, 24x24, 3, 64, 3x3 |1.51X, 1.45X, 1.45X, 1.67X, 1.57X 32, 14x14, 24, 64, 5x5 |1.21X, 1.19X, 1.16X, 1.70X, 1.17X 128, 128x128, 3, 96, 11x11 |2.17X, 2.18X, 2.19X, 2.20X, 2.18X AVX2: Parameters | Speedup (Num of threads: 1, 2, 4, 8, 16) ----------------------------|------------------------------------------ 128, 24x24, 3, 64, 5x5 | 1.66X, 1.65X, 1.61X, 1.56X, 1.49X 32, 24x24, 3, 64, 5x5 | 1.71X, 1.63X, 1.77X, 1.58X, 1.68X 128, 24x24, 1, 64, 5x5 | 1.44X, 1.40X, 1.38X, 1.37X, 1.33X 128, 24x24, 3, 64, 3x3 | 1.68X, 1.63X, 1.58X, 1.56X, 1.62X 128, 128x128, 3, 96, 11x11 | 1.36X, 1.36X, 1.37X, 1.37X, 1.37X In the higher level benchmark cifar10, we observe a runtime improvement of around 6% for AVX512 on Intel Skylake server (8 cores). On lower level PackRhs micro-benchmarks specified in TensorFlow tensorflow/core/kernels/eigen_spatial_convolutions_test.cc, we observe the following runtime numbers: AVX512: Parameters | Runtime without patch (ns) | Runtime with patch (ns) | Speedup ---------------------------------------------------------------|----------------------------|-------------------------|--------- BM_RHS_NAME(PackRhs, 128, 24, 24, 3, 64, 5, 5, 1, 1, 256, 56) | 41350 | 15073 | 2.74X BM_RHS_NAME(PackRhs, 32, 64, 64, 32, 64, 5, 5, 1, 1, 256, 56) | 7277 | 7341 | 0.99X BM_RHS_NAME(PackRhs, 32, 64, 64, 32, 64, 5, 5, 2, 2, 256, 56) | 8675 | 8681 | 1.00X BM_RHS_NAME(PackRhs, 32, 64, 64, 30, 64, 5, 5, 1, 1, 256, 56) | 24155 | 16079 | 1.50X BM_RHS_NAME(PackRhs, 32, 64, 64, 30, 64, 5, 5, 2, 2, 256, 56) | 25052 | 17152 | 1.46X BM_RHS_NAME(PackRhs, 32, 256, 256, 4, 16, 8, 8, 1, 1, 256, 56) | 18269 | 18345 | 1.00X BM_RHS_NAME(PackRhs, 32, 256, 256, 4, 16, 8, 8, 2, 4, 256, 56) | 19468 | 19872 | 0.98X BM_RHS_NAME(PackRhs, 32, 64, 64, 4, 16, 3, 3, 1, 1, 36, 432) | 156060 | 42432 | 3.68X BM_RHS_NAME(PackRhs, 32, 64, 64, 4, 16, 3, 3, 2, 2, 36, 432) | 132701 | 36944 | 3.59X AVX2: Parameters | Runtime without patch (ns) | Runtime with patch (ns) | Speedup ---------------------------------------------------------------|----------------------------|-------------------------|--------- BM_RHS_NAME(PackRhs, 128, 24, 24, 3, 64, 5, 5, 1, 1, 256, 56) | 26233 | 12393 | 2.12X BM_RHS_NAME(PackRhs, 32, 64, 64, 32, 64, 5, 5, 1, 1, 256, 56) | 6091 | 6062 | 1.00X BM_RHS_NAME(PackRhs, 32, 64, 64, 32, 64, 5, 5, 2, 2, 256, 56) | 7427 | 7408 | 1.00X BM_RHS_NAME(PackRhs, 32, 64, 64, 30, 64, 5, 5, 1, 1, 256, 56) | 23453 | 20826 | 1.13X BM_RHS_NAME(PackRhs, 32, 64, 64, 30, 64, 5, 5, 2, 2, 256, 56) | 23167 | 22091 | 1.09X BM_RHS_NAME(PackRhs, 32, 256, 256, 4, 16, 8, 8, 1, 1, 256, 56) | 23422 | 23682 | 0.99X BM_RHS_NAME(PackRhs, 32, 256, 256, 4, 16, 8, 8, 2, 4, 256, 56) | 23165 | 23663 | 0.98X BM_RHS_NAME(PackRhs, 32, 64, 64, 4, 16, 3, 3, 1, 1, 36, 432) | 72689 | 44969 | 1.62X BM_RHS_NAME(PackRhs, 32, 64, 64, 4, 16, 3, 3, 2, 2, 36, 432) | 61732 | 39779 | 1.55X All benchmarks on Intel Skylake server with 8 cores.
* Apply SSE's pmin/pmax fix for GCC <= 5 to AVX's pmin/pmaxGravatar Gael Guennebaud2019-03-10
|
* Fix conflicts and mergeGravatar Gael Guennebaud2019-01-30
|\
* | Fix compilation error for logical packet ops with older compilers.Gravatar Rasmus Munk Larsen2019-01-16
| |
* | Rename pones -> ptrue. Use _CMP_TRUE_UQ where appropriate.Gravatar Rasmus Munk Larsen2019-01-09
|\ \
* | | Collapsed revisionGravatar Rasmus Munk Larsen2019-01-09
| | | | | | | | | | | | | | | | | | | | | | | | | | | * Collapsed revision * Add packet up "pones". Write pnot(a) as pxor(pones(a), a). * Collapsed revision * Simplify a bit. * Undo useless diffs. * Fix typo.
| * | Simplify a bit.Gravatar Rasmus Munk Larsen2019-01-09
| | |
| * | Add packet up "pones". Write pnot(a) as pxor(pones(a), a).Gravatar Rasmus Munk Larsen2019-01-09
|/ /
* | Merged eigen/eigen into defaultGravatar Rasmus Larsen2019-01-09
|\ \
| * | bug #1652: implements a much more accurate version of vectorized sin/cos. ↵Gravatar Gael Guennebaud2019-01-09
| | | | | | | | | | | | | | | | | | | | | This new version achieve same speed for SSE/AVX, and is slightly faster with FMA. Guarantees are as follows: - no FMA: 1ULP up to 3pi, 2ULP up to sin(25966) and cos(18838), fallback to std::sin/cos for larger inputs - FMA: 1ULP up to sin(117435.992) and cos(71476.0625), fallback to std::sin/cos for larger inputs
* | | Add support for pcmp_eq and pnot, including for complex types.Gravatar Rasmus Munk Larsen2019-01-07
|/ /
| * Introducing "vectorized" byte on unpacket_traits structsGravatar Gustavo Lima Chaves2018-12-19
|/ | | | | | | | | | | | | | | | | | | | | This is a preparation to a change on gebp_traits, where a new template argument will be introduced to dictate the packet size, so it won't be bound to the current/max packet size only anymore. By having packet types defined early on gebp_traits, one has now to act on packet types, not scalars anymore, for the enum values defined on that class. One approach for reaching the vectorizable/size properties one needs there could be getting the packet's scalar again with unpacket_traits<>, then the size/Vectorizable enum entries from packet_traits<>. It turns out guards like "#ifndef EIGEN_VECTORIZE_AVX512" at AVX/PacketMath.h will hide smaller packet variations of packet_traits<> for some types (and it makes sense to keep that). In other words, one can't go back to the scalar and create a new PacketType, as this will always lead to the maximum packet type for the architecture. The less costly/invasive solution for that, thus, is to add the vectorizable info on every unpacket_traits struct as well.
* Properly set the number of registers for AVX512Gravatar Gael Guennebaud2018-12-11
|
* bug #1641: fix testing of pandnot and fix pandnot for complex on SSE/AVX/AVX512Gravatar Gael Guennebaud2018-12-08
|
* Enable FMA with MSVC (through /arch:AVX2). To make this possible, I also has ↵Gravatar Gael Guennebaud2018-12-07
| | | | to turn the #warning regarding AVX512-FMA to a #error.
* bug #1637: workaround register spilling in gebp with clang>=6.0+AVX+FMAGravatar Gael Guennebaud2018-12-07
|
* AVX512f includes FMA but GCC does not define __FMA__ with -mavx512f onlyGravatar Gael Guennebaud2018-12-06
|
* Implement AVX512 vectorization of std::complex<float/double>Gravatar Gael Guennebaud2018-12-06
|
* Add missing padd for Packet8i (it was implicitly generated by clang and gcc)Gravatar Gael Guennebaud2018-11-30
|
* Several improvements regarding packet-bitwise operations:Gravatar Gael Guennebaud2018-11-30
| | | | | | - add unit tests - optimize their AVX512f implementation - add missing implementations (half, Packet4f, ...)
* CleanupGravatar Gael Guennebaud2018-11-30
|
* Extend the generic psin_float code to handle cosine and make SSE and AVX use ↵Gravatar Gael Guennebaud2018-11-30
| | | | it (-> this adds pcos for AVX)
* Disable fma gcc's workaround for gcc >= 8 (based on GEMM benchmarks)Gravatar Gael Guennebaud2018-11-28
|
* bug #1631: fix compilation with ARM NEON and clang, and cleanup the weird ↵Gravatar Gael Guennebaud2018-11-27
| | | | pshiftright_and_cast and pcast_and_shiftleft functions.
* Update pshiftleft to pass the shift as a true compile-time integer.Gravatar Gael Guennebaud2018-11-27
|
* Unify SSE/AVX psin functions.Gravatar Gael Guennebaud2018-11-27
| | | | | | | | It is based on the SSE version which is much more accurate, though very slightly slower. This changeset also includes the following required changes: - add packet-float to packet-int type traits - add packet float<->int reinterpret casts - add faster pselect for AVX based on blendv
* cleanupGravatar Gael Guennebaud2018-11-26
|
* Unify SSE and AVX pexp for double.Gravatar Gael Guennebaud2018-11-26
|
* Unify SSE and AVX implementation of pexpGravatar Gael Guennebaud2018-11-26
|
* First step toward a unification of packet log implementation, currently only ↵Gravatar Gael Guennebaud2018-11-26
| | | | | | SSE and AVX are unified. To this end, I added the following functions: pzero, pcmp_*, pfrexp, pset1frombits functions.
* Make SSE/AVX pandnot(A,B) consistent with generic version, i.e., "A and not B"Gravatar Gael Guennebaud2018-11-26
|
* Fix compilation with MSVC by reverting to char* for _mm_prefetch except for ↵Gravatar Gael Guennebaud2018-06-07
| | | | PGI (the later being the one that has the wrong prototype).
* Fix compilation and SSE support with PGI compilerGravatar Gael Guennebaud2018-05-29
|
* comment unreachable codeGravatar Gael Guennebaud2018-04-03
|
* Rename predux_downto4 to be more accurate on its semantic.Gravatar Gael Guennebaud2018-04-03
|
* bug #1436: fix compilation of Jacobi rotations with ARM NEON, some ↵Gravatar Gael Guennebaud2017-06-15
| | | | specializations of internal::conj_helper were missing.
* Reverse arguments for pmin in AVX.Gravatar Rasmus Munk Larsen2017-01-25
|
* Make NaN propagatation consistent between the pmax/pmin and ↵Gravatar Rasmus Munk Larsen2017-01-24
| | | | | | std::max/std::min. This makes the NaN propagation consistent between the scalar and vectorized code paths of Eigen's scalar_max_op and scalar_min_op. See #1373 for details.
* Fix compilation with gcc and old ABI versionGravatar Gael Guennebaud2016-11-23
|
* Optimize predux<Packet8f> (AVX)Gravatar Gael Guennebaud2016-11-22
|
* Optimize predux<Packet4d> (AVX)Gravatar Gael Guennebaud2016-11-22
|
* Merged eigen/eigen into defaultGravatar Benoit Steiner2016-11-03
|\
| * Add pinsertfirst function and implement pinsertlast for complex on SSE/AVX.Gravatar Gael Guennebaud2016-11-02
| |
| * Add missing inline keywordsGravatar Gael Guennebaud2016-10-25
| |
| * Add a pinsertlast function replacing the last entry of a packet by a scalar.Gravatar Gael Guennebaud2016-10-25
| | | | | | | | (useful to vectorize LinSpaced)
* | Merged eigen/eigen into defaultGravatar Benoit Steiner2016-10-12
|\|
| * Fix copy-paste error: Must use _mm256_cmp_ps for AVX.Gravatar Rasmus Munk Larsen2016-10-12
| |
* | Renamed predux_half into predux_downto4Gravatar Benoit Steiner2016-10-06
| |
* | Merged latest updates from trunkGravatar Benoit Steiner2016-10-05
|\ \
| | * Update comment for fast sqrt.Gravatar Rasmus Munk Larsen2016-10-04
| | |
generated by cgit v1.2.3 (git 2.39.1) at 2024-07-03 16:28:50 +0000