Commit message (Collapse) | Author | Age | |
---|---|---|---|
* | Get rid of code duplication for conj_helper. For packets where ↵ | Rasmus Munk Larsen | 2021-06-24 |
| | | | | LhsType=RhsType a single generic implementation suffices. For scalars, the generic implementation of pconj automatically forwards to numext::conj, so much of the existing specialization can be avoided. For mixed types we still need specializations. | ||
* | Revert addition of unused `paddsub<Packet2cf>`. This fixes #2242 | Christoph Hertzberg | 2021-05-06 |
| | |||
* | 1)provide a better generic paddsub op implementation | Guoqiang QI | 2021-01-13 |
| | | | | | 2)make paddsub op support the Packet2cf/Packet4f/Packet2f in NEON 3)make paddsub op support the Packet2cf/Packet4f in SSE | ||
* | Don't guard psqrt for std::complex<float> with EIGEN_ARCH_ARM64 | David Tellenbach | 2020-12-11 |
| | |||
* | Fix vectorization of complex sqrt on NEON | David Tellenbach | 2020-12-10 |
| | |||
* | Add support for CastXML on ARM aarch64 | Brad King | 2020-09-16 |
| | | | | | | | | CastXML simulates the preprocessors of other compilers, but actually parses the translation unit with an internal Clang compiler. Use the same `vld1q_u64` workaround that we do for Clang. Fixes: #1979 | ||
* | Remove unused packet op "palign". | Rasmus Munk Larsen | 2020-05-07 |
| | | | | Clean up a compiler warning in c++03 mode in AVX512/Complex.h. | ||
* | Remove unused packet op "preduxp". | Rasmus Munk Larsen | 2020-04-23 |
| | |||
* | Additional NEON packet-math operations | Joel Holdsworth | 2020-03-26 |
| | |||
* | NEON: Implemented half-size vectors | Joel Holdsworth | 2020-03-10 |
| | |||
* | Improvements to the tidiness and completeness of the NEON implementation | Joel Holdsworth | 2020-01-10 |
| | |||
* | Add masked_store_available to unpacket_traits | Eugene Zhulenev | 2019-05-02 |
| | |||
* | Adding lowlevel APIs for optimized RHS packet load in TensorFlow | Anuj Rawat | 2019-04-20 |
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | SpatialConvolution Low-level APIs are added in order to optimized packet load in gemm_pack_rhs in TensorFlow SpatialConvolution. The optimization is for scenario when a packet is split across 2 adjacent columns. In this case we read it as two 'partial' packets and then merge these into 1. Currently this only works for Packet16f (AVX512) and Packet8f (AVX2). We plan to add this for other packet types (such as Packet8d) also. This optimization shows significant speedup in SpatialConvolution with certain parameters. Some examples are below. Benchmark parameters are specified as: Batch size, Input dim, Depth, Num of filters, Filter dim Speedup numbers are specified for number of threads 1, 2, 4, 8, 16. AVX512: Parameters | Speedup (Num of threads: 1, 2, 4, 8, 16) ----------------------------|------------------------------------------ 128, 24x24, 3, 64, 5x5 |2.18X, 2.13X, 1.73X, 1.64X, 1.66X 128, 24x24, 1, 64, 8x8 |2.00X, 1.98X, 1.93X, 1.91X, 1.91X 32, 24x24, 3, 64, 5x5 |2.26X, 2.14X, 2.17X, 2.22X, 2.33X 128, 24x24, 3, 64, 3x3 |1.51X, 1.45X, 1.45X, 1.67X, 1.57X 32, 14x14, 24, 64, 5x5 |1.21X, 1.19X, 1.16X, 1.70X, 1.17X 128, 128x128, 3, 96, 11x11 |2.17X, 2.18X, 2.19X, 2.20X, 2.18X AVX2: Parameters | Speedup (Num of threads: 1, 2, 4, 8, 16) ----------------------------|------------------------------------------ 128, 24x24, 3, 64, 5x5 | 1.66X, 1.65X, 1.61X, 1.56X, 1.49X 32, 24x24, 3, 64, 5x5 | 1.71X, 1.63X, 1.77X, 1.58X, 1.68X 128, 24x24, 1, 64, 5x5 | 1.44X, 1.40X, 1.38X, 1.37X, 1.33X 128, 24x24, 3, 64, 3x3 | 1.68X, 1.63X, 1.58X, 1.56X, 1.62X 128, 128x128, 3, 96, 11x11 | 1.36X, 1.36X, 1.37X, 1.37X, 1.37X In the higher level benchmark cifar10, we observe a runtime improvement of around 6% for AVX512 on Intel Skylake server (8 cores). On lower level PackRhs micro-benchmarks specified in TensorFlow tensorflow/core/kernels/eigen_spatial_convolutions_test.cc, we observe the following runtime numbers: AVX512: Parameters | Runtime without patch (ns) | Runtime with patch (ns) | Speedup ---------------------------------------------------------------|----------------------------|-------------------------|--------- BM_RHS_NAME(PackRhs, 128, 24, 24, 3, 64, 5, 5, 1, 1, 256, 56) | 41350 | 15073 | 2.74X BM_RHS_NAME(PackRhs, 32, 64, 64, 32, 64, 5, 5, 1, 1, 256, 56) | 7277 | 7341 | 0.99X BM_RHS_NAME(PackRhs, 32, 64, 64, 32, 64, 5, 5, 2, 2, 256, 56) | 8675 | 8681 | 1.00X BM_RHS_NAME(PackRhs, 32, 64, 64, 30, 64, 5, 5, 1, 1, 256, 56) | 24155 | 16079 | 1.50X BM_RHS_NAME(PackRhs, 32, 64, 64, 30, 64, 5, 5, 2, 2, 256, 56) | 25052 | 17152 | 1.46X BM_RHS_NAME(PackRhs, 32, 256, 256, 4, 16, 8, 8, 1, 1, 256, 56) | 18269 | 18345 | 1.00X BM_RHS_NAME(PackRhs, 32, 256, 256, 4, 16, 8, 8, 2, 4, 256, 56) | 19468 | 19872 | 0.98X BM_RHS_NAME(PackRhs, 32, 64, 64, 4, 16, 3, 3, 1, 1, 36, 432) | 156060 | 42432 | 3.68X BM_RHS_NAME(PackRhs, 32, 64, 64, 4, 16, 3, 3, 2, 2, 36, 432) | 132701 | 36944 | 3.59X AVX2: Parameters | Runtime without patch (ns) | Runtime with patch (ns) | Speedup ---------------------------------------------------------------|----------------------------|-------------------------|--------- BM_RHS_NAME(PackRhs, 128, 24, 24, 3, 64, 5, 5, 1, 1, 256, 56) | 26233 | 12393 | 2.12X BM_RHS_NAME(PackRhs, 32, 64, 64, 32, 64, 5, 5, 1, 1, 256, 56) | 6091 | 6062 | 1.00X BM_RHS_NAME(PackRhs, 32, 64, 64, 32, 64, 5, 5, 2, 2, 256, 56) | 7427 | 7408 | 1.00X BM_RHS_NAME(PackRhs, 32, 64, 64, 30, 64, 5, 5, 1, 1, 256, 56) | 23453 | 20826 | 1.13X BM_RHS_NAME(PackRhs, 32, 64, 64, 30, 64, 5, 5, 2, 2, 256, 56) | 23167 | 22091 | 1.09X BM_RHS_NAME(PackRhs, 32, 256, 256, 4, 16, 8, 8, 1, 1, 256, 56) | 23422 | 23682 | 0.99X BM_RHS_NAME(PackRhs, 32, 256, 256, 4, 16, 8, 8, 2, 4, 256, 56) | 23165 | 23663 | 0.98X BM_RHS_NAME(PackRhs, 32, 64, 64, 4, 16, 3, 3, 1, 1, 36, 432) | 72689 | 44969 | 1.62X BM_RHS_NAME(PackRhs, 32, 64, 64, 4, 16, 3, 3, 2, 2, 36, 432) | 61732 | 39779 | 1.55X All benchmarks on Intel Skylake server with 8 cores. | ||
* | Fix compilation on ARM. | Gael Guennebaud | 2019-02-22 |
| | |||
* | Add a few missing packet ops: cmp_eq for NEON. pfloor for GPU. | Rasmus Munk Larsen | 2019-02-21 |
| | |||
* | Fix conflicts and merge | Gael Guennebaud | 2019-01-30 |
|\ | |||
* | | bug #1652: fix position of EIGEN_ALIGN16 attributes in Neon and Altivec | Gael Guennebaud | 2019-01-14 |
| | | |||
| * | Introducing "vectorized" byte on unpacket_traits structs | Gustavo Lima Chaves | 2018-12-19 |
|/ | | | | | | | | | | | | | | | | | | | | | This is a preparation to a change on gebp_traits, where a new template argument will be introduced to dictate the packet size, so it won't be bound to the current/max packet size only anymore. By having packet types defined early on gebp_traits, one has now to act on packet types, not scalars anymore, for the enum values defined on that class. One approach for reaching the vectorizable/size properties one needs there could be getting the packet's scalar again with unpacket_traits<>, then the size/Vectorizable enum entries from packet_traits<>. It turns out guards like "#ifndef EIGEN_VECTORIZE_AVX512" at AVX/PacketMath.h will hide smaller packet variations of packet_traits<> for some types (and it makes sense to keep that). In other words, one can't go back to the scalar and create a new PacketType, as this will always lead to the maximum packet type for the architecture. The less costly/invasive solution for that, thus, is to add the vectorizable info on every unpacket_traits struct as well. | ||
* | Fix compilation with NEON+MSVC | Gael Guennebaud | 2018-04-26 |
| | |||
* | fix const cast in NEON | Gael Guennebaud | 2018-04-18 |
| | |||
* | bug #1436: fix compilation of Jacobi rotations with ARM NEON, some ↵ | Gael Guennebaud | 2017-06-15 |
| | | | | specializations of internal::conj_helper were missing. | ||
* | bug #1325: fix compilation on NEON with clang | Gael Guennebaud | 2016-10-12 |
| | |||
* | Merged kmargar/eigen/tip into default, Altivec/VSX port should be working ok ↵ | Konstantinos Margaritis | 2016-07-10 |
|\ | | | | | | | now. | ||
* | | Avoid global variables with static constructors in NEON/Complex.h | Benoit Jacob | 2016-06-28 |
| | | |||
* | | bug #1240: Remove any assumption on NEON vector types. | Gael Guennebaud | 2016-06-09 |
| | | |||
* | | Fix compile errors initializing packets on ARM DS-5 5.20 | Sean Templeton | 2016-06-03 |
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The ARM DS-5 5.20 compiler fails compiling with the following errors: "src/Core/arch/NEON/PacketMath.h", line 113: Error: #146: too many initializer values Packet4f countdown = EIGEN_INIT_NEON_PACKET4(0, 1, 2, 3); ^ "src/Core/arch/NEON/PacketMath.h", line 118: Error: #146: too many initializer values Packet4i countdown = EIGEN_INIT_NEON_PACKET4(0, 1, 2, 3); ^ "src/Core/arch/NEON/Complex.h", line 30: Error: #146: too many initializer values static uint32x4_t p4ui_CONJ_XOR = EIGEN_INIT_NEON_PACKET4(0x00000000, 0x80000000, 0x00000000, 0x80000000); ^ "src/Core/arch/NEON/Complex.h", line 31: Error: #146: too many initializer values static uint32x2_t p2ui_CONJ_XOR = EIGEN_INIT_NEON_PACKET2(0x00000000, 0x80000000); ^ The vectors are implemented as two doubles, hence the too many initializer values error. Changed the code to use intrinsic load functions which all compilers implementing NEON should have. | ||
| * | add name to copyrights | Konstantinos Margaritis | 2016-04-28 |
|/ | |||
* | bug #1103: fix neon vectorization of pmul(Packet1cd,Packet1cd) | Gael Guennebaud | 2015-12-10 |
| | |||
* | Let unpacket_traits<> exposes the required alignment and make use of it ↵ | Gael Guennebaud | 2015-08-07 |
| | | | | everywhere | ||
* | Fix uninitialized var warning. The compiler was clearing the register ↵ | Benoit Jacob | 2015-05-15 |
| | | | | anyway, so this does not change resulting code | ||
* | must also disable complex<double> when disabling double vectorization | Benoit Jacob | 2015-03-03 |
| | |||
* | The usage of DenseIndex is deprecated, so let's replace DenseIndex by Index | Gael Guennebaud | 2015-02-16 |
| | |||
* | Introduce unified macros to identify compiler, OS, and architecture. They ↵ | Gael Guennebaud | 2014-11-04 |
| | | | | are all defined in util/Macros.h and prefixed with EIGEN_COMP_, EIGEN_OS_, and EIGEN_ARCH_ respectively. | ||
* | Added ARMv8 support | Konstantinos Margaritis | 2014-10-22 |
| | |||
* | Fix many long to int implicit conversions | Gael Guennebaud | 2014-07-08 |
| | |||
* | Fix ptranspose overload prototypes for NEON | Gael Guennebaud | 2014-04-25 |
| | |||
* | Enable vectorization of pack_rhs with a column-major RHS. | Gael Guennebaud | 2014-04-25 |
| | | | | Rename and generalize Kernel<*> to PacketBlock<*,N>. | ||
* | Created a NEON version of the ptranspose packet primitives | Benoit Steiner | 2014-04-23 |
| | |||
* | Implemented the pgather/pscatter packet primitives for the arm/NEON architecture | Benoit Steiner | 2014-04-17 |
| | |||
* | New gebp kernel handling up to 3 packets x 4 register-level blocks. Huge ↵ | Gael Guennebaud | 2014-04-16 |
| | | | | | | speeup on Haswell. This changeset also introduce new vector functions: ploadquad and predux4. | ||
* | Add a mechanism to recursively access to half-size packet types | Gael Guennebaud | 2014-03-28 |
| | |||
* | bug #677: fix usage of pld instrinsics for ccomplexes | Gael Guennebaud | 2013-11-02 |
| | |||
* | Fix bug #591: minor optimization in NEON vectorization support | Simon Pilgrim | 2013-06-10 |
| | |||
* | Automatic relicensing to MPL2 using Keirs script. Manual fixup follows. | Benoit Jacob | 2012-07-13 |
| | |||
* | Get rid of include directives inside namespace blocks (bug #339). | Jitse Niesen | 2012-04-15 |
| | |||
* | Patches to support ARM NEON with Clang 3.0 and LLVM-GCC | Marton Danoczy | 2011-11-04 |
| | |||
* | fix compilation on ARM NEON (missing AlignedOnScalar) | Gael Guennebaud | 2011-05-06 |
| | |||
* | implement ploaddup for complex and SSE/NEON even though they are not used in ↵ | Gael Guennebaud | 2011-02-23 |
| | | | | practice | ||
* | fix a couple of issues with pcplxflip | Gael Guennebaud | 2011-02-23 |
| | |||
* | same for neon | Gael Guennebaud | 2011-02-23 |
| |