| Commit message (Collapse) | Author | Age |
|---|
| | |
|
| |
|
|
| |
This commit addresses this.
|
| |
|
|
| |
CJMADD, which were effectively unused, apart from on x86, where the change results in identically performing code.
|
| |
|
|
| |
LhsType=RhsType a single generic implementation suffices. For scalars, the generic implementation of pconj automatically forwards to numext::conj, so much of the existing specialization can be avoided. For mixed types we still need specializations.
|
| |
|
|
|
|
|
|
|
| |
We can't make guarantees on alignment for existing calls to `pset`,
so we should default to loading unaligned. But in that case, we should
just use `ploadu` directly. For loading constants, this load should hopefully
get optimized away.
This is causing segfaults in Google Maps.
|
| |
|
|
| |
slowdown) when used with Tensorflow. Changing to EIGEN_ALWAYS_INLINE where appropiate.
|
| | |
|
| |
|
|
| |
optimization changing sign with --ffast-math enabled.
|
| |
|
|
| |
Unified implementation using only `vzip`.
|
| |
|
|
| |
should fix #2242 .
|
| | |
|
| |
|
|
|
|
| |
The original produced NaNs when dividing 0/b for subnormal b.
The `complex_divide_stable` was changed to use the more common
Smith's algorithm.
|
| | |
|
| | |
|
| |
|
|
| |
warnings).
|
| | |
|
| |
|
|
|
|
|
| |
Wrong shuffle was used. Need to interleave low/high halves with a
`permute` instruction.
Fixes #2215.
|
| |
|
|
| |
efficient `movsd` instruction for `pset1<Packet2cf>`.
|
| |
|
|
| |
This fixes the problem with taking the address of temporary objects which clang11 treats as errors.
|
| | |
|
| | |
|
| |
|
| |
This reverts commit f019b97aca82071f35726b1aaebf1c598770f0f5
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
| |
Replace usage of `std::numeric_limits<...>::min/max_exponent` in
codebase where possible. Also replaced some other `numeric_limits`
usages in affected tests with the `NumTraits` equivalent.
The previous MR !443 failed for c++03 due to lack of `constexpr`.
Because of this, we need to keep around the `std::numeric_limits`
version in enum expressions until the switch to c++11.
Fixes #2148
|
| | |
|
| |
|
|
| |
This reverts commit 75ce9cd2a7aefaaea8543e2db14ce4dc149eeb03.
|
| |
|
|
|
|
|
|
| |
Replace usage of `std::numeric_limits<...>::min/max_exponent` in
codebase. Also replaced some other `numeric_limits` usages in
affected tests with the `NumTraits` equivalent.
Fixes #2148
|
| | |
|
| |
|
|
| |
This is to support TensorFlow's `tf.math.floormod` for half.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
NVCC does not understand `__forceinline`, so we need to use `inline`
when compiling for GPU.
ICC specializes `std::complex` operators for `float` and `double`
by default, which cannot be used on device and conflict with Eigen's
workaround in CUDA/Complex.h. This can be prevented by defining
`_OVERRIDE_COMPLEX_SPECIALIZATION_` before including `<complex>`.
Added this define to the tests and to `Eigen/Core`, but this will
not work if the user includes `<complex>` before `<Eigen/Core>`.
ICC also seems to generate a duplicate `Map` symbol in
`PlainObjectBase`:
```
error: "Map" has already been declared in the current scope
static ConstMapType Map(const Scalar *data)
```
I tracked this down to `friend class Eigen::Map`. Putting the `friend`
statements at the bottom of the class seems to resolve this issue.
Fixes #2180
|
| |
|
|
|
| |
This is for consistency with bfloat16, and to support initialization
with `std::iota`.
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
| |
Both CUDA and HIP require trivial default constructors for types used
in shared memory. Otherwise failing with
```
error: initialization is not supported for __shared__ variables.
```
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Currently, when compiling with HIP, Eigen::half is derived from the `__half_raw` struct that is defined within the hip_fp16.h header file. This is true for both the "host" compile phase and the "device" compile phase. This was causing a very hard to detect bug in the ROCm TensorFlow build.
In the ROCm Tensorflow build,
* files that do not contain ant GPU code get compiled via gcc, and
* files that contnain GPU code get compiled via hipcc.
In certain case, we have a function that is defined in a file that is compiled by hipcc, and is called in a file that is compiled by gcc. If such a function had Eigen::half has a "pass-by-value" argument, its value was getting corrupted, when received by the function.
The reason for this seems to be that for the gcc compile, Eigen::half is derived from a `__half_raw` struct that has `uint16_t` as the data-store, and for hipcc the `__half_raw` implementation uses `_Float16` as the data store. There is some ABI incompatibility between gcc / hipcc (which is essentially latest clang), which results in the Eigen::half value (which is correct at the call-site) getting randomly corrupted when passed to the function.
Changing the Eigen::half argument to be "pass by reference" seems to workaround the error.
In order to fix it such that we do not run into it again in TF, this commit changes the Eigne::half implementation to use the same `__half_raw` implementation as the non-GPU compile, during host compile phase of the hipcc compile.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This is a new version of !423, which failed for MSVC.
Defined `EIGEN_OPTIMIZATION_BARRIER(X)` that uses inline assembly to
prevent operations involving `X` from crossing that barrier. Should
work on most `GNUC` compatible compilers (MSVC doesn't seem to need
this). This is a modified version adapted from what was used in
`psincos_float` and tested on more platforms
(see #1674, https://godbolt.org/z/73ezTG).
Modified `rint` to use the barrier to prevent the add/subtract rounding
trick from being optimized away.
Also fixed an edge case for large inputs that get bumped up a power of two
and ends up rounding away more than just the fractional part. If we are
over `2^digits` then just return the input. This edge case was missed in
the test since the test was comparing approximate equality, which was still
satisfied. Adding a strict equality option catches it.
|
| |
|
| |
This reverts commit e72dfeb8b9fa5662831b5d0bb9d132521f9173dd
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
It seems *sometimes* with aggressive optimizations the combination
`psub(padd(a, b), b)` trick to force rounding is compiled away. Here
we replace with inline assembly to prevent this (I tried `volatile`,
but that leads to additional loads from memory).
Also fixed an edge case for large inputs `a` where adding `b` bumps
the value up a power of two and ends up rounding away more than
just the fractional part. If we are over `2^digits` then just return
the input. This edge case was missed in the test since the test was
comparing approximate equality, which was still satisfied. Adding
a strict equality option catches it.
|
| |
|
|
|
|
|
|
|
|
| |
In SSE, by adding/subtracting 2^MantissaBits, we force rounding according to the
current rounding mode.
For NEON, we use the provided intrinsics for rint/floor/ceil if
available (armv8).
Related to #1969.
|
| |
|
|
|
|
|
|
|
|
|
|
| |
Since `numeric_limits<half>::max_exponent` is a static inline constant,
it cannot be directly passed by reference. This triggers a linker error
in recent versions of `g++-powerpc64le`.
Changing `half` to take inputs by value fixes this. Wrapping
`max_exponent` with `int(...)` to make an addressable integer also fixes this
and may help with other custom `Scalar` types down-the-road.
Also eliminated some compile warnings for powerpc.
|
| |
|
|
| |
(cherry picked from commit c22c103e932e511e96645186831363585a44b7a3)
|
| |
|
|
|
|
|
|
|
|
|
|
| |
With !406, we accidentally broke arm 32-bit NEON builds, since
`vsqrt_f32` is only available for 64-bit.
Here we add back the `rsqrt` implementation for 32-bit, relying
on a `prsqrt` implementation with better handling of edge cases.
Note that several of the 32-bit NEON packet tests are currently
failing - either due to denormal handling (NEON versions flush
to zero, but scalar paths don't) or due to accuracy (e.g. sin/cos).
|
| |
|
|
| |
Forgot to test this. Fixes bug introduced in !416.
|
| |
|
|
|
|
|
|
|
|
| |
The original will saturate if the input does not fit into an integer
type. Here we fix this, returning the input if it doesn't have
enough precision to have a fractional part.
Also added `pceil` for NEON.
Fixes #1969.
|
| | |
|
| |
|
|
| |
in certain situations.
|
| |
|
|
| |
to non-const when using vector_pair with certain built-ins.
|
| |
|
|
|
|
|
| |
Accuracy is too poor - requires at least two Newton iterations, but then
it is no longer significantly faster than `vsqrt`.
Fixes #2094.
|
Rasmus Munk Larsen
Rohit Santhanam
Antonio Sanchez
Chip-Kerchner
guoqiangqi
Christoph Hertzberg
Jakub Lichman
David Tellenbach
Steve Bronder
Deven Desai