Commit message (Collapse) | Author | Age | |
---|---|---|---|
* | evalSubExprsIfNeededAsync + async TensorContractionThreadPool | Eugene Zhulenev | 2019-08-30 |
| | |||
* | Revert accidentally removed <memory> header from ThreadPool | Eugene Zhulenev | 2019-08-30 |
| | |||
* | Asynchronous expression evaluation with TensorAsyncDevice | Eugene Zhulenev | 2019-08-30 |
| | |||
* | Const correctness in TensorMap<const Tensor<T, ...>> expressions | Eugene Zhulenev | 2019-08-28 |
| | |||
* | Remove shadow warnings in TensorDeviceThreadPool | Eugene Zhulenev | 2019-08-28 |
| | |||
* | Merged in ezhulenev/eigen-01 (pull request PR-683) | Rasmus Larsen | 2019-08-26 |
|\ | | | | | | | Asynchronous parallelFor in Eigen ThreadPoolDevice | ||
* | | Fix get_random_seed on Native Client | maratek | 2019-08-23 |
| | | | | | | | | | | Newlib in Native Client SDK does not provide ::random function. Implement get_random_seed for NaCl using ::rand, similarly to Windows version. | ||
| * | Asynchronous parallelFor in Eigen ThreadPoolDevice | Eugene Zhulenev | 2019-08-22 |
|/ | |||
* | Remove XSMM support from Tensor module | Eugene Zhulenev | 2019-08-19 |
| | |||
* | [Eigen] Vectorize evaluation of coefficient-wise functions over tensor ↵ | Rasmus Munk Larsen | 2019-08-07 |
| | | | | | | | | | | | | blocks if the strides are known to be 1. Provides up to 20-25% speedup of the TF cross entropy op with AVX. A few benchmark numbers: name old time/op new time/op delta BM_Xent_16_10000_cpu 448µs ± 3% 389µs ± 2% -13.21% (p=0.008 n=5+5) BM_Xent_32_10000_cpu 575µs ± 6% 454µs ± 3% -21.00% (p=0.008 n=5+5) BM_Xent_64_10000_cpu 933µs ± 4% 712µs ± 1% -23.71% (p=0.008 n=5+5) | ||
* | Clean up unnecessary namespace specifiers in TensorBlock.h. | Rasmus Munk Larsen | 2019-08-07 |
| | |||
* | Fix performance regressions due to ↵ | Rasmus Munk Larsen | 2019-08-02 |
| | | | | | | | | | | | | | | | | | https://bitbucket.org/eigen/eigen/pull-requests/662. The change caused the device struct to be copied for each expression evaluation, and caused, e.g., a 10% regression in the TensorFlow multinomial op on GPU: Benchmark Time(ns) CPU(ns) Iterations ---------------------------------------------------------------------- BM_Multinomial_gpu_1_100000_4 128173 231326 2922 1.610G items/s VS Benchmark Time(ns) CPU(ns) Iterations ---------------------------------------------------------------------- BM_Multinomial_gpu_1_100000_4 146683 246914 2719 1.509G items/s | ||
* | Fix expression evaluation heuristic for TensorSliceOp | Eugene Zhulenev | 2019-07-09 |
| | |||
* | Add outer/inner chipping optimization for chipping dimension specified at ↵ | Eugene Zhulenev | 2019-07-03 |
| | | | | runtime | ||
* | adding the EIGEN_DEVICE_FUNC attribute to the constCast routine. | Deven Desai | 2019-07-02 |
| | | | | | | | | | | | | | | | | | | | | | | Not having this attribute results in the following failures in the `--config=rocm` TF build. ``` In file included from tensorflow/core/kernels/cross_op_gpu.cu.cc:20: In file included from ./tensorflow/core/framework/register_types.h:20: In file included from ./tensorflow/core/framework/numeric_types.h:20: In file included from ./third_party/eigen3/unsupported/Eigen/CXX11/Tensor:1: In file included from external/eigen_archive/unsupported/Eigen/CXX11/Tensor:140: external/eigen_archive/unsupported/Eigen/CXX11/src/Tensor/TensorChipping.h:356:37: error: 'Eigen::constCast': no overloaded function has restriction specifiers that are compatible with the ambient context 'data' typename Storage::Type result = constCast(m_impl.data()); ^ external/eigen_archive/unsupported/Eigen/CXX11/src/Tensor/TensorChipping.h:356:37: error: 'Eigen::constCast': no overloaded function has restriction specifiers that are compatible with the ambient context 'data' external/eigen_archive/unsupported/Eigen/CXX11/src/Tensor/TensorAssign.h:148:56: note: in instantiation of member function 'Eigen::TensorEvaluator<const Eigen::TensorChippingOp<1, Eigen::TensorMap<Eigen::Tensor<int, 2, 1, long>, 16, MakePointer> >, Eigen::Gpu\ Device>::data' requested here return m_rightImpl.evalSubExprsIfNeeded(m_leftImpl.data()); ``` Adding the EIGEN_DEVICE_FUNC attribute resolves those errors | ||
* | Merged in codeplaysoftware/eigen (pull request PR-667) | Gael Guennebaud | 2019-07-02 |
|\ | | | | | | | | | | | | | [SYCL] : Approved-by: Gael Guennebaud <g.gael@free.fr> Approved-by: Rasmus Larsen <rmlarsen@google.com> | ||
* | | Allocate non-const scalar buffer for block evaluation with DefaultDevice | Eugene Zhulenev | 2019-07-01 |
| | | |||
| * | [SYCL] : | Mehdi Goli | 2019-07-01 |
|/ | | | | | | | * Modifying TensorDeviceSYCL to use `EIGEN_THROW_X`. * Modifying TensorMacro to use `EIGEN_TRY/CATCH(X)` macro. * Modifying TensorReverse.h to use `EIGEN_DEVICE_REF` instead of `&`. * Fixing the SYCL device macro in SpecialFunctionsImpl.h. | ||
* | Fix TensorReverse on GPU with m_stride[i]==0 | Eugene Zhulenev | 2019-06-28 |
| | |||
* | Fix preprocessor condition to only generate a warning when calling ↵ | Rasmus Munk Larsen | 2019-06-28 |
| | | | | eigen::GpuDevice::synchronize() from device code, but not when calling from a non-GPU compilation unit. | ||
* | Remove comma causing warning in c++03 mode. | Rasmus Munk Larsen | 2019-06-28 |
| | |||
* | Merge with Eigen head | Eugene Zhulenev | 2019-06-28 |
|\ | |||
* | | Add block access to TensorReverseOp and make sure that TensorForcedEval uses ↵ | Eugene Zhulenev | 2019-06-28 |
| | | | | | | | | block access when preferred | ||
| * | [SYCL] This PR adds the minimum modifications to the Eigen unsupported ↵ | Mehdi Goli | 2019-06-28 |
|/ | | | | | | | | | | module required to run it on devices supporting SYCL. * Abstracting the pointer type so that both SYCL memory and pointer can be captured. * Converting SYCL virtual pointer to SYCL device memory in Eigen evaluator class. * Binding SYCL placeholder accessor to command group handler by using bind method in Eigen evaluator node. * Adding SYCL macro for controlling loop unrolling. * Modifying the TensorDeviceSycl.h and SYCL executor method to adopt the above changes. | ||
* | Remove extra comma (causes warnings in C++03) | Christoph Hertzberg | 2019-06-26 |
| | |||
* | Optimize evaluation strategy for TensorSlicingOp and TensorChippingOp | Eugene Zhulenev | 2019-06-25 |
| | |||
* | Clean up CUDA/NVCC version macros and their use in Eigen, and a few other ↵ | Rasmus Munk Larsen | 2019-05-31 |
| | | | | CUDA build failures. | ||
* | Merged in rmlarsen/eigen (pull request PR-643) | Rasmus Larsen | 2019-05-20 |
|\ | | | | | | | | | | | Make Eigen build with cuda 10 and clang. Approved-by: Justin Lebar <justin.lebar@gmail.com> | ||
* | | Prevent potential division by zero in TensorExecutor | Eugene Zhulenev | 2019-05-17 |
| | | |||
* | | Always evaluate Tensor expressions with broadcasting via tiled evaluation ↵ | Eugene Zhulenev | 2019-05-16 |
| | | | | | | | | code path | ||
| * | Make Eigen build with cuda 10 and clang. | Rasmus Munk Larsen | 2019-05-15 |
|/ | |||
* | A) fix deadlocks in thread pool caused by EventCount | Rasmus Munk Larsen | 2019-05-08 |
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This fixed 2 deadlocks caused by sloppiness in the EventCount logic. Both most likely were introduced by cl/236729920 which includes the new EventCount algorithm: https://github.com/eigenteam/eigen-git-mirror/commit/01da8caf003990967e42a2b9dc3869f154569538 bug #1 (Prewait): Prewait must not consume existing signals. Consider the following scenario. There are 2 thread pool threads (1 and 2) and 1 external thread (3). RunQueue is empty. Thread 1 checks the queue, calls Prewait, checks RunQueue again and now is going to call CommitWait. Thread 2 checks the queue and now is going to call Prewait. Thread 3 submits 2 tasks, EventCount signals is set to 1 because only 1 waiter is registered the second signal is discarded). Now thread 2 resumes and calls Prewait and takes away the signal. Thread 1 resumes and calls CommitWait, there are no pending signals anymore, so it blocks. As the result we have 2 tasks, but only 1 thread is running. bug #2 (CancelWait): CancelWait must not take away a signal if it's not sure that the signal was meant for this thread. When one thread blocks and another submits a new task concurrently, the EventCount protocol guarantees only the following properties (similar to the Dekker's algorithm): (a) the registered waiter notices presence of the new task and does not block (b) the signaler notices presence of the waiters and wakes it (c) both the waiter notices presence of the new task and signaler notices presence of the waiter [it's only that both of them do not notice each other must not be possible, because it would lead to a deadlock] CancelWait is called for cases (a) and (c). For case (c) it is OK to take the notification signal away, but it's not OK for (a) because nobody queued a signals for us and we take away a signal meant for somebody else. Consider: Thread 1 calls Prewait, checks RunQueue, it's empty, now it's going to call CommitWait. Thread 3 submits 2 tasks, EventCount signals is set to 1 because only 1 waiter is registered the second signal is discarded). Thread 2 calls Prewait, checks RunQueue, discovers the tasks, calls CancelWait and consumes the pending signal (meant for thread 1). Now Thread 1 resumes and calls CommitWait, since there are no signals it blocks. As the result we have 2 tasks, but only 1 thread is running. Both deadlocks are only a problem if the tasks require parallelism. Most computational tasks do not require parallelism, i.e. a single thread will run task 1, finish it and then dequeue and run task 2. This fix undoes some of the sloppiness in the EventCount that was meant to reduce CPU consumption by idle threads, because we now have more threads running in these corner cases. But we still don't have pthread_yield's and maybe the strictness introduced by this change will actually help to reduce tail latency because we will have threads running when we actually need them running. B) fix deadlock in thread pool caused by RunQueue This fixed a deadlock caused by sloppiness in the RunQueue logic. Most likely this was introduced with the non-blocking thread pool. The deadlock only affects workloads that require parallelism. Most computational tasks don't require parallelism. PopBack must not fail spuriously. If it does, it can effectively lead to single thread consuming several wake up signals. Consider 2 worker threads are blocked. External thread submits a task. One of the threads is woken. It tries to steal the task, but fails due to a spurious failure in PopBack (external thread submits another task and holds the lock). The thread executes blocking protocol again (it won't block because NonEmptyQueueIndex is precise and the thread will discover pending work, but it has called PrepareWait). Now external thread submits another task and signals EventCount again. The signal is consumed by the first thread again. But now we have 2 tasks pending but only 1 worker thread running. It may be possible to fix this in a different way: make EventCount::CancelWait forward wakeup signal to a blocked thread rather then consuming it. But this looks more complex and I am not 100% that it will fix the bug. It's also possible to have 2 versions of PopBack: one will do try_to_lock and another won't. Then worker threads could first opportunistically check all queues with try_to_lock, and only use the blocking version before blocking. But let's first fix the bug with the simpler change. | ||
* | Fix stupid shadow-warnings (with old clang versions) | Christoph Hertzberg | 2019-05-07 |
| | |||
* | Restore C++03 compatibility | Christoph Hertzberg | 2019-05-07 |
| | |||
* | Check if gpu_assert was overridden in TensorGpuHipCudaDefines | Eugene Zhulenev | 2019-04-25 |
| | |||
* | Remove deprecation annotation from typedef Eigen::Index Index, as it would ↵ | Rasmus Munk Larsen | 2019-04-24 |
| | | | | generate too many build warnings. | ||
* | Add missing EIGEN_DEPRECATED annotations to deprecated functions and fix few ↵ | Eugene Zhulenev | 2019-04-23 |
| | | | | other doxygen warnings | ||
* | Adding lowlevel APIs for optimized RHS packet load in TensorFlow | Anuj Rawat | 2019-04-20 |
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | SpatialConvolution Low-level APIs are added in order to optimized packet load in gemm_pack_rhs in TensorFlow SpatialConvolution. The optimization is for scenario when a packet is split across 2 adjacent columns. In this case we read it as two 'partial' packets and then merge these into 1. Currently this only works for Packet16f (AVX512) and Packet8f (AVX2). We plan to add this for other packet types (such as Packet8d) also. This optimization shows significant speedup in SpatialConvolution with certain parameters. Some examples are below. Benchmark parameters are specified as: Batch size, Input dim, Depth, Num of filters, Filter dim Speedup numbers are specified for number of threads 1, 2, 4, 8, 16. AVX512: Parameters | Speedup (Num of threads: 1, 2, 4, 8, 16) ----------------------------|------------------------------------------ 128, 24x24, 3, 64, 5x5 |2.18X, 2.13X, 1.73X, 1.64X, 1.66X 128, 24x24, 1, 64, 8x8 |2.00X, 1.98X, 1.93X, 1.91X, 1.91X 32, 24x24, 3, 64, 5x5 |2.26X, 2.14X, 2.17X, 2.22X, 2.33X 128, 24x24, 3, 64, 3x3 |1.51X, 1.45X, 1.45X, 1.67X, 1.57X 32, 14x14, 24, 64, 5x5 |1.21X, 1.19X, 1.16X, 1.70X, 1.17X 128, 128x128, 3, 96, 11x11 |2.17X, 2.18X, 2.19X, 2.20X, 2.18X AVX2: Parameters | Speedup (Num of threads: 1, 2, 4, 8, 16) ----------------------------|------------------------------------------ 128, 24x24, 3, 64, 5x5 | 1.66X, 1.65X, 1.61X, 1.56X, 1.49X 32, 24x24, 3, 64, 5x5 | 1.71X, 1.63X, 1.77X, 1.58X, 1.68X 128, 24x24, 1, 64, 5x5 | 1.44X, 1.40X, 1.38X, 1.37X, 1.33X 128, 24x24, 3, 64, 3x3 | 1.68X, 1.63X, 1.58X, 1.56X, 1.62X 128, 128x128, 3, 96, 11x11 | 1.36X, 1.36X, 1.37X, 1.37X, 1.37X In the higher level benchmark cifar10, we observe a runtime improvement of around 6% for AVX512 on Intel Skylake server (8 cores). On lower level PackRhs micro-benchmarks specified in TensorFlow tensorflow/core/kernels/eigen_spatial_convolutions_test.cc, we observe the following runtime numbers: AVX512: Parameters | Runtime without patch (ns) | Runtime with patch (ns) | Speedup ---------------------------------------------------------------|----------------------------|-------------------------|--------- BM_RHS_NAME(PackRhs, 128, 24, 24, 3, 64, 5, 5, 1, 1, 256, 56) | 41350 | 15073 | 2.74X BM_RHS_NAME(PackRhs, 32, 64, 64, 32, 64, 5, 5, 1, 1, 256, 56) | 7277 | 7341 | 0.99X BM_RHS_NAME(PackRhs, 32, 64, 64, 32, 64, 5, 5, 2, 2, 256, 56) | 8675 | 8681 | 1.00X BM_RHS_NAME(PackRhs, 32, 64, 64, 30, 64, 5, 5, 1, 1, 256, 56) | 24155 | 16079 | 1.50X BM_RHS_NAME(PackRhs, 32, 64, 64, 30, 64, 5, 5, 2, 2, 256, 56) | 25052 | 17152 | 1.46X BM_RHS_NAME(PackRhs, 32, 256, 256, 4, 16, 8, 8, 1, 1, 256, 56) | 18269 | 18345 | 1.00X BM_RHS_NAME(PackRhs, 32, 256, 256, 4, 16, 8, 8, 2, 4, 256, 56) | 19468 | 19872 | 0.98X BM_RHS_NAME(PackRhs, 32, 64, 64, 4, 16, 3, 3, 1, 1, 36, 432) | 156060 | 42432 | 3.68X BM_RHS_NAME(PackRhs, 32, 64, 64, 4, 16, 3, 3, 2, 2, 36, 432) | 132701 | 36944 | 3.59X AVX2: Parameters | Runtime without patch (ns) | Runtime with patch (ns) | Speedup ---------------------------------------------------------------|----------------------------|-------------------------|--------- BM_RHS_NAME(PackRhs, 128, 24, 24, 3, 64, 5, 5, 1, 1, 256, 56) | 26233 | 12393 | 2.12X BM_RHS_NAME(PackRhs, 32, 64, 64, 32, 64, 5, 5, 1, 1, 256, 56) | 6091 | 6062 | 1.00X BM_RHS_NAME(PackRhs, 32, 64, 64, 32, 64, 5, 5, 2, 2, 256, 56) | 7427 | 7408 | 1.00X BM_RHS_NAME(PackRhs, 32, 64, 64, 30, 64, 5, 5, 1, 1, 256, 56) | 23453 | 20826 | 1.13X BM_RHS_NAME(PackRhs, 32, 64, 64, 30, 64, 5, 5, 2, 2, 256, 56) | 23167 | 22091 | 1.09X BM_RHS_NAME(PackRhs, 32, 256, 256, 4, 16, 8, 8, 1, 1, 256, 56) | 23422 | 23682 | 0.99X BM_RHS_NAME(PackRhs, 32, 256, 256, 4, 16, 8, 8, 2, 4, 256, 56) | 23165 | 23663 | 0.98X BM_RHS_NAME(PackRhs, 32, 64, 64, 4, 16, 3, 3, 1, 1, 36, 432) | 72689 | 44969 | 1.62X BM_RHS_NAME(PackRhs, 32, 64, 64, 4, 16, 3, 3, 2, 2, 36, 432) | 61732 | 39779 | 1.55X All benchmarks on Intel Skylake server with 8 cores. | ||
* | Tweak cost model for tensor contraction when parallelizing over the inner ↵ | Rasmus Munk Larsen | 2019-04-12 |
| | | | | | | dimension. https://bitbucket.org/snippets/rmlarsen/MexxLo | ||
* | Update TheadPoolDevice example to include ThreadPool creation and passing ↵ | Jonathon Koyle | 2019-04-10 |
| | | | | pointer into constructor. | ||
* | adding EIGEN_DEVICE_FUNC to the recently added TensorContractionKernel ↵ | Deven Desai | 2019-04-08 |
| | | | | constructor. Not having the EIGEN_DEVICE_FUNC attribute on it was leading to compiler errors when compiling Eigen in the ROCm/HIP path | ||
* | Add missing semicolon | Eugene Zhulenev | 2019-04-02 |
| | |||
* | Add support for custom packed Lhs/Rhs blocks in tensor contractions | Eugene Zhulenev | 2019-04-01 |
| | |||
* | Merged eigen/eigen into default | Deven Desai | 2019-03-19 |
|\ | |||
| * | Fix segfaults with cuda compilation | Eugene Zhulenev | 2019-03-11 |
| | | |||
| * | Fix a bug in TensorGenerator for 1d tensors | Eugene Zhulenev | 2019-03-11 |
| | | |||
| * | Fix a data race in NonBlockingThreadPool | Eugene Zhulenev | 2019-03-11 |
| | | |||
| * | Merge. | Rasmus Munk Larsen | 2019-03-06 |
| |\ | |||
| * | | Add macro EIGEN_AVOID_THREAD_LOCAL to make it possible to manually disable ↵ | Rasmus Munk Larsen | 2019-03-06 |
| | | | | | | | | | | | | the use of thread_local. | ||
| | * | Fix placement of "#if defined(EIGEN_GPUCC)" guard region. | Rasmus Munk Larsen | 2019-03-06 |
| | |\ | | | | | | | | | | | | | | | | | | | | | Found with -Wundefined-func-template. Author: tkoeppe@google.com |