| commit | 39c865b3aa8587d426e726380e5e6f70ac4b6f6d | [log] [tgz] |
|---|---|---|
| author | Xusong Wang <xusongw@google.com> | Tue Aug 28 16:50:01 2018 -0700 |
| committer | Miao Wang <miaowang@google.com> | Thu Sep 06 21:41:06 2018 +0000 |
| tree | fa15d895c67143c0e578351d87bdf0808506b279 | |
| parent | 391eb588cf3132b5de8d290154d0ffbc4135a981 [diff] |
Fix logics for floating-point comparision in VTS test. Set the acceptable error range based on both absolute tolerance and relative tolerance. Currently, absolute tolerance is set to 1e-5 for FP32 and 5 epsilon (~5e-3) for FP16 relaxed computation. The relative tolerance is set to 5ULP of the corresponding precision. Add a TODO mark for potential future adjustment on error limit based on testing. Bug: 111768023 Test: none Change-Id: Idedcec3e09fd7de9696811b93c81d0f180e896ef