hwc/overlay: Fix decimation / bwc bugs
1) While setting BWC check for 90 transform and swap crop
2) Calculate decimation in the getDecimationFactor API itself.
Currently, a residual downscale is calculated and the clients have
to take logs and apply forcible decimation like in MdpCtrl
3) Fix bug where transform gets typecasted to bool
b/14225977
Change-Id: I3c99c571e02e2cf7822342516b6a87d97be553d1
diff --git a/libhwcomposer/hwc_mdpcomp.cpp b/libhwcomposer/hwc_mdpcomp.cpp
index b0953da..44a1317 100644
--- a/libhwcomposer/hwc_mdpcomp.cpp
+++ b/libhwcomposer/hwc_mdpcomp.cpp
@@ -347,7 +347,7 @@
hwc_rect_t crop = integerizeSourceCrop(layer->sourceCropf);
hwc_rect_t dst = layer->displayFrame;
- bool rotated90 = (bool)layer->transform & HAL_TRANSFORM_ROT_90;
+ bool rotated90 = (bool)(layer->transform & HAL_TRANSFORM_ROT_90);
int crop_w = rotated90 ? crop.bottom - crop.top : crop.right - crop.left;
int crop_h = rotated90 ? crop.right - crop.left : crop.bottom - crop.top;
int dst_w = dst.right - dst.left;