Button element not resetting to Texture asset on end hover


I have a button, which uses a HoverSprite asset.
It works correctly when the project starts by showing the texture asset. Once I hover the mouse over it, the hoverSprite shows up. But when the hovering ends, the button does not revert to the original texture asset, Instead it shows the Color value of the element.

Any workaround?