Your wall isn’t really a wall either. It consists of different layers that are not equal to each other. Because of this you create the opportunity to climb on it. This would work the same in real life.
You can improve this by equalizing the parts or making it one whole (a parent entity with the collision and rigidbody component and some children with a render component).