Okay, I understand. Thanks for your effort though!
The problem is mainly that the raycast starts from the player and then first hit the player. That prevents me from using raycastFirst.
Maybe someone knows how to work around this without using raycastAll because using raycastAll again causes problems like the problem of this post.
And yes, that can come in handy anyway.