PS4 FIFA 20 Pro Clubs LAN keeps crashing

So I mostly play Pro Clubs with my friend and my little brother, he plays on his PS4 located at his room, I play on my own PS4 located at my playroom, we are connected to the same network (at home, it is called JARVIS), but very rarely we get to play together, not issues with the rest of the team, it is just my brother and me, one of us, gets kicked out of the match when FIFA20 finds a rival for the matchmaking, it takes us both to the uniform selection screen, but for one of us, it gets stuck loading that certain screen, until it tells us that the connection was lost with the rival. If 7 of us are playing together (5 friends, my brother and me), only 6 are able to play (5 friends, and either my bother or me)

We have tried every LAN connectivity method possible, wiring both PS4's, Wifi-2G and Wifi-5G, we just can't get it to work

We have played together in many different games and we have not had any issues like we have here, our conclusion was, it's got to be FIFA 20

PLEASE HELP ME OUT