I recall that if you use two simultaneously directly into the Xbox it causes problems. It’s worth it to get a 4 port USB hub and plug them into that. It works for sure and as @zerowhitemonkey says it prevents any issues with putting strain in the socket.
I am using this setup with two PS3 - XBoxOne Brooks adapters and it works great.