Hey Rob
I can sure understand the temptation to go back to DX with the D500. But having said that, I am delighted with the results from my D800 and sorry to hear you are having sharpness problems.
I did have that issue using a Nikkor 16-35 with an 8" (Subal) domeport and +2 diopter. Edges were awful. But the 9" is WAY better (still with the +2 diopter) but, agreed, not the easiest to travel with; and with a Sigma 15mm in the 9" dome the images are terrific. I never tried the Sigma in an 8" dome but others have said it works well.
I was happy with a Nikkor 12-24 in an 8" port with a +2 diopter when I had a D300 - so DX sensor. As I said earlier the Tokina 10-17 with either an 8" (with +2 diopter) or 4" port were also really good. As was the Nikkor 10.5 in either port. Sorry, no experience of the Nikkor 10-24.
Did you get any idea at Boot of how Subal will trigger fibre optics with the D500? Are they building-in some sort of trigger?