Well, the real problem is really "server that works on behalf of somebody
X is just the worst *practical* example of this, since not only is it the
most common such server, it's also a case where people see interactive
issues really easily.
And the problem is that a lot of clients actually end up doing *more* in
the X server than they do themselves directly. Doing things like showing a
line of text on the screen is a lot more expensive than just keeping track
of that line of text, so you end up with the X server easily being marked
as getting "too much" CPU time, and the clients as being starved for CPU
time. And then you get bad interactive behaviour.
So "good fairness" really should involve some notion of "work done for
others". It's just not very easy to do..