The part that is counterintuitive to most people when it comes to the "server" terminology is that, with X, your end-user workstation (which may be an incredibly dumb X terminal) is the "display server", which means you remote into a server (in the traditional sense) elsewhere, which then acts as an X client by making requests to your local machine to display windows.
The way most people think about it, "client" is your local machine and "server" is the remote machine that has lots of applications and is potentially multi-user, but X turns that backwards. The big iron is the client and the relatively dumb terminal is the server.
I think the confusion is obvious, given a little empathy for the range of people who use computers.
The server is usually a remote machine, especially back in the time when "client-server" architecture was emerging in mainstream (business) vernacular.
Please don't imagine that I don't fully understand this.
Nevertheless, X11 "server" and "client" have confused very smart and highly technical people. I have had the entertainment of explaining it dozens of times, though rarely recently.
And honestly, still, a server is usually a remote machine in all common usage. When "the server's down", it is usually not a problem on your local machine.
Yes, it’s simultaneously logical if you look at how it works and immensely strange if you don’t understand the architecture. (As has been noted all the way back to the UNIX-HATERS Handbook[1], although, pace 'DonHopkins, the NeWS Book uses the same terminology—possibly because it was written late enough to contain promises of X11/NeWS.)
There is nothing at all strange about the terminology. Go run ps on macOS and marvel at the "WindowServer" process. The generic architectural term is "display server".
This is true, although entertainingly, the "server" part has always been easily confused.
In X11, the "server" runs on your local machine, and the "client" frequently runs on a remote system.