I’ve got a hunch how — at least in some people’s minds — “tray” ceiling became “trey”: “très” is French for “very” (as in “very beautiful”).
From there, it’s just a small step (and single letter) to “trey.”
Of course, that’s also how the word sounds phonetically.
However, the correct adjective for a recessed ceiling — often directly over a bed or Dining Room table, and marked with dramatic, custom moldings — is “tray.”
As in, it resembles an inverted tray.