Start by reading the large commentary at the beginning of xdisp.c,
which providers a general overview of the display engine and its basic
structure and algorithms.  After you digest that, we'd need to talk
about the details, because they are non-trivial.  Let me show you the
tip of the iceberg (and it could be that what I say below won't make
sense until you read that commentary).

Then there will be issues with scrolling: do we swap horizontal and
vertical scrolling for vertical-layout text, or do we keep the scroll
direction (in which case we need to decide what it means to scroll up
or down)?

AFAIU, some vertical scripts are written in rows left-to-right and
some right-to-left, so you actually have 2 different layouts on your
hands.

This is an even tougher issue.  Is support for such mixed scripts a
necessity without which the entire feature won't make sense?  If not,
I'd suggest to limit yourself to either vertical or horizontal layout,
leaving the mix of the two unsupported for starters.  If mixing them
is a must, everything I said above needs to be rethought, and it could
even be that the current design of the display engine simply cannot be
extended to support that.

Do vertical-layout scripts require complex text shaping
(a.k.a. "character composition")?  If so, does HarfBuzz support such
scripts and their corresponding fonts?  I don't know.  We may need a
different shaping engine (if one exists).