Introduction

The main purpose of this article is to gather information in the field of transformation in computer graphics and put it in one place. All the information contained in this article can be found in the depths of the Internet. Unfortunately, the information contained there about matrices and transformations is not exhaustive and does not always give details of the subtleties involved. In this article, I want to discuss these subtleties and give you a full picture of the problem.

One of the three culprits associated with transformations in computer graphics, and invisible at first glance, is the property of matrix multiplication, which in general is not commutative. Of course, there are examples of matrices whose multiplication is commutative, such pairs are just the exception to the rule.

\begin{array}{c} \mathbf{A} * \mathbf{B} \neq \mathbf{B} * \mathbf{A} \end{array}

where A and B are matrices.

We’re used to the concept of commutative multiplication of numbers (scalars). Unfortunately this property does not apply to matrix multiplication. This fact gives rise to some surprising rules that must be known and consistently adhered to.

Secondly, in a 3D Cartesian coordinate system, the result of the cross product of two non-collinear vectors is anticommutative.

\begin{array}{c} \vec{a} \times \vec{b} = - \vec{b} \times \vec{a} \end{array}

This property means the cross-product doesn’t have one solution in 3D Cartesian coordinate system but two equally correct solutions. To solve this ambiguity we need a choice of orientation (or “handedness”) for the 3D Cartesian coordinate system, and by choosing it we will prefer one cross-product solution over another. Again, at first glance, we might say “it’s just adding a minus sign”, however it’s not. This property not only determines the direction of the surface normal vectors, but also determines the direction in which the points rotate around the axis.

The third hides in the shadows of linear algebra textbooks and is the concept of basis vectors. The order of these basis vectors, which defines the coordinate system (and therefore the direction of the axes of the coordinate system and more), is very important. To be able to talk about transformation matrices in 2D or 3D spaces, which is not often addressed in books or articles on the subject, it is necessary to define the base of the linear space on which these transformations will operate. If this is not done, discussions about transformations will not have a common basis, which can lead to unprovable disputes.

From the algebraic and computational point of view, there is no “best” choice of basis vector order, however, e.g. physicists have chosen one convention and consistently apply it in their work. In this way, there are no understatements that hinder cooperation and, what is very important, all calculations are consistent even if they are performed by different people. What cannot be said about computer graphics, where this “common language” is lacking. Therefore, this article aims to first introduce the problem to start a process for developing a “common language” for issues related to transformations in 3D spaces.

Problem complexity

In the real world, many problems often look very simple at first glance, and only after a closer look the complexity reveals itself very quickly. In this article, we want to get to the bottom of this problem. To deal with this, let’s use the old and well-known divide and conquer technique. So let’s break this complex problem down into separate sub-problems, then tackle and solve them separately. This approach will help us ultimately create a neat cheat sheet for all of us and hopefully it finally sums up all the “mess” we encounter when working with transformations in computer graphics.

These subproblems can be divided not only from the point of view of mathematics (theoretical), but also from the point of view of implementation (practical) into:

Matrix storage in memory as a multidimensional array.
Matrix multiplication order.
3D Cartesian coordinate system “handedness”.
3D basis vectors.

Matrix storage in memory as a multidimensional array

In mathematics, a matrix is defined as a rectangular array of numbers arranged in rows and columns.

For example, the matrix below has 3 rows and 5 columns, and can be referred to as a $\mathbf{3 \times 5}$ matrix.

\begin{array}{c} \begin{bmatrix} m_{11} & m_{12} & m_{13} & m_{14} & m_{15}\\ m_{21} & m_{22} & m_{23} & m_{24} & m_{25}\\ m_{31} & m_{32} & m_{33} & m_{34} & m_{35}\\ \end{bmatrix} \end{array}

But how can we represent this matrix in computer memory?

A common solution is to use multidimensional arrays. However computer memory is linear. In other words, it’s a one-dimensional array. Consequently, multidimensional arrays can be stored using two “simple” layout orders:


Row layout order
Column layout order

These “simple” layouts are used to store multidimensional arrays(matrices) depending on the programming language:


Row layout order	C C++ HLSL (by default or with `row_major` modifier) GLSL (with `layout(row_major)` qualifier)
Column layout order	Fortran Matlab HLSL (with `column_major` modifier) GLSL (by default or with `layout(column_major)` qualifier)

Multidimensional arrays in C/C++

As the above table describes in C/C++, multidimensional arrays are stored in row-major order which can be understood as storing multidimensional arrays in memory row by row.

For example, the following matrix:

\begin{array}{c} \begin{bmatrix} m_{11} & m_{12} & m_{13} & m_{14} & m_{15}\\ m_{21} & m_{22} & m_{23} & m_{24} & m_{25}\\ m_{31} & m_{32} & m_{33} & m_{34} & m_{35}\\ \end{bmatrix} \end{array}

in memory is stored as follows:

\begin{array}{|c|c|c|c|c|c|c|c|c|c|c|c|c|c|c|c|c|} \hline & m_{11} & m_{12} & m_{13} & m_{14} & m_{15} & m_{21} & m_{22} & m_{23} & m_{24} & m_{25} & m_{31} & m_{32} & m_{33} & m_{34} & m_{35} & \\ \hline \end{array}

The same multidimensional array in C/C++, can be written as follows:

int foo [3][5];

where the variable foo is an array containing 3 arrays which all contain 5 variables of type int.

How do we interpret this multidimensional array as a matrix, especially those two numbers 3 and 5? There are unfortunately two possibilities again.

[Row][Column] order

In this possibility e.g. the foo[3][5] is interpreted as $\mathbf{3 \times 5}$ matrix, which would be more intuitive to mathematicians because is “closer” to matrix definition (order of arrays dimension [3][5] coincides with matrix dimensions $3 \times 5$ ).

To access the elements of the matrix, we specify the desired row in the first square bracket, and the desired column in the second. Also in this possibility, the rows of the matrix are the rows of the multidimensional array.
[Column][Row] order

In this possibility e.g. the foo[3][5] is interpreted as $\mathbf{5 \times 3}$ matrix.

To access the elements of the matrix, we specify the desired column in the first square bracket, and the desired row in the second. Also in this possibility, the columns of the matrix are the rows of the multidimensional array.

The output of this possibility is that even though C++ in multidimensional arrays are stored as Row-Major layout we can “pretend” that they are stored as Column-Major.

Matrices in HLSL

In HLSL there are special data types for representing matrices up to $4 \times 4$ and vectors up to $4$ components.

For example, the float2x4 (with any modifier) data type is used to represent a $2 \times 4$ matrix and float2 to represent a $2$ component row/column vector.

HLSL does have an overloaded * operator but this operator is doing component-wise multiplication (each element of first matrix is multiplied by the corresponding element of the second matrix). Developers need to use mul(x, y) function to multiply vector/matrices. Sample HLSL source code might be as follows:

Copied!

float4 v;
float4x3 m; 
float3 vp = mul(v,m);

Vectors v and vp are treated as row vectors and this operation can be written in mathematical form as

\begin{array}{c} \vec{v_p} = \vec{v} * \mathbf{m} \\ \end{array} \\ \begin{array}{c} \begin{bmatrix} x & y & z \end{bmatrix} = \begin{bmatrix} v_1 & v_2 & v_3 & v_4 \end{bmatrix} * \begin{bmatrix} m_{11} & m_{12} & m_{13}\\ m_{21} & m_{22} & m_{23}\\ m_{31} & m_{32} & m_{33}\\ m_{41} & m_{42} & m_{43}\\ \end{bmatrix} \end{array}

When we do it the other way:

Copied!

float4 v;
float3x4 m; 
float3 vp = mul(m,v);

vectors v and vp are treated as column vectors and the operation can be written in mathematical form as

\begin{array}{c} \vec{v_p} = \mathbf{m} * \vec{v}\\ \end{array} \\ \begin{array}{c} \begin{bmatrix} x \\ y \\ z \end{bmatrix} = \begin{bmatrix} m_{11} & m_{12} & m_{13} & m_{14}\\ m_{21} & m_{22} & m_{23} & m_{24}\\ m_{31} & m_{32} & m_{33} & m_{34}\\ \end{bmatrix} * \begin{bmatrix} v_1 \\ v_2 \\ v_3 \\ v_4 \end{bmatrix} \end{array}

and lastly, when multiplying two vectors using the mul function:

Copied!

float4 a;
float4 b;
float d = mul(a,b);

it is the equivalent of the dot product of those vectors (i.e. dot(x, y)).

\begin{array}{c} d = \vec{a} \cdot \vec{b} \\ \end{array} \\ \begin{array}{c} d = \begin{bmatrix} a_1 & a_2 & a_3 & a_4\\ \end{bmatrix} * \begin{bmatrix} b_1 \\ b_2 \\ b_3 \\ b_4 \end{bmatrix} \end{array}

Matrices in GLSL

In GLSL there are special data types for representing matrices up to $4 \times 4$ and vectors with up to $4$ components.

For example, the mat2x4 (with any modifier) data type is used to represent a $4 \times 2$ matrix and with vec2 representing a $2$ component row/column vector.

GLSL has an overloaded * operator which is used to multiply scalars as well as multiply matrices and vectors. Sample GLSL source code might be as follows:

Copied!

vec4 v;
mat3x4 m;   
vec3 vp = v * m;

Vectors v and vp are treated as row vectors. This operation can be written in mathematical form as:

\begin{array}{c} \vec{v_p} = \vec{v} * \mathbf{m} \\ \end{array} \\ \begin{array}{c} \begin{bmatrix} x & y & z \end{bmatrix} = \begin{bmatrix} v_1 & v_2 & v_3 & v_4 \end{bmatrix} * \begin{bmatrix} m_{11} & m_{12} & m_{13}\\ m_{21} & m_{22} & m_{23}\\ m_{31} & m_{32} & m_{33}\\ m_{41} & m_{42} & m_{43}\\ \end{bmatrix} \end{array}

When we do it the other way:

Copied!

vec4 v;
mat4x3 m;   
vec3 vp = m * v;

vectors v and vp are treated as column vectors. This operation we can written in mathematics form as

\begin{array}{c} \vec{v_p} = \mathbf{m} * \vec{v}\\ \end{array} \\ \begin{array}{c} \begin{bmatrix} x \\ y \\ z \end{bmatrix} = \begin{bmatrix} m_{11} & m_{12} & m_{13} & m_{14}\\ m_{21} & m_{22} & m_{23} & m_{24}\\ m_{31} & m_{32} & m_{33} & m_{34}\\ \end{bmatrix} * \begin{bmatrix} v_1 \\ v_2 \\ v_3 \\ v_4 \end{bmatrix} \end{array}

and lastly, when multiply two vectors using the * operator

Copied!

vec4 a;
vec4 b;
vec4 d = a * b;

we do not compute the dot product. Instead, component-wise multiplication is done.

To compute the dot product, the dot(x, y) function needs to be used:

Copied!

vec4 a;
vec4 b;
float d = dot(a,b);

Matrix multiplication order

In 3D applications, it is important to combine transformations such as translation, rotation or scaling to create realistic virtual worlds. Since all affine and protective transformations used in computer graphics can be represented as a $4 \times 4$ matrix and the concatenation of a transformation is just a matrix multiplication, it’s obvious that the order of the matrices in the multiplication is crucial.

The rule of matrix multiplication states that the number of columns in the left operand must be equal to the number of rows in the right operand.

\begin{array}{c} \begin{bmatrix} a_{11} & a_{12} \\ a_{21} & a_{22} \\ a_{31} & a_{32} \\ a_{41} & a_{42} \\ \end{bmatrix} * \begin{bmatrix} b_{11} & b_{12} & b_{13}\\ b_{21} & b_{22} & b_{23}\\ \end{bmatrix} = \begin{bmatrix} c_{11} & a_{12} & c_{13}\\ c_{21} & c_{22} & c_{23}\\ c_{31} & c_{32} & c_{33}\\ c_{41} & c_{42} & c_{43}\\ \end{bmatrix} \end{array}

Matrix multiplication (where A, B, C, D, E, Z are appropriate matrices) can be written as follows:

\begin{array}{c} \mathbf{Z} = \mathbf{A} * \mathbf{B} * \mathbf{C} * \mathbf{D} * \mathbf{E} \end{array}

Matrix multiplication in general is not commutative. So, to show our intentions regarding the sequence (order) of actions, it is helpful to use parentheses. Although in mathematics it is assumed that multiplication is from left to right.

Matrix multiplication is associative such that $(\mathbf{A} * \mathbf{B}) * \mathbf{C} = \mathbf{A} * (\mathbf{B} * \mathbf{C})$ is true.

What is described in the subsection below is only to mark/underline the order (direction) in which the transformations are combined.

Post-multiplication Transformation

The matrix post-multiplication order convention can be described as placing parentheses in this way (shown below) to indicate the intention of a particular order in which an operation (e.g. a combination of transformations) is performed.

\begin{array}{c} \mathbf{Z} = \mathbf{A} * \mathbf{B} * \mathbf{C} * \mathbf{D} * \mathbf{E} = \mathbf{A} * (\mathbf{B} * (\mathbf{C} * (\mathbf{D} * \mathbf{E}))) \end{array}

The consequence of this convention is that the right-to-left transformations are composed. Thus, implicitly forcing the matrix to be stored in the column-major layout and treating the vertices as column vectors.

In common usage in 3D applications this means:

Transform vector from one space to another

$\vec{V_w}$ = Transformed vector in another space. A vector is a column vector

$\vec{V_c}$ = Transformed vector in original space. A vector is a column vector

$\mathbf{M}$ = Transform Matrix

\begin{array}{c} \vec{V_w} = \mathbf{M} * \vec{V_c} \\ \end{array} \\ \begin{array}{c} \begin{bmatrix} x \\ y \\ z \\ w \end{bmatrix} = \begin{bmatrix} m_{11} & m_{12} & m_{13} & m_{14}\\ m_{21} & m_{22} & m_{23} & m_{24}\\ m_{31} & m_{32} & m_{33} & m_{34}\\ m_{41} & m_{42} & m_{43} & m_{44}\\ \end{bmatrix} * \begin{bmatrix} v_1 \\ v_2 \\ v_3 \\ v_4 \end{bmatrix} \end{array}

Computing Final Transformation Matrix

$\mathbf{M_f}$ = Final Matrix

$\mathbf{M_m}$ = Model transform, transforming from local model space to global world space

$\mathbf{M_v}$ = View matrix, transforming from world space to camera space

$\mathbf{M_p}$ = Projection Matrix, transforming from camera space to clipping space

\begin{array}{c} \mathbf{M_f} = \mathbf{M_p} * \mathbf{M_v} * \mathbf{M_m} \\ \end{array}

World Matrix Node Graph composition

$\mathbf{M_w}$ = World transformation matrix

$\mathbf{M_l}$ = Current local transformation matrix

$\mathbf{M_p}$ = Transformation matrix of the parent node

\begin{array}{c} \mathbf{M_w} = \mathbf{M_p} * \mathbf{M_l} \end{array}

Computing Local Transformation Matrix

$\mathbf{M_l}$ = Node current local transformation matrix

$\mathbf{M_s}$ = Scaling matrix

$\mathbf{M_r}$ = Rotation matrix

$\mathbf{M_t}$ = Translation matrix

\begin{array}{c} \mathbf{M_l} = \mathbf{M_t} * \mathbf{M_r} * \mathbf{M_s} \\ \end{array}

It is worth mentioning that in scientific works in the field of, for example, physics, this convection is preferred.

Pre-multiplication transformation

The matrix pre-multiplication order convention can be described as placing parentheses in this way (shown below) to indicate the intention of a particular order in which an operation (e.g. a combination of transformations) is performed.

\begin{array}{c} \mathbf{Z} = \mathbf{A} * \mathbf{B} * \mathbf{C} * \mathbf{D} * \mathbf{E} = (((\mathbf{A} * \mathbf{B}) * \mathbf{C}) * \mathbf{D}) * \mathbf{E} \end{array}

The consequence of this convention is that the left-to-right transformations are composed. Thus, implicitly forcing the matrix to be stored in the row-major layout and treating the vertices as row vectors.

In common usage in 3D applications this means:

Transform vector from one space to another

$\vec{V_w}$ = Transformed vector in another space. A vector is a row vector.

$\vec{V_c}$ = Transformed vector in original space. A vector is a row vector.

$\mathbf{M}$ = Transform Matrix.

\begin{array}{c} \vec{V_w} = \vec{V_c} * \mathbf{M} \\ \end{array} \\ \begin{array}{c} \begin{bmatrix} x & y & z & w\\ \end{bmatrix} = \begin{bmatrix} v_1 & v_2 & v_3 & v_4\\ \end{bmatrix} * \begin{bmatrix} m_{11} & m_{21} & m_{31} & m_{41}\\ m_{12} & m_{22} & m_{32} & m_{42}\\ m_{13} & m_{23} & m_{33} & m_{43}\\ m_{14} & m_{24} & m_{34} & m_{44}\\ \end{bmatrix} \end{array}

Computing Final Transformation Matrix

$\mathbf{M_f}$ = Final Matrix

$\mathbf{M_m}$ = Model transform, transforming from local model space to global world space

$\mathbf{M_v}$ = View matrix, transforming from world space to camera space

$\mathbf{M_p}$ = Projection Matrix, transforming from camera space to clipping space

\begin{array}{c} \mathbf{M_f} = \mathbf{M_m} * \mathbf{M_v} * \mathbf{M_p} \\ \end{array}

World Matrix Node Graph composition

$\mathbf{M_w}$ = World transformation matrix

$\mathbf{M_l}$ = Current local transformation matrix

$\mathbf{M_p}$ = Transformation matrix of the parent node

\begin{array}{c} \mathbf{M_w} = \mathbf{M_l} * \mathbf{M_p} \end{array}

Computing Local Transformation Matrix

$\mathbf{M_l}$ = Node current local transformation matrix

$\mathbf{M_s}$ = Scaling matrix

$\mathbf{M_r}$ = Rotation matrix

$\mathbf{M_t}$ = Translation matrix

\begin{array}{c} \mathbf{M_l} = \mathbf{M_s} * \mathbf{M_r} * \mathbf{M_t} \\ \end{array}

3D Cartesian coordinate system “handedness”

Before discussing the topic of the orientation of the 3D Cartesian coordinate system, one more issue should be mentioned and described, which is the “Up Direction” axial convention.

“Up Direction”

Mathematically, there is no preferred axis to be taken as the “Up Direction” for the 3D worlds. In 3D applications (fortunately) only two conventions have developed over the years. And they are:

“Up Direction” is $Z$ -Axis

This convention is very useful e.g. in architecture and when the “working space” is a plane that is the surface of e.g. the earth, then the two main working directions are $X$ -Axis and $Y$ -Axis, so then “Up Direction” (elevation) is $Z$ -Axis.
“Up Direction” is $Y$ -Axis

In this convention, the “working plane” is the monitor screen, the $X$ -Axis and $Y$ -Axis directions are the same as the width and height of the screen, respectively, and the $Z$ -Axis is the “depth” of the screen. So when a 3D virtual world is displayed on the screen, it simulates our everyday perception of the world, which we might experience in FPS games, for example. Therefore, the $Y$ -Axis monitor screen describes “Up Direction”.

Coordinate system “handedness”

To determine the 3D Cartesian coordinate system, we need to select three vectors (axes) that are perpendicular to each other. Then, by selecting the $X$ -Axis and the $Y$ -Axis (or $Z$ -Axis) perpendicular to it, we define a plane dividing the space into two half-spaces. The half-space which is “above” the $XY$ -Plane (or $XZ$ -Plane) is named positive, and “below” is named negative. This fact correlates with the cross-product in three-dimensional space. To get a unique and consistent result, we need to choose the direction of the resulting vector. This can be done by using the right or left-hand rule.

Left-hand rule	Right-hand rule

The right or left-hand rule can be briefly described as:

Let the straightened index finger of the right/left-hand point in the direction of the first vector ( $\vec{a}$ ), and the bent middle finger point in the direction of the second vector ( $\vec{b}$ ). Finally, depending on chosen hand the extended thumb then indicates the direction of the cross-product vector $\vec{a}\times\vec{b}$ .

To determine the directions of the $X , Y , Z$ axes of the 3D Cartesian coordinate system, replace the first vector with the direction of the $X$ -Axis, the second vector with the direction of the $Y$ -Axis, then the thumb will indicate the direction of the $Z$ -Axis.

In this way, we can designate a 3D Cartesian coordinate system with the desired “handedness” (orientation) which can be represented in the table below.

	Left-hand rule	Right-hand rule
Y-Up
Z-Up

It’s worth mentioning is that in mathematics and physics the right-hand orientation is the most common preferred orientation. That orientation is used to find directions e.g. in magnetic fields, electromagnetic fields, and enantiomers in chemistry.

3D basis vectors

If we want to talk about matrices which are representing geometric transformation (explicitly rotation matrices), we need to establish what in Linear Algebra is called the basis vectors. This topic should be presented before Coordinate system “handedness” as is taught in the Linear Algebra classes. However, describing it now will make it easier to understand. To properly define coordinate system basis vector, it is not enough to define the direction of those basis vectors (in other words the direction of the $X$ , $Y$ , and $Z$ axes in 3D space) but in addition, it is necessary to specify ORDER of those axes (determine which axis is the first basis vector, which second and which is third).

How this order is impacting transformations which are used in Computer Graphics?

It is most visible and impactful for Rotations. Because the order is naturally defining “Positive Direction” as what is a direction of rotation when angles of rotation are increasing (for specific rotation matrix). This is independent of the direction of basis vectors (coordinate system axis) as in the following pictures:

\begin{bmatrix}\cos(\theta) & -\sin(\theta) \ \sin(\theta) & \cos(\theta) \ \end{bmatrix}*\begin{bmatrix}x \ y \end{bmatrix}

\begin{bmatrix}\cos(\theta) & \sin(\theta) \ -\sin(\theta) & \cos(\theta) \ \end{bmatrix}*\begin{bmatrix}x \ y \end{bmatrix}

In 3D space, having a set containing three axes let us call them traditionally $X$ , $Y$ and $Z$ there are total of six permutations in which order we can use them to create basis vectors ${X,Y,Z}$ , ${Y,Z,X}$ , ${Z,X,Y}$ , ${X,Z,Y}$ , ${Y,X,Z}$ , ${Z,Y,X}$ . Luckily those six permutations can be split into two groups (which can be created by even or odd permutation of the first basis vector ${X,Y,Z}$ ) which have common properties. For feature recognition, let us represent them as follows:


XYZ order	${X,Y,Z}$ , ${Y,Z,X}$ , ${Z,X,Y}$
XZY order	${X,Z,Y}$ , ${Y,X,Z}$ , ${Z,Y,X}$

For these groups, direction of rotation (for the same rotation matrix) for increasing angles perform rotations


XYZ order	from direction $X$ -Axis to $Y$ -Axis, from direction $Y$ -Axis to $Z$ -Axis, from direction $Z$ -Axis to $X$ -Axis
XZY order	from direction $X$ -Axis to $Z$ -Axis, from direction $Z$ -Axis to $Y$ -Axis, from direction $Y$ -Axis to $X$ -Axis

this direction can be called “Positive Direction” and it is worth highlighting as independent of coordinate system “handedness”!

Depending on chosen basis vector order “Positive Direction” is changed and is shown in the following tables:

“Up Direction” is $Z$ -Axis

	Left-hand rule	Right-hand rule
XYZ order
XZY order

“Up Direction” is $Y$ -Axis

	Left-hand rule	Right-hand rule
XYZ order
XZY order

In order to better understand and use information about rotations, it is useful to introduce the term axis of rotation. The axis of rotation can be briefly described:

Bent fingers point in the direction “Positive Direction”, and then your thumb will point in the axis of rotation or

Point thumb in the axis of rotation direction, and then bent fingers point the direction of “Positive Direction”

This will uniquely destination right or left hand as the axis of rotation helper as shown in following table:

Left-hand axis of rotation	Right-hand axis of rotation

It is very tempting to use concepts that can be found in 2D space rotation like clockwise and counter-clockwise rotation. Unfortunately, in 3D space, this concept is hard to adopt. Because perceiving direction as clockwise or contra-clockwise is depending on the position of the observer.

Using this description of axis rotation of basis vector order defined coordinate system following rotation direction can be presented:

“Up Direction” is $Z$ -Axis

	Left-hand axis of rotation	Right-hand axis of rotation
XYZ order
XZY order

“Up Direction” is $Y$ -Axis

	Left-hand axis of rotation	Right-hand axis of rotation
XYZ order
XZY order

Depending on chosen order of the basis vectors, the direction of the axis of rotation can have the same direction as the result of the cross-product vector or the opposite:


XYZ order	one hand is used for determinate coordinate system “handedness” and the same hand is used determinate rotation
XZY order	one hand is used for determinate coordinate system “handedness” and other hand is used determinate rotation

It is very surprising but in 3D space there is possibly to distinguish two types of “handedness” of its coordinate system.

the first “handedness” determine the direction of cross-product thus the direction of $X$ , $Y$ , and $Z$ or vice versa.
the second “handedness” specifies the “positive direction”, that is, it determines the order of the basis vectors or vice versa.

We can assume (is very likely) that when the 3D application is not specified basis vector order it implicitly chooses XYZ order.

On this assumption in the following table contains coordinate system “handedness” for some of the 3D applications

	Left-hand rule	Right-hand rule
Y-Up	LightWave ZBrush Cinema 4D Unity	Autodesk Maya Modo Houdini 3D Animation Tool Substance Painter Marmoset Toolbag Godot Object-Oriented Graphics Rendering Engine (OGRE)
Z-Up	Unreal Engine* Open 3D Engine	Autodesk 3ds Max Blender SketchUp Autodeck AutoCAD CRYENGINE UNIGINE

Unreal Engine* – The FRotator (and editor rotation gadget) rotates around the $X$ and $Y$ -Axis using the right-hand axis of rotation but rotates around the $Z$ -Axis using the left-hand axis of rotation.

Matrix multiplication order vs Basis vectors order

To introduce even more complication to an already confusing problem let’s introduce the following problem. For example, the 2D coordinate system which we are most familiar with, with basis vectors ${X,Y}$ where the $X$ axis points to the right and the $Y$ axis point up.

Using the following Rotation Matrix $R$

R = \begin{bmatrix} \cos(\phi) & -\sin(\phi)\\ \sin(\phi) & \cos(\phi)\\ \end{bmatrix}

what will happen when transforming vector $V$ (column vector) using post-multiplication order.

R * V = V'\newline \begin{bmatrix} \cos(\phi) & -\sin(\phi)\\ \sin(\phi) & \cos(\phi)\\ \end{bmatrix} * \begin{bmatrix} x\\ y\\ \end{bmatrix} = \begin{bmatrix} \cos(\phi)*x - \sin(\phi)*y\\ \sin(\phi)*x + \cos(\phi)*y\\ \end{bmatrix}

The result of this transformation can be shown in following animation

As can be observed rotation matrix describes rotation in “Positive Direction” for ${X,Y}$ basis vectors (in this case counterclockwise direction).

Now let’s first examine what happens to the motion when the rotation matrix $R$ transpose is used without changing the multiplication order.

R^T * V_{1} = V_{1}'\newline \begin{bmatrix} \cos(\phi) & \sin(\phi)\\ -\sin(\phi) & \cos(\phi)\\ \end{bmatrix} * \begin{bmatrix} x\\ y\\ \end{bmatrix} = \begin{bmatrix} \cos(\phi)*x + \sin(\phi)*y\\ \sin(\phi)*x - \cos(\phi)*y\\ \end{bmatrix}

and secondly when we change an order to pre-multiplication without transposing the rotation matrix $R$ .

V_{2} * R = V_{2}'\newline \begin{bmatrix} x & y\\ \end{bmatrix} * \begin{bmatrix} \cos(\phi) & -\sin(\phi)\\ \sin(\phi) & \cos(\phi)\\ \end{bmatrix} = \begin{bmatrix} \cos(\phi)*x + \sin(\phi)*y & \sin(\phi)*x - \cos(\phi)*y\\ \end{bmatrix}

The components of the resulting vector are computed using the same formulas (they are the same if one of them is transposed).

V_{1} = V_{2}^T

And the result of this transformation can be shown in following animation

As can be observed rotation matrix describes rotation in “Positive Direction” for ${Y,X}$ basis vectors (in this case clockwise direction). What is important the same motion can be achieved by two different matrices (one is the transpose of the second) depending on matrix multiplication order.

The same phenomenon also occurs in 3D space, which clearly presents that in the case of a rotation matrix, its final form also depends on the choice of matrix multiplication order.

Summarizing all the information about the rotation matrix, we can say that:

To properly describe desired rotation in 3D (or 2D) space by just specifying its matrix form is not enough. Two additional pieces of information (or two if basis vectors are in use) MUST be detailed

XYZ order
Matrix order convention

only then can you fully describe your intentions about it.

Quaternions and basis vectors order

Unfortunately, the topic of basis vectors order does not avoid quaternions either.

Hamilton multiplication convention

W.R. Hamilton, the inventor of quaternions, defined their properties (more precisely, the product of their basic elements $\mathbf{ij = k}$ ) and thus defined their “Positive Direction”. So now rotating vectors using quaternions boils down to their so-called “sandwich product” and is defined as

v' = QvQ^{-1}

where $v$ is a vector which is rotated by unit-quaternion $Q$ and $Q^{-1}$ is the conjugate.

The rotation matrix created using Hamilton’s definition can be identified for the ${X,Y,Z}$ basis vectors order and it is as follows:

R = \begin{bmatrix} 1 - 2(Q_y^2 + Q_z^2) & 2(Q_xQ_y - Q_zQ_w) & 2(Q_xQ_z + Q_yQ_w)\\ 2(Q_xQ_y + Q_zQ_w) & 1 - 2(Q_x^2 + Q_z^2) & 2(Q_yQ_z - Q_xQ_w)\\ 2(Q_xQ_z - Q_yQ_w) & 2(Q_yQ_z + Q_xQ_w) & 1 - 2(Q_x^2 + Q_y^2)\\ \end{bmatrix}

in $R*V$ or post-multiplication order.

The above definitions can be found in all mathematics textbooks dealing with this subject.

Shuster multiplication convention

However, there is an alternative convention proposed by M.D. Shuster and used in NASA’s Jet Propulsion Laboratory and, importantly in computer graphics, in the Microsoft DirectX Math Library. In Shuster’s convention, the quaternion product of the basis elements is now $\mathbf{ij = -k}$ thus changing the basis vectors order (versus Hamilton multiplication convention). Consequently it is also necessary to change the order of quaternions in a “sandwich product”

v' = Q^{-1}vQ

where $v$ is a vector which is rotated by unit-quaternion $Q$ and $Q^{-1}$ is the conjugate.

The rotation matrix created using Shuster’s definition can be identified for the ${X,Z,Y}$ basis vectors order and it is as follows:

R = \begin{bmatrix} 1 - 2(Q_y^2 + Q_z^2) & 2(Q_xQ_y + Q_zQ_w) & 2(Q_xQ_z - Q_yQ_w)\\ 2(Q_xQ_y - Q_zQ_w) & 1 - 2(Q_x^2 + Q_z^2) & 2(Q_yQ_z + Q_xQ_w)\\ 2(Q_xQ_z + Q_yQ_w) & 2(Q_yQ_z - Q_xQ_w) & 1 - 2(Q_x^2 + Q_y^2)\\ \end{bmatrix}

in $R*V$ or post-multiplication order.

Graphics APIs and implicit coordinate system “handedness”

When 3D Graphics APIs were first introduced, they came with specific coordinate system conventions:

OpenGL® is using ${X,Y,Z}$ basis vectors, right-hand coordinate system with post-multiplication convention.
DirectX® is using ${X,Y,Z}$ basis vectors, left-hand coordinate system with pre-multiplication convention.

But with the evolution of the GPU hardware and graphics APIs, this has changed. Right now, almost everything in the API is the coordinate system and multiplicative order agnostic, except for two places where ultimately the API must commit to (and cannot be changed by the user) the “handedness” of the 2D or 3D coordinate system used in:

Clipping Space (normalized device coordinate (NDC) space or homogeneous screen space).

API	Clipping Space
DirectX®	$X$ -Axis points to the right, $Y$ -Axis points up, $Z$ -Axis points into the screen, $Z$ in the range $[0,1]$
Vulkan®	$X$ -Axis points to the right, $Y$ -Axis points down, $Z$ -Axis points into the screen, $Z$ in the range $[0,1]$
OpenGL®	$X$ -Axis points to the right, $Y$ -Axis points up, $Z$ -Axis points into the screen, $Z$ in the range $[-1,1]$
Metal	$X$ -Axis points to the right, $Y$ -Axis points up, $Z$ -Axis points into the screen, $Z$ in the range $[0,1]$
WebGPU	$X$ -Axis points to the right, $Y$ -Axis points up, $Z$ -Axis points into the screen, $Z$ in the range $[-1,1]$
WebGL™	$X$ -Axis points to the right, $Y$ -Axis points up, $Z$ -Axis points into the screen, $Z$ in the range $[0,1]$

Texture space

API	Texture space
DirectX®	origin Top-Left corner, $X$ -Axis points to the right, $Y$ -Axis points down
Vulkan®	origin Top-Left corner, $X$ -Axis points to the right, $Y$ -Axis points down
OpenGL®	origin Bottom-Left corner, $X$ -Axis points to the right, $Y$ -Axis points up (by default)
Metal	origin Top-Left corner, $X$ -Axis points to the right, $Y$ -Axis points down
WebGPU	origin Top-Left corner, $X$ -Axis points to the right, $Y$ -Axis points down
WebGL™	origin Bottom-Left corner, $X$ -Axis points to the right, $Y$ -Axis points up

Transformation matrices

All the above considerations regarding firstly, the non-commutative nature of matrix multiplication, and secondly the anticommutativity of a vector’s cross product has the following consequences in four alternative ways of constructing the 3D transformation as a 4×4 matrix. All four of those alternatives are described in the individual sections below.

Tips about the above conventions:

By transposing the entire 4×4 matrix you will only change the pre- and post-multiplication order (without changing the coordinate system “handedness” or basis vector order).
By transposing just, the upper-left 3×3 matrix you will change only the basis vector order (without changing pre- and post-multiplication order).

How to detect LH/RH Row/Column-Major Pre/Post Multiplication convention?

Which convention is used in a particular application is unknown. Answering the following questions will help to identify it:

How is the translation matrix constructed? It is necessary to check how the components of the translation vector $XYZ$ are stored in memory. Are they stored next to each other (1a) or are they away from each other (1b)?
Determination of the order in which the transformations (Scale, Rotation, and Translation) are composed. Whether it is executed in the order T * R * S (2a) or S * R * T (3b)? Or if Matrix Vector multiplication is computed in order M * V (2a) or V * M (2b)?
It is necessary to inspect how rotation around $X$ -Axis or $Z$ -Axis is stored in memory. Is the first in memory stored $-sin$ element (3a) or $sin$ (3b), or in case of rotation around $Y$ -Axis the first in memory is stored $sin$ element (2a) or $-sin$ (2b)?
Straightened index finger of the right/left-hand point in the direction of the $X$ -Axis, and the bent middle finger point in the direction of the $Y$ -Axis. Depending on chosen hand, extended of left-hand thumb points in the direction of $Z$ -Axis (4a) or right-hand (4b).

With the answers to the above questions, the conventions we are looking for can be found in the table below:

Question 1	Question 2	Question 3	Question 4	$\rightarrow$	Convention
1a	2a	3a	4a	$\rightarrow$	Post-multiplication	Column-Major	${X,Z,Y}$	Left-handed
1b	2a	3a	4a	$\rightarrow$	Post-multiplication	Row-Major	${X,Y,Z}$	Left-handed
1a	2b	3a	4a	$\rightarrow$	Pre-multiplication	Row-Major	${X,Y,Z}$	Left-handed
1b	2b	3a	4a	$\rightarrow$	Pre-multiplication	Column-Major	${X,Z,Y}$	Left-handed
1a	2a	3b	4a	$\rightarrow$	Post-multiplication	Column-Major	${X,Y,Z}$	Left-handed
1b	2a	3b	4a	$\rightarrow$	Post-multiplication	Row-Major	${X,Z,Y}$	Left-handed
1a	2b	3b	4a	$\rightarrow$	Pre-multiplication	Row-Major	${X,Z,Y}$	Left-handed
1b	2b	3b	4a	$\rightarrow$	Pre-multiplication	Column-Major	${X,Y,Z}$	Left-handed
1a	2a	3a	4b	$\rightarrow$	Post-multiplication	Column-Major	${X,Z,Y}$	Right-handed
1b	2a	3a	4b	$\rightarrow$	Post-multiplication	Row-Major	${X,Y,Z}$	Right-handed
1a	2b	3a	4b	$\rightarrow$	Pre-multiplication	Row-Major	${X,Y,Z}$	Right-handed
1b	2b	3a	4b	$\rightarrow$	Pre-multiplication	Column-Major	${X,Z,Y}$	Right-handed
1a	2a	3b	4b	$\rightarrow$	Post-multiplication	Column-Major	${X,Y,Z}$	Right-handed
1b	2a	3b	4b	$\rightarrow$	Post-multiplication	Row-Major	${X,Z,Y}$	Right-handed
1a	2b	3b	4b	$\rightarrow$	Pre-multiplication	Row-Major	${X,Z,Y}$	Right-handed
1b	2b	3b	4b	$\rightarrow$	Pre-multiplication	Column-Major	${X,Y,Z}$	Right-handed

What’s coming next

Yaw Pitch Roll Angles.
Composition order results of rotations around the coordinate axes.
Gimbal Lock problem.
Transition from one convention to another. Necessary transformations and their consequences on the data.
Oblique and Axonometric Projections.
More Miscellaneous Matrices.