2.1. Vectors

Consider the set \(\mathbb R\) of all real numbers. The Cartesian product of two sets \(A\) and \(B\) is defined as:

\[A\times B = \{(a,b) : a\in A \mbox{ and } b\in B\}\]

The Cartesian product can be used to define powers of sets as follows:

\[A^n = \underbrace{A \times A\times \ldots \times A}_{\mbox{$n$ times}}\]

This allows us to define the space \(\mathbb R^n\) of all \(n\)-dimensional vectors as follows:

\[\mathbb R^n = \{(a_1,a_2,\ldots,a_n) : a_i\in\mathbb R \mbox{ for all } 1\leq i\leq n\}\]

Thus, an \(n\)-dimensional vector \(v\) is represented as an ordered set of \(n\) real numbers \(a_i\), where \(1\leq i\leq n\). Each element \(a_i\) is called a component of \(v\). In what follows, we will exclusively refer to the \(i\)-th component of a vector \(v\) as \(v_i\), so \(v\) can be written as \(v=(v_1,v_2,\ldots,v_n)\). A vector of dimension \(1\) is commonly referred to as a scalar. In Python, vectors are particularly easy to define with NumPy using arrays. The example below defines a \(4\)-dimensional vector x:

>>> import numpy as np
>>> x = np.array([2,3,1,0])
>>> x
array([2, 3, 1, 0])

Two vectors can be added together in a component-wise fashion to define a new vector, i.e., if \(v\) and \(w\) are two vectors, we can define a new vector \(v+w\) as:

\[v+w = (v_1+w_1,v_2+w_2,\ldots,v_n+w_n)\]

It follows from the above definition that \(v+w = w+v\), showing that vector addition is commutative. Vector addition is supported in Python, as shown below:

>>> v = np.array([2,3,1,0])
>>> w = np.array([5,4,2,7])
>>> v+w
array([7, 7, 3, 7])

Note that vector addition is not defined when the two vectors being added together have different dimensions. If you tried to add together two vectors of different dimensions in Python, you will get an error:

>>> u = np.array([3,4,2])
>>> v = np.array([2,3,1,0])
>>> u+v
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: operands could not be broadcast together with shapes (3,) (4,)

A vector \(v\) can also be multiplied with a scalar \(c\) to define a new vector as follows:

\[c\cdot v = (cv_1,cv_2,\ldots,cv_n)\]

Once again, the order does not matter for scalar multiplication, i.e., \(c\cdot v = v\cdot c\). Scalar multiplication is also supported in Python, as shown here:

>>> v = np.array([2,3,1,0])
>>> v*5
array([10, 15,  5,  0])
>>> 5*v
array([10, 15,  5,  0])

Vector addition and scalar multiplication can together be used to define vector subtraction, i.e., if \(v\) and \(w\) are two vectors, then they can be subtracted together to define a new vector \(v-w\) as:

\[v - w = v + (-1)\cdot w = (v_1-w_1,v_2-w_2,\ldots,v_n-w_n)\]

Again, note that vector subtraction is not defined when the two vectors being subtracted have different dimensions.

2.1.1. Linear Combination, Span, and Linear Independence

Together, these operations allow us to define a linear combination of a set \(S\) of vectors \(\{v^k\}\) as a weighted sum:

\[w = \sum_k a_k v^k\]

where \(a_k\in\mathbb R\) for all \(k\).

Example 1

Consider the unit vectors \((1,0,0)\), \((0,1,0)\), and \((0,0,1)\) in three dimensions. Any vector \((u,v,w)\) can be written as:

\[(u,v,w) = u(1,0,0) + v(0,1,0) + w(0,0,1)\]

Thus, any \(3\)-dimensional vector is a linear combination of the three unit vectors along the X, Y, and Z axes.

The set of all linear combinations of vectors in \(S\) constitute the span of \(S\), denoted as \(\textsf{span } S\), i.e.,

\[\textsf{span }S \equiv \{a_1v^1+\ldots+a_kv^k : v^i\in S\mbox{ and }a_i\in\mathbb R\mbox{ for all }1\leq i\leq k\}\]

From Example 1, any \(3\)-dimensional vector \(v\) can be written as a linear combination of the three unit vectors \((1,0,0)\), \((0,1,0)\), and \((0,0,1)\). Thus, the span of these three unit vectors is the entire \(3\)-dimensional Euclidean space \(\mathbb R^3\). The notion of linear combination, in turn, allows us to define linear independence. A set \(S\) of vectors \(\{v^k\}\) is linearly independent if every non-trivial linear combination of vectors in \(S\) is non-zero, i.e., \(0\notin\textsf{span }S\). More formally, whenever the sum \(a_1v^1+\ldots+a_kv^k = 0\), then the coefficients \(a_i = 0\) for all \(1\leq i\leq k\). A set of vectors is said to be linearly dependent if they are not linearly independent.

Example 2

Consider the three vectors \((2,1,4)\), \((5,2,8)\), and \((1,0,0)\) in \(\mathbb R^3\). These vectors are not linearly independent (i.e., they are linearly dependent) because of the following identity:

\[2\cdot(2,1,4) - 1\cdot(5,2,8) + 1\cdot(1,0,0) = (0,0,0)\]

In this particular case, all the coefficients \(2\), \(-1\), and \(1\) are non-zero, so the zero-vector \(0\in\textsf{span }S\).

2.1.2. Dot Product

Finally, the space \(\mathbb R^n\) also comes equipped with a function \(f:\mathbb R^n\times\mathbb R^n\rightarrow\mathbb R\) called the dot product, denoted as \(v\cdot w\) for two vectors \(v\) and \(w\), and formally defined as:

\[v\cdot w = \sum_{i=1}^k v_iw_i\]

NumPy has a function dot that allows us to easily compute the dot product of two vectors, as shown below:

>>> v = np.array([3,1,4])
>>> w = np.array([2,5,7])
>>> np.dot(v,w)
39

In what follows, we will always denote an \(n\)-dimensional vector as an \(n\times 1\) column vector.