PSC 128/NPB 163

Linear Algebra (a primer)

Why linear algebra?

• What if more than one value is going in and coming out of a system? For example, we have many inputs coming into the retina (photoreceptors) and many outputs (retinal ganglion cells). Any single input affects many different outputs, and any given output is influenced by many different inputs. How do we even begin to understand such a system?
  
  
• Linear algebra provides a useful tool for characterizing the behavior of systems where many values must be represented simultaneously. - e.g., the brain.
  
  

• A vector is simply a list of numbers. A vector is usually denoted in boldface, or underlined:
        or .
  
  An “n-dimensional vector” has n elements. One can think of an n-dimensional vector as either a point in an n-dimensional space, or as an arrow drawn from the origin to the point with coordinates .
  

• The addition of two vectors is simply the addition of their elements:
  
        ,
  Or, we could write the elements of z as . Geometrically, vectors are added by placing them end to end. The vector from the origin to the tip of the last vector is the sum vector.
  

• The inner-product (or dot product) of two vectors takes the sum of products of the elements of each vector: . This provides a measure of the similarity of two vectors (provided you know the length of each vector). If you divide the inner product by the length of each vector, it gives you the cosine of the angle between them
  
         
  where is the angle between the two vectors, and . An alternative geometric interpretation of the inner product is that gives you the length of after it has been projected onto y. Thus, orthogonal vectors have an inner product of zero.The inner product is oftentimes also denoted , where the superscript T denotes “transpose.” The transpose of a vector simply tilts it on its side so it is written as a row of numbers, . 

• A matrix is just a 2D array of numbers. It is used to map a vector x into a new vector y via the relation . It takes the place of the constant, k, in a simple one-dimensional (scalar) linear system. An m x n matrix M has elements
  
         
  and we can write the resulting vector, y, elementwise as . In this case the vector y will be m-dimensional.
  

• Analysis/synthesis: the row vectors of a matrix tell you what part of the input space (vectors on the right side of the matrix) the matrix analyzes. The column vectors tell you what part of the output space (i.e., vectors coming out the left side) the matrix can synthesize from any input. In each case, it is the number of linearly independent vectors that define these spaces.
  

• Just as a linear scalar system can produce only a limited set of remappings of scalar values, so too is a matrix operation limited in the set of remappings it can produce. What do these remappings look like? - rotations (orthonormal matrices) and scalings (diagonal matrices) and combinations thereof.
  
  
• If y=Mx and z=Ay, then z=AMx=Cx, where C=AM is just another matrix. Thus, the concatenation of two matrix operations is just another matrix operation. The combined matrix C is obtained via matrix multiplication:
  
       
  Note however that matrix multiplication is not commutative - i.e, .
  
  
• Any matrix M can be decomposed in terms of a rotation, a scaling, and another rotation. i.e., , where and are orthonormal matrices (rotation) and is a diagonal matrix (scaling).