Linear Regression

Introduction to Linear Regression

• Simple linear regression is useful for finding relationship between two continuous variables. One is predictor or independent variable and other is response or dependent variable.
• It looks for statistical relationship but not deterministic relationship.
• The following figure illustrates the deterministinc relationship between temperature in Degree celsius and Degree Fahrenmheit where the data points (x,y) fall directly on a line

• In a statistical relartionship the relationship between two variables x and y is not perfect and hence the line of relationship doesnot pass through all the points as depicted in the figure below.

• Linear regression attempts to model the relationship between two variables by fitting a linear equation to observed data.
• A linear regression line has an equation of the form Y = a + bX, where X is the explanatory variable and Y is the dependent variable. The slope of the line is b, and a is the intercept (the value of y when x = 0).
• It should be noted that linear regression analysis can only be done if the correlation coefficient is close to -1 or +1 and not possible if the same is close to 0.

Performing Linear Regression in python.

• The scikit-learn package can be used with DataFrames or NumPy arrays to perform linear regression analysis. We need to install the package by using the command pip install -U scikit-learn while we are online.
• Our task is to perform Linear Regression on a dataset comprising of area of land and its price (training dataset). After we perform the linear regression we should be able to predict the optimum price of a piece of land not given in the dataset (test data).

  area	price
  2600	550000
  3000	565000
  3200	610000
  3600	680000
  4000	725000

• To perform linear regression we create the dataset to train our model. (Training of the model here means to find the parameters so that the model best fits the data.)

import numpy as np

area = np.array([2600,3000,3200,3600,4000]).reshape((-1,1))

price = np.array([550000,565000,610000,680000,725000])

print(area)

print(price)

output:

[[2600] [3000] [3200] [3600] [4000]] [550000 565000 610000 680000 725000]

• You can see above that the area has been reshaped into a 2D array comprising of each value as rows. This is required to be like this since the operations to follow needs the explanatory variable X to ba passed as a 2D array to our training function reg.fit()
• We are going to import the machine learning routines available with linear_model from sklearn to help us to create our model.
• The following example illustrates as to how we train our model.

import numpy as np

from sklearn import linear_model

area = np.array([2600,3000,3200,3600,4000]).reshape((-1,1))

price = np.array([550000,565000,610000,680000,725000])

# Instantiatin our linear regression model 

reg = linear_model.LinearRegression()

# Now we train the our regression model with training data (area, price) 

reg.fit(area,price)

#We can now predict the price of area(test data) not present in our training dataset

#observe that predict function too takes a 2D array

p = reg.predict([[3400]])

print('Price for area 3400:',p)

output:

Price for area 3400: [642294.52054795]

• We can also find the slope(m) and intercept(c) using the following example

print('The slope of the curve:',reg.coef_)

print('The intercept of the curve:',reg.intercept_)

 

output:  

The slope of the curve: [135.78767123] The intercept of the curve: 180616.43835616432

Plotting linear regression graph using matplotlib and numpy

• We shall superimpose the scatterplot of (area, price) below the line plot(regression line) using matplotlib

import numpy as np

import matplotlib.pyplot as plt

from sklearn import linear_model

#Exclusively for jupyter to allow the plots to be inline and not require plt.show()

%matplotlib inline

# Our training dataset

area = np.array([2600,3000,3200,3600,4000]).reshape((-1,1))

price = np.array([550000,565000,610000,680000,725000])

# labels for x and y axis

plt.xlabel('area')

plt.ylabel('price')

# Scatter plot for the dataset in red colour

plt.scatter(area,price, color='red')

# Performing linear regression

reg = linear_model.LinearRegression() # instantiaing a linear regression model

reg.fit(area,price) # training the model with data

# Plotting the regression line

plt.plot(area,reg.predict(area), color='blue')

output:

Plotting linear regression graph using matplotlib and pandas

import pandas as pd

import matplotlib.pyplot as plt

from sklearn import linear_model

%matplotlib inline

df = pd.DataFrame([[2600,550000],[3000,565000],[3200,610000],\

                   [3600,680000],[4000,725000]],\

                   columns=['area','price'])

plt.xlabel('area')

plt.ylabel('price')

plt.scatter(df.area,df.price, color='red')

reg = linear_model.LinearRegression() # instantiaing a linear regression model

reg.fit(df[['area']],df.price) # training the model with data

plt.plot(df.area,reg.predict(df[['area']]), color='blue')

output:

Reference:

• https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html
• https://www.youtube.com/watch?v=8jazNUpO3lQ

Correlation

Introduction to Correlation

• Covariance and correlation both primarily assess the relationship between variables.
• Covariance measures the total variation of two random variables from their expected values. However, it does not indicate the strength of the relationship, nor the dependency between the variables.
• Correlation measures the strength of the relationship between variables. Correlation is the scaled measure of covariance.
• The relationship between the two concepts can be expressed using the formula below:

• The main concept of correlation is correlation coeficient . It can have values between -1 to +1. The closer coeficient is to +1 or -1, the more closely the two variables are related.
• If correlation coeficient is close to 0, it means there is no relationship between the variables. If coeficient is positive, it means that as one variable gets larger the other gets larger. If coeficient is negative it means that as one gets larger, the other gets smaller (often called an "inverse" correlation).
• Numpy calculates Linear correlation coefficient using Pearson method.
• The function np.corrcoef(x,y) gives the matrix with following values

import numpy as np

x = np.array([1.2,2.4,3.6,4.8,7.2])

y = np.array([4,6,8,10,12])

print(np.corrcoef(x,y))

output:

[[1. 0.98639392] [0.98639392 1. ]]

Depiction of correlation through scatter plots

• Let us try to visualize correlation between two variables with visual representation. We shall use some random numbers and scatter plots to visualize the concept.

Positive correlation

import numpy as np

import matplotlib

import matplotlib.pyplot as plt

%matplotlib inline

matplotlib.style.use('ggplot')

# 1000 random integers between 0 and 50

x = np.random.randint(0, 50, 1000)

# Positive Correlation with respect to x

y = x + np.random.randint(0,10,1000)

print(np.corrcoef(x,y))

plt.scatter(x, y)

plt.show()

OUTPUT:

Negative correlation:

import numpy as np

import matplotlib

import matplotlib.pyplot as plt

%matplotlib inline

matplotlib.style.use('ggplot')

# 1000 random integers between 0 and 50

x = np.random.randint(0, 50, 1000)

# Negative Correlation with respect to x

y = 100 - x + np.random.randint(0,10,1000)

print(np.corrcoef(x,y))

plt.scatter(x, y)

plt.show()

OUTPUT:

Weak or no correlation

import numpy as np

import matplotlib

import matplotlib.pyplot as plt

%matplotlib inline

matplotlib.style.use('ggplot')

# 1000 random integers between 0 and 50

x = np.random.randint(0, 50, 1000)

# No Correlation with respect to x

y = np.random.randint(0,30,1000)

print(np.corrcoef(x,y))

plt.scatter(x, y)

plt.show()

Co-variance

Covariance 

Introduction:

• In mathematics and statistics, covariance is a measure of the relationship between two random variables.
• It measures the degree of change in the variables, i.e. when one variable changes, will there be the same/a similar change in the other variable.
• Covariance does not assess the dependency between variables.
• Positive covariance: Indicates that two variables tend to move in the same direction.
• Negative covariance: Reveals that two variables tend to move in inverse directions. 

• In numpy we use the function np.cov(x,y) to calculate covariance between two variavles. The result is depicted in the figure below. You can see the 2x2 result matrix consists of covariance of a and b as well as the variance of variable a and variable b. note that np.cov(x,y)=np.cov(y,x)

• To refresh your understanging of variance let us learn it once again. Variance is a measure of how much the data varies from the mean. The following figure illustrates how to calculate variance.

The following example is actually illustrating the computation of covariance using numpy

import numpy as np

x = np.array([1.2,2.4,3.6,4.8,7.2])

y = np.array([4,6,8,10,12])

print(np.cov(x,y))

Output:

[[ 5.328 7.2 ] [ 7.2 10. ]]

Covariance for three variables

• In case of three variables the covriance is calculated in the similar way as discussed above and yields a 3x3 matrix containing the covariance of (x,y), (x,z) and (y,z) and variance of x, y and z.

• The follwoing example demonstrates this feature. Since you cannot pass three variables to cov() function, we need to pass a multidimensional array to the cov() function to get the results.
• It is interesting to note that there is negative covariance between (x,z) and (y,z)

import numpy as np

x = np.array([[1.2,2.4,3.6,4.8,7.2],[4,6,8,10,12],[10, 7, 5, 4, 2]])

print(np.cov(x))

OUTPUT:

[[ 5.328 7.2 -6.78 ] [ 7.2 10. -9.5 ] [-6.78 -9.5 9.3 ]]

Getting only the covariance and not the full matrix

• The following example illustrates how to just print the covariance of x and y and not the whole matrix

import numpy as np

x = np.array([1.2,2.4,3.6,4.8,7.2])

y = np.array([4,6,8,10,12])

print(np.cov(x,y)[0][1])

OUTPUT:

7.199999999999999

NUMPY

INTRODUCTION  TO AN ARRAY IN PROGRAMMING LANGUAGE:-

Array is a container which can hold a fix number of items and these items should be of the same type. Most of the data structures make use of arrays to implement their algorithms. Following are the important terms to understand the concept of Array.

• Element − Each item stored in an array is called an element.
• Index − Each location of an element in an array has a numerical index,                      which is used to identify the element.
• Thus, An array is a special variable, which can hold more than one value at a time. i.e. An array can hold many values under a single name, and you can access the values by referring to an index number.
• Note: Python does not have built-in support for Arrays, but Python Lists can be used instead.

Introduction to NumPy

• NumPy, which stands for Numerical Python, is a library consisting of multidimensional array objects and a collection of routines for processing those arrays. Using NumPy, mathematical and logical operations on arrays can be performed.
• The most important object defined in NumPy is an N-dimensional array type called ndarray. It describes the collection of items of the same type. Items in the collection can be accessed using a zero-based index.

Creating 1D ndarray

The following example creates an 1D ndarray using a list.

import numpy as np

a = np.array([1,2,3,4])

print(a)

print('Dimension:',a.shape)# dimension of the array- (4 rows for 1 D array)

print('Datatype:',a.dtype)

print('Size of each element:', a.itemsize, 'bytes')

OUTPUT:

[1 2 3 4] Dimension: (4,) Datatype: int64 Size of each element: 8 bytes

We can create an array with smaller size by mentioning the dtype specifically to int16 or int8

import numpy as np

a = np.array([1,2,3,4], dtype='int8')

print(a.dtype)

print('Size of each element:', a.itemsize, 'byte')

OUTPUT:

int8 Size of each element: 1 byte

Creating ndarrays using fromiter()

• fromiter() function can be create a numpy array using any iterable (that can be traversed through) object like string, list or a dictionary
• The following examples illustrate the use of fromiter() method

#This example creates an array of unicode charactersrs from a string

#The dtype needs to be exclusively given (Unicode 2 bytes)

import numpy as np

st ="NUMPY"

arr = np.fromiter(st, dtype='U2')

print(arr)

#This example creates an array of integers from a loop

itr = (x*x for x in range(1,8))

arr = np.fromiter(itr, dtype='int8')

print(arr)

#This example will create a dictionary of keys of a dictionary

dic = {'A':10, 'B':20, 'C':20}

arr = np.fromiter(dic, dtype='U2')

print(arr)

#The last example will create an arry selecting only first 5 charecters of the string

#For this we need to set the count attribute to a desired value

st = "PYTHONLANG"

arr = np.fromiter(st, dtype='U2', count=5)

print(arr)

OUTPUT :

['N' 'U' 'M' 'P' 'Y'] [ 1 4 9 16 25 36 49] ['A' 'B' 'C'] ['P' 'Y' 'T' 'H' 'O']

Accessing elements of 1D array

We use 0 based indexing for accessing the elements of an ndarray

import numpy as np

a = np.array([1,2,3,4], dtype='int8')

print('The first element:',a[0])

print('The last element:',a[3])

OUTPUT:

The first element: 1 The last element: 4

Creating 2D ndarray

We can pass a 2D list in the array() function of numpy to create a 2D ndarray as shown below.

import numpy as np

b = np.array([[1,2,3],[4,5,6],[7,8,9]])

print(b)

print('The shape is:',b.shape)

OUTPUT:

[[1 2 3] [4 5 6] [7 8 9]] The shape is: (3, 3)

Accessing the elements of 2D array

The following examples illustrates the use of 0 based indexing on 2D array

import numpy as np

b = np.array([[1,2,3],[4,5,6],[7,8,9]])

print(b)

print('first row:',b[0])

print('First row 1st element:',b[0][0])

print('Third row 3rd element:',b[2][2]) 

print('Second row 1st element:',b[1,0])

OUTPUT:

[[1 2 3] [4 5 6] [7 8 9]] first row: [1 2 3] First row 1st element: 1 Third row 3rd element: 9 Second row 1st element: 4

Creating matrices with numpy functions

Creating matrix of 0s using the zeros() function.

Observe that the dtype of the resulting matrix of zeros is float64

import numpy as np

b = np.zeros((3,3)) # Creates a 3x3 matrix with zeros

print(b)

print(b.dtype)

OUTPUT:

[[0. 0. 0.] [0. 0. 0.] [0. 0. 0.]] float64

Creating the matrix of 1s using the ones() function

import numpy as np

b = np.ones((3,3)) # Creates a 3x3 matrix with ones

print(b)

print(b.dtype)

OUTPUT:

[[1. 1. 1.] [1. 1. 1.] [1. 1. 1.]] float64

Creating a matrix using full() function

This function accepts two parameters - size as tuple and the data value as a number

import numpy as np

b = np.full((2,2), 8) # Creates a 2x2 constant matrix with 8

print(b)

OUTPUT:

[[8 8] [8 8]]

Creating an identity matrix using eye() function

This method accepts one parameter as the size of the identity matrix.

import numpy as np

b = np.eye(3) # Creates an identity matrix

print(b)

OUTPUT:

[[1. 0. 0.] [0. 1. 0.] [0. 0. 1.]]

Creating matrix with random() funcion

• Random numbers can be generated using the random library of numpy.
• The following statement creates a a 2x2 matrix using random.random() funcion

import numpy as np

b = np.random.random((2,2)) # Creates an random number matrix

print(b)

OUTPUT:

[[0.53352925 0.48243526] [0.51794348 0.46761645]]

Using arange() function to create numpy array

#Using arange function to create a numpy array

import numpy as np

# creates an array of integers from 0 to 9

b = np.arange(10) 

print(b)

# creates an array of floats from 1 to 10

a = np.arange(1,11, dtype=float) 

print(a)

# Creates an array with interval of 0.1

c = np.arange(2,3, 0.1)

print(c) 

OUTPUT:

[0 1 2 3 4 5 6 7 8 9]

[ 1. 2. 3. 4. 5. 6. 7. 8. 9. 10.] [2. 2.1 2.2 2.3 2.4 2.5 2.6 2.7 2.8 2.9]

Slicing of numpy arrays

• Slicing of numpy arrays creates sub-arrays from and existing array
• Any modification done in the subarray is reflected back in the main array

#Slicing of numpy arrays

#Resultant array will always be a sub-array of the original array

import numpy as np

a = np.array([[1,2,3,4],[5,6,7,8],[9,10,11,12]])

b = a[0:2] # slice consists of only first two row (excludes row three)

print('b:',b)

c = a[1:2] # Slice consists of only second row

print('c:',c)

d = a[0:1,0:4] # Slice consists of entire first row (first row all columns)

print('d:',d)

e = a[0:1,0:2] # Slice consists of first row and first two columns

print('e:',e)

f = a[0:1,2:4] # Slice consists of first row and columns three and four

print('f:',f)

OUTPUT:

b: [[1 2 3 4] [5 6 7 8]] c: [[5 6 7 8]] d: [[1 2 3 4]] e: [[1 2]] f: [[3 4]]

Some more examples of slicing in numpy arrays

#Slicing of numpy arrays

import numpy as np

a = np.array([[1,2,3,4],[5,6,7,8],[9,10,11,12]])

b = a[1:2,0:4]                 # Slice consists of second row all columns

print('b:',b)

c = a[1:2,1:2]                 # Slice consists of second row second column

print('c:',c)

d = a[:2,0:2]                   # Slice consists of two rows first[1,2] and second[5,6]

print('d:',d)

e = a[:2,1:3]                  # Slice consists of two rows first[2,3] and second[6,7]

print('e:',e)

f = a[1:3,2:4]                              # Slice consists of two rows first[7,8] and second[11,12]

print('f:',f)

OUTPUT:

b: [[5 6 7 8]] c: [[6]] d: [[1 2] [5 6]] e: [[2 3] [6 7]] f: [[ 7 8] [11 12]]

Creating a new arbitry array from an existing one using indexing

#Indexing will create a new arbitry array from the original array

import numpy as np

a = np.array([[1,2], [3, 4], [5, 6]])

print(a.shape)

b = a[[0,1,2],[0,1,0]] # The resultant array will have shape (3,)

print(b)

print(b.shape)

OUTPUT:

(3, 2) [1 4 5] (3,)

Changing the value in a slice will also create a change in original array

#Changing the values in an array resulting from slice

import numpy as np

a = np.array([[1,2,3,4],[5,6,7,8],[9,10,11,12]])

b = a[0:2,1:3] #Sliced array

print(b)

b[0,0]=20 # Changing the value in the slice

print(b)

print(a) # The change is getting reflected in the original array

OUTPUT:

[[2 3] [6 7]] [[20 3] [ 6 7]] [[ 1 20 3 4] [ 5 6 7 8] [ 9 10 11 12]]

Joining two numpy arrays using concatenate() function

#Joining two numpy arrays

import numpy as np

a = np.array([[1,2],[2,4]])

b = np.array([[7,8]])

#Joing as row

c = np.concatenate((a,b), axis=0)

print(c)

#Joinging as column

d = np.concatenate((a,b.T), axis=1)

print(d)

OUTPUT:

[[1 2] [2 4] [7 8]] [[1 2 7] [2 4 8]]

Arithmatic operations on numpy arrays

• Unlike lists the aritmatic operations on numpy arrays are vector operations
• Addition, subtraction, simple multipliction, matrix multiplication, division, square root
• Addition of two arrays: np.add(a,b) or a+b
• Subtraction of two arrays: np.subtract(a,b) or a-b
• Simple multiplication: np.multiply(a,b) or a*b
• Division of two arrays: np.divide(a,b) or a/b
• Matrix multiplication: np.dot(a,b) or a.dot(b)

#Numpy arithmatic operations

import numpy as np

a = np.array([[1,2],[3,4]], dtype=np.int32)

b = np.array([[5,6],[7,8]], dtype=np.int32)

print(a+b)

print(np.add(a,b))

print(a-b)

print(np.subtract(a,b))

print(a*b) # Multiplies corresponding elements

print(np.multiply(a,b))

print(a/b)

print(np.divide(a,b))

print(np.sqrt(a))

# Actual Matrix multiplication

print(a.dot(b))

print(np.dot(a,b))

OUTPUT:

[[ 6 8] [10 12]] [[ 6 8] [10 12]] [[-4 -4] [-4 -4]] [[-4 -4] [-4 -4]] [[ 5 12] [21 32]] [[ 5 12] [21 32]] [[0.2 0.33333333] [0.42857143 0.5 ]] [[0.2 0.33333333] [0.42857143 0.5 ]] [[1. 1.41421356] [1.73205081 2. ]] [[19 22] [43 50]] [[19 22] [43 50]]

Row and column sum of array elements using sum() function

• for axis=0 there will be row sum
• for axis=1 there will be column sum

#numpy sum() function

a = np.array([[1,2],[3,4]], dtype=np.int32)

print(np.sum(a, axis=0))

print(np.sum(a, axis=1))

[4 6] [3 7]