A Complete Road map to Deep learning 2021 – Part 1

    Welcome ! Thanks for visiting my blog site. In my previous article I have listed down the way to kick start the Data Science career Journey

    In this article I am going to list down the concepts to learn to carry forward your career toward Artificial Intelligence

    This Article Contains 5 parts in which we you have to concentrate your learning process

    Don’t be hurry ! Take your Sweet time! Lay the strong foundation at your self pace of time!

    • 1. Prerequisites – Mathematics and Programming
    • 2. History of Artificial Neural Networks
    • 3. Important Architectures to be learnt and Back propagation and Optimization
    • 4. Deep learning Development lifecycle
    • 5. Libraries and Frameworks

    Prerequisites – Mathematics and Programming

    I have been working on neural networks from my 7th semester 2018.I had a subject called “Soft computing ” in which I started learning about neural networks and other soft computing algorithms like Fuzzy logics Evolutionary algorithms

    My interest towards soft computing has brought me to this Artificial intelligent domain

    Lets start with Mathematics and programming required for deep learning

    Mathematics for Deep learning :

    Three Major mathematics topics , we need to learn ( we might have learnt them already in our college curriculam)

    You may have a question, Do We need Mathematics for Deep learning ? Yes we do need , If you want to build your own models and diagnose what happened in the pipeline ? .If you want to improve the model performance , under the hood math is important.

    1. Linear algebra
    2. Multivariate Calculi
    3. Probability and statistics

    Linear Algebra:

    Linear algebra is a basic to many of the machine learning algorithms. It speed up the normal calculations by using vectorization method . Important concepts to learn in Linear algebra

    Topic list

    1. Scalers , Vectors , Matrix , Tensors — These are the basic entities of linear algebra ,variables can be either of the format. Machine learning / deep learning functions will be represented using function with above variables
    2. Matrix arithmetic operations — Multiplication , Addition , Subtraction , Division , Transpose , Inverse , Norms
    3. Different types of Matrices and tensors ( a multidimensional vector is a Tensor)
    4. Eigen Decomposition – Concept of Eigen vectors , Eigen values are useful in Dimensionality reduction methodologies
    5. singular value Decomposition
    6. Determinant , Rank operations

    Follow this Resource link to learn basics of linear algebra along with python (numpy)

    1. https://d2l.ai/chapter_preliminaries/linear-algebra.html
    2. https://jonathan-hui.medium.com/machine-learning-linear-algebra-a5b1658f0151
    3. https://builtin.com/data-science/basic-linear-algebra-deep-learning
    4. https://www.deeplearningbook.org/contents/linear_algebra.html
    5. https://mml-book.github.io/
    6. https://www.math.ubc.ca/~pwalls/math-python/linear-algebra/linear-algebra-scipy/

    Multivariate Calculus :

    When it comes to algorithms , Role of Calculus is inevitable. Tasks like Gradients Descent ,Numerical Optimization , Back propagation , Mathematical Simulations need Calculus as their back bone to computation

    Following Concepts are required in Deep learning

    1. Differential Calculus – Helps in Back propagation , Numerical Optimization , Solution to equations
    2. Integral Calculus – Helps in Area calculation , Monte-carlo simulation,etc
    3. Partial Differentiation — Multi variate equations – solving and optimization
    4. Automatic Differentiation – – Simplify the Differential computation – all the Deep learning frameworks use auto grad function ( automatic differentiation)
    5. Gradients – Hessian , Jababian ,Jacobian – Helps in Gradient based optimization
    6. Back propagation and chain rule – BP is the actual learning process of Deep learning

    List of resources to learn Calculus along with Programming them in python

    1. https://mml-book.github.io/book/mml-book.pdf
    2. https://d2l.ai/chapter_appendix-mathematics-for-deep-learning/multivariable-calculus.html
    3. https://www.sas.upenn.edu/~jesusfv/Lecture_NM_1_Numerical_Differentiation_Integration.pdf
    4. https://ocw.mit.edu/courses/mathematics/18-01sc-single-variable-calculus-fall-2010/download-course-materials/
    5. https://towardsdatascience.com/automatic-differentiation-explained-b4ba8e60c2ad
    6. https://towardsdatascience.com/automatic-differentiation-15min-video-tutorial-with-application-in-machine-learning-and-finance-333e18c0ecbb
    7. https://medium.com/swlh/gradient-based-optimizations-jacobians-jababians-hessians-b7cbe62d662d

    Probability and Statistics

    Probability and Statistics is needed for all the tasks in deep learning and machine learning

    1. To define the type of error to be considered ,
    2. Classification models involves Probability distribution at the end
    3. To measure the performance metrics of the models ,
    4. To define the noise distribution in generative models ,
    5. Deep Belief networks are based on Probability distributions

    I have included extensive list of topics to be learnt in Probability and statistics in my previous blog — https://prabakaranchandran.com/2021/04/22/an-ultimate-data-science-starter-kit-for-beginners/

    Few more :

    1. https://towardsdatascience.com/probability-and-statistics-explained-in-the-context-of-deep-learning-ed1509b2eb3f
    2. https://d2l.ai/chapter_preliminaries/probability.html
    3. https://towardsdatascience.com/machine-learning-probability-statistics-f830f8c09326?gi=9dc0045dff84
    4. https://towardsdatascience.com/probability-theory-for-deep-learning-9551b9255cf0?gi=c0663ceac3ad
    Programming for Deep learning : – Prerequisites
    1. Python is a major programming language which is widely used across different applications
    2. If you are new to python , check out my previous blog . I have listed out resources and topics to be learnt
    3. Learning pacakages like numpy , scipy , statsmodels , pandas , matplotlib is required for initial learning stage of deep learning
    4. Practice the Above listed mathematical topics using the python and its frameworks
    5. If you are coming from Computer science background , you can consider c++ for deep learning ( C++ is faster than Python in computation)

    Check these resources to learn python and its frameworks in terms of deep learning

    1. https://towardsdatascience.com/all-the-numpy-you-need-to-supercharge-your-deep-learning-code-e7a22fe4ede2
    2. https://scipy-lectures.org/packages/sympy.html
    3. https://docs.scipy.org/doc/scipy/reference/tutorial/integrate.html
    4. https://towardsdatascience.com/a-simple-method-for-numerical-integration-in-python-7906c1703af8
    5. https://www.kaggle.com/borisettinger/gentle-introduction-to-automatic-differentiation

    History of Artificial Neural Networks – 1940 – 2000 – Important Architectures

    “Deep Learning waves have lapped at the shores of computational linguistics for several years now, but 2015 seems like the year when the full force of the tsunami hit the major Natural Language Processing (NLP) conferences.” -Dr. Christopher D. Manning, Dec 2015
    
    I have listed down the important historical events happened in the AI domain

    In the above Time line , I have given Resource links to important concepts. We will learn about modern deep learning architectures which are popular in Computer vision , Natural language processing and Reinforcement learning separately in the upcoming blogs

    Since this is a introductory blog on deep learning , I don’t want to put all the complex architecture in one place.

    Optimization in Deep learning

    Optimizer is the function or method to change the weights and biases in order to make the model to converge much faster . Optimizers are the important building block of back propagation. Without optimizer , Back propagation is just gonna be a error and trail method. Optimizers pave the correct path to convergence. The very basic optimizer is Gradient descent which is based on very simple slope calculation along with learning rate.

    List of available optimizers for neural networks . Note : they are not for hyper parameter tuning , optimizers are predominantly used in model training ( at every back ward passes)

    1. Gradient Descent (GD)
    2. Stochastic Gradient Descent
    3. Mini-Batch Gradient Descent
    4. Momentum Based Gradient Descent
    5. Nesterov Accelerated Gradient (NAG)
    6. Adagrad
    7. RMSProp
    8. Adam
    9. AdaDelta

    Resources list :

    1. https://towardsdatascience.com/optimizers-for-training-neural-network-59450d71caf6
    2. https://www.kdnuggets.com/2020/12/optimization-algorithms-neural-networks.html
    3. https://medium.datadriveninvestor.com/overview-of-different-optimizers-for-neural-networks-e0ed119440c3
    4. https://www.upgrad.com/blog/types-of-optimizers-in-deep-learning/
    5. https://keras.io/api/optimizers/
    6. https://www.kaggle.com/getting-started/164479
    7. https://ruder.io/optimizing-gradient-descent/

    Deep learning Development life cycle

    So Far ,we have learnt all the necessary concepts in neural networks , architectures and allied mathematics

    If we need to build a neural network model for a simple regression work , we should maintain several steps to achieve that. The final model has to be available to the end user , it should not sleep in our development environment. That is called as Deep learning development life cycle

    1. Data collection and Data preparation
    2. Deep learning Model development
    3. Training and validation of Deep learning model – Transfer learning if opted
    4. Testing of Deep learning model
    5. Compare performance metrics of multiple model
    6. Deployment of model
    7. Model monitoring

    Read more about MLDC/DLDC

    1. https://www.educba.com/machine-learning-life-cycle/
    2. https://www.datarobot.com/wiki/machine-learning-life-cycle/
    3. https://towardsdatascience.com/transfer-learning-with-convolutional-neural-networks-in-pytorch-dd09190245ce?gi=ede0164dc869
    4. https://subscription.packtpub.com/book/big_data_and_business_intelligence/9781838550356/1/ch01lvl1sec13/ml-development-life-cycle
    5. https://towardsdatascience.com/rethinking-ai-machine-learning-model-management-8afeaa31d8f8
    6. https://towardsdatascience.com/the-machine-learning-lifecycle-in-2021-473717c633bc

    Deep learning Frameworks

    There are several frameworks available for Deep learning tasks , List of popular DL frameworks which are available in python

    1. Tensorflow
    2. PyTorch
    3. Keras
    4. Pytorch Lightning
    5. MxNet
    6. Theano
    7. Chainer
    8. Caffe2
    9. Microsoft CNTK

    Among them Tensorflow and Pytorch are used by most of the deep learning engineer. I don’t want to put a comparison between Tensorflow and pytorch ( that might offend one of them 🙂 )

    Keras is the high level api for tensorflow , Pytorch Ligtning is the high level api wrapper for pytorch.

    Keras and Lightnings are very easy to learn, so beginners / non cs grads can easily learn them and move forward.

    Books and YouTube channels to follow

    1. Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems
    2. Deep Learning Yoshua Bengio
    3. Pattern Recognition and Machine Learning
    4. Grokking Deep Learning
    5. Deep Learning Illustrated: A Visual, Interactive Guide to Artificial Intelligence
    6. https://www.youtube.com/watch?v=6M5VXKLf4D4
    7. https://www.youtube.com/watch?v=9jA0KjS7V_c&list=PLZoTAELRMXVPGU70ZGsckrMdr0FteeRUi
    8. https://www.youtube.com/watch?v=aircAruvnKk
    9. https://www.youtube.com/watch?v=5tvmMX8r_OM&list=PLtBw6njQRU-rwp5__7C0oIVt26ZgjG9NI
    10. https://www.youtube.com/watch?v=zfiSAzpy9NM

    Here by I conclude this article , So far I have plotted the road map for fundamental deep learning concepts

    In the upcoming blogs , I will discuss more about Road map for Computer vision , Generative models ,Natural Language processing, Reinforcement learning

    I hope , This blog helps you! Happy Learning!

    Signing off Prabakaran Chandran

    Advertisement

    3 thoughts on “A Complete Road map to Deep learning 2021 – Part 1

    Leave a Reply

    Fill in your details below or click an icon to log in:

    WordPress.com Logo

    You are commenting using your WordPress.com account. Log Out /  Change )

    Facebook photo

    You are commenting using your Facebook account. Log Out /  Change )

    Connecting to %s

    %d bloggers like this: