PySpark

Apache Spark is an open-source powerful distributed querying and processing engine. It provides flexibility and extensibility of MapReduce but at significantly higher speeds: Up to 100 times faster than Apache Hadoop when data is stored in memory and up to 10 times when accessing disk. Apache Spark allows the user to read, transform, and aggregate data, as well as train and deploy sophisticated statistical models with ease. The Spark APIs are accessible in Java, Scala, Python, R and SQL. Apache Spark can be used to build applications or package them up as libraries to be deployed on a cluster or perform quick analytics interactively through notebooks (like, for instance, Jupyter, Spark-Notebook, Databricks notebooks, and Apache Zeppelin).

Learn about Apache Spark and the Spark 2.0 and PySpark architecture • Build and interact with PySpark DataFrames • Read, transform, and understand data and use it to train machine learning models • Build machine learning models with MLlib and ML • Learn how to submit your applications programmatically using spark-submit • ML Algorithms: common learning algorithms such as classification, regression, clustering, and collaborative filtering • Features : feature extraction, transformation, dimensionality reduction, and selection • Pipelines: tools for constructing, evaluating, and tuning ML Pipelines • Persistence: saving and load algorithms, models, and Pipelines • Utilities: linear algebra, statistics, data handling, etc.

• Big Data and Hadoop • Basic Python data structures • Basic knowledge of Pandas dataframes and SQL • Entry-level Data Science

Hardware : Intel Core 5 processor with 16GB Recommended RAM. OS : Ubuntu Server ( Latest Version ) or Cent OS or Mac OS or Windows 64 bit 7/8/10 ( Latest preferable version ) High Speed Internet Connection ( Open Port for Installations ) Software Prerequisites Java ( Latest Version ) , Scala ( Latest Version) Apache Spark [ Latest Version ] (Downloadable from http://spark.apache.org/downloads.html) A Python distribution containing IPython, Pandas and Scikit-learn Anaconda with Python3.6, PySpark Local Environment www.anaconda.com [ Local Machine ] Hadoop, PySpark PySpark on Hadoop Cloud Environment OR Cloudera Hadoop or Online Databriks Cloud

Course Feedback