Thời lượng : 36 giờ

Chứng chỉ: Big Data Analytics do IT Viet Academy cấp

Giảng viên: IT Viet Academy

Mô tả: Khóa học này cung cấp những kiến thức nền tảng về đặc điểm và các thành phần của Big Data, giá trị mà Big Data mang lại doanh nghiệp,cung cấp các phương pháp phân tích khoa học dữ liệu.

Nội dung khóa học:

  • Introduction To Big Data
    • What is Big Data?
    • The Vs’ of Big Data
  • PySpark
    • PySpark: Spark with Python
    • Spark context, Spark Session, PySpark cell
    • Lambda with map(), filter()
    • Spark DataFrame
  • Programming in PySpark RDD’s
    • Abstraction Spark – resilient distributed dataset (RDD)
    • Abstracting Data with RDDs
    • RDDs from Parallelized collections, RDDs from External Datasets
    • Partitions in your data
    • Basic RDD Transformations and Actions
    • Map and Collect, Filter and Count
    • Pair RDDs in PySpark
    • ReduceBykey and Collect, SortByKey and Collect
    • Advanced RDD Actions
    • CountingBykeys
    • Create a base RDD and transform it
    • Remove stop words and reduce the dataset
    • Print word frequencies
  • PySpark SQL & DataFrames
    • Abstracting Data with DataFrames
    • RDD to DataFrame
    • Loading CSV into DataFrame
    • Operating on DataFrames in PySpark
    • Inspecting data in PySpark DataFrame
    • PySpark DataFrame subsetting and cleaning
    • Filtering your DataFrame
    • Interacting with DataFrames using PySpark SQL
    • Running SQL Queries Programmatically
    • SQL queries for filtering Table
    • Data Visualization in PySpark using DataFrames
    • PySpark DataFrame visualization
    • Create a DataFrame from CSV file
    • SQL Queries on DataFrame
    • Data visualization
  • Manipulating data
    • SQL in nutshell
    • Filter, Select
    • Aggregating, Group & Aggregating
    • Join
  • Data Analysis
    • EDA, Corr
    • Visualization: distplot, implot
  • Wrangling with Spark Functions
    • Drop, Filter, Scaling
    • Working with missing data
    • Join
  • Feature Engineering
    • Feature Generation
    • Differences
    • Ratios
    • Deeper Features, Time Features
    • Time Components, Joining On Time Components
    • Date Math
    • Extracting Features, Extracting Text to New Features
    • Splitting & Exploding
    • Pivot & Join
    • Binarizing, Bucketing & Encoding
    • Binarizing Day of Week
    • Bucketing
    • One Hot Encoding
  • Building a Model
    • Choosing the Algorithm
    • Which MLlib Module?
    • Creating Time Splits
    • Adjusting Time Features
    • Feature Engineering Assumptions for RFR
    • Feature Engineering For Random Forests
    • Dropping Columns with Low Observations
    • Naively Handling Missing and Categorical Values
    • Building a Model
    • Evaluating & Comparing Algorithms
    • Interpreting Results
  • Machine Learning with PySpark MLlib
    • PySpark MF libraries
    • PySpark MLlib algorithms
    • Collaborative filtering
    • Loading Movie Lens dataset into RDDs
    • Model training & predictions
    • Model evaluation using MSE
    • Classification
    • Loading spam & non-spam data
    • Feature hashing & LabelPoint
    • Logistic Regression model training
    • Clustering
    • Loading & parsing the 5000 points data
    • K-means training