1 of 36

Sparkitecture

Welcome to Sparkitecture!

Created by: Colby T. Ford, Ph.D.

PySpark Edition | A work in progress... | Created using

Cloud Service Integration

Azure Storage

Storage is a managed service in Azure that provides highly available, secure, durable, scalable, and redundant storage for your data. Azure Storage includes both Blobs, Data Lake Store, and others.

Databricks-Specific Functionality

Mounting Blob Storage

Once you create your blob storage account in Azure, you will need to grab a couple bits of information from the Azure Portal before you mount your storage.

You can find your Storage Account Name (which will go in below) and your Key (which will go in below) under Access Keys in your Storage Account resource in Azure.

Go into your Storage Account resource in Azure and click on Blobs. Here, you will find all of your containers. Pick the one you want to mount and copy its name into below.

As for the mount point (/mnt/<FOLDERNAME> below), you can name this whatever you'd like, but it will help you in the long run to name it something useful along the lines of storageaccount_container.

Once you have the required bits of information, you can use the following code to mount the storage location inside the Databricks environment

You can then test to see if you can list the files in your mounted location

Resources:

To learn how to create an Azure Storage service, visit

Mounting Data Lake Storage

For finer-grained access controls on your data, you may opt to use Azure Data Lake Storage. In Databricks, you can connect to your data lake in a similar manner to blob storage. Instead of an access key, your user credentials will be passed through, therefore only showing you data that you specifically have access to.

Pass-through Azure Active Directory Credentials

To pass in your Azure Active Directory credentials from Databricks to Azure Data Lake Store, you will need to enable this feature in Databricks under New Cluster > Advanced Options.

Note: If you create a High Concurrency cluster, multiple users can use the same cluster. The Standard cluster mode will only allow a single user's credential at a time.

Azure SQL Data Warehouse / Synapse

Set up Azure SQL DW connection parameters

Define a query

Data Preparation

Reading and Writing Data

Reading in Data

...from Mounted Storage

Machine Learning

About Spark MLlib

MLlib is Apache Spark's scalable machine learning library.

MLlib works with Spark's APIs and with NumPy in Python and with R libraries. Since Spark excels at iterative computation, MLlib runs very fast with highly-scalable, high-quality algorithms that leverage iteration.

Included Functionality:

ML algorithms include:

Classification: logistic regression, naive Bayes,...
Regression: generalized linear regression, survival regression,...
Decision trees, random forests, and gradient-boosted trees

ML workflow utilities include:

Feature transformations: standardization, normalization, hashing,...
ML Pipeline construction
Model evaluation and hyper-parameter tuning

Other utilities include:

Distributed linear algebra: SVD, PCA,...
Statistics: summary statistics, hypothesis testing,...

Resources

Logistic Regression

Setting Up a Logistic Regression Classifier

Note: Make sure you have your training and test data already vectorized and ready to go before you begin trying to fit the machine learning model to unprepped data.

Naïve Bayes

Setting Up a Naïve Bayes Classifier

Note: Make sure you have your training and test data already vectorized and ready to go before you begin trying to fit the machine learning model to unprepped data.

Decision Tree

Setting Up a Decision Tree Classifier

Note: Make sure you have your training and test data already vectorized and ready to go before you begin trying to fit the machine learning model to unprepped data.

Random Forest

Setting Up a Random Forest Classifier

Note: Make sure you have your training and test data already vectorized and ready to go before you begin trying to fit the machine learning model to unprepped data.

Gradient-Boosted Trees

Setting Up Gradient-Boosted Tree Classifier

Note: Make sure you have your training and test data already vectorized and ready to go before you begin trying to fit the machine learning model to unprepped data.

Regression

Decision Tree

Setting Up Decision Tree Regression

Note: Make sure you have your training and test data already vectorized and ready to go before you begin trying to fit the machine learning model to unprepped data.

Load in required libraries

Initialize Decision Tree object

Create a parameter grid for tuning the model

Define how you want the model to be evaluated

Define the type of cross-validation you want to perform

Fit the model to the data

Score the testing dataset using your fitted model for evaluation purposes

Evaluate the model

Note: When you use the CrossValidator function to set up cross-validation of your models, the resulting model object will have all the runs included, but will only use the best model when you interact with the model object using other functions like evaluate or transform.

Random Forest

Setting Up Random Forest Regression

Note: Make sure you have your training and test data already vectorized and ready to go before you begin trying to fit the machine learning model to unprepped data.

Load in required libraries

Initialize Random Forest object

Create a parameter grid for tuning the model

Define how you want the model to be evaluated

Define the type of cross-validation you want to perform

Fit the model to the data

Score the testing dataset using your fitted model for evaluation purposes

Evaluate the model

Gradient-Boosted Trees

Setting Up Gradient-Boosted Tree Regression

Note: Make sure you have your training and test data already vectorized and ready to go before you begin trying to fit the machine learning model to unprepped data.

MLflow

MLflow is an open source library by the Databricks team designed for managing the machine learning lifecycle. It allows for the creation of projects, tracking of metrics, and model versioning.

Install mlflow using pip

MLflow can be used in any Spark environmnet, but the automated tracking and UI of MLflow is Databricks-Specific Functionality.

Feature Importance

Extract important features using Gini

## Based on: https://www.timlrx.com/2018/06/19/feature-selection-using-feature-importance-score-creating-a-pyspark-estimator/
import pandas as pd

def ExtractFeatureImportance(featureImp, dataset, featuresCol):
    list_extract = []
    for i in dataset.schema[featuresCol].metadata["ml_attr"]["attrs"]:
        list_extract = list_extract + dataset.schema[featuresCol].metadata["ml_attr"]["attrs"][i]
    varlist = pd.DataFrame(list_extract)
    varlist['score'] = varlist['idx'].apply(lambda x: featureImp[x])
    return(varlist.sort_values('score', ascending = False))
  
  
# ExtractFeatureImportance(model.stages[-1].featureImportances, dataset, "features")
dataset_fi = ExtractFeatureImportance(model.bestModel.featureImportances, dataset, "features")
dataset_fi = sqlContext.createDataFrame(dataset_fi)
display(dataset_fi)

Extract important features using p-values

Extract coefficients from a model

Model Saving and Loading

Model Saving

Save model(s) to mounted storage

Streaming Data

Structured Streaming

Read in Streaming Data

Reading JSON files from storage

from pyspark.sql.types import *

inputPath = "/mnt/data/jsonfiles/"

# Define your schema if it's known (rather than relying on Spark to infer the schema)
jsonSchema = StructType([StructField("time", TimestampType(), True),
                         StructField("id", IntegerType(), True),
                         StructField("value", StringType(), True)])

streamingInputDF = spark.readStream \
                        .schema(jsonSchema) \
                        .option("maxFilesPerTrigger", 1) \ # Treat a sequence of files as a stream by picking one file at a time
                        .json(inputPath)

References

Databricks Structured Streaming:

Operationalization

Natural Language Processing

Model Evaluation

Multiclass classification evaluator

from pyspark.mllib.evaluation import MulticlassMetrics
evaluator = MulticlassClassificationEvaluator(predictionCol="prediction")

for model in ["lrpredictions", "nbpredictions", "rfpredictions"]:
    
    df = globals()[model]
    ########################################
    # Compute raw scores on the test set
    predictionAndLabels = df.select("prediction", "label").rdd

    # Instantiate metrics object
    metrics = MulticlassMetrics(predictionAndLabels)

    # Overall statistics
    precision = metrics.precision()
    recall = metrics.recall()
    f1Score = metrics.fMeasure()
    print("Summary Stats for: ", model)
    #print(metrics.confusionMatrix())
    print("Accuracy = %s" % evaluator.evaluate(df))
    print("Precision = %s" % precision)
    print("Recall = %s" % recall)
    print("F1 Score = %s" % f1Score)

    # Weighted stats
    #print("Weighted recall = %s" % metrics.weightedRecall)
    #print("Weighted precision = %s" % metrics.weightedPrecision)
    #print("Weighted F(1) Score = %s" % metrics.weightedFMeasure())
    #print("Weighted F(0.5) Score = %s" % metrics.weightedFMeasure(beta=0.5))
    #print("Weighted false positive rate = %s" % metrics.weightedFalsePositiveRate)
    print("\n")

Bioinformatics and Genomics

Azure Data Factory

Transformation with Azure Databricks

Using Azure Databricks with Azure Data Factory, notebooks can be run from an end-to-end pipeline that contains the Validation, Copy data, and Notebook activities in Azure Data Factory.

Validation ensures that your source dataset is ready for downstream consumption before you trigger the copy and analytics job.
Copy data duplicates the source dataset to the sink storage, which is mounted as DBFS in the Azure Databricks notebook. In this way, the dataset can be directly consumed by Spark.
Notebook triggers the Databricks notebook that transforms the dataset. It also adds the dataset to a processed folder or Azure SQL Data Warehouse.

Import a notebook for Transformation

To import a Transformation notebook to your Databricks workspace:

Sign in to your Azure Databricks workspace, and then select Import. Your workspace path can be different from the one shown, but remember it for later.
Select Import from: URL. In the text box, enter https://adflabstaging1.blob.core.windows.net/share/Transformations.html.
Now let's update the

How to use this template

Go to the Transformation with Azure Databricks template and create new linked services for following connections.
- Source Blob Connection - to access the source data.
  For this exercise, you can use the public blob storage that contains the source files. Reference the following screenshot for the configuration. Use the following SAS URL to connect to source storage (read-only access):

Pipeline introduction and configuration

In the new pipeline, most settings are configured automatically with default values. Review the configurations of your pipeline and make any necessary changes.

In the Validation activity Availability flag, verify that the source Dataset value is set to SourceAvailabilityDataset that you created earlier.
In the Copy data activity file-to-blob, check the Source and Sink tabs. Change settings if necessary.

In below datasets, the file path has been automatically specified in the template. If any changes required, make sure that you specify the path for both container and directory in case any connection error.

SourceAvailabilityDataset - to check that the source data is available.
SourceFilesDataset - to access the source data.
DestinationFilesDataset - to copy the data into the sink destination location. Use the following values:

Select Debug to run the pipeline. You can find the link to Databricks logs for more detailed Spark logs.
You can also verify the data file by using Azure Storage Explorer.

For correlating with Data Factory pipeline runs, this example appends the pipeline run ID from the data factory to the output folder. This helps keep track of files generated by each run.

Sparkitecture

Welcome to Sparkitecture!

hashtag

Cloud Service Integration

Azure Storage

hashtagMounting Blob Storage

hashtagResources:

hashtagMounting Data Lake Storage

hashtagPass-through Azure Active Directory Credentials

Azure SQL Data Warehouse / Synapse

hashtagSet up Azure SQL DW connection parameters

hashtagDefine a query

Data Preparation

Reading and Writing Data

hashtagReading in Data

hashtag...from Mounted Storage

Machine Learning

About Spark MLlib

hashtagIncluded Functionality:

hashtagML algorithms include:

hashtagML workflow utilities include:

hashtagOther utilities include:

hashtagResources

Logistic Regression

hashtagSetting Up a Logistic Regression Classifier

Naïve Bayes

hashtagSetting Up a Naïve Bayes Classifier

Decision Tree

hashtagSetting Up a Decision Tree Classifier

Random Forest

hashtagSetting Up a Random Forest Classifier

Gradient-Boosted Trees

hashtagSetting Up Gradient-Boosted Tree Classifier

Regression

Decision Tree

hashtagSetting Up Decision Tree Regression

hashtagLoad in required libraries

hashtagInitialize Decision Tree object

hashtagCreate a parameter grid for tuning the model

hashtagDefine how you want the model to be evaluated

hashtagDefine the type of cross-validation you want to perform

hashtagFit the model to the data

hashtagScore the testing dataset using your fitted model for evaluation purposes

hashtagEvaluate the model

Random Forest

hashtagSetting Up Random Forest Regression

hashtagLoad in required libraries

hashtagInitialize Random Forest object

hashtagCreate a parameter grid for tuning the model

hashtagDefine how you want the model to be evaluated

hashtagDefine the type of cross-validation you want to perform

hashtagFit the model to the data

hashtagScore the testing dataset using your fitted model for evaluation purposes

hashtagEvaluate the model

Gradient-Boosted Trees

hashtagSetting Up Gradient-Boosted Tree Regression

MLflow

hashtagInstall mlflow using pip

Feature Importance

hashtagExtract important features using Gini

hashtagExtract important features using p-values

hashtagExtract coefficients from a model

Model Saving and Loading

hashtagModel Saving

hashtagSave model(s) to mounted storage

Streaming Data

Structured Streaming

hashtagRead in Streaming Data

hashtagReading JSON files from storage

hashtagReferences

Operationalization

Natural Language Processing

Model Evaluation

hashtagMulticlass classification evaluator

Bioinformatics and Genomics

Azure Storage

hashtagMounting Blob Storage

hashtagResources:

hashtagMounting Data Lake Storage

hashtagPass-through Azure Active Directory Credentials

Mounting Blob Storage

Resources:

Mounting Data Lake Storage

Pass-through Azure Active Directory Credentials

Set up Azure SQL DW connection parameters

Define a query

Reading in Data

...from Mounted Storage

Included Functionality:

ML algorithms include:

ML workflow utilities include:

Other utilities include:

Resources

Setting Up a Logistic Regression Classifier

Setting Up a Naïve Bayes Classifier

Setting Up a Decision Tree Classifier

Setting Up a Random Forest Classifier

Setting Up Gradient-Boosted Tree Classifier

Setting Up Decision Tree Regression

Load in required libraries

Initialize Decision Tree object

Create a parameter grid for tuning the model

Define how you want the model to be evaluated

Define the type of cross-validation you want to perform

Fit the model to the data

Score the testing dataset using your fitted model for evaluation purposes

Evaluate the model

Setting Up Random Forest Regression

Load in required libraries

Initialize Random Forest object

Create a parameter grid for tuning the model

Define how you want the model to be evaluated

Define the type of cross-validation you want to perform

Fit the model to the data

Score the testing dataset using your fitted model for evaluation purposes

Evaluate the model

Setting Up Gradient-Boosted Tree Regression

Install mlflow using pip

Extract important features using Gini

Extract important features using p-values

Extract coefficients from a model

Model Saving

Save model(s) to mounted storage

Read in Streaming Data

Reading JSON files from storage

References

Multiclass classification evaluator

Mounting Blob Storage

Resources:

Mounting Data Lake Storage

Pass-through Azure Active Directory Credentials

How to Cite

Included Functionality:

ML algorithms include:

ML workflow utilities include:

Other utilities include:

Resources

Setting Up a Naïve Bayes Classifier

Reading in Data

...from Mounted Storage

Set up Azure SQL DW connection parameters

Define a query

Model Saving

Save model(s) to mounted storage

Install mlflow using pip

Multiclass classification evaluator

Initialize Naïve Bayes object

Create a parameter grid for tuning the model

Define how you want the model to be evaluated

Define the type of cross-validation you want to perform

Fit the model to the data

Score the testing dataset using your fitted model for evaluation purposes

Evaluate the model

Writing out Data

Other Resources

Score new data using a trained model

Load in required libraries

Load in the transformation pipeline

Load in the trained model

Remove unnecessary columns from the scored data