Run the notebook. Compare the results of DummyClassifier, LogisticRegression and the DistilBERT model. Which accuracy is reached by them? Which model performed best?
The notebook produces visualization of the last hidden state from the DistilBERT model before and after training. Compare both visualizations and describe the differences between them. Which effect did training have?
The default value for the train_feature_extractor variable is False which means the DistilBERT model's feature extractor isn't trained, only the classification head will be trained during training. Now set train_feature_extractor to True, which will now train the complete model including the feature extractor, train the model again and compare the accuracy and the hidden state visualization to the previous training without training the feature extractor. Does training the feature extractor improves the model's accuracy? What effect does it have on the training time?
Your Submission
Make two copies of the notebook, one where train_feature_extractor is set to False and one where train_feature_extractor is set to True
Run both notebooks
Answer the questions at the end of the notebook by edting the cell and writing your answers there
Save the notebooks and send both notebooks with all cell outputs to us