Odtnhj

I'm using a dataset with categorical features in PySpark which are indexed and one-hot encoded. After fitting the pipeline I extract the encoded features by using the metadata of the features column. When I include a normalizer in my pipeline I lose the metadata of my categorical features. See example below:

train.show()
+-----+---+----+----+
|admit|gre| gpa|rank|
+-----+---+----+----+
| 0.0|380|3.61| 3|
| 1.0|660|3.67| 3|
| 1.0|800| 4.0| 1|
| 1.0|640|3.19| 4|
| 0.0|520|2.93| 4|
+-----+---+----+----+

from pyspark.ml.feature import StringIndexer, OneHotEncoder, VectorAssembler, Normalizer
#indexer for categorical features
rank_indexer = StringIndexer(inputCol = 'rank', outputCol = 'rank_ind', handleInvalid="skip")
#encoder for categorical features
rank_encoder = OneHotEncoder(inputCol = 'rank_ind', outputCol = 'rank_enc')
# assembler
assembler = VectorAssembler(inputCols=['gre','gpa','rank_enc'], outputCol="featuresVect")
# Create the normalizer
normalizer = Normalizer(inputCol="featuresVect", outputCol="features", p=1.0)

stages = [rank_indexer] + [rank_encoder] + [assembler] + [normalizer]

from pyspark.ml import Pipeline
final_pipeline = Pipeline(
 stages = stages
)

pipelineModel = final_pipeline.fit(train)
data = pipelineModel.transform(train)

data.schema['features'].metadata

## empty dictionary

## excluding the normalizer results in this metadata:
u'ml_attr': u'attrs': u'binary': [u'idx': 2, u'name': u'rank_enc_2',
 u'idx': 3, u'name': u'rank_enc_3',
 u'idx': 4, u'name': u'rank_enc_4'],
 u'numeric': [u'idx': 0, u'name': u'gre', u'idx': 1, u'name': u'gpa'],
 u'num_attrs': 5

Is this normal behavior? How can I include a normalizer without losing this metadata?

In my opinion it doesn't make much sense to use Normalizer on One-hot encoded data in the first place. In Spark, OHE is useful for two type models:

• Multinomial Naive Bayes.


• Linear models.

In the first case normalization will render features completely useless (multinomial model can fully utilize only binary features). In the second case it will make interpretation of the model close to impossible.

Even if you ignore the above normalized data cannot be interpreted as binary features anymore, therefore discarding metadata seems to be a valid behavior.

Related to Why does StandardScaler not attach metadata to the output column?