Cannot clone object <tensorflow.python.keras.wrappers.scikit_learn.KerasClassifier object

Question

This is with regards to TF 2.0.

Please find below my code that performs GridSearch along with Cross Validation using sklearn.model_selection.GridSearchCV for the mnist dataset that works perfectly fine.

# Build Function to create model, required by KerasClassifier

    def create_model(optimizer_val='RMSprop',hidden_layer_size=16,activation_fn='relu',dropout_rate=0.1,regularization_fn=tf.keras.regularizers.l1(0.001),kernel_initializer_fn=tf.keras.initializers.glorot_uniform,bias_initializer_fn=tf.keras.initializers.zeros):
        model = tf.keras.models.Sequential([
        tf.keras.layers.Flatten(input_shape=(28, 28)),    
        tf.keras.layers.Dense(units=hidden_layer_size, activation=activation_fn,kernel_regularizer=regularization_fn,kernel_initializer=kernel_initializer_fn,bias_initializer=bias_initializer_fn), 
        tf.keras.layers.Dropout(dropout_rate),
        tf.keras.layers.Dense(units=hidden_layer_size,activation='softmax',kernel_regularizer=regularization_fn,kernel_initializer=kernel_initializer_fn,bias_initializer=bias_initializer_fn) 
          ])
        optimizer_val_final=optimizer_val
        model.compile(optimizer=optimizer_val, loss='sparse_categorical_crossentropy', metrics=['accuracy'])
        return model

    #Create the model with the wrapper
    model = tf.keras.wrappers.scikit_learn.KerasClassifier(build_fn=create_model, epochs=100, batch_size=10, verbose=2)

    #Initialize the parameter grid
    nn_param_grid = {
        'epochs': [10],     
        'batch_size':[128],
        'optimizer_val': ['Adam','SGD'],
        'hidden_layer_size': [128],
        'activation_fn': ['relu'],     
        'dropout_rate': [0.2],    
        'regularization_fn':['l1','l2','L1L2'],    
        'kernel_initializer_fn':['glorot_normal', 'glorot_uniform'],    
        'bias_initializer_fn':[tf.keras.initializers.zeros]    
    }
    #Perform GridSearchCV
    grid = GridSearchCV(estimator=model, param_grid=nn_param_grid, verbose=2, cv=3,scoring=precision_custom,return_train_score=False,n_jobs=-1) 
    grid_result = grid.fit(x_train, y_train)

My idea is to pass different optimizers with different learning rates , say Adam for learning rates 0.1,0.01 and 0.001. I also want to try out SGD with different learning rates and momentum values.

In that case , when I pass 'optimizer_val': [tf.keras.optimizers.Adam(0.1)], , I get the error as given below:

Cannot clone object <tensorflow.python.keras.wrappers.scikit_learn.KerasClassifier object at 0x7fe08b210e10>, as the constructor either does not set or modifies parameter optimizer_val

Please advise as to how can I rectify this error.

Answer 1

This is sklearn bug. You should reduce the version of sklearn:

conda install scikit-learn==0.21.2

It's OK!

Answer 2

You can fix the issue with changing the list into tuple. If there is any single valued instance then you can use list.

    #Initialize the parameter grid
    nn_param_grid = {
        'epochs': [10],     
        'batch_size':[128],
        'optimizer_val': ('Adam','SGD'),
        'hidden_layer_size': [128],
        'activation_fn': ['relu'],     
        'dropout_rate': [0.2],    
        'regularization_fn':('l1','l2','L1L2'),
        'kernel_initializer_fn':('glorot_normal', 'glorot_uniform'),
        'bias_initializer_fn':[tf.keras.initializers.zeros]    
    }

Answer 3

Found this comment online and it helped!

For those who are getting following error due to above statement: Cannot clone object <keras.wrappers.scikit_learn.KerasClassifier object at 0x7f93ddc5d1d0>, as the constructor either does not set or modifies parameter layers

Change the layers from array of list to array of tuple: layers => [(20,), (45, 30, 15), (40, 20)]
Don't forget to add comma after (20,) otherwise another error/warning will appear - FitFailedWarning: Estimator fit failed. The score on this train-test partition for these parameters will be set to nan. Details: TypeError: 'int' object is not iterable Because single tuple without comma is treated as int.

Answer 4

Only installing TensorFlow 2.8 helped with this issue. Notice that it is available only via pip Anaconda TensorFlow 2.7 vs. Pypi TensorFlow 2.8

To check your version of Tensorflow type: conda list tensorflow

(base) C:\Users\User> conda list tensorflow-gpu
# Name           Version          Build  Channel
tensorflow-gpu   2.4.1     pyhd8ed1ab_3  conda-forge

To uninstall type: conda uninstall tensorflow and to install version 2.8 type:

pip install tensorflow-gpu==2.8