1
$\begingroup$

I have a NN for PINNs, which takes an 2D input and provides an 1D output. I need to normalise the input data(lets say between -1 to 1). I can think of two approaches where

Approach 1: one approach computes the normalised data outside the gradient scope(NN) and passes the normalised input value into the NN

x = np.linspace(-5,5,100)
y = np.linspace(-5,5,100)

# concat the x and y values
x, y = np.meshgrid(x, y)

# normalisation performed outside the scope
input_tensor = np.column_stack((x.flatten(), y.flatten()))

# normalize the input tensor
input_tensor  = (input_tensor - np.mean(input_tensor, axis=0))/np.std(input_tensor, axis=0)

# train step within custom model
def train_step():
    with tf.GradientTape() as tape:
        # Compute the predicted values from the model
        predicted_values = model(input_tensor)

        # Compute the loss
        loss = loss_function(predicted_values, actual_values)

    # Compute the gradients of the loss function wrt the trainable variables
    gradients = tape.gradient(loss, model.trainable_variables)

    # Apply the gradients to the optimizer
    optimizer.apply_gradients(zip(gradients, model.trainable_variables))

    return loss

Appraoch 2: Pass the raw data to NN and perform normalisation within the gradient scope using lambda layers or other operations.

# train step within custom model
x = np.linspace(-5,5,100)
y = np.linspace(-5,5,100)

# concat the x and y values
x, y = np.meshgrid(x, y)

input_tensor = np.column_stack((x.flatten(), y.flatten()))

mean_x, std_x = get_mean_std(x.flatten())
mean_y, std_y = get_mean_std(y.flatten())

# train step within custom model
def train_step():
    with tf.GradientTape() as tape:
        
        # normalize the input tensor
        normalized_x_values = (input_tensor[:, 0:1] - mean_x)/std_x
        normalized_y_values = (input_tensor[:, 1:2] - mean_y)/std_y

        normalized_input_tensor = tf.concat([normalized_x_values, normalized_y_values], axis=1)

        # Compute the predicted values from the model
        predicted_values = model(input_tensor)

        # Compute the loss
        loss = loss_function(predicted_values, actual_values)

    # Compute the gradients of the loss function wrt the trainable variables
    gradients = tape.gradient(loss, model.trainable_variables)

    # Apply the gradients to the optimizer
    optimizer.apply_gradients(zip(gradients, model.trainable_variables))

    return loss

  1. In general, Which one of the approach is preferred and why?
  2. Though Approach-2 is compute intensive when compared to approach 1( since normalisation is performed at every training epoch), is there any advantage in using the same?
  3. Especially for Physics Informed Neural Networks which one of them is preferred and why?
$\endgroup$
1
  • $\begingroup$ Both look same to me. You are normalizing outside or inside your training loop, but the result of both should be same. $\endgroup$
    – cfdlab
    Commented Feb 3 at 4:16

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.