Transfer learning for predicting source terms of principal component transport in chemically reactive flowJournal: -

In a spatially-homogeneous constant volume reactor, the temporal evolution of species and temperature starting from the initial time, $t$ = 0, is computed by solving the system of ordinary differential equations (ODEs) defined by

where $\boldsymbol{\mathrm{\theta}}$ represents the thermochemical state vector (i.e., species mass fraction and temperature), $\dot{\boldsymbol{\mathrm{\omega}}}_{\boldsymbol{\mathrm{\theta}}}$ the reaction rate vector of $\boldsymbol{\mathrm{\theta}}$, and $t_{f}$ the end time. For the initial conditions, the initial pressure of the system, $p_{0}$, is fixed to be atmospheric, and different values for the initial temperature, $T_{0}$, are used including 1000, 1050, 1100, 1300, and 1400 K. The initial mass fractions of the hydrogen/air mixture are determined by an equivalence ratio, $\phi$, which ranges from 0.1 to 3.0. A detailed chemical kinetic mechanism for hydrogen/air mixtures, developed by Li et al. Li04, is used where the dimension of the original thermochemical state vector is 10. A six-stage, fourth-order Runge-Kutta method Kennedy94 with a uniform time step, dt, of 0.2 ns is adopted for time integration. A CHEMKIN library Kee96 is used to compute the chemical kinetics and thermodynamic properties of the mixture.

Figure 1 shows the ignition delay time, $\tau_{\rm{ig}}$, of the hydrogen/air mixture for various $\phi$ and $T_{0}$. As expected, the variation in $\tau_{\rm{ig}}$ exhibits a “U”-shaped profile as a function of $\phi$. Here, $\tau_{\rm{ig}}$ is defined as the time at which the temperature gradient is maximum. In accordance with Arrhenius Law, $\tau_{\rm{ig}}$ notably changes with changes in $T_{0}$.

In the present study, the objective of the PC-transport ROM is to replicate the ignition characteristics of the hydrogen/air mixture over a wide range of $\phi$ at a specific $T_{0}$, meaning that training samples, the low-dimensional manifold, and the corresponding ANN model are separated by $T_{0}$. To reasonably provide a data-sparse scenario, an underlying assumption of the present study is that there exists a sufficient number of training samples spanning over $\phi$ at $T_{0}$ of 1000 K, while the training data size for the cases where $T_{0}>$ 1000 K is assumed to be sparse (see the symbols in Fig. 1 as an example). Specifically, at $T_{0}=$ 1000 K, the training dataset is collected by carrying out 30 different 0-D simulations varying $\phi$ (i.e., $\Delta\phi$ = 0.1; $\phi$ ranging from 0.1 to 3.0), and then the low-dimensional manifold is defined by applying PCA to the training dataset. Training of the ANN model by using a sufficient number of training samples at $T_{0}=$ 1000 K is considered as the “source task” for the present study. Here, $N_{\phi}$ is defined as the number of 0-D simulations at a given $T_{0}$ such that $N_{\phi}$ of the source task is 30. For the “target tasks” where $T_{0}$ is higher than 1000 K (i.e., 1050, 1100 1300, and 1400 K), $N_{\phi}$ is set to be less than or equal to 4, such that the number of training data for the target task is forced to be sparse. In this study, a “sparse dataset” refers to a dataset with insufficient training samples such that the corresponding ROM is unable to replicate the ignition characteristics of a fuel/air mixture with a wide range of $\phi$ (i.e., $\phi=0.1-3.0$) at a given $T_{0}$. The description of the dataset with different $N_{\phi}$ is summarized in Table 1.

Note that the present study provides a data-sparse scenario based on a series of 0-D simulations, but such an imbalance in the number of training samples can also be observed from multi-dimensional simulations and experiments Humbird20, Subel21. It is also noted that transfer learning methods used in the present study are not limited to specific source and target tasks. Rather, these methods can be applied to various scenarios (e.g., different pressure or equivalence ratio conditions between source and target tasks), provided that there is a task similarity between two tasks.

For each 0-D simulation, the thermochemical state vector and their reaction rate, $\boldsymbol{\mathrm{\theta}}$ and $\dot{\boldsymbol{\mathrm{\omega}}}_{\boldsymbol{\mathrm{\theta}}}$, respectively, are uniformly sampled from $t$ of 0 to 2$\tau_{\rm{ig}}$. The number of samples for each 0-D simulation is set to be 20,000 such that the first 10,000 samples are assigned to the pre-ignition zone and the remaining 10,000 samples are related to the post-ignition zone. The “test dataset” at a given $T_{0}$ is also prepared to evaluate the accuracy of the PC-transport ROM. It consists of 29 different 0-D simulation results at a given $T_{0}$ ($\Delta\phi$ = 0.1; $\phi$ ranging from 0.15 to 2.95) and is separated from the training dataset.

Consistent with previous studies applying PC-transport ROM Sutherland09, Parente09, Mirgolbabaei13, Mirgolbabaei14, the dimension of the original thermochemical vector is reduced by applying PCA. Assuming that $M$ number of samples of the $N-$dimensional thermochemical state vectors are collected by performing multiple 0-D simulations at a given $T_{0}$, the dataset of the thermochemical vector, $\boldsymbol{\mathrm{\Theta}}$ = [$\boldsymbol{\mathrm{\theta}}_{1}$, $\boldsymbol{\mathrm{\theta}}_{2}$, …, $\boldsymbol{\mathrm{\theta}}_{M}$], is composed of a $N\times M$ dimensional matrix. After normalizing $\boldsymbol{\mathrm{\Theta}}$ based on its component-wise range, the $N\times N$ dimensional matrix of orthonormal eigenvectors, Q^T, of the covariance matrix of $\boldsymbol{\mathrm{\Theta}}$ is constructed, and subsequently, the dataset of the PC vector, $\boldsymbol{\mathrm{\Psi}}$, can be defined as

where $\boldsymbol{\mathrm{\Psi}}\in\mathbb{R}^{N\times M}$ represents the $M$ numbers of collections of the PC vector, $\boldsymbol{\mathrm{\psi}}=$ [$\psi_{1}$, $\psi_{2}$, …, $\psi_{N}$]^T.

Note that the first PC, $\psi_{1}$, is a linear combination of the original thermochemical state vector that captures the maximum variance of the dataset. The second PC, $\psi_{2}$, is then orthogonal to the first PC, and all the subsequent PCs follow the same concept. In the present study, the leading first five PCs (i.e., $N_{\rm{PC}}$ = 5) are retained from $\boldsymbol{\mathrm{\psi}}$ such that the dimensionality of the system is reduced from 10 to 5, which captures over 99% of the original total variance. In other words, a $N\times N_{\rm{PC}}$ matrix of A is constructed that contains the leading $N_{\rm{PC}}$ eigenvectors of Q. The low-dimensional manifold then becomes

where $\boldsymbol{\mathrm{\Psi}}^{red}\in\mathbb{R}^{N_{\rm{PC}}\times M}$ represents the dataset of the truncated PC vector, $(\boldsymbol{\mathrm{\psi}}^{red})$ = [$\psi_{1}$, $\psi_{2}$, …, $\psi_{N_{\rm{PC}}}$]^T. Hereinafter, $\boldsymbol{\mathrm{\Psi}}^{red}$ and $\boldsymbol{\mathrm{\psi}}^{red}$ are referred to as $\boldsymbol{\mathrm{\Psi}}$ and $\boldsymbol{\mathrm{\psi}}$, respectively, for the sake of brevity.

The system of ODEs for the low-dimensional manifold can be defined by projecting Eq. 1 on the matrix $\textbf{A}^{\rm{T}}$:

where $\dot{\boldsymbol{\mathrm{\omega}}}_{\boldsymbol{\mathrm{\psi}}}$ is the reaction rate term for $\boldsymbol{\mathrm{\psi}}$, defined by $\dot{\boldsymbol{\mathrm{\omega}}}_{\boldsymbol{\mathrm{\psi}}}$ = $\textbf{A}^{\rm{T}}\dot{\boldsymbol{\mathrm{\omega}}}_{\boldsymbol{\mathrm{\theta}}}$ Sutherland09. In the framework of PC-transport ROM, the time integration of Eq. 4 is solved instead of solving Eq. 1, and then a conversion from $\boldsymbol{\mathrm{\psi}}$ into $\boldsymbol{\mathrm{\theta}}$ is carried out as a post-processing step. An ANN model is used for the tabulation of $\dot{\boldsymbol{\mathrm{\omega}}}_{\boldsymbol{\mathrm{\psi}}}$ as a function of $\boldsymbol{\mathrm{\psi}}$.

Figure 2 shows the PC modes with respect to the original thermochemical vector defined by using different training dataset in terms of $T_{0}$. It is readily observed from the figure that the PC modes show a similar trend irrespective of the dataset. Specifically, the first PC mode is negatively correlated with the fuel and oxidizer, while it is positively correlated with the product (i.e., H₂O) and temperature. Accordingly, the first PC represents the oxidation progress of the hydrogen/air mixture. The second PC mode is primarily correlated with the fuel, and the third PC mode is correlated with the formation of HO₂. The results indicate that the PCs obtained through the data-driven approach are linked to a physical interpretation of the combustion system, consistent with previous findings Owoyele17, Malik21.

Nonetheless, it is important to note that the PC modes are slightly altered with change of $T_{0}$, which can have a significant impact on the application of transfer learning to the ROM. In other words, a unified definition of the low-dimensional manifold throughout tasks would be preferred to transfer the knowledge efficiently. In the present study, $\textbf{A}^{\rm{T}}$ defined from the source task (i.e., $N_{\phi}=30$ and $T_{0}$ = 1000 K) is applied to all target tasks to ensure consistency in the definitions of $\boldsymbol{\mathrm{\psi}}$ and $\dot{\boldsymbol{\mathrm{\omega}}}_{\boldsymbol{\mathrm{\psi}}}$. Such an approach is based on the observation that despite the presence of slight differences in the PC modes, the first PC, which also accounts for most of the data variance, also exhibits the least difference when $T_{0}$ is varied.

Note, however, that using a unified definition of $\textbf{A}^{\rm{T}}$ has a potential risk of introducing noticeable errors during the conversion from $\boldsymbol{\mathrm{\psi}}$ to $\boldsymbol{\mathrm{\theta}}$ in the target task, especially if the reconstruction is carried out by using a matrix conversion step, $\boldsymbol{\mathrm{\theta}}\approx\textbf{A}\boldsymbol{\mathrm{\psi}}$. To address this issue, another non-linear ANN model is employed to convert from $\boldsymbol{\mathrm{\psi}}$ to $\boldsymbol{\mathrm{\theta}}$ for all cases, instead of using the matrix inversion. This ensures that the performance of the reconstruction is mainly affected by the number of samples, $M$, in the target task rather than the choice of $\textbf{A}^{\rm{T}}$. As will be discussed, such an ANN model is found to require far fewer parameters compared to the other ANN models that predict $\dot{\boldsymbol{\mathrm{\omega}}}_{\boldsymbol{\mathrm{\psi}}}$. Hence, this ANN model shows a reasonable accuracy even trained with a sparse dataset. It is also noted that the non-linear ANN was found to reduce the reconstruction error during the conversion from $\boldsymbol{\mathrm{\psi}}$ to $\boldsymbol{\mathrm{\theta}}$ as compared to the matrix inversion, $\boldsymbol{\mathrm{\theta}}\approx\textbf{A}\boldsymbol{\mathrm{\psi}}$ Mirgolbabaei15.

Figure 3 shows the temporal evolution of the first three PCs with three different $\phi$ of 0.85, 1.35, and 1.85 at $T_{0}$ of 1000 K, obtained by projecting $\textbf{A}^{\rm{T}}$ onto the FOM result. As discussed earlier, the first PC represents the progress variable of the mixture, such that the first PC switches from negative to positive values near the ignition delay time. For the third PC, its mode is mainly correlated with the intermediate species, namely HO₂, such that it is maximum just before ignition of the mixture.

A fully connected, multi-input, and multi-output ANN model is used to predict the reaction rates of the PCs. A PC vector is used as an input of the ANN, and the reaction rates of the PCs are the output of the ANN model. The architecture of the ANN model is determined by performing a grid search method, from which the number of hidden layers and nodes are set to 3 and 30, respectively. The hyperbolic tangent activation function is adopted for all hidden layers. For cases without the application of transfer learning methods, the Xavier normal initialization method Xavier10, which is a commonly used initialization scheme that is compatible with the hyperbolic tangent activation function, is employed.

As discussed in Section 2.2, the other ANN model is also employed and trained for the reconstruction from $\boldsymbol{\mathrm{\psi}}$ to $\boldsymbol{\mathrm{\theta}}$. One hidden layer with 10 nodes is found to be sufficient for this ANN model to reconstruct the original thermochemical scalars with reasonable accuracy. Note that such an ANN model requires considerably fewer number of training samples, and therefore, transfer learning is not applied to this model.

For both source and target tasks, 80% of the dataset is used as a training dataset, and the remaining 20% is allocated as the validation set to assess the model’s performance and to prevent overfitting. Mean absolute error (MAE) loss function is applied for the ANN training, consistent with previous studies showing that MAE shows a better performance than RMSE and MSE in capturing an ignition process of a fuel/air mixture Zhang20, Han22. The Adam optimizer Adam is used for stochastic optimization. Once the ANN model is optimized using the training dataset, it is both a priori and a posteriori evaluated against the test dataset, which is not involved in the training process.

To efficiently capture the ignition process of the hydrogen/air mixture, the dataset is decomposed into three clusters (Cl#1–Cl#3): (Cl#1) earlier ignition period ($\psi_{3}-\psi_{3,0}<$ 0.005 and $\psi_{1}<0.0$), (Cl#2) later ignition period ($\psi_{3}-\psi_{3,0}\geq$ 0.005 and $\psi_{1}<0.0$), and (Cl#3) post ignition period ($\psi_{1}\geq 0.0$), where the $\psi_{3,0}$ denotes the magnitude of $\psi_{3}$ at the initial condition. The clustering criteria are based on the observation that $\psi_{1}$ and $\psi_{3}$ effectively represent the progress variable and evolution of intermediate species, respectively, as depicted in Fig. 3. Note that the data clustering method has been proven as an effective way to capture the ignition process of various fuel/air mixtures Zhang20, Han22.

In this study, four different transfer learning methods are applied to the target tasks. Let h^s denote the parameter vector extracted from the pre-trained source task. The first transfer learning method (TL1) is that the knowledge of the pre-trained ANN model obtained from the source task is fully shared with the target task. In other words, the parameter vector in the target task, denoted by h, is identical to h^s, and no fine-tuning step is performed in TL1. Therefore, it can be conjectured that TL1 is likely to show good performance only when the task similarity between the source and target is very high. The second transfer learning method (TL2) is to set the initial parameter vector in the target task, h₀, with h^s, and then fine-tune the model using the sparse dataset in the target task. In other words, TL2 serves to initialize the ANN model in the target task by using h^s, and as such, h will be different from h^s after fine-tuning.

As discussed in Li20, a drawback of TL2 is that previous knowledge obtained from the source task may be lost during the fine-tuning step. To resolve this issue, the third transfer learning method (TL3), which is associated with parameter restriction, is applied. The total loss function, $\mathcal{L}$, in TL3 includes the regularization term Li18, Li20, which is slightly different from the conventional $l_{2}$ regularizer, as follows:

where MAE represents the mean absolute error loss function, and $\lambda_{1}$ is the regularization parameter. Here, the magnitude of $\lambda_{1}$ mainly controls the degree of knowledge transferred from source to target task during the fine-tuning step in the target task. Consistent with TL2, h₀ in TL3 is set to be h^s, and subsequently, the ANN model is fine-tuned by using the dataset in the target task. It is evident that h will be identical to h^s as the magnitude of $\lambda_{1}$ is very high, whereas there is no penalty for h to change during the fine-tuning step at $\lambda_{1}=0$. Therefore, it can be considered that TL3 becomes equivalent to TL1 and TL2 as the magnitude of $\lambda_{1}$ approaches infinity and zero, respectively.

Lastly, we introduce a novel transfer learning method called “Parameter control via Partial Initialization and Regularization (PaPIR)”. The central idea of PaPIR is to provide a unified transfer learning framework in terms of initialization and regularization. In addition to applying $\lambda_{1}$ to adjust the effect of the regularization during training, another variable, $\lambda_{2}$, is introduced in PaPIR so that the amount of previous knowledge transferred to the target task in terms of the initialization can also be controlled by changing the magnitude of $\lambda_{2}$. The initialization method of the PaPIR is a combination of two initialization schemes, namely Xavier normal initialization Xavier10 and initialization with h^s. Xavier normal initialization is a family of the Gaussian-based initialization technique with zero mean and a determined variance. Thus, it is a sort of random initialization strategy and in unrelated to the pre-trained knowledge. On the other hand, initialization with h^s is categorized as a data-driven initialization strategy Narkhede22.

The initialization process in PaPIR follows a normal distribution function, $N$, which is expressed as follows:

where w₀ and b₀ represent the initial weights and biases vector of the ANN model in the target task (i.e., h₀ = [w₀, b₀]), and w^s and b^s represent the weight and biases vector extracted from the source task (i.e., h^s = [w^s, b^s]). Here, $f_{\rm{i}}$ and $f_{\rm{o}}$ represent the number of incoming and outgoing nodes at each layer, respectively, which are identical to those used in the Xavier normal initialization scheme Xavier10.

As $\lambda_{2}$ in Eqs. 6–7 approaches zero, the initialization scheme becomes equivalent to the Xavier normal initialization method. As $\lambda_{2}$ approaches unity, on the other hand, h₀ simply becomes identical to h^s. Thus, the degree of knowledge transfer for the initialization of the target task can be adjusted by varying the value of $\lambda_{2}$ between zero and unity, which equivalently represents a bound between the Xavier normal initialization method and h^s, respectively. As will be demonstrated in Section 3, PC-transport ROM with a sparse dataset generally fails to capture the overall ignition process of a hydrogen/air mixture if the ANN model is trained from scratch. This shortcoming is mainly attributed to the propensity of the ANN models to get stuck in local minima, especially with a sparse dataset. Given that an appropriate initialization scheme can help avoid local minima Narkhede22, PaPIR has the potential advantage of enhancing the performance of transfer learning by introducing some degree of randomness during the initialization process. Table 2 summarizes the four transfer learning methods used in the present study.

In the source task, the ANN models for predicting the reaction rate of PCs are trained by using the training dataset with $N_{\phi}$ of 30 ($\phi=0.1-3.0$; $\Delta\phi$ = 0.1) at $T_{0}$ of 1000 K. A system of ODEs, Eq 4, is solved for 29 different 0-D simulations listed in the test dataset, and then the performance of the PC-transport ROM is evaluated against the FOM by comparing $\tau_{\rm{ig}}$ between two different simulations. As mentioned earlier, $\tau_{\rm{ig}}$ is defined by the time at which the temperature gradient reaches its maximum value, and $\tau_{\rm{ig}}$ in the PC-transport ROM can be predicted after reconstructing the temperature profile from the results of PCs.

Figure 4 shows the variations in $\tau_{\rm{ig}}$ for the hydrogen/air mixture at $T_{0}$ = 1000 K with various $\phi$ predicted by the PC-transport ROM and FOM. As shown in the figure, $\tau_{\rm{ig}}$ predicted by the PC-transport ROM shows excellent agreement with the FOM. The relative percentage error is below 2 % for the entire range of $\phi$, demonstrating that PC-transport ROM with a sufficient number of training samples can accurately reproduce the ignition process of a hydrogen/air mixture over a wide range of $\phi$ at a given $T_{0}$. Note that the relative percent error for the fuel-lean mixture is slightly higher than that for the fuel-rich mixture, which is attributed to the fact that the number of training samples assigned to the fuel-lean mixture is fewer than that assigned to the fuel-rich mixture. Also, $\tau_{\rm{ig}}$ shows a steeper variation with $\phi$ as $\phi$ becomes less than 0.5, which also affects the result.

Figure 5 presents the temporal evolution of the original thermochemical state vector of the hydrogen/air mixture at $T_{0}$ of 1000 K and $\phi$ of 1.35, as predicted by the PC-transport ROM and FOM. It is readily observed that the profiles of the thermochemical state variables reconstructed from the PC-transport ROM are in good agreement with the results from FOM for both major and minor species. This finding indicates that the number of PCs retained in this study ($N_{\rm{PC}}=$ 5) is sufficient to recover the original thermochemical state scalars, together with the successful validation of using the ANN model to convert from $\boldsymbol{\mathrm{\psi}}$ to $\boldsymbol{\mathrm{\theta}}$.

The results of the PC-transport ROM for a target task where $T_{0}$ is 1050 K are investigated depending on the number of training samples. In this task, ANN models are trained by using each of the different training datasets, each with a different numbers of training samples (i.e., $N_{\phi}$ = 2 $-$ 30), and both a priori and a posteriori evaluations are carried out to assess the performance of the ROM depending on $N_{\phi}$. At a given $N_{\phi}$, the ANN model training is repeated 10 times to take into account the sensitivity of the model arising from the randomness of the initial parameters and the stochastic nature of the optimization process. The normalized root mean squared error (NRMSE) is adopted to a priori quantify the error of the ANN for predicting $\dot{\boldsymbol{\mathrm{\omega}}}_{\boldsymbol{\mathrm{\psi}}}$ and is defined by

where $\hat{\boldsymbol{\mathrm{\omega}}}_{\boldsymbol{\mathrm{\psi}},p}$ and $\hat{\boldsymbol{\mathrm{\omega}}}_{\boldsymbol{\mathrm{\psi}}}$ represent the normalized reaction rates of the PC vector predicted by the ANN model and obtained from the FOM, respectively.

Figure 6 shows the variations in NRMSE of the test set in the target task with $T_{0}$ of 1050 K and various $N_{\phi}$. The NRMSE of the test set generally shows a decreasing trend with an increase of $N_{\phi}$, such that the optimal value of NRMSE for the case with $N_{\phi}=30$ approaches $O$(10^-1) [%] for all the clusters. This outcome clearly indicates that the number of training samples plays a crucial role in determining the performance of the PC-transport ROM. In addition, the variations in the NRMSE as a result of repeating the ANN model training 10 times exhibit a noticeable fluctuation at a given $N_{\phi}$. Consequently, for Cl#1 and Cl#3, the worst cases with $N_{\phi}=30$ have a similar magnitude of NRMSE compared to the best cases with $N_{\phi}=15$. This result suggests that a multi-start based optimization algorithm would be necessary to obtain the nearest optimal neural network model at a given $N_{\phi}$.

Next, a series of 0-D simulations is carried out by using the ANN models trained with different numbers of training samples. For the case with $N_{\phi}=30$, $\tau_{\rm{ig}}$ is predicted by using the best and worst ANN models out of 10 repetitions of the ANN model training. For the other cases, $\tau_{\rm{ig}}$ is predicted by using the best ANN model only. Figure 7 summarizes the variations in $\tau_{\rm{ig}}$ for the hydrogen/air mixture with $T_{0}$ of 1050 K and various $\phi$ listed in the test set, predicted by using the ANN models with different $N_{\phi}$.

Figure 7 illustrates that the PC-transport ROM fails to capture the overall ignition characteristics of a hydrogen/air mixture with $N_{\phi}\leq$ 15. In this regard, the datasets with $N_{\phi}\leq 15$ are regarded as “sparse datasets”. Note that the PC-transport ROM performs relatively well when the target equivalence ratio is adjacent to one of the equivalence ratios listed in the training dataset. For instance, the training dataset with $N_{\phi}=$ 3 consists of the 0-D simulation results with $\phi$ of 0.5 ,1.5, and 2.5, where the relative-error of the PC-transport ROM is relatively small near $\phi$ of 0.5, 1.5, and 2.5, while the performance of the PC-transport ROM declines as the target equivalence ratio moves farther from the training dataset. It is also important to note that even when a large amount of training samples are used (i.e., $N_{\phi}=$ 30), the simulation results occasionally do not agree well with the results from the FOM, consistent with the a priori evaluation in Fig. 6. This result not only highlights that the number of training samples is a crucial part of optimizing the ANN model but also indicates that the uncertainty of the ANN model training is noticeable, and is primarily due to the stochastic nature of the training process and/or the randomness of the initial parameter.

As a baseline case, the result of applying three transfer learning methods (TL1, TL2, and TL3) to the target task where $T_{0}$ = 1050 K and $N_{\phi}=$ 4 is discussed first. In this case, the difference of $T_{0}$ between source and target tasks is relatively small (i.e., $\Delta T$ = 50K), such that task similarity between the two tasks is considered to be high. The parameters obtained from the source task are used to train the ANN model in the target task in various ways.

As shown in Table 2, TL3 (parameter restriction) becomes equivalent to TL1 (parameter sharing) and TL2 (fine-tuning) as the value of the regularization parameter, $\lambda_{1}$, in Eq. 5 approaches near infinity and zero, respectively. Thus, the performance of the ANN model using three different transfer learning methods can be evaluated by adjusting the magnitude of $\lambda_{1}$. Similar to the previous cases where transfer learning is not employed, ANN training is repeated 10 times to evaluate the uncertainty of the ANN model training.

Figure 8 shows the NRMSE values against the training and test sets, along with the percentage differences in the optimized parameters between the source and target tasks, represented by $\lVert\textbf{{h}}-\textbf{{h}}^{s}\rVert_{2}^{2}$/$\lVert\textbf{{h}}^{s}\rVert_{2}^{2}\times 100$, as a function of $\lambda_{1}$ for the case where $T_{0}$ = 1050 K and $N_{\phi}$ = 4. Several points are noted from the figure.

First, given that $\lambda_{1}$ serves as a penalty term during the fine-tuning, the NRMSE of the training set generally decreases with a decrease of $\lambda_{1}$ for all the clusters. On the other hand, $\lVert\textbf{{h}}-\textbf{{h}}^{s}\rVert_{2}^{2}$/$\lVert\textbf{{h}}^{s}\rVert_{2}^{2}$ continues to increase as $\lambda_{1}$ decreases, indicating that the parameters in the target task become more dissimilar to those in the source task with a decrease of $\lambda_{1}$. This trend demonstrates that the magnitude of $\lambda_{1}$ mainly controls the degree of knowledge transfer from the source to the target task.

Second, as the magnitude of $\lambda_{1}$ becomes sufficiently large (i.e., greater than 10⁴), h becomes nearly identical to $\textbf{{h}}^{s}$, illustrating that the transfer learning method for this case is equivalent to TL1, where the knowledge gained from the source task is fully transferred to the target task. In that case, the NRMSE of the test set can be notably higher than the case without applying transfer learning (see Figs. 8b and d as an example), indicating that TL1 would play a negative role in predicting the ignition delay of the hydrogen/air mixture for the target task even if the difference in $T_{0}$ is relatively small. This would be primarily attributed to the non-linear nature of chemical kinetics, where reaction rates are highly sensitive to temperature change.

Third, as the magnitude of $\lambda_{1}$ approaches near zero, the corresponding transfer learning method represents TL2, where the parameters obtained from the source task are used to initialize the parameters in the target task. In this scenario, $\lVert\textbf{{h}}-\textbf{{h}}^{s}\rVert_{2}^{2}$/$\lVert\textbf{{h}}^{s}\rVert_{2}^{2}$ is relatively high, indicating that the knowledge acquired from the source task is prone to be lost during the fine-tuning process. Nonetheless, it is worth mentioning that the NRMSE of the test set using TL2 nearly equals that obtained from the best ANN model without applying transfer learning. Furthermore, the results of TL2 show less fluctuation from the 10 repetitions of training compared to the results without applying transfer learning. This suggests that initializing the ANN model of the target task with the parameters gained from the source task can be considered an appropriate initialization scheme, provided that the task similarity between the source and target task is high.

Lastly, the NRMSE of the test set reaches its minimum as the magnitude of $\lambda_{1}$ has a finite value (i.e., $\lambda_{1}=$ $O$(10^-4 $-$ 10^-1)) for all the clusters. In this case, the resultant NRMSE is approximately an order of magnitude lower than that obtained from training the ANN model from scratch, clearly demonstrating that TL3 with an optimal value of $\lambda_{1}$ can remarkably improve the performance of the neural network model for the target task. Note that in order to achieve a comparable level of NRMSE as that obtained from TL3, the ANN model trained from scratch requires a larger number of $N_{\phi}$, ranging from 15 to 30, indicating that TL3 can reduce the requisite number of training samples up to eight times. Furthermore, the ANN model training with the use of the optimal value of $\lambda_{1}$ is nearly insensitive to the number of training repetitions. This is attributed to the well-known effect of regularization on the stochastic optimization process. Hence, the overall number of training repetitions required to find the optimal ANN model in the target task can be significantly reduced by applying TL3 with an optimal value of $\lambda_{1}$, provided that the source and target tasks are similar in parameter space.

Based on the observations in Fig. 8, it can be inferred that the loss function in the target task is likely to contain multiple local minima such that the training result of the ANN model may not reach the global minima of the loss function, especially with the sparse dataset. One method to address this issue and improve the performance of the ANN model is to utilize the pre-trained ANN model. If the source and target tasks are similar to each other in the parameter space, then initializing the parameter in the target task ($\textbf{{h}}_{0}$) with that from the source task ($\textbf{{h}}^{s}$) can assist in searching for the optimal parameters. Hence, the performance of the ANN model with TL2 has the potential to show better performance compared to the ANN model trained from scratch. Furthermore, TL3 with the optimal value of $\lambda_{1}$ can help prevent h from significantly deviating from $\textbf{{h}}^{s}$ during the fine-tuning step, which can enhance the accuracy in predicting $\tau_{\rm{ig}}$ in the target task.

Next, a posteriori evaluation of the PC-transport ROM with different transfer learning strategies is carried out by performing a series of 0-D simulations of a hydrogen/air mixture using the PC-transport ROM for the target task, where $T_{0}$ = 1050 K and $N_{\phi}$ = 4. Figure 9 shows the variations in $\tau_{\rm{ig}}$ as a function of $\phi$ depending on the different transfer learning methods. The PC-transport ROM fails to predict the overall ignition trend when the ANN is trained from scratch or trained with TL1. Note that the PC-transport ROM with TL1 inaccurately captures the early stage of 0-D ignition, leading to error accumulation over time. Consequently, the PC-transport ROM with TL1 fails to undergo a thermal runaway crossing for the full range of $\phi$, and hence, $\tau_{\rm{ig}}$ approaches infinity.

The overall performance of the PC-transport ROM with TL2 is better than the PC-transport ROM without applying the transfer learning method or with TL1, demonstrating the importance of the initialization scheme and the fine-tuning step on the result, respectively. Here, $\tau_{\rm{ig}}$ predicted by the PC-transport ROM with TL2 is in relatively good agreement with that from the FOM when $\phi$ is near those in the target task training dataset (i.e., $\phi$ of 0.2, 1.0, 2,0, and 3.0), while the performance of the ANN model decreases as $\phi$ becomes farther removed from those in the training dataset of the target task.

For the PC-transport ROM with the optimal value of $\lambda_{1}$ (i.e., TL3), it is readily observed that the PC-transport ROM shows a good performance of predicting $\tau_{\rm{ig}}$ over a wide range of $\phi$. This result substantiates that the regularization-based transfer learning framework can increase the accuracy of the ANN model with the sparse training dataset. Figure 10 presents the temporal evolution of the PCs with four different values of $\phi$ of 0.15, 0.65, 1.55, and 2.55, predicted by the FOM and the PC-transport ROM with an optimal value of $\lambda_{1}$. Although a slight time lag exists between the FOM and the PC-transport ROM results due to data sparsity, the PC-transport ROM can reasonably capture the onset of ignition and the subsequent equilibrium period of the PCs.

Additional parametric studies are carried out by varying $N_{\phi}$ or increasing $T_{0}$ to 1300 and 1400 K. Figure 11 shows the variations in $\tau_{\rm{ig}}$ as a function of $\phi$ for the target task with $T_{0}$ of 1050 K and decreasing $N_{\phi}$ to 2 and 3 by applying different transfer learning methods. Note that as $N_{\phi}$ decreases to 2 and 3, the training dataset is intended not to cover the entire range of the test dataset (i.e., $\phi$ = 0.5 and 1.5 for $N_{\phi}$ = 2, and $\phi$ = 0.5, 1.5, and 2.5 for $N_{\phi}$ = 3) such that there exist several test cases where $\phi$ is outside of the range of the training dataset (see the highlighted regions in Fig. 11). The overall variations in $\tau_{\rm{ig}}$ predicted by applying different transfer learning methods show a similar trend regardless of the change of $N_{\phi}$. The PC-transport ROM with TL1 fails to capture the onset of ignition of the hydrogen/air mixture for the entire range of $\phi$, while the results with TL2 show a better performance than those without applying transfer learning. The PC-transport ROM with the optimal value of $\lambda_{1}$ in TL3 outperforms all the other models.

As expected, the accuracy of the PC-transport ROM notably decreases as the target $\phi$ of the 0-D simulation is outside of the range of the training dataset, which is a well-known drawback of machine learning models for extrapolation. Nonetheless, the result of the PC-transport ROM with TL3 shows a relatively-good performance even for the cases where $\phi$ is outside of the range of the training set. This result implies that the previous knowledge obtained from the source task helps increase the accuracy of the extrapolation of the ANN model, consistent with previous findings Humbird20.

Next, target tasks are considered where $T_{0}$ is increased further (i.e., $T_{0}$ = 1300, and 1400 K) such that the task similarity between source and target tasks decreases. Figure 12 shows the variations in $\tau_{\rm{ig}}$ for the hydrogen/air mixture with $T_{0}$ of 1300 and 1400 K and $N_{\phi}$ of 4 by using various ANN models with or without applying transfer learning methods. For the cases with $T_{0}$ = 1300 K, it is found that the PC-transport ROM with the optimal value of $\lambda_{1}$ in TL3 shows a reasonable performance over the entire range of $\phi$, while the results without applying transfer learning or with applying TL2 exhibit a noticeable error in predicting ignition of a lean mixture. As $T_{0}$ in the target task further increases to 1400 K, on the other hand, the result with TL3 shows only a marginal improvement compared to the other cases. This result demonstrates that the performance of the regularization-based transfer learning method decreases with a decrease of task similarity between source and target tasks.

Lastly, the performance of the unified transfer learning method, PaPIR, in the different target tasks is investigated. As discussed in Section 2.4, the central idea of PaPIR is to control the degree of knowledge transfer from the source to target task by adjusting the magnitudes of $\lambda_{1}$ and $\lambda_{2}$, which are associated with the regularization and initialization of the ANN model in the target task, respectively. Unlike TL3, h₀ in PaPIR can be distributed by either a normal distribution function following the Xavier normal initialization method ($\lambda_{2}=0$), or h² ($\lambda_{2}=1.0$), or inbetween the two ($0<\lambda_{2}<1$). In this regard, the effect of the initialization on the performance of transfer learning can be investigated by varying $\lambda_{2}$ in PaPIR.

Figure 13 presents the a priori result of the best achievable NRMSE of $\dot{\boldsymbol{\mathrm{\omega}}}_{\boldsymbol{\mathrm{\psi}}}$ of the test set for three different target tasks (i.e., $T_{0}$ = 1050, 1300, and 1400 K with $N_{\phi}$ of 4) as a function of $\lambda_{1}$ and $\lambda_{2}$, conditional on each of the three different clusters. Consistent with the previous results, the best achievable (or minimum) value of NRSME is evaluated by repeating the ANN model training 10 times at a given $\lambda_{1}$ and $\lambda_{2}$. As shown in Fig. 13a, PaPIR covers all the transfer learning methods discussed in the present study, namely, TL1, TL2, and TL3. In general, the results with TL1 exhibit a large error and increase with an increase of $T_{0}$ in the target task. In TL3, there exists an optimal value of $\lambda_{1}$ that results in a lower NRMSE than for the results either without applying transfer learning or with TL2.

For the case where task similarity is relatively high (i.e., $T_{0}=1050$ K in the target task; see Figs. 13a–c), the value of NRMSE is mainly governed by the regularization parameter $\lambda_{1}$, whereas it is largely unaffected by a change of the initialization parameter, $\lambda_{2}$. Since the source and target tasks are similar to each other in this case, the optimal value of $\lambda_{1}$ is relatively large (e.g., $\lambda_{1}$ = 10^-1 in Cl#1). Given that a regularization term serves to convexify the objective function, a relatively large magnitude of $\lambda_{1}$ leads the ANN model to be insensitive to a change in the initialization scheme. Consequently, PaPIR does not outperform TL3 when the task similarity between source and target tasks is high. The best achievable values of the NRMSE depending on the different transfer learning methods are summarized in Table 3.

As $T_{0}$ in the target task increases to 1300 K, results with Cl#1 (Fig. 13d) show that the NRMSE of the test dataset attains its minimum at a relatively low magnitude of $\lambda_{1}$ (= 10^-3). Although the overall variations of the NRMSE are still mainly governed by $\lambda_{1}$, the NRMSE is no longer invariant to a change of $\lambda_{2}$ at the optimal value of $\lambda_{1}$, indicating that $\lambda_{2}$ starts to play a role in the optimization of the ANN model. Since the magnitude of the optimal value of $\lambda_{1}$ decreases compared to the case with $T_{0}=1050$ K, the complexity of the loss function at the optimal value of $\lambda_{1}$ increases, and consequently, the training result can be varied with the different initialization schemes. This finding indicates that an initialization scheme becomes important in the framework of transfer learning as the task similarity between the source and target task becomes relatively low. Note that in Fig. 13d, the ANN model exhibits a slightly-lower magnitude of NRMSE at $\lambda_{2}$ of 0.7 compared to that of 1.0, illustrating the potential advantage of PaPIR over TL3. Readers are referred to Table 3 to quantify the difference of NRMSE between PaPIR and TL3. For the results of Cl#2 and Cl#3, on the other hand, $\lambda_{2}$ still plays a major role in determining the best achievable value of NRMSE (see Figs. 13e and f). This would be because the decrease of task similarity for these clusters is not as pronounced as for Cl#1.

As $T_{0}$ in the target task further increases to 1400 K, it is found that $\lambda_{1}$ still shows a dominant effect on the NRSME compared to $\lambda_{2}$, demonstrating that the primary factor of determining the performance of transfer learning is a regularization parameter in general. Nonetheless, there are several cases where the ANN model trained with $\lambda_{2}<1$ exhibits a lower magnitude of NRMSE compared to the best candidate obtained from TL3 (see Fig. 13g). Note that at $T_{0}$ = 1400 K, the ratio of the best achievable NRMSE obtained from PaPIR to TL3 is 0.827, 0.984, and 0.866, for Cl#1, Cl#2, and Cl#3, respectively. This result shows that adjusting the initial values of the parameters in the target task can further enhance the performance of transfer learning in the target task with a sparse dataset, especially when the task similarity between source and target tasks is low such that the optimal value of the regularization parameter, $\lambda_{1}$, becomes relatively low.

To further investigate the advantage of PaPIR over other transfer learning methods, especially when the task similarity is relatively low, Figure 14 presents the variations in $\tau_{\rm{ig}}$ for a hydrogen/air mixture with $T_{0}$ of 1400 K and $N_{\phi}$ of 4, predicted by the FOM and the PC-transport ROMs with different transfer learning methods. This figure clearly shows that $\tau_{\rm{ig}}$ predicted by PaPIR shows excellent agreement with that from the FOM over the entire range of $\phi$, which is clearly distinct from the other models. Although the relative-error obtained from PaPIR is slightly higher than that from TL3 or from the PC-transport ROM without applying transfer learning at $\phi>1$, the PC-transport ROM with PaPIR shows a more robust performance for predicting the oxidation process of hydrogen/air mixture over a wide range of $\phi$.

One may argue that under the data-sparse scenario, the number of test datasets is also likely to be insufficient, rendering it infeasible to find the optimal values of $\lambda_{1}$ and $\lambda_{2}$ by relying on the test dataset. As a future work, a systematic way of estimating the optimal values of those two parameters without relying on a test dataset will be investigated. One practical example would be adopting the L-curve criterion, a well-known heuristic method to find the optimal regularization parameter without relying on the test dataset Hansen92. Also as future work, we will further investigate different methods to “partially” transfer the knowledge of the pre-trained neural network model in the target task, such as applying Bayesian transfer learning methods.